Definition
Top-p sampling, also called nucleus sampling, is a method for controlling how a language model chooses its next word. At each step the model has a ranked list of candidate words with probabilities, and top-p keeps only the smallest group of top candidates whose probabilities add up to a threshold you set, then picks randomly from just that group. If you set the threshold to 0.9, the model gathers the most likely words until their combined probability reaches ninety percent, throws away the rest, and samples from what remains.
The reason this method exists is that it adapts to how confident the model is, which fixed alternatives do not. When the model is very sure what comes next, a few words already cover most of the probability, so the kept group is small and the output stays focused. When the model is genuinely uncertain, the probability is spread across many words, so the group is larger and the output has more room to vary. The randomness scales with the situation instead of being applied blindly.
What distinguishes top-p from simply always taking the most likely word is that it preserves useful variety without letting in the junk. Language models assign tiny probabilities to a huge number of implausible words, and if you sample from the full list you occasionally pick nonsense. Top-p cuts off that long tail of unlikely options, so the model can still surprise you with a reasonable alternative but almost never blurts out something it barely believed in.
By 2026, top-p is one of the standard sampling controls offered alongside temperature in nearly every language model interface, usually as a value between zero and one. It is often left at a sensible default because it behaves well without much tuning, and many people use it without fully understanding it, which is fine until they start adjusting it together with temperature and get confused about why the output changed in ways they did not expect.
This page covers how top-p sampling works, how it compares to temperature and to the related top-k method, what makes it different from a hard cap on choices, where it fits and where it does not, and how to set it sensibly. The idea worth keeping is that top-p is a dynamic filter on the model's options. It does not add randomness so much as decide which options are allowed to be random, and that adaptiveness is exactly what makes it useful and what makes it interact awkwardly with controls that assume a fixed amount of randomness.
Key Takeaways
- Top-p sampling keeps only the top candidate words whose probabilities add up to a set threshold, then samples from that group.
- It exists to adapt the amount of randomness to the model's confidence, keeping a narrow set when sure and a wider one when uncertain.
- It preserves useful variety while cutting off the long tail of very unlikely words that produce nonsense.
- By 2026 it is a standard control, usually a value between zero and one, often left at a well-behaved default.
- Top-p decides which options are allowed rather than simply adding randomness, which is why it interacts awkwardly with temperature.
How Top-p Sampling Works
At each generation step the model produces a probability for every possible next word, then top-p sorts those candidates from most to least likely and starts adding them up. It keeps including words until the running total of their probabilities reaches the threshold you chose, and everything past that point is discarded. The model then samples its next word only from the group that made the cut, ignoring the vast tail of words it considered but assigned little chance.
The size of that kept group is not fixed, and that is the whole point. If the model is confident and the top word alone carries most of the probability, the group might contain just a handful of options. If the model is torn between many plausible continuations, the group grows to include all of them. The same threshold produces a tight set in confident moments and a loose one in uncertain moments, which is why the method is called nucleus sampling: it keeps the dense core of probability.
A lower threshold makes the model more conservative, since it keeps fewer words and leans harder on the front-runners. A higher threshold lets more of the distribution through, allowing more variety. At a threshold of one the method keeps everything and effectively does nothing to filter, while at a very low threshold it collapses toward always picking the single most likely word, behaving almost deterministically.
The practical effect is a model that stays coherent while still being able to vary its output naturally. Because the long tail of nonsense words is cut before sampling, the surprises top-p produces tend to be sensible alternatives rather than gibberish. This is why it often gives more natural-feeling text than cranking up a blunt randomness dial, which can reach into the tail and pull out something the model never should have said.
Top-p Sampling Compared to Temperature
Temperature and top-p both influence randomness, but they work on different parts of the problem. Temperature rescales the probabilities of all candidates, sharpening or flattening the whole distribution before anything is chosen. Top-p leaves the probabilities as they are and instead decides which candidates are even eligible, by trimming the low-probability tail. One reshapes the odds, the other restricts the field.
The consequence is that top-p adapts to the model's confidence while temperature does not. Temperature applies the same reshaping whether the model is certain or uncertain, so a high temperature will happily flatten a confident distribution and let unlikely words compete. Top-p, by contrast, keeps the eligible set small exactly when the model is sure, which tends to protect coherence in the moments that matter most.
Because they act through different mechanics, they can be combined, but doing so makes the result hard to predict. Raising temperature adds randomness by flattening the distribution, while tightening top-p removes options by trimming it, so the two can partly cancel or compound in ways that are not obvious. This is why the usual advice is to adjust one and leave the other at its default rather than tuning both at once.
If you only touch one, which to pick depends on how you think about the task. Temperature maps cleanly onto the intuitive creativity-versus-reliability dial, so it is the friendlier choice for most people. Top-p is the better choice when you specifically want the randomness to track the model's own uncertainty, keeping output tight where the model is confident and looser where it is genuinely unsure.
What Makes Top-p Different From Top-k
Top-p is often confused with top-k, another sampling method that also restricts the model's choices, and the difference is worth getting straight. Top-k keeps a fixed number of the most likely candidates, say the top forty, regardless of their probabilities, and samples from those. Top-p keeps a variable number of candidates based on their combined probability. One fixes the count, the other fixes the probability mass.
That distinction changes how each behaves when the model's confidence shifts. Top-k always keeps the same number of options, so when the model is very sure it still keeps forty candidates even though most of them are near-hopeless, and when the model is uncertain it cuts off good options once it hits the count. Top-p sidesteps both problems by letting the group shrink or grow with the actual shape of the distribution.
In practice this makes top-p the more forgiving default, because it does not need to guess the right number of candidates in advance. A fixed count that works well when the model is confident can be badly wrong when it is uncertain, whereas a probability threshold behaves reasonably across both cases without adjustment. That adaptiveness is the main reason top-p largely displaced top-k as the go-to filtering method.
Neither is wrong, and they can even be used together, with top-k as a hard ceiling on the number of candidates and top-p as the probability-based filter within it. But for most purposes top-p alone captures what people want from top-k while avoiding its main weakness, so reaching for top-k specifically is usually only worthwhile when you have a concrete reason to cap the raw number of options.
Where Top-p Sampling Fits and Where It Does Not
Top-p fits well as a general-purpose randomness control for open-ended generation, especially conversational and creative text where you want natural variation without the risk of the occasional nonsense word. Its adaptiveness means you can set a reasonable threshold once and get sensible behavior across a wide range of prompts, which is why it is a comfortable default for chat-style applications that see unpredictable inputs.
It also fits well when you want variety but cannot tolerate incoherence, which is a common combination. Because it trims the tail before sampling, top-p gives you diversity that stays inside the bounds of what the model actually finds plausible. For tasks like drafting varied marketing copy or generating alternative phrasings, this is often more useful than a blunt randomness dial that can occasionally reach into the tail and break the output.
Top-p fits poorly when you need strict determinism, because any threshold above the point of collapse still leaves room for the model to vary between runs. For tasks where the same input must produce exactly the same output, such as feeding a downstream parser, you want to remove randomness entirely rather than merely restrict it, and top-p is not the tool for that. A near-zero temperature is the cleaner way to pin the output down.
It also fits poorly as a knob for improving accuracy or format compliance, which it does not control. Tightening top-p reduces variation, so it can make faulty output more consistent, but it will not make the model more correct or better at following a structure. When the problem is wrong answers or broken formatting, top-p is a distraction, and the fix lies in the prompt, the context, or the choice of model.
How to Set Top-p Sampling Well
Leave it at its default unless you have a specific reason to change it, because top-p is designed to behave well without tuning. Most interfaces ship with a threshold that produces natural output across common tasks, and moving it without a clear goal usually just trades one reasonable behavior for another while adding a variable you now have to reason about. Change it when the output is too repetitive or too loose, not on general principle.
If output feels bland and repetitive, raise the threshold to let more of the distribution through, which gives the model more room to choose alternatives. If output wanders or occasionally goes off the rails, lower the threshold to keep the eligible set tighter. Small moves matter here, so nudge it and observe rather than making large jumps, since the effect varies with how confident the model tends to be on your prompts.
Do not adjust top-p and temperature at the same time. Because they interact, changing both leaves you unable to tell which one caused a difference, so pick the one whose behavior you understand best for the task and hold the other at its default. For most people that means tuning temperature and leaving top-p alone, but the reverse is fine as long as you only move one.
For strict, repeatable output, do not rely on top-p at all. Set temperature to zero or near it to make the model effectively deterministic, rather than trying to squeeze determinism out of a low top-p threshold. Top-p restricts randomness but does not eliminate it, so using it for reproducibility gives you output that is close to stable but not reliably identical, which tends to surface as an intermittent bug.
Test on realistic inputs before settling a value, especially for anything that sees varied prompts. Because top-p adapts to the model's confidence, a threshold that looks perfect on one type of prompt can behave differently on another, so a setting chosen from a single example is not evidence it works in general. Trying it across the range of inputs your application actually receives is what tells you whether the value holds up.
Best Practices
- Leave top-p at its default unless output is clearly too repetitive or too loose.
- Raise the threshold for more variety and lower it for tighter, more focused output, in small steps.
- Adjust only top-p or temperature at a time, since the two interact and obscure each other's effects.
- Use a near-zero temperature, not a low top-p, when you need strictly repeatable output.
- Test any chosen threshold across the realistic range of prompts, not just one example.
Common Misconceptions
- Top-p does not keep a fixed number of words; it keeps as many as needed to reach a probability threshold.
- Top-p is not the same as temperature; it trims which candidates are eligible rather than reshaping their probabilities.
- Top-p is not the same as top-k; one fixes probability mass, the other fixes the count of candidates.
- Top-p does not make a model more accurate or better at following formats; it only controls variation.
- A low top-p does not guarantee identical output between runs, so it is not a reliable way to get determinism.
