Top-p sampling

At each step a language model assigns every next option a probability, then has to choose¹. Always taking the single most likely option, or searching for the highest-probability continuation, often produces bland text that repeats itself¹. Drawing from the whole list is no better. Holtzman and colleagues call the long low-probability end an unreliable tail, and pure sampling from it comes out incoherent¹.

Top-p sampling, which they named nucleus sampling, keeps the smallest set of options whose probabilities add up to at least p, then samples inside that set¹. The set is rebuilt at every step. It grows when many options are plausible and shrinks when the model is sure. They say the nucleus usually holds somewhere between one option and about a thousand¹.

Temperature reshapes the whole distribution, and they argue it does not by itself cut that tail enough¹. Top-k sampling, used by Fan, Lewis and Dauphin in 2018, always keeps a fixed count. Their story model kept the 10 most likely next words². A fixed count is clumsy: too small and the text goes generic, too large and unlikely words are boosted when the remaining probabilities are rescaled¹.

They had GPT-2 Large write 5,000 passages of up to 200 tokens from held-out WebText prefixes¹. At p = 0.95, nucleus sampling scored perplexity 13.13, near human text at 12.38, and repeated 0.36% of the time against 0.28% for humans¹. Beam search with 16 beams repeated 28.94% of the time¹. On HUSE, their combined human and statistical score, nucleus was highest at 0.97. That figure used a smoothed stand-in, because plain truncation gave the method probability zero on most tokens¹.

APIs expose the same control as top_p. OpenAI's completions API calls it nucleus sampling: 0.1 keeps tokens in the top 10% of probability mass, and the docs say to change top_p or temperature, not both³. Gemini's topP is the maximum cumulative probability of tokens to consider. The docs say Gemini uses nucleus sampling, sometimes with a fixed top-k, and that some models disallow top-k⁵. Anthropic's Messages API still defines top_p as nucleus sampling, but models released after Claude Opus 4.6 reject any value below 0.99⁴.