Sampling Strategies & Beam Search

Two people call the same model with the same prompt. One gets a crisp, sensible paragraph. The other gets:

The best way to learn programming is to learn programming. The best wayto learn programming is to learn programming. The best way to learn...

Identical weights. Identical prompt. The difference is entirely in what happened after the model produced its output, in the twenty lines of code that turn a vector of scores into a chosen token.

That step is called decoding, and it is the most under-appreciated knob in the whole stack. The model's job ends when it emits a probability distribution over the vocabulary. Something else has to pick. How it picks determines whether you get a loop, a hallucination, a bland answer, or something good.

The obvious strategy, and the two ways it fails

Greedy decoding is the simplest possible rule: take the highest-probability token, every time.

next_id = logits.argmax(dim=-1)

It is deterministic, fast, and wrong for most purposes. Here is why, in two separate ways.

Failure one: locally best is not globally best

Greedy decoding commits to a token before knowing what it makes possible. Consider a two-step choice with these probabilities:

Step 1:   token A: 0.60      token B: 0.40Step 2 (best continuation available after each):  after A: 0.20             joint probability = 0.60 x 0.20 = 0.12  after B: 0.70             joint probability = 0.40 x 0.70 = 0.28

Greedy takes A, because 0.60 > 0.40, and ends up on a sequence with joint probability 0.12. The path through B was worth 0.28 — more than twice as likely — but greedy could not see it, because seeing it requires looking one step further ahead than greedy ever looks.

Failure two: the repetition trap

The looping output above is not a bug in the model. It is a stable state that greedy decoding walks into and cannot leave. Once the model has produced a phrase, that phrase in its own context makes repeating it slightly more likely — text genuinely does repeat, and the model has learned that. Greedy always takes the most likely token, so the moment repetition becomes the single most likely continuation, it is locked in. There is no randomness available to escape.

Underlying both failures is a deeper point. High-probability text is not the same as good text. Human writing is not the maximally probable sequence — it contains surprises, specific word choices, and information. A decoder that relentlessly maximises probability produces prose that is bland where it is not broken.

The model's distribution is a good description of what text looks like. Always taking its mode is a bad way to produce text, because real writing does not sit at the mode.

Temperature: reshaping the distribution

Temperature divides the logits before the softmax:

Take five candidate tokens with logits $[4.0,\, 3.0,\, 2.0,\, 1.0,\, 0.0]$ and compute all three cases fully.

T = 1.0  (unchanged)  exp: 54.598, 20.086, 7.389, 2.718, 1.000    sum = 85.791  p:    0.636,  0.234, 0.086, 0.032, 0.012T = 0.5  (logits doubled: 8, 6, 4, 2, 0)  exp: 2980.96, 403.43, 54.598, 7.389, 1.000  sum = 3447.38  p:    0.8647,  0.1170, 0.0158, 0.0021, 0.0003T = 2.0  (logits halved: 2.0, 1.5, 1.0, 0.5, 0)  exp: 7.389, 4.482, 2.718, 1.649, 1.000      sum = 17.238  p:   0.4287, 0.2600, 0.1577, 0.0956, 0.0580

Low temperature sharpens: the top token's share rises from 63.6% to 86.5%, and the tail is crushed by more than an order of magnitude. High temperature flattens: the fifth-ranked token goes from a 1-in-83 chance to nearly 1-in-17.

The two limits are worth naming. As $T \to 0$ the distribution becomes one-hot and sampling becomes exactly greedy decoding. As $T \to \infty$ it becomes uniform over the entire vocabulary — sampling pure noise. Note that temperature never changes the ranking, only the gaps between ranks.

The danger with high temperature is precisely that fattened tail. Raising $T$ to 1.5 does not mainly make the model more creative among good options; it mainly hands probability to the thousands of tokens the model correctly judged to be nonsense. That is where most temperature-driven gibberish comes from.

Truncation: cut the tail off first

The insight behind the next two methods: rather than reweighting the bad options, remove them entirely, and only then sample.

Top-k

Keep the $k$ highest-probability tokens, discard the rest, renormalise. With $k = 2$ on the $T=1$ distribution above:

keep:        0.636, 0.234       (sum 0.870)renormalise: 0.636/0.870 = 0.731             0.234/0.870 = 0.269

Simple and effective, but $k$ is a fixed number applied to a distribution whose shape changes constantly. Two cases show the problem:

A fixed $k$ is either too permissive or too restrictive, depending on how confident the model happens to be at that moment.

Top-p (nucleus sampling)

Instead of a fixed count, use a probability budget. Sort tokens by probability, walk down the list accumulating mass, and stop once the running total reaches $p$. With $p = 0.9$:

token   p       cumulative1st     0.636   0.636      keep  (below 0.9, continue)2nd     0.234   0.870      keep  (below 0.9, continue)3rd     0.086   0.956      keep  (crossed 0.9 - stop here)4th     0.032   -          discard5th     0.012   -          discardrenormalise the kept three by 0.956:  0.636/0.956 = 0.665  0.234/0.956 = 0.245  0.086/0.956 = 0.090

The size of the kept set now adapts automatically. On "The capital of France is" with Paris at 0.97, top-p with $p=0.9$ keeps one token — the threshold is crossed immediately. On a genuinely open continuation it might keep two hundred. The model's own confidence decides how much freedom the sampler gets, which is exactly the behaviour you want.

Top-k asks "how many options should I consider?" Top-p asks "how much probability should I trust?" — and only the second question has an answer that stays right as the context changes.

A newer variant, min-p, sets the cutoff relative to the top token: keep any token whose probability is at least $\text{min\_p} \times p_{\max}$. With $\text{min\_p} = 0.1$ and $p_{\max} = 0.636$, the threshold is 0.0636, which keeps the top three tokens here. It behaves similarly to top-p but is more forgiving at high temperature, where top-p can end up including a long stretch of individually implausible tokens that collectively fit under the budget.

Repetition penalties

Truncation does not by itself stop loops. Three related mechanisms attack repetition directly:

Set any of these too high and you get a characteristic failure: the model starts avoiding common function words, because the and of have naturally appeared many times. Output becomes strained and slightly foreign. If generated text is drifting into odd phrasing, an over-aggressive repetition setting is a likely culprit.

Beam search: keeping options open

Beam search directly attacks greedy's first failure. Instead of one running candidate, keep the best $B$ partial sequences at every step, scored by cumulative log-probability.

Work through beam width 2 on a three-token vocabulary:

STEP 1 - score every first token  A: log p = -0.5      B: log p = -0.9      C: log p = -2.3  Keep the top 2 beams:  A (-0.5),  B (-0.9)STEP 2 - expand every beam by every token  From A (-0.5):            From B (-0.9):    A->X  -1.6  =>  -2.1      B->X  -0.4  =>  -1.3    A->Y  -0.9  =>  -1.4      B->Y  -1.8  =>  -2.7    A->Z  -2.0  =>  -2.5      B->Z  -2.2  =>  -3.1  All six candidates ranked:    BX -1.3   <-- best    AY -1.4    AX -2.1    AZ -2.5    BY -2.7    BZ -3.1  Keep the top 2:  BX (-1.3),  AY (-1.4)

Greedy would have chosen A at step 1 and never reconsidered, finishing at AY with −1.4. Beam search found BX at −1.3 — a better sequence reached through a worse first token. That is the entire value proposition.

Logs are used rather than raw probabilities for a practical reason: multiplying fifty probabilities each around 0.1 gives $10^{-50}$, which underflows in floating point. Adding fifty log-probabilities is numerically stable.

The length problem

Every extra token adds a negative number to the score, so longer sequences always score worse. Left uncorrected, beam search systematically prefers to stop early. The standard fix divides by a power of the length:

The exponent is a compromise: $\alpha = 0$ is no correction and favours short outputs; $\alpha = 1$ is a plain average and over-rewards long rambling ones.
Where beam search belongs — and where it does not
Beam search shines when there is essentially one right answer and the task is to find it: translation, speech transcription, constrained code generation, structured extraction. It is actively harmful for open-ended generation, and the reason is the point made earlier — it is a better maximiser, and maximising probability is the wrong target for creative text. Beam search output on a story prompt is famously flat and repetitive, precisely because it succeeds at finding high-probability text.
Choosing, and combining
Task
Settings that work
Reasoning
Factual question answering
Greedy, or $T = 0$–0.2
One correct answer; randomness only introduces error
Code generation
$T = 0.2$, top-p 0.95
Syntax is unforgiving; slight variation helps escape a bad first line
Summarisation
$T = 0.3$–0.5, top-p 0.9
Faithfulness matters more than style
General conversation
$T = 0.7$–0.8, top-p 0.9
The default balance for most assistants
Creative writing
$T = 0.9$–1.1, top-p 0.95
Surprise is the point; occasional oddity is acceptable
Brainstorming variants
$T = 1.0$, top-p 0.98, sample $n$ times
You want spread across the space, then pick
Translation, transcription
Beam search, width 4–5, $\alpha = 0.6$
Single target; global optimisation pays
In real implementations these filters compose, and the order matters. The conventional pipeline is:
Python
Copy
1import torch23def sample_next(logits, generated_ids, temperature=0.8, top_k=0, top_p=0.9,4                repetition_penalty=1.1):5    # 1. repetition penalty on raw logits6    for tok in set(generated_ids):7        if logits[tok] > 0:8            logits[tok] /= repetition_penalty9        else:10            logits[tok] *= repetition_penalty      # sign-correct1112    # 2. temperature13    logits = logits / temperature1415    # 3. top-k truncation16    if top_k > 0:17        kth = torch.topk(logits, top_k).values[-1]18        logits[logits < kth] = float("-inf")1920    # 4. top-p truncation21    if top_p < 1.0:22        sorted_logits, sorted_idx = torch.sort(logits, descending=True)23        cum = torch.softmax(sorted_logits, dim=-1).cumsum(dim=-1)24        remove = cum - torch.softmax(sorted_logits, dim=-1) >= top_p25        logits[sorted_idx[remove]] = float("-inf")2627    # 5. sample from what survives28    probs = torch.softmax(logits, dim=-1)29    return torch.multinomial(probs, num_samples=1)
Applying temperature before truncation means the truncation sees the reshaped distribution — a high temperature widens the nucleus, so temperature and top-p interact rather than acting independently. Reversing the order gives materially different behaviour, which is one reason the same nominal settings can produce different results across libraries.
Sampling on reasoning models
Everything above assumes you control the sampler and that the model answers straight away. Reasoning models — trained to write a long chain of thought before the answer, as the lesson on training stages describes — change both assumptions.

Where you do control the sampler — open-weight reasoning models you host yourself — the advice for factual tasks flips. Qwen3's model card recommends temperature 0.6, top-p 0.95 and top-k 20 in thinking mode, and warns against greedy decoding because it "can lead to performance degradation and endless repetitions". That is the repetition trap from earlier in this lesson at a larger scale: a thinking trace runs to thousands of tokens, and greedy decoding gets thousands of chances to fall into a loop it cannot leave. Moderate temperature is what keeps the trace moving.
So the task table above describes non-reasoning models. For a reasoning model, start from the settings on its model card, and control cost and depth with its thinking or effort setting rather than with temperature.
Making this useful
When output quality disappoints, change the decoder before you change the model or the prompt. It is free, instant, and a large share of "the model is bad at this" complaints are actually "the sampler is misconfigured for this". Loops mean temperature is too low or a repetition penalty is missing. Nonsense and invented facts mean temperature is too high or top-p is too permissive. Flat, generic, identical-every-time answers mean temperature is too low.
When you need reproducibility — regression tests, evaluations, anything with a fixed expected output — use greedy or $T = 0$ on models that allow it and are not reasoning models. But be aware that even greedy decoding is not bit-identical across hardware or batch sizes, because floating-point reduction order changes and two logits that differ in the eighth decimal place can swap ranks. Test against behaviour, not against exact strings.
And when you are measuring anything about a model — benchmark scores, hallucination rates, a comparison between two systems — record the decoding parameters alongside the numbers. A model at $T = 0$ and the same model at $T = 1.0$ can differ by many points on the same benchmark. A comparison that does not hold decoding fixed is not measuring the models.