Temperature and Top-P Explained: Controlling LLM Randomness

Pramod Dutta · Jun 28, 2026 · 13 min read

You send the exact same prompt to a large language model twice and get two different answers. The first time it feels magical. The second time, when you are trying to ship a feature that needs to behave predictably, it feels like a bug. It is not a bug. It is sampling, and the two dials that control it are called temperature and top-p. Almost every developer who works with the OpenAI API, Anthropic's Claude, Google Gemini, or a local model through Ollama eventually bumps into these parameters, tweaks them semi-randomly, and hopes for the best. That trial-and-error approach works until it does not, and then you are debugging why your extraction pipeline occasionally hallucinates a field or why your creative writing assistant keeps producing the same bland sentence.

This article takes the mystery out of both parameters. We will look at what actually happens inside the model when it picks the next token, how temperature reshapes the probability distribution, how top-p (also called nucleus sampling) trims the set of candidate tokens, why you almost never want to crank both at once, and what values to reach for in real production scenarios. By the end you will be able to set these knobs on purpose instead of by superstition.

How A Language Model Actually Picks The Next Word

Before you can control randomness, you need to know where it comes from. A large language model does not "decide" to write a word the way a human does. At every step it produces a giant list of scores, one score for every token in its vocabulary. Depending on the model, that vocabulary might hold anywhere from around fifty thousand to a few hundred thousand tokens. Those raw scores are called logits.

Logits by themselves are not probabilities. They can be negative, they can be large, and they do not sum to anything meaningful. To turn them into a proper probability distribution, the model runs them through a function called softmax. Softmax exponentiates each logit and then divides by the sum of all the exponentials, which guarantees every value lands between zero and one and the whole set adds up to one.

Here is the core of softmax written out in plain Python so you can see there is no magic involved.

import math

def softmax(logits):
    # Subtract the max for numerical stability, then exponentiate.
    max_logit = max(logits)
    exps = [math.exp(x - max_logit) for x in logits]
    total = sum(exps)
    return [e / total for e in exps]

logits = [3.2, 1.1, 0.7, -0.4]
probs = softmax(logits)
print([round(p, 3) for p in probs])
# [0.777, 0.095, 0.064, 0.021]

Once the model has this probability distribution, it has to actually choose a token. The simplest strategy is called greedy decoding: always pick the single most likely token. Greedy decoding is fully deterministic, but it tends to produce repetitive, dull text and can get stuck in loops. The alternative is sampling: treat the probabilities as a weighted lottery and draw a token according to those weights. Sampling is where randomness enters, and temperature and top-p are the two levers that shape the lottery before the draw happens.

The key insight to hold onto is this. The model's raw opinion about what comes next is fixed once it has processed your prompt. Temperature and top-p do not change what the model knows. They change how much freedom the sampler has to pick something other than the safest, highest-probability option.

Temperature: Reshaping The Whole Distribution

Temperature is a single number, usually somewhere between zero and two, that you divide the logits by before softmax runs. That one small change has an outsized effect on the shape of the resulting distribution.

When temperature is low, close to zero, dividing by a tiny number blows the logits up and makes the gaps between them enormous. After softmax, the top token gobbles up almost all the probability mass and everything else is starved. The output becomes sharp, focused, and nearly deterministic. When temperature is high, above one, dividing shrinks the logits toward each other, the gaps flatten out, and after softmax the probability is spread more evenly across many tokens. The output becomes diverse, surprising, and sometimes incoherent.

Here is temperature applied to the same logits from before.

import math

def softmax_with_temp(logits, temperature):
    scaled = [x / temperature for x in logits]
    max_logit = max(scaled)
    exps = [math.exp(x - max_logit) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

logits = [3.2, 1.1, 0.7, -0.4]

for t in [0.2, 1.0, 1.8]:
    probs = softmax_with_temp(logits, t)
    print(f"temp {t}: {[round(p, 3) for p in probs]}")

# temp 0.2: [1.0, 0.0, 0.0, 0.0]
# temp 1.0: [0.777, 0.095, 0.064, 0.021]
# temp 1.8: [0.53, 0.185, 0.152, 0.083]

Notice what happens across the three rows. At temperature 0.2 the leading token is a near-certainty and the tail collapses to nothing. At temperature 1.0 you get the model's native distribution untouched, because dividing by one changes nothing. At temperature 1.8 the mass has spread out, the second and third tokens are now genuinely in contention, and even the long-shot fourth token has a real chance of being chosen.

A useful mental picture is a landscape of hills. Low temperature carves the terrain into one towering peak surrounded by flat plains, so the sampler almost always rolls to the summit. High temperature erodes everything into gentle rolling hills of similar height, so the sampler could end up almost anywhere. There is no universally correct height. The right terrain depends entirely on the job.

Top-P: Sampling From The Nucleus

Top-p, also known as nucleus sampling, takes a completely different approach to controlling randomness. Instead of reshaping the probability of every token, it draws a cutoff line and throws away the unlikely tail entirely before sampling.

The procedure is straightforward. Sort every token by probability from highest to lowest. Walk down the sorted list adding up probabilities as you go. The moment the running total reaches or exceeds your top-p value, stop. Everything you have accumulated so far forms the nucleus, and everything below the line is discarded. You then renormalize the survivors so they sum to one and sample only from that smaller set.

The clever part is that the size of the nucleus is dynamic. It adapts to how confident the model is at each step.

def top_p_filter(probs, p):
    # Pair each probability with its original index, sort descending.
    indexed = sorted(enumerate(probs), key=lambda pair: pair[1], reverse=True)
    nucleus = []
    cumulative = 0.0
    for idx, prob in indexed:
        nucleus.append((idx, prob))
        cumulative += prob
        if cumulative >= p:
            break
    # Renormalize the survivors so they sum to one.
    total = sum(prob for _, prob in nucleus)
    return [(idx, prob / total) for idx, prob in nucleus]

probs = [0.6, 0.2, 0.12, 0.05, 0.03]
print(top_p_filter(probs, 0.9))
# [(0, 0.638), (1, 0.213), (2, 0.128), (3, 0.053)]

In this example, with top-p set to 0.9, the first four tokens together cross the 0.9 threshold, so the fifth token is cut out completely. Now imagine a different step where the model is extremely confident and the top token alone already carries 0.95 probability. With the same top-p of 0.9, the nucleus would contain just that one token, and sampling would be effectively deterministic. That adaptiveness is the whole point of nucleus sampling. When the model is sure, the pool of candidates shrinks automatically. When the model is genuinely uncertain and probability is spread thin, the pool grows to include more options.

This is precisely why many practitioners prefer top-p over a fixed alternative called top-k. Top-k always keeps exactly the k most likely tokens no matter what, which means it can either strangle a step where the model wanted variety or wave through junk on a step where the model was already certain. Top-p sidesteps that rigidity by keying off cumulative probability instead of a fixed count.

Temperature Versus Top-P: What Is The Real Difference

It is easy to lump these two together because both influence how random the output feels, but they operate on different things and it pays to keep them straight.

Temperature rescales the logits and therefore reshapes the entire distribution, including the long tail. Every token stays eligible, but their relative odds shift. Low temperature makes the top choice more dominant. High temperature levels the playing field.
Top-p leaves the relative probabilities alone within the kept set and instead truncates the distribution, deleting the least likely tokens outright before any sampling happens. It is a gate, not a rescaling.

Put differently, temperature is a volume knob on how strongly the model favors its top pick, while top-p is a filter that decides which picks are even allowed into the room. Temperature can never fully eliminate a bad token, only make it rare. Top-p can eliminate a bad token completely by pushing it outside the nucleus.

There is a subtle ordering detail worth knowing. When both are active in a typical inference stack, temperature is applied first to the logits, and then top-p filtering runs on the resulting probabilities. That order matters, because a high temperature flattens the distribution and pushes more tokens above the cutoff line, which effectively widens the nucleus. This coupling is exactly why turning both dials to extreme values at the same time so often produces mush. You are widening the candidate pool with temperature and simultaneously asking top-p to be permissive, and the two effects compound.

Practical Settings For Real Workloads

Enough theory. Here is where these numbers actually land when you are building something. Treat these as sensible starting points, not commandments, and always test against your own prompts and data.

Factual question answering, classification, data extraction, and structured JSON output. Reach for a low temperature, roughly 0.0 to 0.3, and leave top-p near 1.0 or slightly lower. You want the model to commit to its single best answer. Randomness here is not creativity, it is error. When you need the model to return a specific field or a valid category label, determinism is a feature.
Code generation. A low-to-moderate temperature, around 0.1 to 0.4, tends to work well. Code has strict syntax, and a high temperature invites subtle bugs, invented function names, and broken structure. A little variety can help when you are asking for alternative implementations, but the default should lean conservative.
General chat, summarization, and explanation. The middle band, roughly 0.5 to 0.8, gives you fluent, natural prose that does not read like a robot reciting the same sentence every time, while still staying grounded and on topic. This is the comfortable default for most assistant-style applications.
Creative writing, brainstorming, marketing copy, and idea generation. Push temperature up toward 0.8 to 1.1 to unlock genuine variety and unexpected turns of phrase. This is the one place where surprise is the goal rather than the risk.

A few operational rules keep you out of trouble. Change one parameter at a time so you can actually attribute the effect. Do not stack a high temperature on top of an aggressive top-p, because you will amplify the noise. Most model providers recommend tuning either temperature or top-p, not both, and holding the other at its neutral default. If reproducibility matters, for example in automated tests or evaluation harnesses, drive temperature to zero and, where the API supports it, pin a fixed seed so you get the same output run after run.

Calling The Knobs From Code

Setting these parameters in practice is usually a one-line change in your API request. The exact field names are consistent across most of the major providers, which makes switching between them relatively painless. Here is a compact example using the OpenAI-style Python client, which many other services also mimic.

from openai import OpenAI

client = OpenAI()

def ask(prompt, temperature=0.7, top_p=1.0):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        top_p=top_p,
    )
    return response.choices[0].message.content

# Deterministic, focused answer for an extraction task.
strict = ask("Extract the company name from: Acme Corp reported Q3 earnings.",
             temperature=0.0)

# Varied, imaginative output for a brainstorm.
creative = ask("Give me an unusual metaphor for machine learning.",
               temperature=1.0)

print(strict)
print(creative)

The pattern is the same whether you are calling Anthropic's Claude, Google Gemini, Mistral, or a local model served through Ollama or vLLM. The field names may differ slightly, but temperature and top-p are near-universal. If you run a model locally, you often get even finer-grained control over decoding, including top-k, repetition penalties, and the sampling seed, all exposed as request parameters.

One caution about defaults. Every provider ships its own default temperature, and it is frequently around 0.7 or 1.0, which is fine for chat but far too loose for anything that needs to be deterministic. Never assume the default is safe for your use case. If you are building an extraction service and you leave temperature at the provider default, you will eventually get a surprising output and spend an afternoon confused about why. Set the value explicitly in every request so your intent is visible in the code.

Common Mistakes And How To Avoid Them

Having watched a lot of developers wire up their first LLM feature, the same handful of errors show up again and again.

Cranking temperature to fight repetition. If a model keeps repeating itself, the instinct is to raise temperature until the loop breaks. That works, but it is a blunt instrument that degrades quality everywhere. A repetition penalty or a frequency penalty, where available, is a far more surgical fix that targets the actual problem without turning the rest of your output into chaos.
Tuning both temperature and top-p aggressively at once. As covered above, the two interact, and pushing both to extremes compounds the randomness in ways that are hard to reason about. Pick one as your primary control and leave the other at its neutral value.
Expecting temperature zero to be perfectly reproducible. Temperature zero gets you greedy decoding, which is deterministic in principle, but you can still see tiny variations across runs due to floating-point non-determinism on GPUs, load balancing across different hardware, and silent model updates on the provider side. If you need bit-for-bit reproducibility, combine temperature zero with a fixed seed and, ideally, a pinned model version.
Believing higher temperature makes the model smarter or more creative in a meaningful sense. It does not add knowledge or reasoning ability. It only widens the range of what the model is willing to say. Sometimes that reads as creativity, and sometimes it reads as the model confidently stating something false. The underlying competence is unchanged.
Ignoring the interaction with your prompt. Sampling parameters are not a substitute for a clear prompt. A vague prompt at low temperature just gives you a consistently mediocre answer. Fix the prompt first, then tune the sampling to taste.

Building A Mental Model You Can Trust

If you strip away all the detail, here is the compact version worth remembering. A language model turns your prompt into a probability distribution over its entire vocabulary. Temperature stretches or compresses that distribution: low makes the top choice dominate, high evens everyone out. Top-p slices off the improbable tail before sampling: low keeps only the most likely handful, high lets more candidates through. Lower settings buy you consistency and precision. Higher settings buy you variety and surprise. Neither is better in the abstract, and the entire skill is matching the setting to the task in front of you.

The developers who ship reliable LLM features are the ones who treat these parameters as deliberate engineering choices rather than magic incantations. They set temperature to zero for extraction and evaluation, they leave it in the comfortable middle for chat, and they open it up for creative work, and they document why in the code so the next person is not left guessing. That habit alone separates a flaky prototype from a dependable product.

If you want to go deeper into decoding strategies, sampling, prompt design, evaluation, and the full stack of skills needed to build production-grade AI systems, that is exactly what we cover in the AI Engineering Roadmap course on teachyou.ai. It walks you from the fundamentals of how these models generate text all the way through building, testing, and shipping real applications, with hands-on projects at every step. Temperature and top-p are just the first two dials. There is a whole console waiting once you know how the machine works underneath.