What Is Model Distillation? Smaller Models, Similar Performance

Picture this. You have a massive language model that answers customer questions beautifully, reasons through tricky problems, and rarely embarrasses you in production. It is also expensive to run, slow to respond, and impossible to deploy on anything smaller than a rack of GPUs. Now your product team wants that same quality on a mobile app, at a fraction of the cost, with responses in under a second. That gap between what your best model can do and what you can actually afford to ship is exactly the problem model distillation was built to solve. It lets you take the knowledge locked inside a large, capable model and pour it into a smaller one that runs faster and cheaper while behaving almost the same. In this article we will walk through what distillation really is, why it works, the main techniques engineers use, and how you would run a distillation project yourself.

What Model Distillation Actually Means

Model distillation, sometimes called knowledge distillation, is the process of transferring the behavior of a large trained model into a smaller model. The big model is called the teacher. The small model is called the student. The goal is to make the student produce outputs that closely match the teacher, so you can retire the expensive teacher in production and serve the cheap student instead.

The term comes from a 2015 paper by Geoffrey Hinton and colleagues titled "Distilling the Knowledge in a Neural Network." The core insight was simple but powerful. A trained model knows far more than the single label it predicts. When a classifier looks at a photo of a dog, it does not just output "dog." It outputs a full probability distribution across every class, and that distribution carries information. The model might assign a small but non zero probability to "wolf" and an even smaller one to "cat," which tells you something about how the model perceives similarity between classes. Those soft probabilities are the real treasure. Distillation is mostly about teaching the student to reproduce that rich distribution rather than just the final hard answer.

Here is the key distinction. Regular training teaches a model from ground truth labels, which are usually one hot, meaning the correct class gets a value of one and everything else gets zero. Distillation teaches the student from the teacher's soft outputs, which contain shades of gray. Those shades encode what researchers call dark knowledge, the hidden structure the teacher learned that never shows up in the raw labels.

Why Smaller Models Can Keep Similar Performance

A natural objection is that a smaller model has fewer parameters, so surely it must be less capable. That intuition is only partly right. Large models are often heavily over parameterized during training. They need all that capacity to discover good solutions from scratch, sifting through noisy data and finding the patterns that matter. But once the patterns are found, representing them does not require nearly as many parameters. Distillation works because learning is harder than representing.

Think of it like this. A senior engineer spends years reading messy documentation, debugging production incidents, and forming mental models of how a system behaves. A well written onboarding guide can transfer a huge chunk of that hard won understanding to a junior engineer in a few weeks. The guide is far smaller than the sum of the senior's experience, yet it captures the useful conclusions. The teacher model is the senior engineer. The distilled student is the junior who got a great onboarding guide.

There are three concrete reasons the student can stay close to the teacher.

• The soft labels are more informative than hard labels, so the student learns from a richer signal per example and needs less data to reach good accuracy.
• The teacher has already filtered out most of the noise in the training data, so the student learns a cleaner target than the raw dataset would provide.
• The student is being asked to match a function that is known to be learnable, since the teacher already learned it, rather than to discover that function on its own.

None of this means distillation is magic. A student that is far too small will still lose accuracy, and some tasks that genuinely need enormous capacity resist compression. But for a large range of practical problems, a student that is five to ten times smaller can retain most of the teacher's quality.

The Core Mechanics: Soft Targets and Temperature

Let us get into how distillation actually works at the level of the loss function, because this is where most of the interesting engineering lives.

When a neural network produces a classification, the final layer outputs raw scores called logits. A softmax function turns those logits into probabilities. In normal use, softmax tends to produce a very peaky distribution, where the winning class gets something like 0.98 and everything else is squeezed near zero. Those near zero values are exactly the dark knowledge we care about, but they are so tiny that the student barely notices them during training.

The trick Hinton introduced is temperature. You divide the logits by a temperature value before applying softmax. A higher temperature softens the distribution, spreading probability mass so the smaller values become large enough to teach from. Here is what that looks like in practice.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.5):
    # Soft targets: match the teacher's softened distribution
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)

    # KL divergence pulls the student toward the teacher
    # Scaled by temperature squared to keep gradient magnitudes stable
    soft_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean")
    soft_loss = soft_loss * (temperature ** 2)

    # Hard targets: also learn from the real ground truth labels
    hard_loss = F.cross_entropy(student_logits, labels)

    # Blend the two objectives
    return alpha * soft_loss + (1 - alpha) * hard_loss

Two things deserve attention here. First, the loss blends two objectives. The soft loss pulls the student toward the teacher's full distribution, and the hard loss keeps the student anchored to the actual correct answers. The alpha parameter controls the balance. Second, the temperature squared factor exists because softening the logits shrinks the gradients, so multiplying by temperature squared restores them to a useful scale. Forget that factor and your student trains far too slowly.

Temperature is a knob you tune. A value between two and ten is common. Too low and you lose the dark knowledge. Too high and the distribution becomes so flat it stops carrying meaningful class relationships.

Major Types of Distillation

Distillation is not a single algorithm. It is a family of techniques, and knowing which one fits your situation matters. Here are the main varieties engineers reach for.

• Response based distillation. The student matches the teacher's final output, whether that is a probability distribution over classes or generated text tokens. This is the classic approach from the original paper and the easiest to implement.
• Feature based distillation. Instead of only matching final outputs, the student also matches the teacher's intermediate layer activations. This gives the student more guidance about how the teacher represents information internally, not just what it concludes. It usually needs an adapter layer to align the different shapes of teacher and student hidden states.
• Relation based distillation. The student learns to preserve relationships between examples or between layers, such as which inputs the teacher considers similar. Rather than matching absolute values, it matches the structure of the teacher's representation space.

There is also a practical split based on where the teacher's knowledge comes from.

1. Offline distillation. You start with a fully trained, frozen teacher and train the student against it. This is by far the most common setup because it is simple and the teacher never changes mid training.
2. Online distillation. Teacher and student train at the same time, which helps when no strong pretrained teacher exists yet, though it is more complex to orchestrate.
3. Self distillation. A model teaches itself, often by having deeper layers guide shallower ones, or by distilling from an earlier checkpoint of the same architecture into a later one.

For most teams shipping a product, offline response based distillation is the starting point. It is well understood, easy to debug, and produces strong results without exotic machinery.

Distilling Large Language Models

The classic theory was written for image classifiers, but the technique that everyone talks about today is distilling large language models. The mechanics shift a little when your model generates text rather than picking one label out of a thousand.

With a language model, every generated token is itself a classification over the entire vocabulary, which might be a hundred thousand possible tokens. So distillation happens token by token. At each position, the teacher produces a distribution over the vocabulary, and the student learns to match it. The temperature and KL divergence ideas carry straight over.

But there is a second, increasingly popular flavor specific to LLMs, and it does not need access to the teacher's internal logits at all. It works purely through generated text.

• You take a large, capable teacher model and prompt it to produce high quality answers, often with detailed step by step reasoning.
• You collect those answers into a dataset of prompt and response pairs.
• You fine tune a smaller student model on that dataset using ordinary supervised learning.

This is often called sequence level distillation or, more casually, data distillation. The student learns to imitate the teacher's outputs even though it only ever sees text, never probabilities. A huge advantage is that you can distill from a model you only access through an API, since you never need the raw logits. When people say they built a small model that "reasons like" a frontier model, this is frequently how they did it. They generated a large collection of worked examples from the big model and trained the small one to copy the style and substance.

A closely related idea is chain of thought distillation, where the teacher is prompted to show its reasoning, and the student is trained to reproduce both the reasoning steps and the final answer. Learning the intermediate steps, not just the conclusion, tends to make small models dramatically better at multi step problems than training on final answers alone.

A Practical Distillation Workflow

Enough theory. Here is how a distillation project actually unfolds, step by step, in a realistic setting where you want to shrink a costly model into something you can afford to serve.

1. Pick your teacher and confirm it is genuinely good. The student can only inherit quality the teacher already has. If your teacher is mediocre, distillation gives you a smaller mediocre model. Distillation copies behavior, it does not invent new capability.
2. Choose a student architecture and size. This is your main cost and latency lever. A common move is to keep the same model family but pick a version with far fewer parameters, so the tokenizer and general design stay compatible.
3. Assemble a transfer dataset. For LLM data distillation, this means gathering a large, diverse set of prompts that resemble real production traffic, then generating teacher responses for each. Coverage matters more than raw volume. The student will be weak on anything the transfer set never touched.
4. Train the student against the teacher. Use the blended loss for logit level distillation, or plain supervised fine tuning for text level distillation. Watch both the distillation loss and a held out quality metric.
5. Evaluate on tasks you actually care about, not just training loss. Build an evaluation set that mirrors your real use cases and compare student against teacher head to head. Track the gap explicitly so you know exactly what you traded away.
6. Iterate on the weak spots. Where the student underperforms, add more transfer examples covering those cases, adjust temperature or the alpha blend, or bump the student size slightly. Distillation is rarely one and done.

A short pseudo loop for the text level approach looks like this.

# Text-level distillation, high level sketch
teacher = load_teacher_model()          # large, expensive, frozen
student = load_small_base_model()        # the model you will actually ship

# 1. Generate the transfer dataset from real-looking prompts
transfer_data = []
for prompt in production_like_prompts:
    answer = teacher.generate(prompt, reasoning=True)
    transfer_data.append({"prompt": prompt, "target": answer})

# 2. Fine-tune the student to imitate the teacher's answers
for epoch in range(num_epochs):
    for batch in make_batches(transfer_data):
        outputs = student(batch["prompt"])
        loss = supervised_loss(outputs, batch["target"])
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

# 3. Compare student vs teacher on a held-out eval set
report = evaluate(student, teacher, held_out_eval)
print(report.quality_gap, report.cost_reduction, report.latency_reduction)

The important habit is measuring the trade every step of the way. You are exchanging some accuracy for large gains in cost and speed, and you want that exchange rate to be a decision, not a surprise.

Benefits, Costs, and When to Skip It

Distillation is a tool, not a religion. It shines in specific situations and wastes your time in others. Here is an honest accounting.

The benefits are compelling when they apply.

• Lower serving cost, since a smaller model uses less compute per request, which compounds across millions of calls.
• Lower latency, because fewer parameters mean faster inference, which matters enormously for interactive applications.
• Deployability, since a small student can run on edge devices, phones, or modest hardware where the teacher never could.
• Privacy and control, because a distilled model you own can run on your own infrastructure instead of calling an external API.

The costs and limits are just as real.

• You need a strong teacher first. If you do not already have one, distillation is not your starting point.
• You spend engineering effort building transfer data and evaluation harnesses, which is not free.
• The student almost always loses some accuracy. Usually a little, sometimes more, occasionally too much for the task.
• The student inherits the teacher's biases and mistakes, so a flawed teacher produces a flawed student.
• There can be legal and licensing questions when distilling from a model whose terms restrict using its outputs to train competitors, so check before you build.

When should you skip distillation? If your production model is already small and cheap enough, do not bother. If your accuracy requirements leave no room to give anything up, be cautious, and test aggressively before committing. And if you do not yet have a teacher whose behavior you would be happy to copy, invest in building that first. Distillation multiplies whatever quality you start with. It cannot create quality from nothing.

Distillation Versus Other Compression Methods

Distillation is one of several ways to make models smaller and faster, and it helps to know where it sits among the alternatives so you can combine them wisely.

• Quantization reduces the numerical precision of the model's weights, for example from sixteen bit floats down to eight or four bit integers. It shrinks memory and speeds up math without changing the architecture. It is often the easiest first win.
• Pruning removes parts of the network that contribute little, such as near zero weights or entire unimportant channels, leaving a sparser model.
• Distillation changes the model itself, training a genuinely different, smaller architecture to reproduce the behavior of the larger one.

These techniques are not rivals. In fact they compose beautifully. A very common production recipe is to distill a large teacher into a compact student, then quantize that student to shrink it further, and possibly prune it too. Each method attacks the size and speed problem from a different angle, and stacking them can yield a model that is a small fraction of the original size while holding on to most of the quality. If you remember one thing, let it be that distillation is about transferring learned behavior into a new architecture, while quantization and pruning shrink an existing one in place.

Bringing It All Together

Model distillation is one of the most practical ideas in applied machine learning because it directly attacks the tension every team faces between quality and cost. You train or obtain a large, capable teacher, then teach a small, efficient student to mimic it closely, using soft targets and temperature to transfer the rich dark knowledge that ordinary labels leave behind. For language models, you can go even further and distill purely through generated text, letting a compact model inherit the reasoning style of a frontier system it only ever saw through an API. The result is a model that runs faster, costs less, and fits where the original never could, while giving up only a modest slice of accuracy that you measured and chose to trade.

The mental model to carry with you is that learning is expensive but representing what was learned is cheap. Distillation exploits that gap. It does not conjure capability out of thin air, so it rewards teams that already have a strong teacher and a clear sense of the tasks they care about. Pair it with quantization and pruning, measure the trade offs at every step, and you have a reliable path from an impressive but impractical model to one you can actually ship.

If you want to go deeper on the engineering skills that make projects like this succeed, from designing evaluation harnesses to fine tuning and deploying efficient models in production, the AI Engineering Roadmap course on teachyou.ai walks through the full journey with hands on projects. It is built for engineers who want to move past theory and ship real systems, and distillation is exactly the kind of technique it prepares you to apply with confidence.