Must the teacher be a single model?

No. An ensemble can provide the teaching signal.

Do I need the teacher's weights?

Not always. Some methods use accessible outputs, though the available signal constrains the method.

Does distillation remove the need for real labels?

Not necessarily. Teacher outputs can complement labels, but trusted labels and independent evaluation remain valuable, especially when the teacher makes systematic mistakes.

AI Glossary / Generative AI and language / Intermediate

Distillation

Definition

Distillation is training a student model using information supplied by a teacher model or ensemble. Often the student is smaller or cheaper to run and learns to approximate the teacher's behavior, although distillation can also be used between models of similar size.

What it means

The teacher can provide more than a single correct-looking answer. In classification, its probability distribution shows how it rates alternatives. That pattern can contain useful information about similarities between classes that a one-hot label does not express.

The student learns from examples and a chosen objective; it is not simply a compressed copy of the teacher's parameter file. Its architecture can differ, and it may not reproduce every capability, confidence pattern or failure mode. What transfers depends on the training data, teacher signal, student capacity and optimization.

The word “knowledge” is a useful shorthand for learned behavior. It should not suggest that a complete, neatly packaged body of facts moves intact between models. The student's performance still needs independent evaluation for the task it will perform.

How it works

Choose representative transfer inputs and obtain teacher outputs, such as class probabilities, logits, intermediate features or generated responses. Train the student to match the selected signal. When trusted labels are available, the training objective can combine teacher-matching loss with ordinary supervised loss.

In classical probability-based distillation, a temperature parameter softens the teacher's softmax distribution, making relative scores among less likely classes more visible. The student uses the same temperature for the matching objective. At deployment, prediction generally uses the ordinary temperature appropriate to the trained model and task. The training temperature should not be confused with a user-facing creativity setting.

Language-model distillation may instead use teacher-generated text as training examples. This transfers a different form of signal from matching full token distributions. Access to only completed answers does not reveal the teacher's entire probability distribution.

Measure the student against both the teacher and an equivalent student trained without distillation. This helps determine whether the transfer provided value and whether the final model meets the required quality, latency and resource constraints.

Three practical examples

A vision team trains a smaller classifier from an ensemble's output distributions for use on a constrained device.
A support team trains a task-focused student on reviewed examples generated by a more capable teacher.
A speech system transfers information from a computationally expensive model into a model designed for faster inference.

Think of it this way

Think of an experienced tutor showing a learner not only the preferred answer but also which alternatives are close and why distinctions matter. That richer teaching signal can help the learner organize the task.

The analogy has limits. A teacher model can be wrong, and matching its outputs does not imply understanding its reasons. A learner with less capacity may also lose distinctions that were important in the teacher's behavior.

Worked example

Comparing a hard label with a soft target

Consider three invented classes: invoice, receipt and letter. The trusted hard label is invoice, represented as (1, 0, 0). A teacher distribution for the same example is (0.70, 0.25, 0.05), showing that receipt is a more plausible alternative than letter.

Suppose the student predicts (0.60, 0.30, 0.10). The cross-entropy against the teacher distribution, using natural logarithms, is −[0.70 ln(0.60) + 0.25 ln(0.30) + 0.05 ln(0.10)], approximately 0.774. Against the hard label alone, it is −ln(0.60), approximately 0.511.

These losses represent different targets and should not be compared as if the smaller number proves a better model. The example illustrates the extra class-relationship information in a soft target. A full distillation setup also specifies temperature, loss weighting and the transfer dataset.

Illustrative case study

A fictional company, Maple Devices, wants a compact image classifier for offline stock checks. It distils a larger teacher using common product images, and the student's average accuracy looks close to the teacher's.

A category-level review reveals a larger gap on uncommon reflective packaging. The team expands the transfer examples for relevant operating conditions, compares against a student trained on labels alone and measures performance on the actual device. It also reviews cases where both models make the same mistake.

The case shows why agreement is not enough. A student can faithfully imitate a teacher's errors, and an overall score can hide weaknesses in the categories that matter to a deployment.

Common misconceptions and mistakes

Distillation is the same as quantization. Quantization changes numerical representation; distillation trains a model from a teacher signal. They can be combined.
The student must use the teacher's architecture. Architecture choices can differ.
The student always performs worse. Results depend on data and training; neither universal superiority nor universal inferiority is guaranteed.
Matching the teacher proves correctness. The teacher can be wrong.
A smaller parameter count guarantees lower end-to-end latency. Hardware, architecture, sequence length and implementation also matter.

Business use

Distillation can support lower-cost deployment, specialized models and reduced dependence on a large model at inference time. Its value depends on the total process, including generating teacher data, training, evaluation and maintaining the student after requirements change.

Define which capabilities must survive the transfer. Test rare but important cases, calibration and relevant behavior boundaries alongside average task scores. A distilled model is a new model version requiring its own evidence, rather than inheriting every assurance claimed for its teacher.

Check your understanding

A student agrees with its teacher on 98% of examples. Does that establish 98% accuracy against the true labels?

Answer: No. Agreement measures imitation. Both models can give the same incorrect answer, so accuracy must be assessed against appropriate reference labels or task outcomes.

Related terms

Logits
Temperature
Knowledge distillation
teacher model
student model
soft targets
model compression
quantisation
synthetic training data

Learn next

Compare a teacher distribution with a hard label on a small classification example. Then define a student evaluation that measures correctness, resource use and important subgroups separately. Read quantization next to understand how changing numerical precision differs from learning through a teacher.

Sources

Sources: Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network: https://arxiv.org/abs/1503.02531 Original paper: https://www.cs.toronto.edu/~hinton/absps/distillation.pdf Keras, knowledge distillation example: https://keras.io/examples/vision/knowledge_distillation/ Google Machine Learning Glossary, distillation: https://developers.google.com/machine-learning/glossary#distillation

Previous: Distant supervision Next: Distributed algorithm