Quick takeaways
- Distillation makes sense when latency, cost, or device constraints matter more than the last point of benchmark performance.
- Use a large teacher to generate soft labels and a small student to learn from them; the student often beats the same model trained from scratch.
- Quantization and pruning compress an existing model; knowledge distillation transfers behavior. They can be stacked, but each adds engineering complexity.
- Evaluate distilled models on your own task distribution, not just public leaderboards.
Model training and distillation
Martin Thissen walks through fine-tuning open-source models, a related skill to model distillation.
When to distill vs. use a larger model
Start with the constraint. If a frontier model already meets your latency and cost budget, distillation is usually not worth the research cycle. Distill when inference volume, latency, privacy, or hardware limits make the large teacher impractical.
Do not distill just to chase a smaller parameter count. The goal is acceptable quality at a better cost or speed profile. Compare against the AI model API pricing guide before committing to a compression project.
Teacher-student training
The teacher is a large, capable model—often a frontier LLM or a fine-tuned specialist. The student is a smaller architecture or a scaled-down version of the same family. The student learns to imitate the teacher's outputs on a representative dataset.
Basic pipeline: Collect task inputs → Run them through the teacher → Capture full outputs (soft labels or reasoning traces) → Train the student on those outputs → Evaluate against a held-out set → Iterate on data quality and temperature sampling.
Data quality matters more than model size. A student trained on a small, clean, diverse dataset often outperforms one trained on a large, noisy dataset. Sample at non-zero temperature to give the student exposure to the teacher's distribution, not just greedy outputs.
Knowledge distillation
Standard fine-tuning trains on hard labels: the correct answer. Knowledge distillation trains on soft labels: the teacher's probability distribution across all possible outputs. This extra signal teaches the student about relationships between classes or tokens.
Loss combination: Most implementations blend distillation loss (student mimics teacher probabilities) with a small amount of supervised loss (student matches ground truth). The weighting depends on how noisy your labels are and how close the student's capacity is to the teacher.
For generative models, distillation can also mean training on the teacher's reasoning traces or chain-of-thought outputs. This works well for structured tasks like JSON extraction, classification, and routing decisions. Pair the result with the how to evaluate LLMs guide to confirm the student preserves reasoning quality.
Quantization
Quantization reduces the numerical precision of model weights and activations. A 32-bit float model can often run as INT8, INT4, or even lower without a meaningful quality drop, depending on the model and task.
Post-training quantization (PTQ): Fastest path. Convert a trained model to lower precision with calibration data. Good for immediate deployment speedups.
Quantization-aware training (QAT): More expensive. Train with simulated low-precision operations so the model learns to tolerate quantization noise. Often recovers more accuracy than PTQ.
Watch out for activation outliers, sensitive layers, and long-context degradation. Not all layers quantize equally. Mixed-precision schemes keep critical layers at higher precision while compressing the rest.
Pruning
Pruning removes weights, attention heads, or entire layers that contribute little to the output. The goal is a sparser or smaller model that runs faster and uses less memory.
Pruning is rarely a standalone solution for modern LLMs. It works best when combined with retraining, quantization, or distillation, and when the target hardware supports sparse operations.
Smaller model tradeoffs
Smaller models win on cost, latency, and deployability. They lose on generalization, long-tail knowledge, and the ability to follow complex, multi-step instructions without breaking.
Throughput
Students serve more requests per GPU and fit into cheaper instance types.
Generalization
Teachers handle out-of-distribution prompts better; students may fail on rare inputs.
Maintenance
Each distilled model is another artifact to version, evaluate, and retrain when the teacher improves.
Treat the tradeoff as a product decision, not just an accuracy metric. A student that is 5% worse on a benchmark but 10x cheaper may be the right choice for a high-volume feature.
Deployment patterns
Distilled models can replace the teacher entirely, act as a fast first-pass filter, or run alongside the teacher in a cascading architecture.
Student-only serving
Use the student for every request when quality and cost targets are both met.
Student-then-teacher
Route easy queries to the student and escalate uncertain ones to the teacher based on confidence or guardrails.
Student pre-filter
Use the student for classification, routing, or retrieval scoring before the teacher handles generation.
On-device student
Deploy quantized or distilled variants to laptops, phones, or private servers where cloud calls are not acceptable.
Start with the simplest pattern that meets your latency and quality bar. A cascade adds operational complexity: you now monitor two models, two failure modes, and a routing policy.
Evaluating distilled models
Benchmarks are a starting point, not the finish line. Evaluate the student on real inputs from production or a representative validation set that reflects your users and task distribution.
Metrics to track: Task accuracy, latency at p50/p95/p99, throughput, memory usage, cost per request, hallucination rate, and output format adherence. Compare each against the teacher and against the business constraint, not just against the original baseline.
Watch for error mode shifts. A student may preserve overall accuracy while becoming more confident on wrong answers or more brittle on edge cases. Human review of a sample of failures is essential before full rollout.
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Inference cost | Frontier models are used for every request, driving high API bills. | Distilled students handle narrow tasks at a fraction of the token cost. |
| Latency | Large models are too slow for real-time or high-volume features. | A smaller distilled model returns answers faster and fits cheaper hardware. |
| Deployment | Applications depend on a single provider's availability and policies. | Distilled weights can run on owned infrastructure, at the edge, or through multiple hosts. |
| Specialization | General models are prompted to behave like specialists. | A student is trained on the teacher's outputs for a specific task, often matching quality on that task. |
| Iteration speed | Teams wait for new frontier releases to improve performance. | Teams can retrain students on their own data and evaluation sets on their own schedule. |
FAQ
When should I choose distillation over prompt engineering or RAG?
Distill when inference cost or latency is the bottleneck and the task is narrow enough that a smaller model can match the teacher. Prompt engineering and RAG are usually faster first steps before committing to a training pipeline.
Can any model be distilled?
In practice, you need access to a capable teacher and a representative dataset. Closed-weight frontier models may restrict how you use their outputs, so check license and terms before distilling from them.
How much smaller can the student be?
It depends on the task. For narrow classification or extraction, students can be 10-100x smaller. For open-ended generation, compression ratios are more modest and usually require quantization or pruning on top.
Does distillation preserve safety guardrails?
Not automatically. A student may inherit the teacher's capabilities without inheriting its refusals or moderation behavior. Evaluate safety and alignment separately.
Should I quantize before or after distillation?
Usually after. Distill the student first to learn from the teacher, then quantize the trained student for deployment. Quantizing a teacher and then distilling can work but adds instability.
What hardware do I need to distill a model?
Training a student is far cheaper than training a frontier model from scratch, but you still need GPUs for any non-trivial LLM. LoRA and other parameter-efficient methods can reduce requirements significantly.
What is model distillation?
Training a smaller student model to imitate a larger teacher model, often preserving much of the teacher's capability on a narrow task.
When should I distill instead of using a frontier model?
When latency, cost, privacy, or device constraints matter more than the last point of benchmark performance.