What is Distillation?
Distillation
Knowledge Distillation is a compression technique where a smaller model (the student) is trained to replicate the behavior and output probabilities of a much larger model (the teacher). This transfers reasoning capabilities into smaller footprints.
Why It Matters for AI Builders
Helps AI builders design and scale robust architectures; mastering the implementation of Distillation improves latency, accuracy, and operational efficiency for edge model creation, mobile-device ai deployment, and inference cost reduction.
Detailed Deep Dive
Knowledge distillation is a model compression technique where a small, efficient model (the student) is trained to replicate the behavior and performance of a large, complex model (the teacher). Instead of training on hard labels, the student model is trained on the "soft probabilities" output by the teacher, capturing dark knowledge about how the teacher generalizes. This enables deploying lightweight, low-latency models on edge devices.
Frequently Asked Questions
Q:Why is a distilled model better than a model trained from scratch?
Because the student model learns from the rich probability distributions (soft labels) of the teacher, capturing nuances that raw text datasets lack.
Q:Give an example of a distilled model.
DistilBERT, which is 40% smaller than BERT but retains 97% of its language understanding performance.
Quick Facts
- CategoryModel Training
- Key ApplicationEdge model creation, mobile-device AI deployment, and inference cost reduction
Coverage Trend12 Weeks
Cite This Term
Reference this definition in your articles, research, or documentation to credit this source:
Distillation Media Coverage & Intelligence
Senior White House official accuses Moonshot AI of copying Anthropic's leading frontier model
A senior White House official has accused Chinese artificial intelligence startup Moonshot AI of using so-called "distillation" techniques to essentially pirate the capabilities of Anthropic PBC's most powerful models in its own, open-source AI systems. "We have information that Moonshot AI...
Treasury Threatens Sanctions After White House Claims Moonshot Distilled Anthropic's Fable
Treasury Secretary Scott Bessent warned the U.S.
Exploring Self-distilled Reasoning for Supervised Fine-tuning with Amazon Nova
In this post, we explore an idea for generating thinking token for dataset that lack reasoning traces in SFT customization.
Building bilingual NER for cargo logistics with Amazon Bedrock
In this post, we share the technical approach using token-based distillation, lessons learned, and deployment architecture. If you face similar bilingual NER...