# What is Distillation? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Model Training  
> **Canonical Citation:** [https://spidits.com/ai-glossary/distillation](https://spidits.com/ai-glossary/distillation)

## Definition
Knowledge Distillation is a compression technique where a smaller model (the student) is trained to replicate the behavior and output probabilities of a much larger model (the teacher). This transfers reasoning capabilities into smaller footprints.

## Architectural Mechanics & Deep Dive
Knowledge distillation is a model compression technique where a small, efficient model (the student) is trained to replicate the behavior and performance of a large, complex model (the teacher). Instead of training on hard labels, the student model is trained on the "soft probabilities" output by the teacher, capturing dark knowledge about how the teacher generalizes. This enables deploying lightweight, low-latency models on edge devices.

## Industry Applications & Real-World Use Cases
Edge model creation, mobile-device AI deployment, and inference cost reduction

## Frequently Asked Questions

### Why is a distilled model better than a model trained from scratch?
Because the student model learns from the rich probability distributions (soft labels) of the teacher, capturing nuances that raw text datasets lack.

### Give an example of a distilled model.
DistilBERT, which is 40% smaller than BERT but retains 97% of its language understanding performance.

## Related Terminology
- [models](https://spidits.com/ai-glossary/models)
- [llm](https://spidits.com/ai-glossary/llm)
- [inference](https://spidits.com/ai-glossary/inference)

---
*From the SPIDITS AI Knowledge Base (300+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*