# What is Mixture of Depths? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Neural Architectures  
> **Canonical Citation:** [https://spidits.com/ai-glossary/mixture-of-depths](https://spidits.com/ai-glossary/mixture-of-depths)

## Definition
Mixture of Depths (MoD) is a compute optimization technique where models dynamically route and process only a fraction of tokens through specific layers, skipping computation for simpler tokens.

## Architectural Mechanics & Deep Dive
Mixture of Depths (MoD) is an optimization technique for Transformer models that dynamically allocates computational resources per token. Rather than passing every token through every layer, MoD uses routing algorithms to skip layers for simple tokens (like punctuation) and reserve full layer depth for complex, semantic tokens, speeding up inference.

## Industry Applications & Real-World Use Cases
Compute budget optimization, inference speedups.

## Frequently Asked Questions

### How does MoD optimize computing resources?
It skips layer operations for easy-to-predict tokens, prioritizing power for complex ones.

### How does MoD differ from Mixture of Experts (MoE)?
MoE routes tokens to different experts within a layer. MoD routes tokens past entire layers.

## Related Terminology
- [mixture-of-experts](https://spidits.com/ai-glossary/mixture-of-experts)
- [inference](https://spidits.com/ai-glossary/inference)
- [transformer](https://spidits.com/ai-glossary/transformer)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*