# What is Induction Heads? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Mechanistic Interpretability  
> **Canonical Citation:** [https://spidits.com/ai-glossary/induction-heads](https://spidits.com/ai-glossary/induction-heads)

## Definition
Induction Heads are a specialized two-head attention circuit mechanism in Transformer neural networks that implement in-context pattern copying. A Previous Token Head attends to prior occurrences of a token, while an Induction Head looks back to locate tokens that succeeded it in previous contexts, allowing models to replicate multi-step patterns and complete zero-shot copying.

## Architectural Mechanics & Deep Dive
Induction Heads are specialized attention circuits in Transformer architectures that execute in-context pattern matching and copying. Discovered by Anthropic researchers, an Induction Head operates in tandem with a Previous Token Head: the Previous Token Head shifts attention to the token immediately following a target, while the Induction Head attends back to prior instances of the target token to predict what comes next. The formation of these heads during early pre-training epochs marks a critical phase change, driving the emergence of zero-shot in-context learning.

## Industry Applications & Real-World Use Cases
Mechanistic interpretability, in-context learning analysis, and model capability diagnostics

## Frequently Asked Questions

### What are Induction Heads in transformer models?
Induction Heads are two-head attention circuits that search context history for prior occurrences of the current token and copy the token that followed it, enabling models to perform pattern repetition.

### Why are Induction Heads important for in-context learning?
Research by Anthropic showed that the emergence of Induction Heads during pre-training directly coincides with a dramatic phase change in a model's ability to perform in-context learning.

## Related Terminology
- [transformer](https://spidits.com/ai-glossary/transformer)
- [self-attention](https://spidits.com/ai-glossary/self-attention)
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [deep-learning](https://spidits.com/ai-glossary/deep-learning)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*