# What is Differential Attention? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Model Architecture  
> **Canonical Citation:** [https://spidits.com/ai-glossary/differential-attention](https://spidits.com/ai-glossary/differential-attention)

## Definition
Differential Attention is an attention operator that calculates the difference between two separate softmax attention maps to cancel out background noise and concentrate focus on contextually critical tokens.

## Architectural Mechanics & Deep Dive
Differential Attention is an attention operator introduced in the Differential Transformer (Diff Transformer) paper by Microsoft Research. Traditional softmax attention often suffers from "over-attention," dispersing attention weights across irrelevant tokens and causing hallucinations. Differential Attention splits query and key projections into two groups and computes the difference between two separate softmax attention maps. This subtraction acts as a high-pass filter, canceling common-mode background noise and amplifying attention toward essential tokens.

## Industry Applications & Real-World Use Cases
Long-context LLM retrieval, hallucination mitigation, and attention noise suppression

## Frequently Asked Questions

### What is Differential Attention in neural networks?
Differential Attention is an attention mechanism created by Microsoft Research that subtracts one softmax attention map from another to eliminate irrelevant noise.

### What problem does Differential Attention solve?
Standard attention maps often assign non-zero probability mass to irrelevant background tokens; differential attention cancels out this noise, improving in-context retrieval and reducing hallucinations.

## Related Terminology
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [transformer](https://spidits.com/ai-glossary/transformer)
- [self-attention](https://spidits.com/ai-glossary/self-attention)
- [needle-in-a-haystack](https://spidits.com/ai-glossary/needle-in-a-haystack)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*