# What is Attention Sink? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Neural Architectures  
> **Canonical Citation:** [https://spidits.com/ai-glossary/attention-sink](https://spidits.com/ai-glossary/attention-sink)

## Definition
An Attention Sink is a phenomenon where autoregressive LLMs focus a massive amount of attention weights on the first few tokens of a sequence, regardless of their semantic meaning. Keeping these tokens in the cache prevents performance collapse in long conversations.

## Architectural Mechanics & Deep Dive
Attention Sinks refer to the phenomenon in autoregressive language models where the first few tokens in a sequence receive disproportionately high attention scores, acting as a mathematical dumping ground for softmax normalization. By preserving these initial tokens in the key-value cache, developers can implement sliding-window context caches that enable infinite sequence generation without crash.

## Industry Applications & Real-World Use Cases
Infinite context window streaming, persistent chatbot hosting, and memory cache tuning

## Frequently Asked Questions

### Why do attention sinks occur?
Because the Softmax function requires attention weights to sum to 1, and the initial tokens act as a "dumping ground" for unnecessary attention.

### How is this phenomenon used in streaming LLMs?
By keeping the first few tokens (the sink) permanently cached along with a sliding window of recent tokens, allowing infinite generation without retraining.

## Related Terminology
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [kv-cache](https://spidits.com/ai-glossary/kv-cache)
- [autoregressive-model](https://spidits.com/ai-glossary/autoregressive-model)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*