# What is Linear Attention? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Neural Architectures  
> **Canonical Citation:** [https://spidits.com/ai-glossary/linear-attention](https://spidits.com/ai-glossary/linear-attention)

## Definition
Linear Attention is a class of attention mechanisms designed to approximate the standard self-attention operation in linear time complexity relative to sequence length, bypassing the quadratic memory scaling limits of standard Transformers.

## Architectural Mechanics & Deep Dive
Linear attention is an optimization of the standard attention mechanism that reduces its computational and memory complexity from quadratic to linear with respect to sequence length. By changing the order of matrix multiplications, linear attention models process extremely long context windows efficiently, making them attractive for long-document and multimodal processing.

## Industry Applications & Real-World Use Cases
Infinite context length LLMs, long-sequence DNA analysis, and fast state-space models.

## Frequently Asked Questions

### Why does standard self-attention scale quadratically?
Because standard attention calculates similarity scores between every token and every other token in the sequence (an N x N matrix).

### What are examples of models utilizing linear complexity layouts?
State Space Models (like Mamba) or Linear Transformer variants that rewrite the attention matrix multiplication order.

## Related Terminology
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [context-window](https://spidits.com/ai-glossary/context-window)
- [transformer](https://spidits.com/ai-glossary/transformer)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*