# What is Multi-head Latent Attention? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Model Architecture  
> **Canonical Citation:** [https://spidits.com/ai-glossary/multi-head-latent-attention](https://spidits.com/ai-glossary/multi-head-latent-attention)

## Definition
Multi-head Latent Attention (MLA) is a memory-efficient attention mechanism that compresses Key-Value (KV) vectors into a low-rank joint latent space during inference, dramatically reducing KV cache memory footprint while maintaining full Multi-Head Attention modeling capacity.

## Architectural Mechanics & Deep Dive
Multi-head Latent Attention (MLA) is an architectural innovation popularized by DeepSeek (in DeepSeek-V2/V3/R1). Standard Multi-Head Attention (MHA) requires storing massive Key and Value tensors in GPU VRAM during inference, creating a memory bottleneck for long context windows. MLA solves this by compressing Key and Value states into a joint low-rank latent vector. During attention computation, the model projects the latent vector back into key and value spaces on the fly, paired with a decoupled Rotary Position Embedding (RoPE) strategy. This reduces KV cache size by over 90% while preserving model expressivity.

## Industry Applications & Real-World Use Cases
High-throughput LLM inference, long-context window optimization, and GPU RAM compression

## Frequently Asked Questions

### What is Multi-head Latent Attention (MLA)?
MLA is an attention architecture introduced by DeepSeek that compresses key and value vectors into a low-rank latent vector, drastically lowering KV cache memory during inference.

### How does MLA compare to Grouped-Query Attention (GQA)?
While GQA reduces KV cache by sharing key/value heads across query heads, MLA projects keys and values into a unified low-rank latent space, achieving even smaller cache memory without sacrificing model performance.

## Related Terminology
- [kv-cache](https://spidits.com/ai-glossary/kv-cache)
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [transformer](https://spidits.com/ai-glossary/transformer)
- [deepseek](https://spidits.com/ai-glossary/deepseek)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*