# What is KV Cache? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Model Operations  
> **Canonical Citation:** [https://spidits.com/ai-glossary/kv-cache](https://spidits.com/ai-glossary/kv-cache)

## Definition
A KV Cache (Key-Value Cache) is an inference-time optimization storing the computed Key and Value attention tensors of past tokens to prevent redundant recalculations in autoregressive decoding.

## Architectural Mechanics & Deep Dive
KV cache (Key-Value cache) is a performance optimization technique used during autoregressive decoding in Transformer models. Because generating text sequentially requires re-evaluating the attention matrices of past tokens, KV cache stores the calculated Key and Value vectors of previous tokens in GPU memory. This avoids redundant calculations at each step, significantly accelerating inference speeds.

## Industry Applications & Real-World Use Cases
Response generation acceleration, chatbot latency reduction, and long context queries.

## Frequently Asked Questions

### What problem does KV caching solve?
It avoids computing attention scores quadratically, reducing the process to linear time during token generation.

### What is the best way to optimize KV cache memory usage for long-context LLM inference?
Optimizations include PagedAttention (dynamic memory allocation), quantization (storing KV cache tensors in FP8 or INT4), and eviction policies (pruning less critical attention weights) to free up VRAM.

## Related Terminology
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [token](https://spidits.com/ai-glossary/token)
- [context-window](https://spidits.com/ai-glossary/context-window)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*