# What is FlashAttention? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Neural Architectures  
> **Canonical Citation:** [https://spidits.com/ai-glossary/flashattention](https://spidits.com/ai-glossary/flashattention)

## Definition
FlashAttention is a memory-efficient, exact self-attention algorithm that speeds up Transformer training and inference by tiling computations in GPU SRAM and avoiding HBM access.

## Architectural Mechanics & Deep Dive
FlashAttention is a highly optimized, hardware-aware exact attention algorithm that accelerates Transformer models. By restructuring attention calculation to process data in blocks and fit within fast GPU SRAM memory (avoiding frequent reads/writes to slower High Bandwidth Memory), FlashAttention dramatically reduces memory footprint and training latency, enabling longer context windows.

## Industry Applications & Real-World Use Cases
Multi-million token context training, attention training acceleration, and memory footprint reduction.

## Frequently Asked Questions

### How does FlashAttention optimize GPUs?
It avoids writing large intermediate attention matrices to GPU High Bandwidth Memory (HBM), utilizing faster SRAM.

### Does FlashAttention lose precision?
No, it computes exact mathematical attention, not an approximation like sparse attention.

## Related Terminology
- [attention-mechanism](https://spidits.com/ai-glossary/attention-mechanism)
- [gpu](https://spidits.com/ai-glossary/gpu)
- [context-window](https://spidits.com/ai-glossary/context-window)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*