# What is Tokenization? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Natural Language Processing  
> **Canonical Citation:** [https://spidits.com/ai-glossary/tokenization](https://spidits.com/ai-glossary/tokenization)

## Definition
Tokenization is the process of breaking down a text string into smaller pieces called tokens (which can be characters, subwords, or full words). Tokenization converts text into numbers that a neural network can process.

## Architectural Mechanics & Deep Dive
Tokenization is the pre-processing step that converts raw text strings into a sequence of integer token IDs that a neural network can process. Using algorithms like Byte-Pair Encoding (BPE), tokenization maps words and characters to an established vocabulary index, balancing vocabulary size against text coverage.

## Industry Applications & Real-World Use Cases
LLM vocabulary building, text pre-processing, and prompt encoding

## Frequently Asked Questions

### Why do LLMs use subword tokenization instead of word tokenization?
To handle spelling mistakes and compound words without generating a massive, unmanageable vocabulary size.

### What are Byte-Pair Encoding (BPE) and WordPiece?
Popular subword tokenization algorithms used by modern models like GPT (BPE) and BERT (WordPiece).

## Related Terminology
- [llm](https://spidits.com/ai-glossary/llm)
- [context-window](https://spidits.com/ai-glossary/context-window)
- [nlp](https://spidits.com/ai-glossary/nlp)

---
*From the SPIDITS AI Knowledge Base (300+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*