# What is Tokenizer? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Natural Language Processing  
> **Canonical Citation:** [https://spidits.com/ai-glossary/tokenizer](https://spidits.com/ai-glossary/tokenizer)

## Definition
A Tokenizer is a pre-processing component that breaks down raw text strings into discrete units called tokens (words, subwords, or characters) and maps them to numerical integer IDs that can be processed by a neural network.

## Architectural Mechanics & Deep Dive
A tokenizer is the software component that implements the tokenization algorithm. It maintains the vocabulary mapping index and splits input text into token IDs before passing them to the model, and conversely decodes predicted token IDs back into readable human language strings during output generation.

## Industry Applications & Real-World Use Cases
Text pre-processing, text generation decoding, and vocabulary indexing.

## Frequently Asked Questions

### What is a subword tokenizer?
A tokenizer that splits unfamiliar words into smaller fragments (e.g., "tokenizing" into "token" and "izing"), helping handle out-of-vocabulary terms.

### What happens during detokenization?
The tokenizer converts the model's output integer IDs back into human-readable text strings.

## Related Terminology
- [token](https://spidits.com/ai-glossary/token)
- [tokenization](https://spidits.com/ai-glossary/tokenization)
- [nlp](https://spidits.com/ai-glossary/nlp)

---
*From the SPIDITS AI Knowledge Base (300+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*