# What is Token? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Natural Language Processing  
> **Canonical Citation:** [https://spidits.com/ai-glossary/token](https://spidits.com/ai-glossary/token)

## Definition
A Token is the fundamental unit of text sequence analyzed or generated by a natural language model (roughly equal to 3/4 of a word). Words are encoded into token IDs before passing into neural layers.

## Architectural Mechanics & Deep Dive
A token is the basic unit of text processed by a Large Language Model. A token can represent a single character, a subword fragment, or a whole word. For example, the word "artificial" might be split into multiple tokens. Models read and generate text token by token, directly impacting prompt limits and cost metrics.

## Industry Applications & Real-World Use Cases
Prompt sizing limits, vocabulary mapping, and cost metric usage billing

## Frequently Asked Questions

### What is a typical token-to-word conversion ratio?
About 100 tokens correspond to approximately 75 words in standard English text.

### Why not use characters as tokens directly?
Character tokenization leads to extremely long sequences for the model to process, increasing compute overhead, while word tokenization creates a vocabulary list too large to index efficiently.

## Related Terminology
- [tokenization](https://spidits.com/ai-glossary/tokenization)
- [llm](https://spidits.com/ai-glossary/llm)
- [context-window](https://spidits.com/ai-glossary/context-window)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*