Token
A Token is the fundamental unit of text sequence analyzed or generated by a natural language model (roughly equal to 3/4 of a word). Words are encoded into token IDs before passing into neural layers.
Frequently Asked Questions
What is a typical token-to-word conversion ratio?▼
About 100 tokens correspond to approximately 75 words in standard English text.
Why not use characters as tokens directly?▼
Character tokenization leads to extremely long sequences for the model to process, increasing compute overhead, while word tokenization creates a vocabulary list too large to index efficiently.
Quick Facts
- CategoryNatural Language Processing
- Key ApplicationPrompt sizing limits, vocabulary mapping, and cost metric usage billing
Coverage Trend12 Weeks
Related AI Terms
Token Media Coverage & Intelligence
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Three-dimensional (3D) brain MRI is central to clinical neurology and neuro-oncology, where generative models could augment under-represented cohorts, simulate
Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese
Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting phys
Choosing the Right NVIDIA Platform for Running Inference on CoreWeave
Explore the ideal NVIDIA GPUs for running inference on CoreWeave-optimize latency, reduce token cost, and match your model to the ideal GPU for real-time performance.
NEA's Tiffany Luck says enterprises are still figuring out their AI ROI
Tokenmaxxing was the hottest trend in Silicon Valley earlier this year, with CEOs encouraging employees to push AI usage as far as it would go. Then the bill ca
Getting more from each token: How Copilot improves context handling and model routing
How GitHub Copilot is making more of each session go toward useful work, so your credits go further. The post Getting more from each token: How Copilot improves
Anthropic ships major Claude Design overhaul with design system imports, code round-trips, and a fix for its token-burning problem
When Anthropic quietly released Claude Design in April as a " research preview ," it generated the kind of instant traction most product teams dream about: more
NEA's Tiffany Luck on AI IPOs, personal agents, and the ROI reckoning
Tokenmaxxing was the hottest trend in Silicon Valley earlier this year, with CEOs encouraging employees to push AI usage as far as it would go. Then the bill ca
Why a bigger context window won't fix your agent's memory
Context windows have grown fast. Models that once capped out at a few thousand tokens now advertise hundreds of thousands, and the natural assumption was that the agent memory problem would shrink as the window grew. Stuff more into the prompt, the th...
Anthropic "pauses" token-based billing for its Claude Agent SDK
Move originally planned for Monday would have heavily increased power users' costs.
'Pretty Crazy' Token Usage Is Testing Bosses' Bet on AI
A Silicon Valley software maker and an ecommerce company reveal to WIRED how they are navigating the emerging challenge of "tokenomics."
Your agent hit the context limit. Here's the playbook
Your agent was humming along, then it stopped mid-task and declared the job done when it wasn't. One common cause is the AI context limit, a hard ceiling on how many tokens a model can hold in its working memory at once. The frustrating part: this oft...
Kimi K2.7-Code cuts thinking tokens 30% - but practitioners say the benchmarks don't check out
Moonshot AI released Kimi K2.7-Code this week, an open-source update to its K2 coding model family, claiming leaner reasoning and double-digit performance gains
PixelRAG beats text parsers on accuracy and cuts AI agent token costs 10x
Most enterprise RAG pipelines start the same way: a text parser converts web pages and documents into plain text so they can be chunked and indexed for retrieva
FinOps AI governance demands new KPIs as token economics reshape enterprise cost models
FinOps AI governance is being stress-tested as AI spending accelerates across enterprises. The familiar levers of cost optimization - tagging, rightsizing and reserved capacity - are proving insufficient against a cost model governed by tokens, opaque billing and architectures that shift faster than
Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit
Context windows are becoming a computational bottleneck. The longer an agent runs, the more tokens accumulate from retrieved documents, reasoning traces and con
Google's DiffusionGemma generates 256 tokens in parallel and self-corrects as it goes
GenAI image generators like Stable Diffusion do not draw a picture pixel by pixel from left to right. They start with noise and iteratively refine the entire im
Context windows in AI: why every token is a budget decision
Some of today's most capable LLMs now support very large context windows. That doesn't mean you should fill them. Context windows have grown fast, but the underlying cost and quality tradeoffs haven't gone away. They've just gotten easier to ignore. ...
Is this the dawn of the Tokenpocalypse?
We're likely to see more price increases as the big AI companies plan to go public.
The token bill comes due: Inside the industry scramble to manage AI's runaway costs
"The whole conversation shifted from tokenmaxxing and 'go fast' to 'we need guardrails, how do we control this?'"
Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation
Modern reasoning models can allocate different amounts of test-time computation, such as thinking tokens, model
Meta's AI agent for WhatsApp Business is now available globally
WhatsApp will charge businesses for using its AI agent based on token usage.
Alibaba's Qwen3.7-Plus supports text, video and imagery inputs at low cost of $0.4/$1.6 per 1M token - but it's proprietary
Alibaba this week released Qwen3.7-Plus , the latest AI large language model (LLM) in its globally beloved and increasingly expansive Qwen family, boasting more