What is Inference?
Inference
Inference is the process of using a trained AI model to make predictions or generate text based on new inputs. During inference, data flows forward through the neural network to produce an output, without modifying the model's weights.
Why It Matters for AI Builders
Helps AI builders design and scale robust architectures; mastering the implementation of Inference improves latency, accuracy, and operational efficiency for real-time chat responses, api query execution, and mobile-device ai features.
Detailed Deep Dive
Inference is the phase where a trained machine learning model runs in production to process new, unseen inputs and generate predictions or content. Unlike the computationally expensive training phase, inference does not update model weights; it simply passes the input through the network to calculate the output. Optimizing inference (e.g., using quantization or cache systems) is essential for reducing serving latency and hosting costs.
Frequently Asked Questions
Q:How does inference differ from training?
Training calculates errors and updates the model's weights. Inference just uses the already-calculated weights to process new data.
Q:Why is inference speed important?
Low latency inference is critical for interactive user experiences (e.g. conversational voice agents).
Quick Facts
- CategoryModel Operations
- Key ApplicationReal-time chat responses, API query execution, and mobile-device AI features
Coverage Trend12 Weeks
Related AI Terms
Cite This Term
Reference this definition in your articles, research, or documentation to credit this source:
Inference Media Coverage & Intelligence
Epistemic Sybil Resistance: Multiplying AI Agents Without Multiplying Evidence
Multi-agent AI systems improve inference by spawning agents and synthesizing reports. But another agent is not another observation: apparently independent...
Accessing OpenAI models on Amazon Bedrock from Australia with global cross-Region inference
Australian teams can now access OpenAI GPT-5.6 Sol, Terra, and Luna models on Amazon Bedrock with global cross-Region inference from the Asia Pacific...
Get Started with Kimi K3 on CoreWeave Dedicated Inference
Follow this step-by-step guide to deploy Kimi K3 on CoreWeave Dedicated Inference on NVIDIA GB300 NVL72.
Active Inference as Context Acquisition for AI Agents
Interactive AI agent must acquire the right context as efficiently as possible. When a user omits a constraint, preference, file, or task variable, an agent...
Introducing cross-Region inference for OpenAI GPT-5.6 models on Amazon Bedrock
Amazon Bedrock now offers OpenAI GPT-5.6 models (Sol, Terra, and Luna) in more than 25 AWS Regions with cross-Region inference. Learn how US geographic and...
Kog is going deeper to squeeze more inference out of GPUs
The idea that GPU are poorly suited for agentic workflows may be a misconception, according to French startup Kog.
Partnering with Preview: Lights, Inference, Action
The post Partnering with Preview: Lights, Inference, Action appeared first on Sequoia Capital .
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered...
Multi-tier storage rewrites the economics of AI inference
As inference becomes the dominant workload in AI infrastructure, multi-tier storage architectures are emerging as a key method for cost control and enhanced performance. These architectures combine flash, object storage and disk-based capacity tiers, enabling enterprises to serve training and...
Google Cloud launches Gemini 3.6 Flash with sub-50ms agentic inference speed
Google Cloud expanded the Gemini 3.6 lineup with Flash edition, engineered for high-frequency tool calls and real-time voice agents.
Powerful Compute So Compact, It's Clutch - Build AI Anywhere With NVIDIA Jetson
NVIDIA announces new Jetson Thor edge modules for autonomous robotics and real-time physical AI inference.
LLM Optimization Integration for Amazon SageMaker Python SDK
The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook.
OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking
Your GPU bill is rising. Your models are serving billions of token. Yet one question remains unanswered: what does each token actually cost? This is not a hypothetical problem. Platform teams today operate in a fog...
Is the future of data centers portable? Runware builds a pod to find out
On Tuesday, AI infrastructure company Runware announced the launch of its own modular data center called Sonic Inference Pod.
Netflix co-founder backs $312M round for optical inference appliance maker Olix
Artificial intelligence hardware startup Olix Computing Ltd. today announced that it has raised $312 million in funding. The Series C round included contributions from Arm Holding plc, Netflix Inc. co-founder Reed Hastings and several others. Olix is now valued at $3.3 billion, about triple what...
Robotics and edge AI put new pressure on computing infrastructure
Physical AI is forcing the technology industry to rethink the entire computing stack. Robots, autonomous systems and intelligent devices need economical inference, secure data access and infrastructure that works beyond conventional clouds. Rafay Systems is addressing those demands through...
Autoscaling endpoints for LLM inference
GPU utilization can read healthy while your queue backs up, and a new replica takes minutes to warm. Here's how to pick autoscaling metrics, tune scale-up/down windows, and budget for cold starts on dedicated inference.
The Token Pricing Illusion: Understanding AI Inference Economics
In production, the sticker price per million token is a poor proxy for real inference cost. The better metric is performance-adjusted cost per useful token: correct, relevant, and usable output.
GLM 5.2 Now Available on CoreWeave Inference
GLM 5.2 is available now on CoreWeave Serverless Inference. Here's why open weights matter, how the model performs, and what you can build with it today.
Production AI Runs on Inference. Are You Ready for It?
Production AI depends on inference. Learn how to evaluate reliability, cost, and control and choose the right inference deployment for every workload.
Kimi K2.7 Code Now Available on Serverless Inference with Leading Benchmark Price-Performance
CoreWeave Inference achieves the highest output speed for the newly-launched Kimi K2.7 Code and ranks in the most attractive price-performance quadrant.
From Experimentation to Production: Why Inference Is the Defining Layer of AI
AI's center of gravity has moved from training to inference. Here is what that shift demands from the infrastructure underneath it.
Inference meta-monitoring for Amazon SageMaker AI endpoints with Amazon Quick
Learn how to build an inference meta-monitoring system for Amazon SageMaker AI endpoints using Amazon Quick. This governance layer sits above production ML...
Cerebras and AMD partner to build the world's fastest disaggregated AI inference solution
Disaggregated AI inference is proving to be more than a complementary answer to the prefill and decode bottleneck slowing enterprise AI at scale, and Cerebras and AMD just announced a partnership to build the fastest version of it in the world. The recent collaboration pairs AMD's Helios...
Configuring Dedicated Model Inference
The three-part resource model behind Together AI Dedicated Model Inference-endpoints, deployments, configs-and how capacity-aware routing ties them together.
ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
ThunderAgent is a program-aware scheduler for agentic inference. By treating each agent workflow as a schedulable program, it eliminates KV cache thrashing to deliver more than 2x single-node throughput and near-linear multi-node scaling.
Introducing Claude Opus 5 on AWS: Anthropic's most capable Opus model
This post covers Opus 5's improvements and practical guidance for AI engineers integrating the model into agentic systems and production inference workloads...
Get started with OpenAI GPT-5.6 Sol, Terra, and Luna on Amazon Bedrock
OpenAI GPT-5.6 Sol, Terra, and Luna are now generally available on Amazon Bedrock. Learn how to select a model, run inference through the Responses API on...
As agentic AI inference surges, tokenomics becomes the enterprise's defining budget constraint
The transition from chatbot to autonomous agent is changing the shape of demand itself, and tokenomics - the economics of AI token consumption - is emerging as the defining constraint on enterprise budgets as round-the-clock inference replaces intermittent usage. Fewer than 1% of potential...
AMD targets outcome-driven enterprise AI infrastructure as token economics reshape deployments
Enterprise AI is entering a new phase as organizations shift their focus from experimentation to production deployments that deliver measurable business outcomes. That transition is bringing AI token economics to the forefront, reshaping infrastructure priorities around inference costs and the...
AI chip startup Etched more than doubles valuation to $10.3B in new $300M round
Etched Inc., a startup with a chip optimized for artificial intelligence inference, today announced that it has raised $300 million in funding. The Series C round was led by Sequoia. SK Hynix Inc., the world's largest supplier of memory for AI chips, participated as well alongside Andreessen...
Partnering with Etched: Building the Inference Machine
The post Partnering with Etched: Building the Inference Machine appeared first on Sequoia Capital .
AI chip startup Etched defies skeptics, hits $10.3B valuation from big-name investors
Etched, founded by three Harvard dropout, has created new chips and memory components that speed up inference on any AI model -- no GPU required, it says.
AI infrastructure systems redefine the AMD-Nvidia rivalry as inference reshapes the market
The race to build AI infrastructure systems has moved beyond chip specifications into a battle over entire rack-scale platforms, as inference and agentic workloads redefine what counts as a computer. That shift is forcing challengers, once judged purely on GPU benchmarks, to prove they can ship...
The production platform for open-weight AI inference
Run open models in production with full control over performance, cost, and quality. Deploy in minutes, roll out safely, and scale to your SLOs.
'Beyond the GPU' video series: What to expect from theCUBE's July 23 coverage
When Advanced Micro Devices Inc. held its earnings call in May, Chief Executive Lisa Su told analysts that the current ratio of 4.5 GPU to 1 CPU will compress toward 1 to 1 as AI agent and inference workloads require more CPU support. Numbers such as these point to a number of factors that go [...
Why Inference Latency and Availability Drift in Production
Latency drift and gradual availability failures are the defining production inference challenge. This blog describes where drift originates and what to measure before it reaches your users.
Infinity raises $15M to run AI inference on any chipset
Early-stage AI infrastructure research company Infinity Inc. said it raised $15 million in seed funding to develop software that automatically prepares new artificial intelligence chips to run inference workloads. The round values the company at $100 million on a post-money basis. Infinity will...
Inference startup Infinity raises $15M from Touring Capital, OpenAI and Anthropic researchers
AI infrastructure company Infinity announced Monday a $15 million raise at a $100 million valuation from investors including Touring Capital, Principal VC...
Inference cloud operator General Compute raises $400M in debt financing
Artificial intelligence infrastructure startup General Compute Inc. today announced that it has secured $400 million in debt financing. Upper90, the investment firm that is underwriting the round, will initially provide the company with $100 million. General Compute plans to draw down additional...
CoreWeave Closes the Loop Between Training and Inference
CoreWeave unifies training and inference so AI agent can learn in production, improve autonomously, and evolve into productive coworkers with serverless RL and observability.
Why the first GPU financiers are turning to inference chips in a $400 million deal
A $400 million chip-backed loan points to the next wave of AI infrastructure deals.
Special Breaking Analysis: Nvidia's AI networking moat is real - but the lock-in debate continues
In a special editorial discussion hosted by Dave Vellante and Bob Laliberte, Nvidia Corp. networking chief Gilad Shainer explains why agentic inference turns the network into part of the computer. We believe Nvidia is materially ahead of the field, but in this Special Breaking Analysis we...
Three insights you may have missed from theCUBE's coverage of RAISE Summit
Agentic inference is reshaping the center of gravity in artificial intelligence infrastructure. What began as a race to scale training has shifted into a phase defined by expanding context window, memory‑augmented reasoning and the need to keep graphics processing units continuously fed with...
What does 99.9% uptime mean for inference?
Reliability numbers are easy to publish. We break down what 99%, 99.9%, and 99.99% uptime actually require, the failure domains each tier has to survive, and the questions to ask any inference provider before you commit.
Boundless taps idle crypto GPUs to cut AI inference costs
Boundless Network Inc., a distributed computing company that built its graphics processing unit network to generate zero-knowledge proofs for the cryptocurrency industry, is expanding its network toward artificial intelligence inference, the process of providing responses to AI queries. The...
Launching UI for generative AI inference recommendations in Amazon SageMaker AI
In this post, we introduce the UI for optimized generative AI inference recommendations in Amazon SageMaker AI Studio, a low-code no-code (LCNC) experience...
Disaggregated prefill and decode for LLM inference on SageMaker HyperPod
In this post, we show how to implement DPD with vLLM on Amazon SageMaker HyperPod using the HyperPod Inference Operator.
VAST Data targets KV cache storage and neoclouds as AI infrastructure enters the exabyte era
As AI infrastructure investment scales globally and inference workloads multiply, cache storage is emerging as the critical data layer that makes AI factories functional, persistent and economically viable in an era of disaggregated computing. VAST Data Inc. has positioned itself at the center of...
Fast token generation emerges as the key differentiator as heterogeneous inference takes hold
The race for fast token generation has moved from benchmark sheets into production data centers, and the hardware blueprint for winning it is no longer a GPU-only story. As agentic AI use cases multiply and users demand real-time interactivity, inference infrastructure is being redesigned from...
Enhancing enterprise inference on Amazon SageMaker HyperPod with data capture, Hugging Face, NVMe, and Route 53 integration
In this post, we walk through five capabilities now available in SageMaker HyperPod inference: multi-tier data capture for auditing and model improvement...
Cerebras Systems positions inference speed as the defining edge in AI infrastructure
The race to build the fastest AI infrastructure is reshaping the semiconductor industry, with inference speed emerging as the defining competitive dimension of the AI era. As AI model wars intensify across OpenAI, Anthropic and Google, the underlying compute layer is under pressure to keep pace...
AMD targets system-level AI infrastructure optimization as agentic workloads reshape enterprise compute
Infrastructure design is being redefined by agentic AI, pushing the industry toward system-level AI infrastructure optimization, balancing performance and cost across diverse workloads rather than focusing on faster chips alone. As inference scales and AI moves closer to users, modular...
Tensordyne targets AI inference market with logarithmic math and Juniper-derived rack architecture
The race to serve AI inference faster and cheaper is exposing the hard limits of conventional chip architecture. As demand for real-time AI responses accelerates, the industry's standard response - stacking more high-bandwidth memory onto power-hungry silicon - is running into a wall, and...
Solidigm targets the intelligence layer as agentic inference pushes storage to center stage
The shift from model training to agentic inference is forcing a fundamental rethink of how artificial intelligence infrastructure is built and which components carry the most strategic weight. What was once treated as commodity plumbing is now being recognized as the intelligence layer where raw...
Hot French startup ZML releases free product to speed inference across lots of AI chips
ZML, a hot French AI startup endorsed by Turing Award winner Yann LeCun, has now released ZML/LLMD, software that could make running AI less costly.
Open, convenient and predictable: Introducing Provisioned Throughput
Provisioned Throughput gives you reserved inference capacity for frontier open models like MiniMax M3 and GLM-5.2. Token-based pricing, a 99% uptime SLA, and up to 90% lower cost than proprietary APIs. No GPU-hour math, no infrastructure to manage.
Report: China's DeepSeek follows OpenAI in developing its own custom inference chips
DeepSeek Ltd., one of China's most visible artificial intelligence companies, is pushing to design its own, in-house silicon aimed at inference workloads, according to a report by Reuters today. The report cites three people familiar with the company's plans as saying that it has been exploring...
Streaming benchmark and recommendation results to MLflow with Amazon SageMaker AI
In this post, you learn how to use the new MLflow integration with Amazon SageMaker AI optimized inference recommendation jobs and Amazon SageMaker AI...
NVIDIA Unlocks AI Compute at Scale, Inviting Partners to Power the AI Infrastructure Buildout
As AI moves from model development to production inference, compute demand is accelerating and shifting toward continuously operating AI factories that...
Inference chip startup Etched launches with $800M in funding
Etched Inc., a developer of artificial intelligence inference chips, launched today with $800 million in funding. The startup raised the capital over multiple rounds. The most recent investment, which closed in December, valued Etched at $5 billion. It included the participation of VentureTech...
How NVIDIA's Inference Software Stack Powers the Lowest Token Cost
As organizations move from AI pilots to production AI factories, infrastructure decisions have shifted from peak chip specifications to cost per token: how...
DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%
Even as the geopolitical conversation around AI continues to grow more fraught following the U.S. government's actions to limit the new models from Anthropic...
Sail Research raises $80M to optimize long-horizon AI agents
Artificial intelligence inference startup Sail Research Inc. today announced that it has raised $80 million in funding at a $450 million valuation. The company received the bulk of the capital in the form of a Series A round led by Sequoia. It earlier raised a seed round led by Kleiner Perkins...
Qualcomm shares jump 14% on Modular acquisition, guidance upgrade
Qualcomm Inc.'s stock jumped 14% in after-hours trading today after it shared a series of updates about its artificial intelligence roadmap. The company announced plans to acquire an inference software startup called Modular Inc. and previewed two upcoming AI chips. Additionally, Qualcomm...
OpenAI and Broadcom unveil LLM-optimized inference chip
OpenAI and Broadcom introduce Jalapeño, a custom AI chip built for LLM inference to improve performance, efficiency, and scale across AI systems.
Upbound open-sources Modelplane to optimize inference clusters
Upbound Inc. today released Modelplane, a new open-source tool for managing artificial intelligence inference clusters. San Francisco-based Upbound is backed by $69 million from Alphabet Inc.'s GV fund, Intel Capital and others. It's best known as the creator of Crossplane, an open-source...
NVIDIA and AWS Collaborate to Bring AI to Production at Scale
Building AI systems at scale is demanding, requiring low-latency inference, fast vector search, strong GPU price-performance and infrastructure that can grow...
HPE and Kamiwaza rethink AI infrastructure for the inference era
As AI factories evolve into "data centers of the future," the infrastructure stack must also transform into a mix of CPU and GPU platforms that can deliver a full set of AI computing solutions. This runs the gamut from application hosting to intelligence generation and from static workflows to...
Groq raises $650M to scale fast AI inference cloud data centers worldwide
Groq closed a $650 million financing round as it pivot to a dedicated AI inference cloud provider operating 13 datacenters.
Monitor and debug generative AI inference with SageMaker detailed metrics and Insights dashboard on CloudWatch
Amazon SageMaker AI provides fully managed real-time inference hosting for machine learning models. You deploy a model to a SageMaker endpoint backed by one...
Context is becoming the missing layer in enterprise AI
The enterprise AI market is entering a new phase. For the past several years, the focus has been on larger models, faster inference and broader deployment of generative AI capabilities. Yet despite growing investment, many organizations continue to struggle with governance, accuracy, operational...
AI inference startup Baseten reportedly raising $1.5B months after its last mega-round
Startup Baseten is reportedly close to finalizing a $1.5 billion round at a $13 billion as the "inference gold rush" marches on.
AI inference provider Baseten reportedly raising $1.5B in funding
Baseten Inc., a startup with a platform for running artificial intelligence inference workloads, is raising $1.5 billion in funding. The Wall Street Journal reported today that Altimeter Capital, Conviction, Spark Capital, Sands Capital and Wellington Management are co-leading the deal. It's...
Lights. Camera. Inference. CoreWeave Powers the Next Act of AI Creation.
CoreWeave expands Conductor into AI workflows, enabling faster, secure training and inference for creative teams across sports, entertainment, and gaming.
Exclusive: Mindbeam touts dramatic performance improvements in CPU-based AI inference
Two-year-old startup Mindbeam AI Inc. today released an open-source artificial intelligence inference framework designed to make large language models run more efficiently on standard consumer processors, a move the company says could reduce reliance on expensive graphics processing units for some...
Evoflux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents
Compact language models (LMs) reduce cost, latency, and deployment risk for tool agents. Yet MCP-style tool use requires more than isolated function calling...
Extract Data with On-demand and Batch Pipelines Dynamically
This post demonstrates an intelligent document processing pipeline that consists of both on-demand inference and batch inference options on Amazon Bedrock to...
Inference Is Your Product's Reliability Layer
Learn how CoreWeave helps AI teams manage latency, cost, and control before production issues reach users.
NVIDIA Confidential Computing to Help Expand Apple's Private Cloud Compute
NVIDIA GPU with Confidential Computing are now used for confidential inference in Apple's Private Cloud Compute (PCC), as it expands beyond Apple's data...
GITCO: Gated Inference-Time Context Optimization in TSFMs
Patch-based Time Series Foundation Model (TSFMs) suffer from context poisoning: structurally anomalous patches capture disproportionate attention and...
NVIDIA Nemotron 3 Ultra now available on Amazon SageMaker JumpStart
Deploy NVIDIA Nemotron 3 Ultra on Amazon SageMaker JumpStart. Get 5x faster inference and 30% lower cost for agentic AI workloads with this frontier reasoning...