What is a GPU?
GPU(Graphics Processing Unit)
A Graphics Processing Unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory. Because training neural networks involves massive matrix multiplication, the parallel processing power of GPUs is critical for modern AI workloads.
Why It Matters for AI Builders
Directly governs the hardware efficiency and hardware-level token throughput when deploying model training compute, high-throughput inference hosting, and heavy image synthesis; optimizing GPU is a major factor in compute cost budgeting.
Detailed Deep Dive
A Graphics Processing Unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images. Due to their massively parallel architecture containing thousands of cores, GPUs are exceptionally efficient at performing the matrix multiplications that form the computational core of deep learning training and inference.
Frequently Asked Questions
Q:Why are GPUs better than CPUs for AI?
CPUs have a few powerful cores optimized for sequential tasks. GPUs have thousands of smaller cores designed to perform simple operations (like matrix multiplication) simultaneously.
Q:What is VRAM?
Video RAM (VRAM) is the fast dedicated memory on the GPU. Modern AI models require massive VRAM (e.g. 24GB to 80GB) to store parameters during training and inference.
Quick Facts
- CategoryHardware & Infrastructure
- Key ApplicationModel training compute, high-throughput inference hosting, and heavy image synthesis
Coverage Trend12 Weeks
Related AI Terms
Cite This Term
Reference this definition in your articles, research, or documentation to credit this source:
GPU Media Coverage & Intelligence
KernelArc: A Multi-Agent Framework for GPU Kernel Optimization
We present KernelArc, a multi-agent framework for autonomous GPU kernel optimization across heterogeneous workloads. Strategy-specialized agents run in...
Lambda prices $926 million senior secured term loan B facility, the first investment-grade-rated term loan B financing by a private neocloud
First private neocloud to execute an investment-grade-rated financing in the term loan B market, further broadening the investor base for AI infrastructure financing. Facility supports the purchase and deployment of GPU infrastructure dedicated to an investment-grade offtaker. Heavily...
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered...
Does Kubernetes DRA Replace HAMi?
Projects that want to share a GPU on Kubernetes have to work around an API instead of with it. The device plugin interface could count devices, and that was the whole vocabulary: nvidia.com/gpu: 1. It meant one...
OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking
Your GPU bill is rising. Your models are serving billions of token. Yet one question remains unanswered: what does each token actually cost? This is not a hypothetical problem. Platform teams today operate in a fog...
Nvidia open-sources cuFile API, accelerating GPU read/write capability for high-speed storage
As artificial intelligence applications become ever hungrier for faster access to data, Nvidia Corp. today announced it is open-sourcing the application programming interface for its powerful cuFile vertical data storage stack, enabling millisecond data access. The company also announced a...
Choosing the right orchestration layer for your AI use cases
Compute scarcity is not the only struggle AI teams face. Optimal utilization is also key. Friction also emerges when they outgrow informal coordination methods such as shared spreadsheets, manual SSH access, or ad hoc GPU allocation, or when their orchestration stack no longer scales with their...
Autoscaling endpoints for LLM inference
GPU utilization can read healthy while your queue backs up, and a new replica takes minutes to warm. Here's how to pick autoscaling metrics, tune scale-up/down windows, and budget for cold starts on dedicated inference.
Stragglers, Synchronization, and Stalled GPUs: How Enterprise AI Training Fails Quietly
Large-scale AI training can fail quietly when stragglers, stalls, and wasted GPU cycles slow progress. CoreWeave helps teams turn compute into predictable model advancement.
CoreWeave Trains DeepSeek-V3 Benchmark in Two Minutes
CoreWeave's MLPerf® Training v6.0 results set new records, demonstrating how customers can train frontier AI model faster, scale more efficiently, and get more value from every GPU deployed.
Protopia and Rafay deliver multi-tenancy for shared GPU AI factories
Enterprise AI infrastructure providers are turning to multi-tenancy paired with upstream data protection to convert idle GPU capacity into secure, token-metered services that enterprises will actually adopt at scale. That tension is pushing infrastructure providers toward a model that treats...
TensorWave targets focused AI cloud strategy to deliver better customer experience
As AI infrastructure matures, AI cloud strategy is increasingly defined by reliability and open ecosystems rather than raw GPU performance. That evolution is creating new opportunities for specialized providers to challenge incumbent players with more focused strategies. TensorWave Inc. was...
AMD's Helios strategy turns the GPU battle into a systems contest
The market for data center GPU is evolving beyond individual chip specifications into a contest over fully integrated rack-scale systems. As AI workloads scale into the gigawatt range, buyers increasingly demand validated infrastructure that can be deployed quickly rather than components...
AI infrastructure systems redefine the AMD-Nvidia rivalry as inference reshapes the market
The race to build AI infrastructure systems has moved beyond chip specifications into a battle over entire rack-scale platforms, as inference and agentic workloads redefine what counts as a computer. That shift is forcing challengers, once judged purely on GPU benchmarks, to prove they can ship...
Anthropic to Buy up to 2 Gigawatts of GPU Capacity From AMD
Anthropic PBC will purchase up to 2 gigawatts' worth of graphics cards from Advanced Micro Devices Inc. as part of a multibillion-dollar deal announced today. The partnership also has several other components.
'Beyond the GPU' video series: What to expect from theCUBE's July 23 coverage
When Advanced Micro Devices Inc. held its earnings call in May, Chief Executive Lisa Su told analysts that the current ratio of 4.5 GPU to 1 CPU will compress toward 1 to 1 as AI agent and inference workloads require more CPU support. Numbers such as these point to a number of factors that go [...
Microsoft and Mistral AI expand strategic agreement for European sovereign cloud AI
Microsoft committed to integrating Mistral's latest frontier models into Copilot Studio and Azure Foundry while leveraging Europe-based GPU data centers.
Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community
No more two-year compute contracts. Together AI and YC just gave YC startups a faster way to get GPU.
Why the first GPU financiers are turning to inference chips in a $400 million deal
A $400 million chip-backed loan points to the next wave of AI infrastructure deals.
Why your Kubernetes scheduler can't handle AI workloads
Imagine this scenario: You have a distributed training job with 16 worker pods, each requesting 1 GPU. 4 GPU are currently available. The default Kubernetes scheduler ( kube-scheduler ) may schedule those 4 pods while the remaining 12 stay pending. Meanwhile, those 4 GPU are reserved by pods...
Fast token generation emerges as the key differentiator as heterogeneous inference takes hold
The race for fast token generation has moved from benchmark sheets into production data centers, and the hardware blueprint for winning it is no longer a GPU-only story. As agentic AI use cases multiply and users demand real-time interactivity, inference infrastructure is being redesigned from...
DDN targets GPU efficiency with AI data infrastructure as the make-or-break layer
The race to build AI factories is well underway, and the winning organizations have learned that AI data infrastructure determines whether or not GPU investments pay off, while others are still scrambling to assemble workable solutions. That divide is the clearest indicator from the field, said...
SpaceX and xAI compute cluster powers Grok 4.5 training run
xAI leverages expanded Colossus GPU supercluster to deliver high-speed token generation for coding workloads.
PyTorch 2.13.0 Released with torch.compile Performance Tuning and AOTInductor Enhancements
PyTorch 2.13 reached general availability, featuring improved Python 3.13 compilation speeds and enhanced multi-GPU distributed training APIs.
Open, convenient and predictable: Introducing Provisioned Throughput
Provisioned Throughput gives you reserved inference capacity for frontier open models like MiniMax M3 and GLM-5.2. Token-based pricing, a 99% uptime SLA, and up to 90% lower cost than proprietary APIs. No GPU-hour math, no infrastructure to manage.
Understanding dynamic resource allocation in Kubernetes
Dynamic Resource Allocation (DRA) recently reached GA in Kubernetes v1.35, and I believe many of us are eager to give it a try. Adding to the momentum, NVIDIA has moved dra-driver-nvidia-gpu into Kubernetes SIGs, with the...
NVIDIA BioNeMo Agent Toolkit Brings Accelerated AI to Life Sciences Researchers in Claude Science
Life sciences has entered an era of computational scale, and for more than a decade, NVIDIA has built the full GPU-accelerated computing stack - spanning...
NVIDIA and AWS Collaborate to Bring AI to Production at Scale
Building AI systems at scale is demanding, requiring low-latency inference, fast vector search, strong GPU price-performance and infrastructure that can grow...
HPE and Kamiwaza rethink AI infrastructure for the inference era
As AI factories evolve into "data centers of the future," the infrastructure stack must also transform into a mix of CPU and GPU platforms that can deliver a full set of AI computing solutions. This runs the gamut from application hosting to intelligence generation and from static workflows to...
Import AI 463: Self-improving robots; a 10k Chinese GPU cluster; and an elegiac essay for the human era
What eras bookend our interregnum?
Rackspace signs definitive agreement to deploy 30MW of AMD AI compute
Rackspace and AMD have completed a deal to roll out massive GPU infrastructure globally.
What AI benchmarks miss about real-world performance
Presented by F5 Enterprise AI teams have spent years solving for compute, securing GPU allocations, negotiating cloud capacity, and benchmarking training...