# What is Distributed Training? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Hardware & Infrastructure  
> **Canonical Citation:** [https://spidits.com/ai-glossary/distributed-training](https://spidits.com/ai-glossary/distributed-training)

## Definition
Distributed Training is the practice of partitioning machine learning workloads (data or parameters) across multiple compute processors (GPUs/TPUs) to accelerate training times for large neural networks.

## Architectural Mechanics & Deep Dive
Distributed training is a paradigm where the training workload of a large AI model is split across multiple processing units (GPUs or TPUs) and physical servers. As model sizes outgrow the memory of a single device, distributed training utilizes techniques like Data Parallelism (splitting the dataset batch), Model Parallelism (splitting layers across devices), and Pipeline Parallelism to coordinate training efficiently.

## Industry Applications & Real-World Use Cases
Foundation model pre-training, parameter scaling operations, and cluster orchestration.

## Frequently Asked Questions

### What is Data Parallelism vs. Model Parallelism?
Data parallelism splits the dataset across devices, running copies of the model. Model parallelism splits the model layers across different GPUs because the model is too large to fit in a single device's VRAM.

### What are popular frameworks for distributed training?
PyTorch Distributed Data Parallel (DDP), Megatron-LM, and DeepSpeed.

## Related Terminology
- [gpu](https://spidits.com/ai-glossary/gpu)
- [tpu](https://spidits.com/ai-glossary/tpu)
- [pytorch](https://spidits.com/ai-glossary/pytorch)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*