# What is Generative Pre-training? Definition & Technical Mechanics

> **Source:** [SPIDITS AI AI Glossary](https://spidits.com/ai-glossary)  
> **Category:** Model Training  
> **Canonical Citation:** [https://spidits.com/ai-glossary/generative-pre-training](https://spidits.com/ai-glossary/generative-pre-training)

## Definition
Generative Pre-training is the initial phase of training a Large Language Model on massive, unlabeled text datasets where the model learns token relationships by predicting the next word in sequence.

## Architectural Mechanics & Deep Dive
Generative Pre-training is the initial self-supervised phase of training a Large Language Model, where it learns language structure and world knowledge by predicting the next token in massive text corpora. This unsupervised phase builds the foundational capabilities of the model (creating a "base model"), which is subsequently aligned using fine-tuning or preference optimization.

## Industry Applications & Real-World Use Cases
Foundation model preparation, unsupervised text pre-training, and base model building.

## Frequently Asked Questions

### Why is pre-training generative?
Because it uses a next-token generation loss objective rather than classifying inputs, teaching the model vocabulary and patterns.

### What is the cost of generative pre-training?
It is highly resource-heavy, requiring clusters of GPUs running for weeks or months on terabytes of raw text.

## Related Terminology
- [supervised-learning](https://spidits.com/ai-glossary/supervised-learning)
- [foundation-model](https://spidits.com/ai-glossary/foundation-model)
- [scaling-laws](https://spidits.com/ai-glossary/scaling-laws)

---
*From the SPIDITS AI Knowledge Base (336+ terms indexed). Explore full technical definitions: [https://spidits.com/ai-glossary](https://spidits.com/ai-glossary)*