Glossary
Pretraining
Also: pre-training · pretraining
The first and most expensive training stage: the model learns to predict the next token over a huge text corpus and gains general knowledge of language and the world.
Example
After pretraining a model continues text but follows instructions poorly. Fine-tuning and RLHF turn it into an assistant.