Glossary

LoRA

Also: Low-Rank Adaptation

A cheap way to fine-tune: the model's weights stay frozen and small low-rank matrices are trained alongside them, cutting trainable parameters for GPT-3 by 10,000x, per the authors.

Example

An open model is tuned to a company's answer format on one GPU by training only a LoRA adapter while the base model stays unchanged.

Related terms

Fine-tuningфайнтюнинг Quantizationквантование

Source

Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (2021)