Glossary
Attention
Also: attention · self-attention
The mechanism by which a model decides, for each token, which other tokens in the context to rely on and how much. The core of the transformer.
Example
In “the agent ran the tests and they failed”, attention links “they” to “tests”. Over long contexts attention spreads thin, hence context rot.