One-Sentence Definition
Attention is a mechanism that lets a neural network weigh the importance of different parts of its input when producing each part of its output, allowing the model to focus on the most relevant information at each step.
How It Works
Imagine translating the English sentence "The bank of the river was muddy" into French. To translate "bank" correctly, the model needs to focus on "river" -- not on every word equally. Attention provides this selective focus.
The standard implementation, called scaled dot-product attention, works with three sets of vectors: queries, keys, and values. For each position in the output, the model creates a query vector that represents what information it needs. Every position in the input provides a key vector (what information it contains) and a value vector (the actual content). The model computes a similarity score between the query and every key, normalizes those scores into weights using a softmax function, and then produces a weighted sum of the values. Positions with high similarity get more weight; irrelevant positions get almost none.
Self-attention applies this process within a single sequence. When processing a sentence, each word attends to every other word in the same sentence, building a contextualized representation. The word "bank" gets a different representation depending on whether the surrounding words are about rivers or about finance.
Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections. Different heads can focus on different types of relationships -- one head might track syntactic dependencies while another captures semantic similarity. GPT-4, Claude, and Gemini all use multi-head self-attention as their core computational building block.
The original 2017 paper "Attention Is All You Need" by Vaswani et al. at Google proposed the transformer architecture, which replaced recurrence entirely with attention. This enabled massive parallelization during training and set the stage for the large language model era.
Why It Matters
Attention is arguably the single most important mechanism in modern AI. Every frontier language model -- GPT-4o, Claude, Gemini, Llama, Mistral -- is built on attention. Vision transformers use attention to process images. Audio models like Whisper use it for speech recognition. Multimodal models use cross-attention to connect text with images, video, and audio.
The mechanism also created a scaling dynamic that defines the current era: attention quality improves with more data and more parameters, which is why labs like OpenAI, Anthropic, Google, and Meta invest billions in training ever-larger attention-based models. The main bottleneck is that standard attention scales quadratically with sequence length, which has driven research into efficient attention variants like FlashAttention, grouped-query attention, and sparse attention.
Key Takeaway
Attention lets AI models dynamically focus on the most relevant parts of their input at each step, and it is the core mechanism inside every transformer-based model powering the current generation of AI.
Part of the AI Weekly Glossary.