Scaled dot-product attention

Mathematical modelAI / MLMathematics

\mathrm{Attention}(\htmlClass{sjt-Q}{Q},\htmlClass{sjt-K}{K},\htmlClass{sjt-V}{V}) = \htmlClass{sjt-softmax}{\mathrm{softmax}}\!\left(\frac{\htmlClass{sjt-QKT}{Q K^{T}}}{\htmlClass{sjt-sqrt}{\sqrt{d_k}}}\right)\htmlClass{sjt-V2}{V}

Tap a symbol above, or choose one here:

Q

QueriesRosetta entry →

Meaning

One vector per position: "what am I looking for?" Computed as Q = XW_Q from the input.

Its job in this equation

One side of every relevance comparison.

Picture it

A search query typed by each word.

Read it, progressively

1. Gist

Each word looks at every other word, decides how relevant each one is, and takes a relevance-weighted blend of their information.

What it does not establish

Attention weights are not explanations: a high weight does not prove that token was "why" the model answered as it did. The equation omits masks, multiple heads, projections, positional information and the rest of the transformer block.

Play with it

Scaled dot-product attention

Choose a query token:

Thecatsatonthematbecauseitwastired

Key dimension

d_k

(illustrative)64

“it” attends most to“cat” · 56%

Raw score (top)2.80

Row entropy (spread)2.07 bits

$\mathrm{softmax}\!\left(\tfrac{QK^{T}}{\sqrt{d_k}}\right)V$ . Each row of the grid is one query token. Its scores are dot products with every key, which softmax turns into weights summing to 1, and the output is that weighted mix of value vectors. These 4-dimensional vectors were written by hand so that “it” looks for an animate noun. Trained models learn their own features, and they are rarely this tidy. Turn off the $\sqrt{d_k}$ scaling and raise $d_k$ : the softmax saturates to one-hot, which is the problem the scaling was introduced to prevent.

Loading interactive…

§ Module 6: The Mathematics Inside Modern AI Next equation: Bayes’ theorem →