~/teaching · machine learning

The attention mechanism and the transformer, step by step

One plain sentence, then the maths, then the numbers by hand: embeddings, queries, keys, values, heads, positions, encoder, decoder

Sitraka Forler, Luxembourg

Revision 12 October 2026 · about 20 minutes

Abstract

A language model needs the meaning of a word, and the meaning of a word depends on the words around it. Attention is the operation that lets each word of a sentence borrow from the others, in proportions the model learns. This article builds it up from the first idea to the full transformer, in the order a student needs: embeddings, the weighted average, queries, keys and values, several heads, the position of words, then the encoder, the decoder and the prediction of the next token.

Every section opens with one plain sentence, then gives the mathematics, then redoes the arithmetic on a four-word sentence in two dimensions, small enough to check with a pencil. Nothing is hidden behind a library call: the twelve lines of NumPy at the end reproduce every number in the text.

How to read

If you only have ten minutes, read the plain sentences and look at the figures: they tell the whole story.
If you are preparing an exam or an interview, follow the step-by-step lists and redo the example on paper; the tables give the numbers to compare with.
If you want to implement it, go to the code at the end, then come back to the sections for the reasons behind each line.

This article accompanies the machine learning course at Centrale Méditerranée.

Part I

Words as vectors

§1Embeddings: a word is a point in spaceF1
§2The problem: one vector per word, whatever the sentence

Embeddings: a word is a point in space

An embedding turns a word into a list of numbers, placed so that words with similar meanings sit close together.

A computer cannot compare the words shares and stock as text: the strings share no letter. It can compare two lists of numbers. An embedding is a function that gives every token (a word, or a piece of a word) a vector in $\R^d$ , with $d$ typically between a few hundred and a few thousand, trained so that geometry mirrors meaning: close vectors for related words, far vectors for unrelated ones.

Step by step

Tokenise. Cut the text into tokens. In this article one token is one word; real models cut into sub-words so that rare words are still covered.
Look up. Each token is a row of a big table $E \in \R^{|V| \times d}$ , where $|V|$ is the size of the vocabulary. The embedding of token $i$ is simply the row $E_i$ .
Compare. Two vectors are compared by the cosine of the angle between them, which ignores their length: $\cos\theta = \dfrac{\vv{a}\cdot\vv{b}}{\lVert\vv{a}\rVert\,\lVert\vv{b}\rVert}$ . A value near 1 means the same direction, near 0 unrelated, near $-1$ opposite.

\cos\theta_{ab} = \frac{\vv{a}\cdot\vv{b}}{\lVert\vv{a}\rVert\,\lVert\vv{b}\rVert} = \frac{\sum_{k=1}^{d} a_k\, b_k}{\sqrt{\sum_k a_k^2}\,\sqrt{\sum_k b_k^2}}

(1)

The table $E$ is learned, not written by hand: a model is trained on a large corpus to predict words from their neighbours, and the rows of $E$ settle where the prediction works best. The famous regularities follow from that training: in a good space, the vector from king to queen is almost the same as the vector from man to woman.

The example used in the whole article

Take the sentence Apple shares fell sharply, four tokens, and a toy space of dimension $d = 2$ . Read axis 1 as "food" and axis 2 as "finance". The embedding table, trained on general text, has put Apple near the fruit, because that is what the word means most of the time:

X = \begin{bmatrix} \vv{x}_{\text{Apple}} \\ \vv{x}_{\text{shares}} \\ \vv{x}_{\text{fell}} \\ \vv{x}_{\text{sharply}} \end{bmatrix} = \begin{bmatrix}1.00 & 0.20 \\ 0.10 & 1.00 \\ 0.30 & 0.60 \\ 0.20 & 0.50\end{bmatrix}

(2)

Figure 1: The four tokens in the toy plane. Apple points to food; shares, fell and sharply point to finance. The dashed arrow is where attention will move Apple in section 3.

The cosine between Apple and shares is 0.29: the plain embeddings say the two words have little to do with each other. In this sentence they have everything to do with each other. That gap is the problem attention solves.

Back to contents.

The problem: one vector per word, whatever the sentence

A fixed embedding gives the same vector to a word in every sentence, so it cannot tell a fruit from a company.

The table $E$ has one row per token. Apple gets the same vector in an apple a day and in Apple shares fell sharply; interest gets the same vector whether it is a feeling or a rate; bond whether it is a promise, a chemical link or a security. Linguists call these words polysemous; for a model they are a single point that should be two.

What we want is a second sequence $\vv{y}_1, \dots, \vv{y}_L$ , one vector per token, where $\vv{y}_i$ is the vector of token $i$ in this sentence: the same word, re-positioned by its neighbours. The representation must be built from the sentence itself, at the moment the sentence is read, not looked up in a table.

Remark (Why a weighted average). The simplest way for a token to take its context into account is to mix in a little of every other token. A mix is a weighted average, and the whole question becomes: how much of each neighbour, and who decides.

Back to contents.

Sitraka FORLERsystems architect, lecturer in machine learning

Attention, step by step

Part I of IIInext: The attention mechanism →

Part II

The attention mechanism

§3Attention as a weighted averageF2
§4Queries, keys and valuesF3
§5Multi-head attentionF4
§6Positional encoding: giving the words an orderF5

Attention as a weighted average

Each word becomes a weighted average of all the words in the sentence, with heavier weights on the words that matter to it.

Let the sentence be the sequence $\vv{x}_1, \dots, \vv{x}_L$ of embeddings, $L$ tokens of dimension $d$ . The contextual vector of token $i$ is a weighted sum of all the tokens:

\vv{y}_i = \sum_{j=1}^{L} w_{ij}\,\vv{x}_j, \qquad w_{ij} \ge 0,\quad \sum_{j=1}^{L} w_{ij} = 1

(3)

The weight $w_{ij}$ says how much token $j$ counts in the new version of token $i$ . The constraints make the weights a probability distribution over the sentence, so $\vv{y}_i$ stays in the same region of space as the inputs: a true average, not an explosion.

Step by step

Score. Measure how relevant $j$ is to $i$ . The cheapest measure is the dot product $s_{ij} = \vv{x}_i\cdot\vv{x}_j$ : large when the two vectors point the same way, zero when they are perpendicular.
Normalise. Turn the $L$ scores of row $i$ into weights that are positive and sum to 1 with the softmax: $w_{ij} = \dfrac{e^{s_{ij}}}{\sum_{\ell=1}^{L} e^{s_{i\ell}}}$ .
Mix. Compute $\vv{y}_i$ as the weighted sum of equation (3).
Repeat for every token. Rows do not interact: the $L$ computations are independent and run in parallel, which is why this operation suits GPUs so well.

w_{ij} = \softmax_j\big(s_{i1}, \dots, s_{iL}\big) = \frac{\exp(\vv{x}_i\cdot\vv{x}_j)}{\sum_{\ell=1}^{L}\exp(\vv{x}_i\cdot\vv{x}_\ell)}

(4)

Stacking the rows gives the matrix form. With $X \in \R^{L\times d}$ , the matrix of scores is $S = XX\T$ (entry $ij$ is $\vv{x}_i\cdot\vv{x}_j$ ), the softmax is applied row by row, and the new sequence is the product of the weights by the inputs:

Y = \softmax\!\left(XX\T\right) X, \qquad Y \in \R^{L\times d}

(5)

By hand, on the example

Row 1 is Apple. Its dot products with the four tokens are $\vv{x}_1\cdot\vv{x}_1 = 1.0\times1.0 + 0.2\times0.2 = 1.04$ , then 0.30 with shares, 0.42 with fell and 0.30 with sharply. Exponentials, then division by their sum, give the weights of the first row of Table 1.

**Table 1:** Attention weights $w_{ij}$ with the plain dot product (no projection, no scaling). Each row sums to 1. **Apple** keeps 40 % of itself and gives **shares** no more than **sharply**.
$i \backslash j$	Apple	shares	fell	sharply
Apple	0.401	0.191	0.216	0.191
shares	0.176	0.359	0.245	0.220
fell	0.238	0.293	0.245	0.224
sharply	0.233	0.290	0.247	0.230

The new vector of Apple is $\vv{y}_1 = 0.401\,\vv{x}_1 + 0.191\,\vv{x}_2 + 0.216\,\vv{x}_3 + 0.191\,\vv{x}_4 = [0.523,\;0.497]$ . It has moved from $[1.00,\;0.20]$ towards the finance axis, which is the right direction, but mostly because three of the four words happen to lie there. Nothing in the computation knew that shares was the word that mattered.

Figure 2: The same weights as a heat map, left with the plain dot product, right after the learned projections of section 4. Darker is heavier. On the right, the row of Apple has moved its weight onto shares.

Remark (Where the method falls short). The dot product rewards vectors that already point the same way. If Apple and shares were perpendicular in the embedding space, their score would be 0 and shares would get the smallest weight, exactly when it should get the largest. The weights must be computed in a space where the relevant pairs are aligned, and that space has to be learned. That is the job of the next section.

Back to contents.

Queries, keys and values

Before comparing, each word is projected three times: into a question it asks, a label it answers to, and the content it hands over.

In the weighted average, one vector $\vv{x}_i$ played three parts at once: it asked for context, it was compared against, and it was mixed in. Separating the three parts gives the model freedom to learn each one. Picture a reading room: a reader hands in a query (what I am looking for), each book carries a key on its spine (what I am about), and the book itself is the value (what you get if you open me). Relevance is measured between the query and the keys; what is returned is a mix of the values.

Step by step

Project. Three learned matrices turn the input into queries, keys and values: $Q = XW^Q$ , $K = XW^K$ , $V = XW^V$ , with $W^Q, W^K \in \R^{d\times d_k}$ and $W^V \in \R^{d\times d_v}$ .
Score. The score of token $j$ for token $i$ is the dot product of the query of $i$ with the key of $j$ : $s_{ij} = \vv{q}_i\cdot\vv{k}_j$ , that is $S = QK\T$ .
Scale. Divide the scores by $\sqrt{d_k}$ so that they stay in the range where the softmax still has a gradient (remark below).
Normalise and mix. Softmax on each row, then multiply by the values.

\Attention(Q, K, V) = \softmax\!\left(\frac{QK\T}{\sqrt{d_k}}\right) V

(6)

This is the scaled dot-product attention of the 2017 paper, and it is the whole mechanism. Everything that follows (heads, positions, encoder, decoder) is built around this one line. The matrices $W^Q$ , $W^K$ , $W^V$ are the parameters that training adjusts: the loss on the next-token prediction pushes them so that the queries of a token line up with the keys of the tokens that help predict what comes next.

Figure 3: The three projections of one token, then the score against every key, the softmax, and the mix of the values. The grey box is applied to every token of the sentence at once, as one matrix product.

Remark (Why divide by the square root of the key dimension). If the coordinates of $\vv{q}$ and $\vv{k}$ are independent with mean 0 and variance 1, their dot product is a sum of $d_k$ terms of variance 1, so it has variance $d_k$ and typical size $\sqrt{d_k}$ . With $d_k = 64$ the raw scores spread over tens of units, the softmax puts all its mass on one token, and its gradient vanishes. Dividing by $\sqrt{d_k}$ brings the variance back to 1 whatever the dimension.

By hand, on the example

First the scaling alone. With $d_k = 2$ , the scores of row 1 are divided by $\sqrt{2}$ : 0.735, 0.212, 0.297, 0.212, and the weights become 0.353, 0.209, 0.228, 0.209: flatter, as expected, since the softmax of smaller numbers is closer to uniform.

Now a projection. We choose $W^Q$ by hand to show what training could discover: a matrix that swaps the two axes. The query of a token then asks for the other reading of itself. Apple, which lives on the food axis, asks for finance. $W^K$ and $W^V$ stay the identity.

W^Q = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \qquad Q = XW^Q = \begin{bmatrix}0.20 & 1.00 \\ 1.00 & 0.10 \\ 0.60 & 0.30 \\ 0.50 & 0.20\end{bmatrix}, \qquad K = V = X

(7)

The query of Apple is now $[0.20,\;1.00]$ . Its scaled scores against the four keys are 0.283, 0.721, 0.467, 0.382: shares is now the best match, as Table 2 shows.

**Table 2:** Attention weights after the projection $W^Q$ and the $\sqrt{d_k}$ scaling. The row of **Apple** now puts its largest weight on **shares**; the row of **shares** looks back at **Apple**.
$i \backslash j$	Apple	shares	fell	sharply
Apple	0.206	0.319	0.247	0.227
shares	0.361	0.202	0.227	0.210
fell	0.296	0.240	0.240	0.225
sharply	0.292	0.238	0.241	0.229

The output for Apple becomes $\vv{y}_1 = [0.358,\;0.623]$ , further along the finance axis than before, and this time because the right word pulled it. One $2\times2$ matrix, chosen on purpose, did what no dot product on the raw embeddings could do. A trained model has thousands of such matrices and chooses them from data.

Back to contents.

Multi-head attention

Several attentions run side by side, each with its own projections, so that one can track meaning while another tracks grammar.

One set of projections captures one kind of relation. The swap matrix above resolves Apple; it says nothing about which verb goes with which subject, or about which word is two places to the left. Rather than ask one projection to do everything, the transformer runs $h$ attentions in parallel, the heads, each with its own $W^Q_i$ , $W^K_i$ , $W^V_i$ of reduced dimension $d_k = d_v = d / h$ , then glues their outputs together.

Step by step

Split. Head $i$ projects the input into its own small space: $\text{head}_i = \Attention(XW^Q_i,\; XW^K_i,\; XW^V_i)$ , each output in $\R^{L\times d_v}$ .
Concatenate. The $h$ outputs are placed side by side, giving a matrix in $\R^{L\times h d_v} = \R^{L\times d}$ .
Project back. A last matrix $W^O \in \R^{h d_v\times d}$ mixes the heads into one representation per token.

\begin{aligned} \MultiHead(X) &= \Concat(\text{head}_1, \dots, \text{head}_h)\,W^O \\ \text{head}_i &= \Attention\!\left(XW^Q_i,\; XW^K_i,\; XW^V_i\right) \end{aligned}

(8)

The cost is the same as one head of full dimension: $h$ heads of dimension $d/h$ do the same amount of arithmetic as one head of dimension $d$ . What is gained is diversity. In trained models, probes find heads that follow syntax (a verb attends to its subject), heads that attend to the previous token, heads that copy a rare word from earlier in the text, and heads whose role nobody has named.

Figure 4: Two heads on the example. Each head sees one axis of the toy space; their outputs are concatenated and projected by

W^O

. In a real model each head sees a learned

d/h

-dimensional slice.

By hand, on the example

Take $h = 2$ heads of dimension $d_k = d_v = 1$ . Head 1 keeps only the finance coordinate of each token ( $W^Q_1 = W^K_1 = W^V_1 = [0, 1]\T$ ), head 2 keeps only the food coordinate ( $[1, 0]\T$ ). In head 2 the row of Apple is 0.425, 0.173, 0.211, 0.191: Apple, the only food word, mostly attends to itself. In head 1 it is 0.232, 0.272, 0.251, 0.246: on the finance axis the four tokens look alike and the weights are nearly uniform. Concatenating the two outputs for Apple gives $[0.591,\;0.544]$ ; with $W^O$ the identity, that is the multi-head output.

Remark (What the two heads say). Head 2 answers who am I, head 1 answers what is the market around me. A single head of dimension 2 would have had to blend both questions into one score. This is the whole point of heads: separate questions, separate weights, one combined answer.

Back to contents.

Positional encoding: giving the words an order

Attention treats the sentence as a bag of words, so a vector describing each position is added to each token before anything else happens.

Look again at equation (6): nothing in it depends on the index $j$ except through $\vv{x}_j$ . Permute the tokens and the outputs are permuted the same way, unchanged. The fund bought the bank and The bank bought the fund give the same set of vectors. Order carries meaning and the mechanism throws it away.

The fix is to add to each embedding a vector that depends only on the position $p$ of the token in the sequence, before the first attention layer:

\vv{x}_p \leftarrow \vv{x}_p + \PE(p), \qquad \PE(p) \in \R^d

(9)

Step by step

Choose a code for each position. The 2017 paper uses sines and cosines of geometrically spaced frequencies: coordinate $2i$ of $\PE(p)$ is $\sin(p / 10000^{2i/d})$ , coordinate $2i+1$ is $\cos(p / 10000^{2i/d})$ . Low coordinates oscillate fast and tell positions apart locally; high coordinates oscillate slowly and order the whole sequence.
Add, do not concatenate. Adding keeps the dimension $d$ and lets the projections $W^Q$ , $W^K$ read position and content together, which is how a head can learn to attend to the token just before me.
Let the layers use it. From then on every query and key carries position, so the scores can depend on distance as well as on meaning.

\PE(p)_{2i} = \sin\!\left(\frac{p}{10000^{2i/d}}\right), \qquad \PE(p)_{2i+1} = \cos\!\left(\frac{p}{10000^{2i/d}}\right)

(10)

Remark (Why sines and cosines). For a fixed offset $k$ , the pair $(\sin\omega p, \cos\omega p)$ at position $p+k$ is a rotation of the pair at position $p$ by the angle $\omega k$ : a linear function, independent of $p$ . A head can therefore learn k positions to the left as a fixed linear map, whatever the absolute position. The encodings are also bounded, and they extend to lengths never seen in training. Many recent models use a rotary variant that applies the same rotation idea inside the queries and keys rather than adding it to the input.

Figure 5: The sinusoidal encoding for

d = 2

: the two coordinates of

\PE(p)

for positions 0 to 3 are points on the unit circle at angles 0, 1, 2 and 3 radians. Moving one position is always the same rotation.

By hand, on the example

With $d = 2$ there is a single frequency, $1/10000^0 = 1$ , so $\PE(p) = [\sin p, \cos p]$ in radians. Table 3 adds it to the four embeddings. The two sentences Apple shares fell sharply and Shares fell sharply, Apple now give Apple two different inputs, $[1.00,\;1.20]$ at position 0 and $[1.14,\;-0.79]$ at position 3, and all the attention weights change with them.

**Table 3:** Sinusoidal positional encoding for $d = 2$ and the position-aware input $X + \PE$ . The encoding depends only on the position, never on the word.
token	$p$	$\sin p$	$\cos p$	$x_1 + \sin p$	$x_2 + \cos p$
Apple	0	0.000	1.000	1.000	1.200
shares	1	0.841	0.540	0.941	1.540
fell	2	0.909	−0.416	1.209	0.184
sharply	3	0.141	−0.990	0.341	−0.490

Back to contents.

Sitraka FORLERsystems architect, lecturer in machine learning

Attention, step by step

Part II of IIInext: The transformer →

Part III

The encoder block

An encoder block lets the words talk to each other (attention), then lets each word think on its own (a small neural network), with shortcuts so that training stays stable.

The transformer of Vaswani et al. (2017) was built to turn a sequence into another sequence, a French sentence into an English one. Its encoder reads the whole input and produces one context-rich vector per input token. It is a stack of $N$ identical blocks (6 in the paper), and each block is two sub-layers wrapped in the same two tricks.

One encoder block, input $X \in \R^{L\times d}$

Multi-head self-attention. $A = \MultiHead(X)$ : queries, keys and values all come from $X$ , which is why it is called self-attention. Information crosses between tokens here, and only here.
Residual connection and normalisation. $X' = \LayerNorm(X + A)$ . Adding the input back (the residual) means a block only has to learn a correction, and gradients flow straight through the stack; normalising each token vector to mean 0 and variance 1 keeps the scale steady from block to block.
Feed-forward network. $F = \FFN(X')$ , the same two-layer network applied to every token independently: $\FFN(\vv{x}) = W_2\,\max(0, W_1\vv{x} + \vv{b}_1) + \vv{b}_2$ , with a hidden width of $4d$ in the paper. No information crosses between tokens here; each vector is transformed on its own.
Residual and normalisation again. $Z = \LayerNorm(X' + F)$ is the output of the block and the input of the next one.

\begin{aligned} X' &= \LayerNorm\big(X + \MultiHead(X)\big) \\ Z &= \LayerNorm\big(X' + \FFN(X')\big) \end{aligned}

(11)

\begin{aligned} \LayerNorm(\vv{x}) &= \gamma \odot \frac{\vv{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \\ \mu &= \frac{1}{d}\sum_{k=1}^{d} x_k, \qquad \sigma^2 = \frac{1}{d}\sum_{k=1}^{d} (x_k - \mu)^2 \end{aligned}

(12)

Attention mixes across the sentence; the feed-forward network works inside each token. The division of labour is deliberate: attention decides where to look, the feed-forward network decides what to make of it. Stacking $N$ blocks lets a token gather context about context: after two blocks, Apple can use what shares itself learned from fell.

Figure 6: The encoder block (left) and the decoder block (right). Each arrow that loops around a sub-layer is a residual connection followed by layer normalisation. The decoder's middle sub-layer takes its keys and values from the encoder's output.

Back to contents.

The decoder block

The decoder writes the output one token at a time, looking at what it has already written and at the encoder's reading of the input.

The decoder produces the output sequence token by token. At step $t$ it has already produced $t-1$ tokens, embedded and position-encoded into $Y_{<t} \in \R^{(t-1)\times d}$ , and it has the encoder output $Z \in \R^{L\times d}$ . Each of its $N$ blocks has three sub-layers, each wrapped in the residual connection and normalisation of the encoder.

One decoder block

Masked self-attention. Multi-head attention over the tokens already written, with one change: a token may only attend to the positions before it. In the score matrix, every entry $s_{ij}$ with $j > i$ is set to $-\infty$ before the softmax, so its weight becomes exactly 0. Without the mask, training would let the model read the answer it is supposed to predict.
Cross-attention. The queries come from the decoder's own vectors; the keys and values come from the encoder output $Z$ . This is where the output looks at the input: when writing the third English word, the decoder asks which French words are relevant to it. It is the only place the two halves of the model meet.
Feed-forward network. The same per-token network as in the encoder.

\begin{aligned} \Attention_{\text{masked}}(Q, K, V) &= \softmax\!\left(\frac{QK\T}{\sqrt{d_k}} + M\right) V \\ M_{ij} &= \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases} \end{aligned}

(13)

\text{Cross}(Y, Z) = \Attention\!\left(YW^Q,\; ZW^K,\; ZW^V\right)

(14)

Remark (Encoder-decoder, encoder only, decoder only). The 2017 model has both halves because it translates. Models that classify or embed text keep only the encoder (BERT is one). The large language models that write text keep only the decoder: there is no separate input to attend to, the prompt and the answer are one sequence, and the mask is what makes generation left-to-right.

Back to contents.

Predicting the next token

The last vector of the decoder is turned into a score for every word of the vocabulary, the scores become probabilities, and one word is chosen.

After the last decoder block, the vector at the latest position, $\vv{h}_t \in \R^d$ , summarises everything written so far and everything read in the input. A linear layer maps it to one score per vocabulary entry, and a softmax turns the $|V|$ scores into a probability distribution:

\begin{aligned} P(\text{next token} = v \mid \text{context}) &= \softmax\!\left(W^{\text{out}}\,\vv{h}_t + \vv{b}\right)_v \\ W^{\text{out}} &\in \R^{|V|\times d} \end{aligned}

(15)

Generation loop

Score the vocabulary. Compute the distribution above for the current prefix.
Pick a token. The most probable one (greedy decoding), or a sample from the distribution, often after sharpening or flattening it with a temperature, or keeping only the top candidates.
Append and repeat. The chosen token joins the prefix, gets its embedding and position, and the decoder runs again. Stop at the end-of-sequence token or at a maximum length.

During training there is no loop: the whole target sentence is fed to the decoder at once, the mask hides the future, and the loss is the cross-entropy between the predicted distribution at each position and the true next token, summed over the sentence. This is called teacher forcing, and it is why training a transformer parallelises over the sequence while generation does not.

Remark (Cost). Self-attention compares every token with every other: $L^2$ scores of $d_k$ multiplications each, so $O(L^2 d)$ time and $O(L^2)$ memory per head and per layer. That quadratic term is the reason long contexts are expensive, and the reason a whole literature exists on sparser or approximate attention.

Back to contents.

Sitraka FORLERsystems architect, lecturer in machine learning

Attention, step by step

Part III of IIInext: The whole forward pass, in order →

The whole forward pass, in order

Tokenise and embed. Cut the text into tokens; look up each one in the table $E$ to get $X \in \R^{L\times d}$ .
Add positions. $X \leftarrow X + \PE$ , so that the order of the words survives the next step.
Project. For each head, $Q = XW^Q$ , $K = XW^K$ , $V = XW^V$ .
Score, scale, mask. $S = QK\T / \sqrt{d_k}$ , plus $-\infty$ above the diagonal in a decoder.
Normalise. Softmax on each row: every token gets a distribution over the sentence.
Mix. Multiply the weights by $V$ ; concatenate the heads; project with $W^O$ .
Residual, normalise, feed-forward, residual, normalise. One block is done; repeat $N$ times.
Read the last vector. Linear layer to the vocabulary, softmax, pick a token, append it, go back to step 1 for the new prefix.

The same arithmetic in twelve lines of NumPy

Twelve lines that redo section 4 on the example matrix $X$ , with the axis-swapping $W^Q$ . The two printed lines are the first row of Table 2 and the output for Apple.

import numpy as np

def attention(Q, K, V, mask=None):
    """Scaled dot-product attention, one line per step of section 4."""
    d_k = K.shape[-1]
    S = Q @ K.T / np.sqrt(d_k)            # scores, scaled
    if mask is not None:
        S = S + mask                      # -inf above the diagonal
    W = np.exp(S - S.max(axis=-1, keepdims=True))
    W = W / W.sum(axis=-1, keepdims=True) # softmax, row by row
    return W @ V, W                       # mix of the values, and the weights

# Apple, shares, fell, sharply
X = np.array([[1.0, 0.2], [0.1, 1.0], [0.3, 0.6], [0.2, 0.5]])
W_Q = np.array([[0.0, 1.0], [1.0, 0.0]])   # the axis swap of section 4
Y, W = attention(X @ W_Q, X, X)
print(W[0].round(3))   # [0.206 0.319 0.247 0.227]: Apple looks at shares
print(Y[0].round(3))   # [0.358 0.623]

Glossary

Embedding: A learned map from tokens to vectors, trained so that distance reflects meaning. One row per token in a table $E$ .
Token: The unit the model reads: a word, a sub-word or a character, depending on the tokeniser.
Attention: A weighted average of the values of a sequence, with weights computed from the match between one query and all the keys.
Query, key, value: Three projections of the same token: what it looks for, what it answers to, what it hands over.
Head: One attention with its own projections; several run in parallel and are concatenated.
Positional encoding: A vector added to each embedding that encodes the position of the token, so that order is not lost.
Residual connection: Adding the input of a sub-layer to its output, so the sub-layer learns a correction and gradients pass through.
Layer normalisation: Rescaling each token vector to mean 0 and variance 1, then a learned scale and shift.
Encoder: The stack of blocks that reads the input sequence and outputs one context vector per token.
Decoder: The stack of blocks that writes the output one token at a time, with a causal mask and cross-attention to the encoder.
Softmax: The function that turns a list of scores into positive weights summing to 1, by exponentiating and dividing by the total.

References

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention Is All You Need, NeurIPS 2017. arXiv:1706.03762.
D. Bahdanau, K. Cho, Y. Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015. arXiv:1409.0473. The first attention mechanism, additive rather than dot-product.
J. L. Ba, J. R. Kiros, G. E. Hinton, Layer Normalization, 2016. arXiv:1607.06450.
J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021. arXiv:2104.09864. The rotary variant mentioned in section 6.
J. Alammar, The Illustrated Transformer, 2018. jalammar.github.io.
P. Moskwa, Attention, please! Demystifying the math behind the attention mechanism and the transformer model, 2025. paulinamoskwa.github.io. The article whose path this one follows, with a PyTorch implementation.
A. Rush et al., The Annotated Transformer, Harvard NLP, 2018, revised 2022. nlp.seas.harvard.edu. The paper re-implemented line by line.

About this article

Written for the students of the machine learning option at Centrale Méditerranée, and for anyone who has read a transformer tutorial and still could not redo the numbers. The path from embeddings to the full model is the classic one; the sentence, the two-dimensional example, the hand-chosen projection and the code are this article's own. The mathematics is that of Vaswani et al. (2017), with the layer normalisation of Ba et al. (2016).

Drafted with the help of Claude Code, an Anthropic product; every formula was checked against the papers and every number is recomputed by the page's own code at each build.

Revision: 12 October 2026. First published the same day.