Level N5 · Foundations · Classic networks · runs in your browser · last part on your computer

Seq2seq and the first attention

How do you turn one sequence into another, and why did attention appear?

Side trip · best after level 10 · Probability and sampling

Boss level

Write attention for every decoder step at once in NumPy, softmax included, and pass every test in your browser. Then, on your computer, put the same three lines into boss.py in PyTorch and train an LSTM encoder–decoder that reverses at least 90% of new 20-digit strings.

Level N4’s LSTM reads a sequence and predicts the next symbol. Many tasks need more: read one whole sequence, then write a different one, often of a different length. Digits in, words out: 427015 → “four hundred twenty seven thousand fifteen”. Or a string in, the same string backwards out. This is sequence to sequence, “seq2seq” for short.

This level builds the first model that did it well, finds its weakness, and fixes it with the idea the whole Transformer uses: attention. The boss has a part that runs on your computer; see Run the code on your computer.

1. An encoder and a decoder

The model is two LSTMs, one after the other:

  1. The encoder reads the input, one token at a time, exactly like level N4. It produces one state per input token. Its last state (h and c) is its summary of the whole input.
  2. The decoder starts from that summary. It writes the output one token at a time: it gets a start token <bos>, predicts the first word, gives that word to itself as the next input, predicts the next, and stops when it writes <eos>.

As everywhere in the course, vectors are rows. With a state of H numbers, the encoder turns an input of L tokens into L states, an array of shape (L, H). The decoder receives only the last one.

Number An LSTM encoder with H = 32 reads a string of 24 digits. The decoder receives only the encoder’s last h and c. How many numbers is that?
🔒 Answer the question above to unlock

This is the problem: a 4-digit input and a 24-digit input both have to fit into the same 64 numbers. Before you look at the measurements, make a guess.

Predict firstGuess before you look. With H = 32, the plain encoder–decoder reverses 4-digit strings perfectly. On new 20-digit strings, about what percent does it get exactly right?
🔒 Answer the question above to unlock

Two of these models were trained to reverse strings of 2 to 24 digits, then tested on 300 new strings of each length. An answer counts only if every digit is right.

Reversing digit strings: exact answers on new strings

Tap a length to read its numbers. Switch the state size H to give the model a bigger state.

Loading the recorded runs…
state size H
Waiting for the recorded runs…
without attention

The short strings are easy. The long ones fail: the summary vector is a bottleneck, a narrow passage. Everything the decoder will ever know about the input has to pass through it.

I got stuck here Why not make the state bigger? Switch H to 128.

It helps: at 16 digits the bigger model gets 62% instead of 4%. But it only moves the length where accuracy drops. At 24 digits it is back to 5%. Any fixed size is not enough for a long enough input, and every extra number in the state costs weights and time. The fix is not a bigger state. It is letting the decoder look at the input again.

2. Attention: read every encoder state

🔒 Answer the question above to unlock

The encoder did not only produce a summary. It produced a state for every input token, and those are still there. Attention lets the decoder use all of them, at every step it writes:

  1. Score each encoder state against the decoder’s current state s with a dot product (level 1). E holds the encoder states as rows, so all the scores at once are scores = E @ s.
  2. Turn the scores into weights that add up to 1, with softmax (level 10).
  3. Take the weighted average of the encoder states: context = weights @ E. That is a vector of H numbers, computed again for this step.

The decoder then predicts the next word from s and the context together. A state that scores high gets a big weight, so the context is mostly made of the input positions that matter for this word.

Here are three encoder states with two numbers each, and a decoder state. The encoder states are the encoder’s hidden states, so we call them h₀, h₁, h₂ (the rows of E). Drag them.

One attention step: the decoder state s looks at the encoder states

Drag s or h₀, h₁, h₂ (or focus one and use the arrow keys). Thicker arrows get more weight.

33h₀h₁h₂s
statehscore h·sweight
h₀[1, 0]??
h₁[0, 1]??
h₂[0, -1]??

s = [2, 0]

context = weights @ E = ?

Drag s or the encoder states. Thicker arrows get more weight.
s, the decoder state encoder states (thicker = more weight) context, the weighted average
Number E = [[1, 0], [0, 1], [0, −1]] (rows h₀, h₁, h₂) and the decoder state s = [2, 0]. What is the score of h₀, that is h₀ · s?
🔒 Answer the question above to unlock

All three scores are [2, 0, 0]. Softmax turns them into weights: raise e to each score, then divide each by the total.

Number The scores are [2, 0, 0]. Use e² ≈ 7.39 and e⁰ = 1. After softmax, what weight does h₀ get? (2 decimals)
🔒 Answer the question above to unlock

The weights are about [0.79, 0.11, 0.11]. The context is the weighted average: 0.79 × h₀ + 0.11 × h₁ + 0.11 × h₂.

ChooseThe weights are [0.79, 0.11, 0.11] and E = [[1, 0], [0, 1], [0, −1]]. What is the second number of context = weights @ E?
🔒 Answer the question above to unlock

In a real model the states have many numbers. Shapes tell you what goes where: E is (L, H), s is (H,), the scores and the weights are (L,): one per input token.

Shape E holds 8 encoder states of 32 numbers each: E is (8, 32). The decoder state s is (32,). What shape is context = weights @ E? (Write one number, like (5).)
🔒 Answer the question above to unlock

Now write the last line of one attention step. softmax from level 10 is ready to use.

CodeFinish one attention step: the context is the weighted average of the encoder states.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

Go deeper Other ways to score

The dot product is the simplest score, and the one the Transformer uses. An older version scores each encoder state h with a small network instead: v · tanh(s @ W1 + h @ W2), where W1, W2 and v are learned. It can compare vectors of different sizes and was popular in the first attention models. Both work. The dot product won because it is a single matrix multiply for all positions at once, and fast hardware is designed for exactly that.

3. Does it fix the bottleneck?

🔒 Answer the question above to unlock

The same reversing experiment, the same LSTM sizes, now with attention added to the decoder.

ChooseThe same reversing task and the same LSTM with H = 32, now with attention. At 20 digits the plain model got 0%. Does the attention model do better than 0%?
🔒 Answer the question above to unlock

Reversing digit strings: exact answers on new strings

Tap a length to read its numbers. Switch the state size H to give the model a bigger state.

Loading the recorded runs…
state size H
Waiting for the recorded runs…
without attentionwith attention

With a 32-number state, the plain model gets 0% at 20 digits. With attention it gets 72%. With 128 numbers, attention keeps 85% at 24 digits, where the plain model is at 5%. The summary vector no longer has to carry everything. The decoder can get what it needs from the input when it needs it.

4. Where does it look?

🔒 Answer the question above to unlock

An attention model leaves a trace you can read: the weights at every output step. This one writes numbers as English words, up to six digits, the same kind of task the Transformers of level 17 and level N7 learn. English reads 427015 in groups of three digits: “four hundred twenty seven” (427), then “thousand”, then “fifteen” (015). It was trained on random numbers and tested on 1,000 it never saw: it reads 100% of them exactly right.

ChooseThe model reads 427015 and writes “four hundred twenty seven thousand fifteen”. When it writes “seven”, which digit should get the most weight?
🔒 Answer the question above to unlock

Where the decoder looks

Point at or tap a cell. Rows: what the decoder wrote. Columns: what it read. Darker = more weight.

Loading the recorded weights…
Waiting for the recorded weights…
attention weight, 0 → 1

Nobody told the model which digit goes with which word. It learned these alignments only because looking at the right digit made its next word easier to predict.

The same weights for 427015, drawn as arcs from each word back to the digits. A brighter, thicker arc is a larger weight.

Each word reads the digits it needs

Press Next to write the words one at a time.

427015fourhundredtwentyseventhousandfifteen<eos>encoder statesdecoder steps
Press Next to start.
Layers (2)

Look at where the bright arcs land: “seven” reaches back to the 7, and “thousand” and “fifteen” to the last three digits.

Gray disc = done; hollow disc = not reached yet. forward (this step) Thicker arc = more attention weight. Select a layer to see its numbers.

Try it

Pick 427,015 and point at the row for “hundred”. It does not look at the 4. It looks two digits ahead, at the 7: the last two digits decide whether “seventeen” or “twenty seven” comes after “hundred”. Then look at “thousand”, which spreads its weight over the last three digits. Switch to “Reversing digits”: the bright cells run along the diagonal from the top right to the bottom left, the last input digit first.

5. Keep the attention, drop the recurrence

🔒 Answer the question above to unlock

Attention fixed the bottleneck, but the model still reads with an LSTM: one token after another, each step waiting for the previous one, and the encoder’s own states still get weaker over long inputs (level N3). The next idea was a big change:

ModelHow the input is readHow the output finds the input
encoder–decoderan LSTM, step by steponly the final summary
+ attention (this level)an LSTM, step by stepattention over every encoder state
encoder–decoder Transformer (level N7)attention, all tokens at onceattention over every encoder state
GPT (levels 17–21)attention over earlier tokens, all at oncethere is no separate encoder

Keep the attention and remove the recurrence. In level 14, every word builds its new vector by attending to every other word in the same sentence, with the same three steps you just computed: dot-product scores, softmax, weighted average.

I got stuck here Is this the same attention as level 14?

The arithmetic is the same: scores from dot products, softmax, weighted average. Two things change. Here the decoder state looks at the encoder states, two different sequences; level 14 starts with words looking at words of their own sentence, which is called self-attention. And level 14 first multiplies each vector by learned matrices (queries, keys, values), so “what I look for”, “what others compare against” and “what I give” can be different. It also divides the scores by √d_k, so long vectors do not make the softmax too sharp. The step from this level to the cross-attention of level N7 is short: the decoder’s query looks at the encoder’s keys and values.

6. The boss

Part 1: every decoder step at once

So far the decoder had one state s. It writes T words, so it has T states, and attention runs once for each. Stack them as the rows of S, shape (T, H), and all the scores come from one matrix product: S @ E.T is (T, L), row t holding decoder state t’s scores against every encoder state.

The softmax then runs along each row separately. Two NumPy tools do that. x.max(axis=-1, keepdims=True) takes the largest number of each row and keeps the result as a column, so it lines up with x; x.sum(axis=-1, keepdims=True) does the same with the sum:

x = np.array([[2., 0, 0], [0, 2, -2]])
x.max(axis=-1, keepdims=True)    # [[2.], [2.]]   one number per row
x - x.max(axis=-1, keepdims=True)  # [[0., -2., -2.], [-2., 0., -4.]]   no number above 0, so exp never overflows

Subtracting each row’s largest score before np.exp does not change the softmax (level 10), and it keeps exp from overflowing.

CodeThe boss, part 1. Write attention for every decoder step at once: one row of weights per decoder state, each a softmax over the encoder states, and one context row per decoder state. Write the softmax yourself. Replace ____ with as many lines as you need.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

🔒 Answer the question above to unlock

Look at what you just wrote: softmax(S @ E.T) @ E. Rename S to Q (the queries) and E to K and V (the keys and the values), and it is level 14’s attention, softmax(Q @ K.T) @ V, minus the scaling by √d_k and the learned matrices that make Q, K and V.

Part 2: on your computer

Download boss.py into a folder of its own and open it there (setup). It has the whole reverser except attend_all(S, E). Write it in PyTorch: the same three steps as part 1, for a whole batch. Now S is (B, T, H) and E is (B, L, H): every tensor has a batch axis in front, and each of the B examples gets its own attention.

Run python boss.py. It checks your attend_all() on random batches, trains the reverser for about a minute and tests it on 300 new 20-digit strings. At 270 or more exactly right it prints a line that starts with N5 PASS. Paste that line here. The code at the end of the line only shows that you pasted it unchanged; this part relies on your honesty.

The level counts as cleared once part 1 (attend_all on this page) and this line both pass.

CodeThe boss, part 2. Paste the PASS line that boss.py printed for your own attend_all(), between the quotes.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

We also trained the same reverser with and without attention, with H = 128 and on this one task only. The plain model does better than in the chart above (50% at 20 digits on our machine), but it is still far below attention (98%).

Recap

a summary for when you finish the level

The key formulas and common mistakes appear here once you clear the level.

You can now

  • Explain the bottleneck: without attention, the whole input must fit into the encoder’s last state.
  • Compute one attention step by hand: dot-product scores, softmax weights, weighted average of the encoder states.
  • Write attention for every decoder step at once in NumPy, softmax included.

Keep in mind

  • One step: scores = E @ s, w = softmax(scores), context = w @ E
  • Shapes: E (L, H), s (H,), scores and weights (L,), context (H,)
  • All steps: softmax(S @ E.T) @ E, with S @ E.T of shape (T, L), softmax along each row
  • Subtract each row’s largest score before np.exp: same softmax, no overflow

Common mistakes

  • Taking the softmax over the wrong axis: each decoder state’s row of weights must add up to 1.
  • Thinking a bigger state fixes the bottleneck: it only moves the length where accuracy drops.
‹ N4. LSTM and GRUN6. Autoencoders ›

Press ? for keyboard shortcuts

Reading mode · every part open, no stars