Level N2 · Foundations · Classic networks · runs in your browser

Residuals and normalization

Why do very deep networks stop learning, and what fixed it?

Side trip · best after level 10 · Probability and sampling

This level uses N1 and levels 6–7.

More layers should mean a smarter network. For years it didn’t: past a few dozen layers, networks got worse, not better. Two small changes fixed it, and both sit inside every Transformer block you will build in level 16:

a residual connection, which adds a layer’s input back to its output;
normalization, which rescales numbers so they stay in a stable range.

1. Deeper should be better. Is it?

Backpropagation (level 6) sends the gradient down one layer at a time. Each layer multiplies it by its own slope. If every slope is a bit below 1, the product of many of them is tiny.

Number A plain stack has 4 layers. Going down, each one multiplies the gradient by its slope, 0.5. The gradient at the top is 1. What reaches the bottom?

🔒 Answer the question above to unlock

Now a real stack. Each layer is h ← tanh(h @ W), 16 numbers wide, with weights drawn from a normal distribution with std 0.8/√16. That is a little smaller than the level-7 rule of 1/√16.

Choose30 plain tanh layers. Going down, each one shrinks the gradient to about three quarters of its size. The gradient at the top has std 1. Roughly how big is it when it reaches layer 0?

🔒 Answer the question above to unlock

Stack 30 tanh layers: how much of the signal and the gradient is left?

Drag depth and weight scale. Compare the plain stack with the residual stack.

Signal going up std of each layer’s output

Gradient coming down std of the gradient reaching each layer

depth 30 weight scale 0.8/√16

gradient at layer 0 ÷ gradient at the top: plain 1.3e-4, residual 11.22

plain: h ← tanh(h @ W) residual: h ← h + tanh(h @ W) log scale; the dashed line is 1

The lines marked “plain” are the plain stack. Going up, the signal (each layer’s outputs) shrinks layer after layer. Coming back down, the gradient does the same: in demo.py, the gradient’s std is 1.02 at the top, 0.098 ten layers down, 0.0059 twenty layers down and 0.000135 at the bottom. The first layers get almost no learning signal, so they barely change. A deeper plain network can finally be worse than a shallow one.

I got stuck here Level 7 fixed this with a good starting scale. Why isn’t that enough?

A good starting scale keeps the numbers in range on step one. Training then changes every weight, and tanh units slowly move into their flat ends, where the slope is near 0. Thirty factors that are each a little different from 1 still multiply into something tiny or huge. The fix has to keep working while the weights change. A residual connection does.

2. The residual path

🔒 Answer the question above to unlock

A residual layer adds its input back:

h_{\text{out}} = h + f(h)

$f$ is the layer that learns, here tanh(h @ W). Take one number to see what this does to the gradient. The slope of $x + f(x)$ is

\frac{d}{dx}\bigl(x + f(x)\bigr) = 1 + f'(x)

Number A residual layer computes y = x + f(x), with f(x) = 0.5 · x. What is the slope dy/dx?

🔒 Answer the question above to unlock

That 1 is the point. Even when $f$ has a slope near 0, the gradient still passes down at full size along the plain $x$ path, the residual path. The layer $f$ is on the other side of the +: we call it the layer side. Ten plain layers with slope 0.1 multiply the gradient by $0.1^{10} = 10^{-10}$ . Ten residual layers with the same $f$ multiply it by $1.1^{10} \approx 2.59$ .

Look at the lines marked “residual” in the lab: the gradient reaching layer 0 never shrinks to 0. In this stack it even grows, to about 11 times its size at the top. The residual signal grows too, from 0.97 at the input to 2.33, 3.46 and 4.39 after 10, 20 and 30 layers in demo.py. Each block adds something to $h$ , so $h$ keeps getting larger. Numbers that keep growing are the next problem, so every residual network also normalizes: sections 4 and 5.

Try it

Set the weight scale to 0.5 and the depth to 50. How far down does the plain gradient fall, and where does the residual one stop? Then try scale 1.5: what goes wrong for the plain stack now?

Go deeper With many numbers per layer, why is it still “1 + something”?

With 16 numbers per layer, the slope of a layer is a 16×16 table of slopes (one for each input and output pair), called the Jacobian. For $h + f(h)$ it is $I + J_f$ , where $I$ is the identity table: 1 on the diagonal, 0 elsewhere. Going down $L$ blocks multiplies $L$ such tables. Expanded, the product contains $I \cdot I \cdots I = I$ as one of its terms, so there is always a route on which the gradient isn’t shrunk at all.

3. A residual block in a convolutional network

🔒 Answer the question above to unlock

Residual networks for pictures stack many blocks like this one. The layer side has two 3×3 convolutions (level N1), each followed by BatchNorm (section 4) and a ReLU. The residual path carries the block’s input around them, unchanged.

One residual block. The output is ReLU(x + F(x)). If F learns nothing useful, the block still passes x through.

The + adds two tensors cell by cell, so $F(x)$ must have exactly the shape of $x$ . A 3×3 convolution with padding 1 keeps the height and width (level N1), and its number of filters sets the number of output channels.

Shape x has shape (64, 8, 8): 64 channels, each 8×8. Both convolutions on the layer side are 3×3 with padding 1 and 64 filters. What shape is F(x)?

Go deeper What if a block changes the number of channels?

Then $x$ and $F(x)$ no longer match. The residual path gets its own small layer, usually a 1×1 convolution: it mixes the channels at each pixel into the new count without looking at neighbors. With stride 2 it also halves the height and width, to match a layer side that shrinks the picture.

4. BatchNorm: one feature, across the batch

🔒 Answer the question above to unlock

Residuals keep the gradient from vanishing, but the numbers inside the network can still grow or shift as training changes the weights. Normalization puts them back in a stable range: subtract the mean, divide by the standard deviation (std = √variance, level 10). Afterwards the numbers have mean 0 and std 1.

The question is which numbers share a mean. Here is a batch of 4 examples (rows) with 4 features each (columns):

	f0	f1	f2	f3
ex 0	1	4	2	9
ex 1	1	0	6	3
ex 2	5	1	5	1
ex 3	5	7	3	1

BatchNorm normalizes each feature (column) using that column’s mean and std over the batch. To get a std by hand: subtract the mean, square each difference, average the squares, take the square root.

Number Feature 0 (column 0) is [1, 1, 5, 5]. Its mean is 3. Find its std, then the BatchNorm value of X[2][0] = (5 − mean) ÷ std. Ignore the tiny ε.

🔒 Answer the question above to unlock

Real BatchNorm adds a tiny number ε to the variance before the square root, so it never divides by 0. Then it applies a learned scale and shift per feature, so the network can undo the normalization if that helps.

During training, the mean and std come from the current batch. At test time there may be only one example, so BatchNorm keeps a running mean: after each batch it keeps 0.9 of the old value and adds 0.1 of the new batch’s mean.

Number BatchNorm’s running mean for a feature is 2. The next batch has mean 4 for that feature. The update keeps 0.9 of the old value and adds 0.1 of the new one. What is the running mean now?

ChooseTraining with a batch of 1 example, x = [3, 7, 1]. BatchNorm subtracts each feature’s mean over the batch. What is the output?

In a convolutional network, like the residual block in section 3, each channel is one feature. Its mean is taken over the batch and over every pixel of that channel.

5. LayerNorm: one example, across its features

🔒 Answer the question above to unlock

LayerNorm uses the other direction of the table: it normalizes each example (row) using that row’s own mean and std. It never looks at the other examples in the batch.

Number LayerNorm uses each row’s own mean and std. Example 2 (row 2) is [5, 1, 5, 1]. What is the LayerNorm value of X[2][1]? Ignore ε.

🔒 Answer the question above to unlock

Which numbers are averaged together?

Pick BatchNorm or LayerNorm, then point at, tap or focus a cell to choose its column or row.

X: 4 examples (rows) × 4 features (columns)

	f0	f1	f2	f3
ex 0
ex 1
ex 2
ex 3

normalized: (x − mean) ÷ std of its column

	f0	f1	f2	f3
ex 0	-1	0.37	-1.26	1.68
ex 1	-1	-1.1	1.26	-0.15
ex 2	1	-0.73	0.63	-0.76
ex 3	1	1.46	-0.63	-0.76

feature 0: [1, 1, 5, 5] → mean 3, std 2

the numbers averaged together

Number A batch has 32 examples with 64 features each: shape (32, 64). LayerNorm computes 32 means, one per example. How many means does BatchNorm compute?

ChooseA model writes text one token at a time, so at each step its input is a single sequence. Which normalization still works there?

Transformers use LayerNorm, not BatchNorm, for three reasons:

Sentences have different lengths. Short sentences in a batch get extra padding tokens (PAD), and BatchNorm would mix them into every feature’s mean.
Generation runs one token at a time. When a model writes, the “batch” is often one sequence, where batch statistics are useless.
The same token should get the same answer. With LayerNorm, a token’s output depends only on that token, not on which other sentences share its batch.

I got stuck here If LayerNorm only looks at one example, how can it help training at all?

It still does the job that matters: whatever the layer below produced, the next layer receives numbers with mean 0 and std 1. The next layer’s weights never have to adjust to inputs that keep getting bigger or shifting. In level 16 you will see it after (or, in today’s models, before) every attention and feed-forward step of the Transformer block, right next to the residual +.

6. Write it yourself

🔒 Answer the question above to unlock

In NumPy, X.mean(axis=0) averages down each column, giving one number per column. X.mean(axis=1) averages across each row, giving one number per row. keepdims=True keeps the averaged axis with size 1, so a (4, 4) table gives a (1, 4) or (4, 1) result that broadcasts back over the table (level 1). X.var(...) gives the variance the same way, and np.sqrt takes the square root.

X = np.array([[1., 3], [5, 7]])
X.mean(axis=0, keepdims=True)   # [[3., 5.]]      one mean per column
X.var(axis=1, keepdims=True)    # [[1.], [1.]]    one variance per row

Here is BatchNorm, with ε added to the variance. Write the whole LayerNorm function: three lines.

CodeWrite layer_norm: normalize each row (one example) by its own mean and variance, with ε added to the variance before the square root, as batch_norm does for columns.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

Recap

a summary for when you finish the level

The key formulas and common mistakes appear here once you clear the level.

You can now

Compute how much gradient reaches the bottom of a plain stack: multiply the slopes of every layer.
Explain why a residual layer $h + f (h)$ keeps the gradient from shrinking to 0: its slope is $1 + f^{'} (x)$ .
Normalize a table by hand and in NumPy, per column (BatchNorm) or per row (LayerNorm).

Keep in mind

$h_{out} = h + f (h)$ , slope $1 + f^{'} (x)$
Normalize: $(x - mean) / var + ε$ , then mean 0 and std 1
BatchNorm: axis=0, one mean per feature over the batch. LayerNorm: axis=1, one mean per example
Running mean: keep 0.9 of the old value, add 0.1 of the new batch’s mean

Common mistakes

Using the wrong axis: BatchNorm averages down each column, LayerNorm across each row.
Thinking a slope a bit below 1 is harmless: 30 layers at 0.75 shrink the gradient to about 0.0002.

‹ N1. ConvolutionsN3. Recurrent networks ›

Press ? for keyboard shortcuts

Residuals and normalization

A short test to skip this level

1. Deeper should be better. Is it?

Stack 30 tanh layers: how much of the signal and the gradient is left?

2. The residual path

3. A residual block in a convolutional network

4. BatchNorm: one feature, across the batch

5. LayerNorm: one example, across its features

Which numbers are averaged together?

6. Write it yourself

Recap

You can now

Keep in mind

Common mistakes

Keyboard shortcuts