# U3. Build your own autograd

> How can a computer find every gradient automatically?

LLM by Hand · Foundations · side trip: Under the hood · runs in your browser · interactive page: https://llm.liko.page/learn/autograd/

In level 6 you wrote the backward pass by hand. In PyTorch (level 9) one line, `loss.backward()`, does all of it.
This level shows what that one line does inside. By the end you will have written the same thing yourself, in about 50 lines of Python.

It uses two ideas:

1. **Record the work.** While computing, remember every result, which operation made it, and from which inputs.
   That record is a **computation graph**.
2. **Each operation knows only its own rule.** Multiplication knows how its output moves when an input moves.
   So does addition. Nothing knows the whole network. The chain rule from level 6 combines the pieces:
   walk the graph backwards and multiply.

## 1. A graph you can step through

Take $L = (a \cdot b + c) \cdot d$ with $a = 2$, $b = 3$, $c = 1$, $d = -2$. Name the in-between results:
$e = a \cdot b$ and $f = e + c$. Press the main button (it starts as **Compute e = a·b**) to compute one node at a time. After $L$, the same button
runs the backward pass.

Backward starts by setting $\partial L / \partial L = 1$. That only says “if L grows by a little bit, L grows by
that same little bit”. It is the starting number; every other gradient is a product that starts from it.
Then backward **visits** nodes from the end to the start.
Visiting a node means one thing: push the node’s gradient into its inputs, using that node’s own rule.

- the product $L = f \cdot d$ sends $d \times$ (L’s grad) to $f$, and $f \times$ (L’s grad) to $d$:
  each input gets **the other input** times the product’s grad
- the sum $f = e + c$ sends f’s grad unchanged to both $e$ and $c$

*[Interactive lab: Autograd — open the page to use it]*

**Question.** a = 2, b = 3, c = 1, d = −2. e = a·b, f = e + c, L = f·d. What is ∂L/∂d?

*Answer it on the page to check your work.*

$a$ is farther away. Its gradient travels $L \to f \to e \to a$, multiplying by one local factor at each node:
1 at the sum, and $b$ at the product $e = a \cdot b$.

**Question.** a = 2, b = 3, c = 1, d = −2. e = a·b, f = e + c, L = f·d. What is ∂L/∂a?

*Answer it on the page to check your work.*

**Try it**

Change $c$ to 10 and press **Run all**. $L$ changes, but every gradient stays the same, because $c$ is only ever **added**.
Now change $b$. Which gradients move, and why does $\partial L / \partial b$ not depend on $b$ itself?

## 2. Each value remembers where it came from

Here is the core of an autograd engine. A `Value` holds a number (`data`), its gradient (`grad`),
the values it was made from (`_children`), and a small function `_backward` that knows the local rule.
Addition is done. Multiplication is your turn: the rule is the one from the lab.

Three pieces of Python that may be new to you:

- `def __add__(self, other)` runs when you write `a + b` with two `Value`s; `__mul__` runs for `a * b`.
  `self` is the left one, `other` the right one.
- `lambda: None` is a tiny function that does nothing. It is the `_backward` of a value that was not made by an operation.
- `_backward` is defined **inside** `__mul__`, so it can still use `self`, `other` and `out` later, when backward calls it.
  The line `out._backward = _backward` stores that function on the result, for later.

**Code question.** Write the gradient that a product sends to its first input.

Fill in the blank (`____`):

```python
class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self._children = children
        self._backward = lambda: None

    def __add__(self, other):
        out = Value(self.data + other.data, (self, other))
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad += ____
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

def grads(a, b, c, d):
    a, b, c, d = Value(a), Value(b), Value(c), Value(d)
    e = a * b
    f = e + c
    L = f * d
    L.grad = 1.0
    for v in [L, f, e]:          # visit from the end, by hand for now
        v._backward()
    return [a.grad, b.grad, c.grad, d.grad]

print(grads(2, 3, 1, -2))
```

*Answer it on the page to check your work.*

**If you are stuck: Why does _backward use out.grad? Where does that come from?**

When `_backward` runs, `out` has already received its own gradient from the nodes after it.
That is what visiting from the end guarantees. `out.grad` is "how much L moves per unit of `out`".
The local rule turns it into "how much L moves per unit of `self`". That one multiplication is the chain rule.

## 3. When a value is used twice

In a real network one weight feeds many places. The simplest case: $y = x \cdot x$. Here $x$ is **both**
inputs of the same multiplication. Calculus says $dy/dx = 2x$.

**Question.** y = x · x with x = 3. What is dy/dx?

*Answer it on the page to check your work.*

In `x * x`, `self` and `other` are the same object: both names point to `x`.

**Predict.** Someone writes self.grad = … and other.grad = … (with =, not +=). For y = x · x at x = 3, what does their backward give for x.grad?

A. 6, still right
B. 3, half the right answer
C. 9

*Answer it on the page to check your work.*

So each of the two lines in `_backward` sends a part of the gradient to the same `x`: `self.grad` gets $x \times 1$,
and `other.grad` gets another $x \times 1$. Only adding both parts gives $2x$.

**If you are stuck: So why += instead of =?**

A value that feeds several places changes $L$ through **all** of them. Its total gradient is the sum of
what each path sends back. `+=` collects them. With `=`, every path overwrites the one before, and only the
last path counts.

This is also why `loss.backward()` in PyTorch adds into `.grad`, and why you call `opt.zero_grad()`
before every step (level 9). Adding is right inside one backward pass. Across steps, old sums must be cleared.

Now give `Value` a new operation yourself: subtraction, `out = self - other`. Its forward part is done. For the backward rule:

- If `self` grows by a little, `out` grows by the same amount. The local factor is 1, as in a sum.
- If `other` grows by a little, `out` shrinks by that amount. The local factor is −1.

So `self` gets `1 × out.grad` and `other` gets `−1 × out.grad`. Write both lines, and keep adding into `.grad`:
`other.grad -= out.grad` is short for `other.grad = other.grad - out.grad`, the same as adding −1 × `out.grad`.

**Code question.** Write the backward rule of subtraction, out = self − other: one line for self, one line for other.

Fill in the blank (`____`):

```python
class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self._children = children
        self._backward = lambda: None

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __sub__(self, other):
        out = Value(self.data - other.data, (self, other))
        def _backward():
            ____
        out._backward = _backward
        return out

def grads(a, b, c):
    a, b, c = Value(a), Value(b), Value(c)
    e = a * b
    L = e - c
    L.grad = 1.0
    for v in [L, e]:             # visit from the end, by hand
        v._backward()
    return [a.grad, b.grad, c.grad]

print(grads(2, 3, 1))   # should be [3, 2, -1]
```

*Answer it on the page to check your work.*

## 4. The order of visits

A node may only be visited after **every** node that uses it has been visited. Otherwise its gradient
is not complete yet. Listing the nodes so that each comes after its inputs is called a **topological order**.
For the lab’s graph one such list is `[a, b, e, c, f, d, L]`: every node appears after the nodes it was made from.

The code builds that list with a small recursive function, `build`: before adding a node, it first builds each of its
children (a **depth-first walk**), so the children always land earlier in the list. Then backward visits the list
**from the end**, starting at L.

One more piece of Python: `reversed(xs)` walks a list from its last item to its first.
For example, `list(reversed([1, 2, 3]))` is `[3, 2, 1]`.

Write the end of `backward`, after `build(self)`: give L its starting gradient (section 1), then visit every node in the
right order and call its `_backward`. The `tanh` method is done too, so the last test is a real neuron.

**Code question.** Write the end of backward, after build(self): set the starting gradient of the output node, then call _backward on every node, in reverse topological order. Several lines.

Fill in the blank (`____`):

```python
import math

class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self._children = children
        self._backward = lambda: None

    def __add__(self, other):
        out = Value(self.data + other.data, (self, other))
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))
        def _backward():
            self.grad += (1 - t ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo, seen = [], set()
        def build(v):
            if v not in seen:
                seen.add(v)
                for child in v._children:
                    build(child)
                topo.append(v)      # a node goes in after all of its inputs
        build(self)
        ____

def grads(a, b, c, d):
    a, b, c, d = Value(a), Value(b), Value(c), Value(d)
    L = (a * b + c) * d
    L.backward()
    return [a.grad, b.grad, c.grad, d.grad]

print(grads(2, 3, 1, -2))
```

*Answer it on the page to check your work.*

**Deeper: This is what loss.backward() does**

PyTorch works the same way, with three differences, all only in size:

- it works on whole tensors, so one node is a matrix multiplication instead of one product of two numbers;
- it knows the local rule for hundreds of operations: `@`, `exp`, `softmax`, `layer_norm`, …;
- the rules are written in fast compiled code, and can run on a GPU.

The order of visits, the `+=`, the starting gradient $\partial L / \partial L = 1$: all the same.
`tanh` shows the pattern for any new operation. Forward computes $t = \tanh(x)$, backward adds $(1 - t^2) \times$ `out.grad`.
Once an operation has those two pieces, the engine can use it anywhere.

## You can now

- Find every gradient of a small graph by hand, visiting the nodes from the end.
- Write the backward rule of a new operation as a `_backward` function that adds into `.grad`.
- Put the nodes in topological order and walk it backwards to run the whole backward pass.