# U1. Tensors in memory

> Where do the numbers of an array really live, and why is a transpose free?

LLM by Hand · Foundations · side trip: Under the hood · runs in your browser · interactive page: https://llm.liko.page/learn/tensors-in-memory/

In level 1 an array was a table with a shape. Inside the computer there is no table. There is one long row of
numbers, and a short description of how to read that row as a table. This level is about that description.

It explains three things you will meet later:

1. why `x.T` costs nothing, even for a huge matrix;
2. why `reshape` sometimes makes a copy and sometimes does not;
3. why level 21 writes `.transpose(1, 2).contiguous().view(...)` when it joins the attention heads.

## 1. One long row

Take twelve numbers and read them as 3 rows of 4:

```python
# np.arange(12.0) is 0.0, 1.0, …, 11.0
x = np.arange(12.0).reshape(3, 4)
# [[ 0.  1.  2.  3.]
#  [ 4.  5.  6.  7.]
#  [ 8.  9. 10. 11.]]
```

`reshape(3, 4)` reads the twelve numbers in order and fills row 0 first, then row 1, then row 2.
In memory, nothing changed: the twelve numbers still lie in one row, 0 to 11.

To find `x[i, j]`, NumPy needs to know how far to step in memory for each axis. These steps are the **strides**.

- Moving one place along a row (`j` grows by 1) moves 1 place in memory.
- Moving down one row (`i` grows by 1) jumps over a whole row of 4: 4 places.

So the strides of `x` are `(4, 1)`, and every element follows one **address rule**:

$$
\text{position of } x[i, j] \;=\; \text{start} + i \cdot s_0 + j \cdot s_1
$$

For `x`, the start is 0. So `x[1, 2]` lives at $0 + 1 \cdot 4 + 2 \cdot 1 = 6$. An array is these four things:
the memory, the shape, the strides and the start.

Pick an array in the lab below, then point at or tap one of its cells. For now stay with `x`; the other buttons
are in the next sections.

*[Interactive lab: Strides — open the page to use it]*

One detail of units. NumPy’s `x.strides` counts **bytes**, not elements. Each float64 number takes 8 bytes,
so NumPy prints `(32, 8)`. Divide by `x.itemsize` (8) to get `(4, 1)`. This page always counts in elements, and
so does PyTorch’s `x.stride()`.

**Question.** An array of shape (3, 5) is contiguous, so its strides are (5, 1) and it starts at memory position 0. At which memory position is element [2, 3]?

*Answer it on the page to check your work.*

With more axes the rule is the same. The last axis has stride 1. Each axis to its left jumps over everything to
its right, so its stride is the product of the sizes to its right. An array whose strides follow this pattern is
**contiguous**: reading it in row order walks through memory one step at a time.

**Question.** An array of shape (2, 3, 4) is contiguous. What are its strides, counted in elements? Type them like a shape, for example (1, 2, 3).

*Answer it on the page to check your work.*

Now write the pattern as code. Walk the shape from the right: the step starts at 1, and after each axis it
grows by that axis’s size. `strides.insert(0, s)` puts `s` at the front of a list, so the list is finally in
left-to-right order.

**Code question.** Write `contiguous_strides`(shape): the strides of a contiguous array, in elements. Walk the shape from the right.

Fill in the blank (`____`):

```python
def contiguous_strides(shape):
    strides = []
    step = 1
    for size in reversed(shape):
        ____
    return strides

print(contiguous_strides((3, 4)))       # [4, 1]
print(contiguous_strides((2, 3, 4)))    # [12, 4, 1]
```

*Answer it on the page to check your work.*

## 2. A view: new strides, same memory

Press **x.T** in the lab. The transpose has shape (4, 3), but it has no memory of its own. It reads the same
twelve numbers, with the shape and the strides swapped. No number is moved, so a transpose of a huge matrix takes
no time.

**Question.** x = np.arange(12.0).reshape(3, 4) has shape (3, 4) and strides (4, 1), in elements. What are the strides of x.T? Type them like a shape.

*Answer it on the page to check your work.*

An array that reads another array’s memory is a **view**. NumPy can tell you when two arrays share memory:

```python
np.shares_memory(x, x.T)    # True
```

**Predict.** x = np.arange(12.0).reshape(3, 4), so x[1, 0] is 4. Then y = x.T and y[0, 1] = 100. What is x[1, 0] now?

A. 4: y is a separate array
B. 100: y and x use the same memory
C. An error: you cannot write into a transpose

*Answer it on the page to check your work.*

Slicing makes views too. `x[:, ::2]` takes every second column, so one step along its row is 2 steps in memory:
strides `(4, 2)`. `x[1:, 1:]` keeps the strides `(4, 1)` but starts at position 5. Both are in the lab.

Some operations always make a **copy**, an array with new memory: `x.copy()`, and indexing with a list such as
`x[[0, 2]]`.

With more axes, a transpose names the two axes to swap. In NumPy that is `np.swapaxes(t, 1, 2)`, and in PyTorch
`t.transpose(1, 2)`. Both swap those two sizes in the shape and the same two strides.

**Question.** t has shape (2, 3, 4) and strides (12, 4, 1). What are the strides of np.swapaxes(t, 1, 2), which has shape (2, 4, 3)? Type them like a shape.

*Answer it on the page to check your work.*

**Try it**

Press **x[:, ::2]** and look at the memory row. The dashed cells are memory this array never reads. Then press
**x[1:, 1:]**: the first five cells are dashed, because the view starts at position 5.

## 3. When reshape has to copy

`reshape` keeps the numbers in **row order**: row 0 left to right, then row 1, and so on.
If the array is contiguous, row order is memory order, and `reshape` only writes new strides. Press
**x.reshape(2, 6)**: strides `(6, 1)`, same memory.

Now try it on `x.T`. Read `x.T` in row order: 0, 4, 8, 1, 5, 9, … In memory these sit at positions 0, 4, 8, 1, 5, 9:
forward 4, forward 4, back 7. No single stride can describe that walk, so `x.T.reshape(12)` cannot be a view.
NumPy makes a copy with the numbers in the new order. Press **x.T.reshape(2, 6)** to see the new memory.

**Question.** small = np.arange(6.0).reshape(2, 3) is [[0, 1, 2], [3, 4, 5]]. What is small.T.reshape(6)[1]?

*Answer it on the page to check your work.*

**Predict.** x = np.arange(12.0).reshape(3, 4). Which of these must be a copy, with its own memory?

A. x.reshape(6, 2)
B. x.T.reshape(12)
C. x[::2]

*Answer it on the page to check your work.*

PyTorch has the same rules, with names for each case:

| PyTorch | What it does |
|---|---|
| `x.stride()` | the strides, in elements |
| `x.is_contiguous()` | `True` when row order is memory order |
| `x.T`, `x.transpose(1, 2)` | a view with swapped strides; never copies |
| `x.view(...)` | a new shape on the same memory; an error if no strides fit |
| `x.reshape(...)` | a view when possible, a copy otherwise |
| `x.contiguous()` | a copy in row order (or `x` itself, if it is already contiguous) |

Level 21 joins the attention heads. The output `out` has shape `(B, h, L, d_k)`. To put the heads side by side for
each word, the code swaps axes 1 and 2 and then joins the last two axes:
`out.transpose(1, 2).contiguous().view(B, L, d_model)`.

**Question.** Attention heads out has shape (2, 4, 10, 16) = (B, h, L, dₖ) and strides (640, 160, 16, 1). What are the strides of out.transpose(1, 2), with shape (2, 10, 4, 16)? Type them like a shape.

*Answer it on the page to check your work.*

**If you are stuck: view says “view size is not compatible with input tensor’s size and stride”. What happened?**

After `transpose(1, 2)` the strides are `(640, 16, 160, 1)`. To join the last two axes into one axis of 64,
the 4 heads × 16 numbers of one word would have to sit in one run of 64 neighbors in memory. They do not:
the next head is 160 places away. `view` never moves numbers, so it refuses.

`.contiguous()` copies the numbers into row order, with strides `(640, 64, 16, 1)`, and then `view` works.
`.reshape(B, L, d_model)` does both steps in one call.

**Deeper: Why contiguous memory is also faster**

Memory is read in blocks of neighbors (64 bytes at a time is common), not one number at a time. Reading a
contiguous array uses every number of every block. Reading down a column of a large matrix stored row by row uses one number
per block and does not use the rest. Many fast functions, on the CPU and on the GPU, first make their input
contiguous for this reason. A copy costs time and memory too, so code that runs often avoids copies it does not need.

## 4. Broadcasting is stride 0

In level 1, broadcasting “stretched” a size-1 axis to match the other array, and the stretched cells were pale
because they were not stored. Here is how NumPy does it: the stretched axis gets **stride 0**.

```python
b = np.broadcast_to(x[0], (3, 4))   # x[0] is [0, 1, 2, 3]
b.strides                           # (0, 8) in bytes → (0, 1) in elements
```

Moving down one row of `b` moves 0 places in memory, so every row reads the same four numbers.
Press **`broadcast_to`(x[0], (3, 4))** in the lab: each used memory cell is read by 3 cells of the array (×3).

**Question.** v is a float32 vector of shape (4,), and each float32 number takes 4 bytes. How many bytes of memory does `np.broadcast_to(v, (1000, 4))` store?

*Answer it on the page to check your work.*

The view from `broadcast_to` is read-only. Writing `b[0, 0] = 5` raises an error, because that one memory cell is
also `b[1, 0]` and `b[2, 0]`.

NumPy can build any view by hand with `as_strided`, if you give it the shape and the strides in bytes:

```python
from numpy.lib.stride_tricks import as_strided
as_strided(x[0], shape=(3, 4), strides=(0, 8))   # the same as broadcast_to(x[0], (3, 4))
```

Use it with care. `as_strided` checks nothing. A stride that is too big makes it read memory past the end of the
array, and you get random numbers, or a crash, with no error message.

**Deeper: Sliding windows with `as_strided`**

With strides `(8, 8)` on `np.arange(6.0)`, each row starts one number after the row before:

```python
as_strided(np.arange(6.0), shape=(4, 3), strides=(8, 8))
# [[0. 1. 2.]
#  [1. 2. 3.]
#  [2. 3. 4.]
#  [3. 4. 5.]]
```

These are all windows of 3 neighbors, with no copy. Convolutions (level N1) read their input this way.
NumPy has a safe function for exactly this: `np.lib.stride_tricks.sliding_window_view(a, 3)`, which checks the
sizes for you.

Now put the whole level into one function. `read` builds an array from memory using only the shape, the strides
and the start. The tests give it a transpose, a step, a slice and a broadcast: every view in this level.

**Code question.** Write the one line that reads element [i, j] from memory buf, given strides and a start offset. The tests use a transpose, a step, a slice and a broadcast.

Fill in the blank (`____`):

```python
def read(buf, shape, strides, offset=0):
    rows, cols = shape
    out = []
    for i in range(rows):
        row = []
        for j in range(cols):
            row.append(____)
        out.append(row)
    return out

# the memory of x = np.arange(12).reshape(3, 4)
buf = list(range(12))
print(read(buf, (3, 4), (4, 1)))         # x itself
print(read(buf, (4, 3), (1, 4)))         # x.T
```

*Answer it on the page to check your work.*

## You can now

- Find where any element lives in memory from the shape, the strides and the start.
- Say whether a transpose, a slice or a reshape is a view or a copy, before running it.
- Explain why `view` fails after `transpose`, and what `.contiguous()` does about it.