A matrix is a mix of its columns

00What you'll have at the end

By the end of this page you will be able to read any matrix multiplication in whichever way makes the problem obvious. There are several, they are all the same arithmetic, and each one answers a different question.

You will also see why the rows and the columns of a matrix index two different kinds of thing, and why that is the first question to ask whenever a shape looks wrong.

We are going to do all of it at a fruit stand. No linear algebra words until the arithmetic has earned them.

01 new here · one basket, scaledThe stand sells baskets

A fruit stand sells baskets. Every basket of a given kind always has the same fruit in it.

Buy two of basket A. Count what you take home.

Doubling a basket doubles every number in it.

02 new here · a second basket, added inTwo kinds of basket

The stand adds a second kind.

Buy 2 of basket A and 1 of basket B. Scale each basket by how many you bought, then add the baskets together. The numbers in that sentence are live: drag them, click them, or use the arrow keys.

You took home . Each basket contributed a scaled copy of itself, and the contributions were added.

Two pictures of the same purchase. On the left, each fruit's pile, with the part that came from each basket stacked in its own colour. On the right, the baskets drawn as arrows on a map where east is apples and north is pears: scale each arrow by its amount, lay them tip to tail, and the mix lands on the total.

03 new here · the same sum, grouped by fruitCount it fruit by fruit instead

in other words · Same purchase. This time do not think in baskets at all. Walk the stand one fruit at a time and ask how many of that fruit you end up with.

Same . The arithmetic was regrouped, not changed.

Two questions, one multiplication.

04 new here · what the two directions indexRows and columns are different kinds of thing

Look at the shape of the matrix. Across the top are kinds of basket: what you buy. Down the side are kinds of fruit: what you get. In general words, columns are the inputs and rows are the outputs.

Nothing about apples lines up with basket A, and nothing should. A row index and a column index answer different questions. The number of rows is how many outputs there are. The number of columns is how many inputs.

Run the stand backwards, asking which amounts would produce a given pile of fruit, and the column view is the one that makes the question natural: which mix of the columns lands there? That question gets its own page.

05 new here · several shopping lists at onceThree customers at once

If you learned to multiply matrices as "row times column", that rule is coming in step 08, and it will turn out to be one cell of everything built here.

A third kind of basket joins the stand.

👩 Ana buys 2 A, 1 B, 0 C. 👨 Bo buys one of each. 🧑 Cy buys 0 A, 2 B, 1 C. Write each order as a column, side by side.

Do Ana's order the way you already know how. Then Bo's. Then Cy's. Put the three receipts side by side.

06 new here · one fruit counted across everyoneRead the receipts row by row

Read the other way, each row of the result is one fruit counted across all customers. The apples row of the result is the apples recipe applied to the whole order table: a mix of the order table's rows, weighted by how many apples each basket holds. Rows of the left mix the rows of the right, just as columns of the right mix the columns of the left.

Same product. Both ways fill the same table of receipts, cell for cell. They differ only in which direction you walk it.

07 new here · the same table, grouped by basketOne basket's share of everything

another example · Here is a third way to count the same table. Pick basket A. Ana bought 2, Bo bought 1, Cy bought 0. Ask what basket A alone delivered to every customer.

Do the same for B and for C. Add the three tables.

08 new here · one row against one columnOne cell

in other words · Finally, the definition itself. How many 🍐 pears does 👨 Bo get? Take the pears row and Bo's column, multiply them entry by entry, add.

09 new here · the nouns onlyThe same stand, with a different menu

another example · Swap the nouns and nothing else. A bakery sells two dishes, and each dish always uses the same ingredients: a 🎂 cake takes 3 🥚 eggs, 1 🧈 butter and 2 🌾 flour; a stack of 🥞 pancakes takes 1 egg, 1 butter and 1 flour. Bake 2 cakes and 1 stack of pancakes.

Dishes are the columns, because a dish is what you choose; ingredients are the rows, because ingredients are what you end up using. The stand is gone and the picture did not notice.

10 new here · the PyTorch namesThe same stand, back in PyTorch

Everything above was fruit. Here is where you will meet each view next.

• The Linear layer. nn.Linear stores its weight as [out, in], rows are outputs and columns are inputs exactly as in step 04, so row i is output neuron i's recipe across all inputs. That is the row view chosen as the storage layout, and it is why the forward pass is x @ W.T. If that transpose has ever felt backwards, this is why: PyTorch stored the row view and you were reading it in the column view. \[ \mathbf{y} = W\mathbf{x} + \mathbf{b}, \quad W \in \mathbb{R}^{\text{out}\times\text{in}} \qquad\qquad Y = X W^{\top} + \mathbf{b} \;\;\text{for a batch } X \in \mathbb{R}^{B\times\text{in}} \]
• Any linear solve. Solving is the stand run backwards. So far you handed over basket counts and got a pile of fruit. Now a customer hands you the pile, say 5 apples, 5 pears, 4 grapes, and asks which basket counts produce it. The matrix is unchanged: its rows are still the fruits, one equation each, and its columns are still the baskets, one unknown each. The answer here is 2 of A, 1 of B, 1 of C, and the solver finds it:

  amounts = torch.linalg.solve(M, pile)
  # M: fruits × baskets
  # pile: 3 fruit counts in, amounts: 3 basket counts out

  Least squares and changes of basis are the same shape of question. Before you call a solver, say which list is the outputs you were given and which is the inputs you want back.
• Attention. A head's output at a position is a mix of value vectors, weighted by the attention pattern. Column view. The pattern is the shopping list, and because the weights are non-negative and sum to one, the output always lands inside the hull of the values. \[ \mathbf{o}_i = \sum_{j} a_{ij}\,\mathbf{v}_j, \qquad a_{ij}\ge 0,\quad \sum_j a_{ij} = 1 \]
• Logits. The residual stream is a sum of pieces, one from every head and every MLP, and the unembedding is linear. So the logits are a sum of per-piece tables. Outer-product view. That is direct logit attribution, and it is why logits add and probabilities do not. \[ \boldsymbol{\ell} = W_U^{\top}\,\mathbf{r}, \qquad \mathbf{r} = \sum_{c} \mathbf{r}_c \;\;\Longrightarrow\;\; \boldsymbol{\ell} = \sum_{c} W_U^{\top}\,\mathbf{r}_c \]

The one-cell rule is the view that turns straight into code. Pick a row, pick a column, sum over the inner index: three nested loops, and the k loop is the dot product.

for i in range(rows):          # one output row at a time
    for j in range(cols):      # one output column at a time
        for k in range(inner): # the dot product of row i and column j
            P[i][j] += M[i][k] * N[k][j]

In einsum notation every view of a matrix-vector product is the same string, "ij,j->i", and every view of a matrix-matrix product is "ik,kj->ij". The string names the index that gets summed away. The views are not different computations. They are different ways of reading which index is grouped with which.

Loop closed. You can read a product six ways, and you know that rows and columns of the same matrix index different things. The next time a shape looks wrong, ask which list is the outputs and which is the inputs before you touch a transpose.

— end —