Transformers from First Principles

Why another explanation of transformers?

Most explanations show you the diagram and ask you to remember it. These notes do the opposite. Every part of a transformer exists because something simpler broke, so each chapter builds the simple thing first, watches it break with real numbers, and only then brings in the fix. Could you have invented attention in 2014, given what people knew then? By the end, I think you will say yes.

One running problem goes through the whole book. A customer replies waste of time, and the model has to answer well. The same three words and the same tiny weights follow you from the first page to the last, small enough to work out by hand. Then the same steps run on real models, GPT-2 and Qwen3-0.6B.

Who it is for

Someone who left school decades ago, and a PhD student, at the same time. Every idea is explained three ways: a picture you can hold in your head, the exact mathematics with every shape written, and the general principle underneath. Skip the rung you already know. Every chapter opens with a short maths check and a refresher, so nobody is left behind.

What you will be able to do

• Run a full forward pass of a small transformer by hand, for a whole batch of sentences.
• Explain what training does to every weight, as arrows turning and stretching.
• Read any model's config.json and work out its parameter count, training cost and KV-cache memory.
• Design a transformer for a given compute, memory and latency budget, and defend every choice.

How the notes work

• Built block by block. A diagram of the whole machine opens every chapter and shows which block you are adding.
• Matrices you can see. Every multiplication is drawn: the row, the column, the cell they produce, and the sum written out.
• Predict first. Before each worked example you write down a guess. A wrong guess shows you exactly what to look at again.
• Every number is real. Nothing is typed by hand. Scripts compute every number and check it, and Appendix B shows how to rebuild them all.
• Check yourself. Each wrong option in a question is a common mistake, and the answers explain which one.

Contents

1. 0Notation
2. 1The problem, and 70 years of attempts
3. 2Tokens, and words as points in space
4. 3Attention: letting tokens talk
5. 4Order: from sinusoids to rotations (RoPE)
6. 5Many heads, the MLP, the residual stream
7. 6Normalisation, and why depth is hard
8. 7From scores to words: temperature and sampling
9. 8Learning: how the arrows get nudged
10. 9Inference: the KV cache and its bill
11. 10The toy scaled up: GPT-2, then Qwen3
12. 11Designing your own transformer
13. AThe calculus, checked numerically
14. BReproducing every number

Read it here

If the viewer does not load on your device, open the PDF directly.