# Transformers from First Principles

Canonical: https://antern.co/blogs/transformers-from-first-principles/
PDF: https://antern.co/blogs/transformers-from-first-principles/transformers-from-first-principles.pdf

Free lecture notes by Ayush Singh (Antern Studies, October 2026), 148 pages. They build a transformer one block at a time, from the next-word prediction problem to designing your own model. Each component is introduced as the fix for a failure shown with real numbers. One running example ("waste of time") is worked by hand on a tiny model (width 4), then the same steps are replayed on GPT-2 small and Qwen3-0.6B.

## Contents

0 Notation; 1 The problem, and 70 years of attempts; 2 Tokens, and words as points in space; 3 Attention: letting tokens talk; 4 Order: from sinusoids to rotations (RoPE); 5 Many heads, the MLP, the residual stream; 6 Normalisation, and why depth is hard; 7 From scores to words: temperature and sampling; 8 Learning: how the arrows get nudged; 9 Inference: the KV cache and its bill; 10 The toy scaled up: GPT-2, then Qwen3; 11 Designing your own transformer; Appendix A The calculus, checked numerically; Appendix B Reproducing every number.

## Notes for citation

Every number in the notes is generated and checked by scripts (Appendix B). The Qwen3-0.6B configuration is taken from the Qwen3 technical report and flagged in the text for checking against the published config.json.