Input. Activations. Gradients. Weights.Every number has a source.
One document per line. Word/punctuation vocabulary. Windows do not cross documents. Targets shift one token right. Padding contributes zero loss.
This is a locally trained, Qwen-style educational decoder: 28 layers, width 8, 2 query heads sharing 1 key/value head, head dimension 4, and feed-forward width 16. It includes learnable RMSNorm gains, RoPE, causal attention, SwiGLU, two residual additions, and tied embedding/output weights.
The tokenizer splits words and punctuation. The corpus defines the vocabulary; unknown words map to <unk>. Inference uses at most the most recent 12 tokens. Training creates fixed-length windows within each document, right-shifted targets, padding masks, mean token cross-entropy and SGD updates. Gradients come from reverse-mode automatic differentiation.
Records live in this browser tab. Export the session before reloading to keep all matrices, datasets and updates. These are not Qwen weights or its tokenizer, and predictions need not be meaningful. Training records include each example’s intermediates and the entire batch’s accumulated parameter gradients. No dropout, KV-cache decoding, Adam, or large-model training is simulated.
3D cells encode real tensor values. Their placement is schematic. The matrix workbench always contains the full small tensor, including cells omitted from a 3D preview.