# Qwen architecture

Canonical: https://antern.co/blogs/qwen-architecture/

An interactive 3D transformer study by Antern. Follow tokens, inspect matrix cells and multiplication terms, rotate the model, select any of 28 layers, and inspect forward passes, gradients and SGD updates.

## The teaching model

A small Qwen-style decoder: hidden width 8, 28 layers, two query heads sharing one key/value head, head dimension 4, feed-forward width 16. Computation includes RMSNorm, rotary positional encoding, causal attention, SwiGLU, residual connections, and tied embedding/output weights.

This is not pretrained Qwen and does not use Qwen weights or its tokenizer. It uses a small word/punctuation vocabulary and a browser-local training corpus. Predictions are educational and need not be meaningful.

## Learning path

Text and tokens → embedding lookup → normalization → query/key/value → rotary position encoding → attention scores → causal mask and softmax → weighted values → feed-forward → all 28 blocks → prediction → dataset and learning.

## Training and privacy

Edit the corpus, inspect within-document windows and right-shifted targets, select batches, and train with valid-token mean cross-entropy and SGD. Reverse-mode automatic differentiation computes gradients. Inputs, datasets and weights remain in the browser tab. Export records before reloading. No dropout, KV-cache decoding, Adam or large-model training is simulated.