transformer

An interactive browser city that visualizes how an LLM works, with live transformer mechanics such as tokenization, attention, KV cache, residual stream, and sampling.

A PyTorch reference implementation of DeepSeek Sparse Attention from the LLMs-from-scratch project, including a lightweight indexer, top-K token selection, and a GPT-style model with KV-cache support.

A technical blog post showing a tiny diffusion transformer running on an RP2350 microcontroller to generate 128×128 face images, with details on model design, quantization, and on-device inference.

A research blog post describing an open-source small transformer trained from scratch to reach 44% on ARC-AGI-1 at very low cost, with architecture notes, ablations, and pointers to code.