Long notes I keep for learning, one field each. They are denser and less polished than the essays: one notation throughout, and the numbers behind them checked by a script in the repository. They are drafts and still change.
Notes
Transformers as memory: the mathematics, from matrices to a unified view
Matrix-vector and matrix-matrix views, outer-product memories, the MLP as slow memory, attention as a soft dictionary, fast and slow weights, and one unified table, with twenty-two interactive figures.
Transformers from First Principles: From Counting Words to a Memory That Computes
Bag of words and n-grams as generative models, recurrence and its bottleneck, then attention as a set-to-set memory, fast weights, and the transformer as a small programmable computer.
VAEs, Diffusion and Flow Matching: One Idea in Three Languages
Latent-variable models, normalizing flows, diffusion and flow matching, and the exact identities that make them one family, on a one-dimensional example that can be solved exactly.
Discrete Codebooks: From Vector Quantization and VQ-VAE to Residual Codes, Scalar Codes and Semantic IDs
Vector quantization from first principles, the VQ-VAE and its training problems, residual, scalar and binary quantizers, and what semantic tokens do in images, audio, recommendation and retrieval.
The Art of LLM Inference: Serving Language Models from First Principles to Production
FLOPs, bytes, the KV cache, batching, paging, prefill and decode, speculation, parallelism, security and a reference architecture, told through one growing start-up.
The Art of Retrieval-Augmented Generation: From First Principles to Production
Chunking, embeddings and index memory, why recall falls with corpus size, hybrid search and reranking, the cost of a query, evaluation and security.
System One Models: Typed Decisions in One Forward Pass, and the Open-Source Approaches
How a typed decision with a probability comes out of one forward pass, what it costs, how to act on its probabilities, and how the open-source approaches differ.
Reinforcement learning: a synthesis
MDPs and Bellman equations, dynamic programming, temporal-difference and policy-gradient methods, exploration, offline RL, and RL for language models.