Structured technical notes on LLMs, multimodal models, quantization, training, inference, and AI systems.https://modelfoundations.com/https://modelfoundations.com/notes/chinchilla-scaling/https://modelfoundations.com/notes/chinchilla-scaling/Chinchilla showed that 2022-era LLMs were dramatically undertrained, replacing 'scale parameters' with 'scale parameters and tokens together.' A 2024 replication attempt found real problems in one of its three analyses — and the headline rule survived anyway.Sun, 12 Jul 2026 00:00:00 GMThttps://modelfoundations.com/notes/gptq/https://modelfoundations.com/notes/gptq/GPTQ quantizes a 175B model to 3–4 bits in a few GPU-hours by turning layer-wise quantization into a sequence of cheap Hessian-guided weight updates — and by noticing that the expensive part of the classic algorithm was never necessary.Sun, 05 Jul 2026 00:00:00 GMThttps://modelfoundations.com/notes/flashattention-2/https://modelfoundations.com/notes/flashattention-2/FlashAttention-1 made attention IO-aware; FlashAttention-2 gets another ~2× by fixing how the work is divided — fewer non-matmul FLOPs, parallelism over sequence length, and warp-level partitioning that stays out of shared memory.Sun, 28 Jun 2026 00:00:00 GMT