Two resources have shaped how I think about what it means to build a large language model from scratch. The first is Andrej Karpathy's "Intro to Large Language Models," a one hour talk that distills the core ideas behind modern LLMs with unusual clarity. The second is the Ultra-Scale Playbook by the Nanotron team at Hugging Face, a deep technical guide to training these models across GPU clusters. This post is how I would use both to re-teach myself pretraining from zero.
Full credit to Andrej Karpathy, founding member of OpenAI and former head of AI at Tesla, for the conceptual framing, and to the Nanotron team at Hugging Face for one of the most transparent pieces of distributed training literature published to date.
Karpathy's framing is useful here. A large language model is, at its core, a next-token prediction machine. Given a sequence of tokens, it predicts what comes next. That's it. The emergent behavior, the reasoning, the apparent world knowledge, all of it falls out of doing this prediction task at scale across enormous amounts of text. The model learns by compression: to predict well, it has to learn structure.
The training process has two stages. Pretraining is where the model learns from raw text at internet scale, no labels, no instructions, just next-token prediction across trillions of tokens. What comes out is a base model, a powerful but raw artifact that completes text rather than follows instructions. Post-training is where you shape that artifact into something useful: supervised fine-tuning on curated data, reinforcement learning from human feedback, preference optimization. The base model is the substrate. Everything useful is built on top of it.
Before you can train anything, you have to understand where memory goes. The Ultra-Scale Playbook breaks it down cleanly. For a transformer, you are storing four things: model weights, gradients, optimizer states, and activations. In mixed precision training with the Adam optimizer, a 7B parameter model already requires around 140GB just for weights, gradients, and optimizer states before you've loaded a single batch of data. A single H100 has 80GB. You are already over budget.
Activations make this worse. Unlike weights and gradients, activation memory scales with batch size and quadratically with sequence length. At short sequences it's negligible. At 4k tokens it starts mattering. At 32k tokens it dominates everything else.
The first tool to handle this is activation recomputation, also called gradient checkpointing. Instead of storing all intermediate activations for the backward pass, you discard them during the forward pass and recompute on the fly during the backward. Selective recomputation, which discards only the attention activations since those are cheapest to recompute and most expensive to store, gives you a 70% reduction in activation memory at around a 2.7% compute cost. Most frameworks using FlashAttention already do this automatically.
The second tool is gradient accumulation. Instead of computing one giant batch, you split it into micro-batches, accumulate gradients across them, and update the weights once. This lets you simulate a large global batch size on a single GPU without blowing memory.
Once you run out of tricks on a single GPU, you scale out. The Ultra-Scale Playbook covers five dimensions of parallelism, each solving a different constraint.
Data parallelism is the obvious one: replicate the model across GPUs, each processes a different batch, sync gradients via all-reduce. It works well until the all-reduce communication overhead starts dominating at hundreds of GPUs.
ZeRO takes data parallelism further by sharding optimizer states, gradients, and parameters across GPUs instead of replicating them. ZeRO-3, which shards everything, lets you fit a model across GPUs even when no single GPU has enough memory, at the cost of extra all-gather communication when you need parameters during the forward pass.
Tensor parallelism shards the weight matrices themselves across GPUs, splitting attention heads and feedforward dimensions so each GPU computes a portion of every layer. The tradeoff is that this adds communication directly in the critical path of every forward pass, so it only works efficiently with fast intra-node interconnects like NVLink. In practice TP is capped at 8, the number of GPUs on a single node.
Pipeline parallelism solves the cross-node problem by splitting layers across nodes and passing activations sequentially. GPU 1 runs layers 1-4, GPU 2 runs layers 5-8, and so on. The cost is idle time, the pipeline bubble, where some GPUs are waiting for others. Sophisticated schedules like 1F1B, interleaved stages, and DeepSeek's DualPipe approach reduce this bubble to near zero by decomposing the backward pass into finer-grained operations and filling idle slots.
Context parallelism handles the case where the sequence is simply too long for even a single node to handle, splitting tokens across GPUs and using Ring Attention to share key-value pairs during the attention computation.
The technical depth of the Ultra-Scale Playbook is only useful once you understand what you are trying to build. Karpathy's talk grounds you in that. A language model at inference time is running one forward pass at a time, predicting the next token, sampling from a distribution, feeding that token back in. The training loop is inverting this: given what came next, adjust the weights so the model would have predicted it better. Do this across a trillion tokens and the model learns structure that generalizes far beyond memorization.
The other thing Karpathy makes clear is that the base model is not the product. The base model is the starting point. The actual useful artifact requires post-training, and post-training requires understanding what the base model learned and where it fell short. Knowing both halves is what lets you reason about why a fine-tuned model behaves the way it does.
Watch Karpathy's intro talk first. It is one hour and it gives you the mental model everything else plugs into. Then read the Ultra-Scale Playbook high-level overview and single GPU training sections. At that point you understand the memory constraints and why they matter. Then read the data parallelism and ZeRO sections, then tensor and pipeline parallelism. By the end you have a complete picture from a single GPU training loop to a 512-GPU cluster.
The goal is not to memorize the schedules or the communication primitives. The goal is to build the intuition that lets you look at a training run, see it is bottlenecked on cross-node communication, and know that pipeline parallelism is probably your lever, not tensor parallelism. That kind of reasoning is what separates someone who has read the papers from someone who can actually make decisions about a training infrastructure.
I am still building that intuition. This post is as much a note to future me as anything else.