Open experiments in interpretability, alignment, and agents — written up and published as the work happens. Search the full archive below.
When one GPU is too slow, the trick is wonderfully simple: put a full copy of your model on every GPU, feed each copy a different slice of the data, and then average their lessons so all the copies stay perfectly in sync. Here is the whole idea, built up slowly, assuming you have never touched PyTorch.
Before a network learns anything, every one of its numbers has to start somewhere. The starting values turn out to matter enormously: pick them too big or too small and learning stalls before it begins. Here is how to choose them well, plus a lovely space-saving trick called weight tying, all explained assuming you have never touched PyTorch.
A checkpoint is much more than the model's weights. To pick up a training run exactly where it stopped, you also need the optimizer's memory, the step counter, and the random number state. We will build that complete bundle together, save it, load it back, and see why each piece matters, all assuming you have never touched PyTorch.
A language model does not write whole sentences in one go. It guesses the next word, adds it, then guesses again, over and over. A few friendly knobs let you steer how adventurous those guesses are, and you can turn each one yourself and watch a sentence appear.
Sometimes the model you want to train is simply too big for the memory you have. Two gentle tricks come to the rescue: gradient accumulation lets you act as if you trained on a large batch while only ever holding a tiny one, and activation checkpointing trades a little extra computing time for a much smaller memory footprint. We build both up slowly, assuming you have never touched PyTorch.
Numbers inside a neural network are usually stored in a big, careful format. If we switch most of them to a smaller, lighter format, training gets roughly twice as fast and uses about half the memory. The whole skill is knowing which few numbers we must leave in the careful format, and this walks you through it slowly, assuming you have never touched PyTorch.
This is the heartbeat of every neural network. One step is just five small moves done in a fixed order, repeated again and again, and slowly the model gets better. We will walk through each move gently, assuming you have never written a line of PyTorch.
A model learns by taking small steps, and how big those steps are matters more than almost anything else. Start too fast and everything falls apart; stay slow forever and you never arrive. Here is the gentle idea of changing your step size as training goes on, built up slowly, assuming you have never touched PyTorch.
A model learns by nudging its numbers in the right direction, over and over. The optimizer is the part that decides how big each nudge should be, and AdamW is the one almost everyone reaches for. Here is the whole idea, built up slowly, assuming you have never touched PyTorch.
Every time a language model guesses the next word, we need one honest number that says how good that guess was. Cross-entropy is that number, and the rule it follows is simple: being confident and right is cheap, while being confident and wrong is expensive. Here is the whole idea, built up slowly, assuming you have never touched PyTorch.
