Plate 2 · 9 min read
Why LoRA works: fine-tuning a giant model with a tiny matrix
I fine-tuned GPT-2 on movie reviews and found the weight update could be rebuilt from a handful of small pieces. This essay lets you rebuild it yourself, one rank at a time.