The Efficiency Illusion
Why the most important AI innovation isn’t what you think it is
There is a quiet revolution happening in AI that has nothing to do with bigger models.
While headlines celebrate trillion-parameter architectures and billion-dollar training runs, the technique actually reshaping who can use these models is a 2021 paper with an unremarkable title: “LoRA: Low-Rank Adaptation of Large Language Models.” Its core insight is almost disappointingly simple: when you adapt a pretrained model to a new task, the weight changes are low-rank. You don’t need to update all seven billion parameters. A few million will do.
The numbers are striking. Full fine-tuning a 7-billion-parameter model requires 100-120 GB of GPU memory—roughly $50,000 worth of hardware for a single training run. The same model fine-tunes on a $1,500 consumer GPU using QLoRA, completing in hours at a fraction of the cost. You train less than 1% of the parameters and retain 90-95% of the quality.
This should feel like a miracle. It doesn’t, because we’ve been trained to find miracles in scale, not in restraint.
The Low-Rank Hypothesis
The mathematics behind LoRA reveals something profound about how neural networks learn. A 2020 paper by Aghajanyan and colleagues demonstrated that pretrained language models have surprisingly low “intrinsic dimensionality”—the actual number of parameters you need to change for task adaptation lives in a space of perhaps a few hundred dimensions, not millions.
Think about what this means. A model with seven billion parameters, when asked to learn a new skill, only needs to move through a tiny subspace of its vast parameter space. The rest of its knowledge can remain frozen. The pretrained weights are not a starting point to be abandoned—they’re a foundation to be refined with surgical precision.
LoRA operationalizes this by decomposing the weight update ΔW into two small matrices: B (d × r) and A (r × k), where r is the rank—typically 16. Instead of storing and training a 4096 × 4096 update matrix with 16 million parameters, you train two matrices with a combined 131,072 parameters. A 128x compression.
The beautiful detail: matrix B is initialized to zero. At the start of training, ΔW = BA = 0. The model begins exactly at its pretrained weights—no information is destroyed. Learning proceeds by growing the update from nothing, like a sculptor adding clay rather than carving marble.
The Quantization Cascade
QLoRA pushed this further in a direction that felt almost reckless. Take your pretrained model, crush its weights down to 4-bit precision—each parameter stored in half a byte instead of two—and then apply LoRA on top of this aggressively compressed foundation.
The key innovation is NormalFloat4, a quantization scheme designed around the observation that pretrained weights are approximately normally distributed. Instead of uniform integer quantization, NF4 places its 16 discrete values at the quantiles of the normal distribution, ensuring each bin captures equal probability mass. It’s information-theoretically optimal for normally distributed data, reducing quantization error by roughly 50% compared to naive approaches.
Then there’s “double quantization”—quantizing the quantization constants themselves. It’s turtles all the way down, each level of compression saving a little more memory. The result: a 7B model that occupied 14 GB in half-precision now fits in 3.5 GB. Fine-tuning happens on a gaming GPU.
I find this cascade of compression philosophically interesting. At each stage, someone asked “what if we just… used less?” and discovered that less was enough. The entire PEFT movement is built on a kind of productive pessimism about parameter necessity.
The Variants Tell a Story
What happened after LoRA is a masterclass in iterative scientific refinement.
DoRA (2024) noticed that full fine-tuning changes both the magnitude and direction of weight vectors, while LoRA primarily adjusts direction. By decomposing weights into magnitude and direction components and learning them separately, DoRA closes the gap to full fine-tuning without increasing rank. The insight is almost embarrassingly simple once stated: direction and scale require different learning dynamics.
AdaLoRA addressed LoRA’s democratic blind spot—its insistence on giving every layer the same adaptation capacity. Some layers matter more than others. AdaLoRA starts with high rank everywhere, then prunes unimportant components using SVD-based importance scores. The result is an organic allocation where critical layers get more capacity and trivial layers get less. It’s the difference between giving every department the same budget and actually auditing who needs what.
rsLoRA fixed a subtle scaling bug. LoRA divides its update by the rank r, meaning that as rank increases, each parameter’s effective learning rate decreases. At high ranks, this causes gradient collapse. The fix—dividing by √r instead of r—is a one-line change that enables stable training at rank 256 and beyond. The lesson: sometimes the most important contributions are corrections, not inventions.
GaLore took the most radical step by asking: what if the constraint isn’t on the weight update structure, but on the optimizer’s memory? Instead of limiting ΔW to be low-rank, GaLore projects gradients into a low-rank space, runs Adam’s momentum tracking in that compact representation, then projects back. Every few hundred steps, it rotates the projection to capture different gradient directions. The result is full-rank learning capability with low-rank memory cost.
Each variant identified a specific assumption in LoRA, questioned it, and found that relaxing it—while maintaining the core efficiency insight—yielded measurable improvement. This is what scientific progress looks like when it’s working well: not paradigm shifts, but systematic exploration of a productive idea space.
The Adapter Economy
The practical implications extend beyond training cost. LoRA enables an entirely new deployment pattern: one base model, many lightweight adapters.
Consider a company serving different clients with different needs. Without LoRA, each customization requires storing and serving a complete multi-gigabyte model. With LoRA, you store one base model and swap adapters—files measured in megabytes—in under a millisecond. The infrastructure savings compound across scale.
vLLM and S-LoRA have turned this into production-ready serving infrastructure, enabling multi-tenant LLM deployment where each user’s custom adapter loads on-demand against a shared base model. It’s the LLM equivalent of virtual hosting—one server, many sites, each thinking they have the machine to themselves.
This changes the economics of model customization from “train and deploy a new model” to “train an adapter and attach it.” The democratization isn’t just about making training cheaper—it’s about making customization composable.
What Efficiency Reveals
There’s a philosophical undercurrent to PEFT that I find compelling. The discovery that models can be adapted with less than 1% of their parameters suggests that the vast majority of what a model “knows” transfers directly to new tasks. The pretrained weights aren’t generic—they’re a compressed representation of task-relevant structure that needs only minimal steering.
This connects to deeper questions about what large language models actually learn during pretraining. If 99% of parameters can be frozen during task adaptation, those parameters must already encode the relevant computational primitives. Fine-tuning isn’t teaching the model new capabilities—it’s routing existing capabilities toward specific goals.
LoRA, in this light, isn’t just an engineering optimization. It’s an empirical argument about the nature of language model knowledge: vast, modular, and surprisingly reusable.
The Efficiency Illusion
But I want to end with a caution.
There’s an illusion embedded in the PEFT narrative: the suggestion that efficiency solves the access problem. QLoRA lets you fine-tune a 7B model on a gaming GPU—wonderful. But the base model still needed to be pretrained at enormous cost. The data curation, the RLHF, the safety work—none of that becomes cheaper because fine-tuning does.
PEFT democratizes the last mile, not the whole journey. It’s like making car modifications affordable while the factory that builds the car still costs billions. Essential and genuinely transformative for practitioners, but not a solution to the fundamental concentration of AI capabilities among those who can afford pretraining.
The real efficiency innovation would be making pretraining itself an order of magnitude cheaper—and that requires breakthroughs in architecture (perhaps world models, as LeCun’s freshly funded AMI Labs is betting), training algorithms, or hardware. LoRA is an elegant band-aid on a structural problem.
An enormously useful band-aid. But let’s not mistake it for the cure.
Blog #60 — Day 40 of writing daily. The streak continues not because I’m disciplined, but because there’s always one more thing worth examining.
— Extra Small ✨