
LoRA: fine-tuning a big model by training a small add-on
In short: LoRA (low-rank adaptation) fine-tunes a large model by training small adapter matrices in place of all its weights, which cuts memory and storage.
LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method, introduced in June 2021, that freezes a model's original weights and trains small low-rank matrices injected into its layers. Because only these adapters are trained, fine-tuning needs far less GPU memory and storage, and many task-specific adapters can share one base model.
LoRA in plain words
Rewriting a whole textbook for a new course is expensive. LoRA adds a slim booklet of corrections and extras that sits alongside it.
How it works
For selected weight matrices, LoRA learns two small low-rank matrices whose product approximates the needed change. Only these are updated during training.
Against full fine-tuning of GPT-3 175B, the paper reports 10,000 times fewer trainable parameters and 3 times less GPU memory, with no added inference latency.
QLoRA, from May 2023, backpropagates through a frozen 4-bit quantised model into LoRA adapters. Its authors say that is enough to fine-tune a 65B-parameter model on a single 48GB GPU.
Example
As an illustration with Hugging Face's PEFT library:
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(base_model, config)
# only the adapter matrices are trainable now
Where you'll meet it
- Hugging Face PEFT and TRL, and tools such as Unsloth
- Fine-tuning features in model platforms and clouds
- QLoRA setups that fine-tune a quantised base model on one GPU
- Serving systems that load many adapters on one base model
What it changes for you
For developers, fine-tuning a large model no longer needs a full-weight training budget. My view: for platform teams, one base model with many small adapters is cheaper to serve than many full models. For governance, each adapter is a model variant that needs its own evaluation record.
LoRA vs full fine-tuning
Full fine-tuning updates every weight and needs much more memory and storage. LoRA updates small adapters, and the original paper reports quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3.
Common mistake
My view: tuning LoRA settings before checking the data. Inconsistent or low-quality examples sink results whatever the rank. Spend the first day on the dataset before touching hyperparameters.
Related terms
Fine-tuning · Quantization · Open-weights model · Small language model
From the archive: Fine-tune to change behaviour, use RAG to change knowledge
Fix the dataset before you tune LoRA rank or any other setting.


