← back to the archiveCover illustration for “LoRA: fine-tuning a big model by training a small add-on”
TERM7mo ago·Published · Updated ·by Andy Padia

LoRA: fine-tuning a big model by training a small add-on

In short: LoRA (low-rank adaptation) fine-tunes a large model by training small adapter matrices in place of all its weights, which cuts memory and storage.

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method, introduced in June 2021, that freezes a model's original weights and trains small low-rank matrices injected into its layers. Because only these adapters are trained, fine-tuning needs far less GPU memory and storage, and many task-specific adapters can share one base model.

LoRA in plain words

Rewriting a whole textbook for a new course is expensive. LoRA adds a slim booklet of corrections and extras that sits alongside it.

How it works

For selected weight matrices, LoRA learns two small low-rank matrices whose product approximates the needed change. Only these are updated during training.

Against full fine-tuning of GPT-3 175B, the paper reports 10,000 times fewer trainable parameters and 3 times less GPU memory, with no added inference latency.

QLoRA, from May 2023, backpropagates through a frozen 4-bit quantised model into LoRA adapters. Its authors say that is enough to fine-tune a 65B-parameter model on a single 48GB GPU.

Example

As an illustration with Hugging Face's PEFT library:

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
                    lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(base_model, config)
# only the adapter matrices are trainable now

Where you'll meet it

  • Hugging Face PEFT and TRL, and tools such as Unsloth
  • Fine-tuning features in model platforms and clouds
  • QLoRA setups that fine-tune a quantised base model on one GPU
  • Serving systems that load many adapters on one base model

What it changes for you

For developers, fine-tuning a large model no longer needs a full-weight training budget. My view: for platform teams, one base model with many small adapters is cheaper to serve than many full models. For governance, each adapter is a model variant that needs its own evaluation record.

LoRA vs full fine-tuning

Full fine-tuning updates every weight and needs much more memory and storage. LoRA updates small adapters, and the original paper reports quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3.

Common mistake

My view: tuning LoRA settings before checking the data. Inconsistent or low-quality examples sink results whatever the rank. Spend the first day on the dataset before touching hyperparameters.

Fine-tuning · Quantization · Open-weights model · Small language model

From the archive: Fine-tune to change behaviour, use RAG to change knowledge

Fix the dataset before you tune LoRA rank or any other setting.

#field-guide#fine-tuning#models#training

Sources

  1. arxiv.org/abs/2106.09685
  2. arxiv.org/abs/2305.14314
  3. huggingface.co/docs/peft/package_reference/lora
← older drop
Agentic workflow: model steps chained by your code
newer drop →
AI governance: who decides how AI is used, and how

related drops

explore all 557 drops →
← back to the archiveday 175