---
title: "LoRA fine-tuning: adapting a model without rewriting it"
description: "See how LoRA trains a task update in two thin matrices while the base model stays frozen, and why rank sets a hard floor on what the adapter can learn."
published: 2026-07-18
updated: 2026-07-18
canonical: https://protrailblazer.com/posts/lora-fine-tuning/
tags: [machine-learning, demo, lora, fine-tuning, peft, adapters, frozen-weights, low-rank]
---

# LoRA fine-tuning: adapting a model without rewriting it

**Answer:** LoRA (low-rank adaptation) fine-tunes a model by freezing all of its original weights and training a small add-on instead. Two thin matrices, A and B, learn an update that gets added on top of a frozen weight matrix, so the base model never changes and only a tiny fraction of parameters train. The demo below trains a real adapter live: watch the loss fall, sweep the rank, and see why an adapter with too little rank hits a floor it can never train past.

The interactive below runs a real low-rank adapter against a seeded 12x12 target weight matrix. Hit train, sweep the rank slider, swap task presets, and tap the merged or adapter-off toggles to see the base model snap back instantly.

*Interactive demo: [open the lora fine tuning demo](https://protrailblazer.com/demos/lora-fine-tuning/)*

## What you're seeing

The canvas is split into three columns. On the left, a 12x12 grid shows the frozen base matrix W in gray, exactly as it was after pretraining. The base cells never change color or value during training. In the center, the low-rank adapter overlays W as a translucent red sheet (ΔW = (α/r)·B·A). Each step that sheet brightens or shifts as the adapter learns. On the right, output bars compare three signals against white target ticks: base only, adapter output, and the white seeded target. The closer the adapter bars land to the ticks, the lower the loss.

The HUD in the top-right tracks rank, training step, current loss, and a white rank-floor line that shows the minimum loss achievable at the current rank. When the live loss curve flattens onto that white line, the adapter has learned everything it can. Below the loss display, the parameter panel lists exact counts for the demo (the tiny 12x12 matrix) alongside an illustrative 7B-scale estimate that updates when you switch the target-modules preset between query, value, or all-attention layers.

A few things to try:

- **Drop the rank to 1 and watch where the loss curve flattens.** The white rank-floor line is above zero, meaning no amount of extra training steps can close the gap. The update you are trying to learn has more structure than a rank-1 matrix can hold.
- **Raise rank to 4 or 8 on the same task and compare.** The floor drops toward zero and the adapter converges faster. More rank means more capacity to represent the target update, not just a dial that makes things better in general.
- **Switch between the task presets (tone, domain vocab, formatting) at the same rank.** Each preset seeds a different target ΔW, so the same rank sits at a different floor depending on how structured the task update actually is.
- **Toggle "merged view" while training.** The display folds ΔW into W and redraws as a single combined matrix (W' = W + ΔW). This is what serving looks like after deployment: one matmul, zero extra overhead. Toggle it off and the two matrices separate again.
- **Hit "adapter off" mid-run.** The output bars snap back to the base-only values instantly, showing that the base model is unchanged and the adapter is a removable layer, not an edit.

## How LoRA actually works

A standard weight matrix in a transformer has dimensions d×d, so its full update during fine-tuning would require training d×d parameters per layer. For a 4096-dimensional model that is roughly 16 million parameters per matrix. LoRA replaces that full update with a product of two thin matrices: A with shape r×d (down-projection) and B with shape d×r (up-projection), where r is far smaller than d. The update ΔW = (α/r)·B·A, and the adapted output becomes (W + ΔW)x.

During training, W is completely frozen. Only A and B receive gradients via backpropagation. Because r is small, the number of trainable parameters collapses. At rank 4 on a 4096-dimensional query matrix, you train 4×4096 + 4096×4 = 32,768 parameters instead of 16 million. That is about 0.2 percent of the original parameter count. The demo's 12x12 matrix makes the ratio more vivid: at rank 2 the adapter is two rows and two columns, a third of the base parameter count on this toy scale, but the same recipe on the illustrative 7B estimate trains around 0.007 percent.

B is initialized to all zeros. This means ΔW = 0 at step zero, so fine-tuning begins from the exact base model behavior and does not need to unlearn random noise first. A is initialized from a random Gaussian. The alpha (α) hyperparameter controls the scale of the adapter's contribution relative to its rank, functioning as a stable learning-rate multiplier that does not need retuning when you change r.

The loss function the demo minimizes is the Frobenius distance between the adapter output (ΔW·x) and the target update (ΔW*·x) for a batch of random inputs. Real LoRA training minimizes a task loss (cross-entropy on next-token prediction, for example) through the same two matrices. The gradient descent update reaches A and B through the chain rule exactly as in full fine-tuning, just with far fewer parameters to update.

## Why rank is capacity, not quality

A rank-r matrix can only represent vectors that lie in an r-dimensional subspace. If the true update ΔW* has a higher rank than r, there is no setting of A and B that can reproduce it exactly. The best you can do is the closest rank-r approximation, which leaves a residual that the training loop can never eliminate. That is the white floor line in the demo.

This is not a bug. Most task-specific updates to a pretrained model are low-rank in practice, which is why LoRA works. A language model already knows grammar, facts, and how to follow instructions; fine-tuning on a narrow task (summarizing legal contracts, for example) shifts the model in a direction that lives in a small subspace of the full weight space. Rank 4 to 16 captures that shift in most published experiments, though the right value depends on the model, the task, and how different the target distribution is from the original training mix.

LoRA is most often applied to the query and value projection matrices inside attention layers (see the target-modules preset in the demo). The demo's 7B illustrative estimate updates when you switch between query-only, value-only, and all-attention modes to show how much the trainable count changes with scope. The QKV framework is also where you can see why targeting different projections might favor different tasks, though the full theory of which layers to adapt and why is still an active area of research.

For the mechanics of what these weights do during inference, the [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text) traces the full path from input token through the weight matrices to a sampled output. And for the optimizer that actually trains A and B step by step, see the [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo).

## Frequently asked questions

### Does LoRA change the base model's weights at all?

No. The base matrix W receives no gradients and never updates. Only the adapter matrices A and B are trained. At serving time you can either pass the input through W and ΔW separately (two matmuls) or merge them into a single W' = W + ΔW (one matmul, same result, zero latency overhead). The separation is permanent: you can always subtract ΔW back out and recover the original base model.

### What is the difference between rank and alpha in LoRA?

Rank (r) sets the dimension of A and B, which determines how much of the update space the adapter can express. Alpha (α) scales the adapter's output by α/r before it is added to the base output. In practice α is often set to a fixed multiple of r (commonly α = r or α = 2r) so that the effective scale stays stable when you change r. Rank controls capacity; alpha controls how strongly the adapter's update is weighted against the frozen base.

### Why does the loss plateau and not keep falling?

The white rank-floor line marks the closest approximation a rank-r matrix can reach to the target update. If the target has a true rank higher than r, no training run of any length can cross that floor. The demo shows this directly: at rank 1 on tasks with a richer target, the loss curve flattens onto the floor line and stays there even if you run for many more steps.

### How many parameters does LoRA actually train versus full fine-tuning?

On the demo's 12x12 matrix at rank 2, the adapter has 2×12 + 12×2 = 48 parameters versus 144 in the full matrix. The illustrative 7B estimate in the parameter panel shows the real-world version: targeting query and value projections across 32 transformer layers trains roughly 0.007 percent of the base parameter count at rank 8, and the adapter fits in about 1 MB at FP16. These illustrative figures are approximate and depend on architecture; the demo labels them as such.

### Can LoRA be applied to layers other than attention?

Yes. LoRA can wrap any linear layer, including the feed-forward blocks, embedding layers, and output projection. In practice most published results apply it to the query and value projections because those are where task-specific behavior tends to concentrate, but the math is the same regardless of which layer you target. The target-modules preset in the demo only affects the 7B illustrative parameter count, not the actual training, since the demo matrix represents a generic weight update.

## Key takeaways

- LoRA never edits the base model. The frozen weights and the adapter are separate objects, which is why you can merge the update into the weights for zero-overhead serving and still subtract it back out later.
- Rank is capacity, not quality. A rank-r adapter can only express a rank-r update, so when the task's true update is richer, training plateaus at a hard floor that no extra steps or patience can cross.
- The trainable fraction collapses at scale. In the tiny 12x12 demo a rank-2 adapter is a third of the base parameters, but the same recipe on the illustrative 7B estimate trains about 0.007 percent and fits the adapter in roughly a megabyte.
- B starts at zero on purpose. That standard init makes the update exactly zero at step 0, so fine-tuning starts from the base model's exact behavior instead of from random noise it must first unlearn.