Fine-Tuning Stable Diffusion — LoRA, DreamBooth, and Textual Inversion

You have 18 photographs of one specific ceramic mug — matte olive glaze, a chipped handle, a hairline crack near the base — and a catalogue that needs it in forty different settings. No prompt gets you that mug. "A matte olive ceramic mug with a chipped handle" gets you a generic mug that is olive-ish and undamaged. The model has never seen this object and has no word for it.

So you fine-tune. You load the SD 1.5 U-Net, point AdamW at all 860 million parameters, and immediately run into the memory arithmetic:

You rent an A100. Eight hundred steps later, something worse than an out-of-memory error happens: it works, and then it stops working. The mug is perfect. But "a photograph of a golden retriever" also returns your mug. So does "a mountain landscape". Eighteen images have overwritten a model trained on billions, and the loss curve looked fine the whole time. That is catastrophic forgetting, and it is the reason nobody fine-tunes these models the obvious way.

And even if it worked, you would have produced a 3.4 GB checkpoint that does exactly one thing. Ten products means 34 GB, and no way to use two of them together.

What is actually in there, and what needs to change

Before adapting a model it helps to know what parts exist and what each one does.

The last row is the key observation. Teaching the model a new subject or a new style is mostly a matter of changing which prompt tokens map to which visual features, and that mapping lives in the cross-attention projection matrices. The convolutions that know what glaze and ceramic look like need no revision at all.

LoRA: learning the change, not the weights

The intuition

A fine-tune produces new weights W. Write those as the original plus a change: W = W₀ + ΔW. Freeze W₀ and learn only ΔW. So far this saves nothing — ΔW is the same size as W₀.

The insight, from the LoRA paper, is that ΔW from fine-tuning is empirically low rank. If you fine-tune fully and then take the singular value decomposition of the update, almost all its energy sits in a handful of singular directions. That makes sense: "make this model draw my mug" is a small, structured change, not 860 million independent adjustments.

The maths

Instead of storing a full d×k matrix, factor it through a narrow bottleneck of rank r:

In plain English: project the input down to r dimensions, transform it there, project it back up, and add the result to the frozen layer's output. The parameter saving is the whole point. Take a cross-attention key projection in SD 1.5's middle blocks: W is 1280×768, so 983,040 parameters. With r = 4 you store 4×(1280 + 768) = 8,192 parameters — a 120× reduction on that one matrix.

Two details in the formula do real work.

The initialisation. A is initialised from a small random normal and B is initialised to exactly zero, so ΔW = BA = 0 at step 0. The adapted model starts out bit-identical to the original. Gradients still flow to B because ∂L/∂B depends on Ax, which is non-zero. You get a guaranteed-safe start with no loss of trainability.

The α/r scaling. Without it, changing the rank changes the magnitude of the update, so every rank change would need a new learning rate. Dividing by r normalises that. In practice people set α = r (scale exactly 1.0) or α = 2r (scale 2.0, a stronger effect). The single most common LoRA bug is training with one α/r ratio and loading with another, which silently doubles or halves the LoRA's strength.

LoRA does not make a smaller model. It makes a smaller diff — and because that diff is additive and starts at zero, it can be scaled, swapped, stacked, or removed at inference with no retraining.

Two ways to use it

At inference you have a choice. Keep the factors separate and compute W₀x + (α/r)BAx as two operations — slightly slower, but you can change the LoRA weight per generation or unload it instantly. Or merge: compute W₀ + (α/r)BA once and overwrite the weights — zero inference overhead, but the LoRA is now baked in and you need to reload the base model to change it. Serve many LoRAs from one process: keep them separate. Serve one fixed style at high volume: merge.

1import torch2from diffusers import StableDiffusionPipeline34pipe = StableDiffusionPipeline.from_pretrained(5    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")67pipe.load_lora_weights("./olive-mug-lora", adapter_name="mug")8pipe.load_lora_weights("./watercolour-lora", adapter_name="paint")910# separate: weights adjustable per call11pipe.set_adapters(["mug", "paint"], adapter_weights=[0.9, 0.5])12img = pipe("a photo of sks mug on a windowsill").images[0]1314# merged: fastest, but baked in15# pipe.fuse_lora(lora_scale=0.9)

Adapter weights compose additively, so they compound. Two LoRAs at 1.0 each behave like one at 2.0: over-saturated, distorted, often unusable. Keep the sum around 1.0–1.5. And LoRAs trained on different base models do not mix — an SD 1.5 LoRA loaded onto SDXL has mismatched tensor shapes at best, and at worst loads partially and produces noise.

The training loop

1from peft import LoraConfig23# freeze everything, then attach adapters to attention projections only4unet.requires_grad_(False)5unet.add_adapter(LoraConfig(6    r=16, lora_alpha=16,7    target_modules=["to_q", "to_k", "to_v", "to_out.0"],8))9params = [p for p in unet.parameters() if p.requires_grad]10print(sum(p.numel() for p in params))        # ~3.2M, not 860M1112opt = torch.optim.AdamW(params, lr=1e-4)     # 10-100x the full-fine-tune LR1314for step, batch in enumerate(loader):15    with torch.no_grad():16        latents = vae.encode(batch["pixels"]).latent_dist.sample() * 0.1821517        cond = text_encoder(batch["input_ids"])[0]18    noise = torch.randn_like(latents)19    t = torch.randint(0, 1000, (len(latents),), device=latents.device)20    noisy = scheduler.add_noise(latents, noise, t)21    pred = unet(noisy, t, encoder_hidden_states=cond).sample22    loss = torch.nn.functional.mse_loss(pred.float(), noise.float())23    loss.backward(); opt.step(); opt.zero_grad()

Note the learning rate: 1e-4, ten to a hundred times higher than a full fine-tune's 1e-5 to 1e-6. With only 3.2M trainable parameters and a zero-initialised B, small steps make no progress at all. Using 1e-6 out of caution is the reason many first LoRA attempts appear to do nothing.

DreamBooth: binding a subject to a token

LoRA is a parameterisation — it says which weights you train. DreamBooth is a method — it says what data and captions you train on. They are routinely combined, and the combination is what most "train a LoRA of my dog" tools actually run.

DreamBooth's recipe: pick a rare token that the tokeniser does not already associate with anything (sks, ohwx, zwx), pair it with a class noun, and caption all your images "a photo of sks mug". The rare token has no prior meaning, so gradient descent is free to load it with your specific object; the class noun anchors it to the right region of the model's existing knowledge, so it starts from "mug-shaped thing" rather than from nothing.

Why prior preservation is not optional

Train on 18 images captioned "a photo of sks mug" and something predictable happens. The word "mug" appears in every caption, alongside your specific mug, every single step. The model learns that "mug" means this mug. Ask for "a blue mug" afterwards and you get yours in blue. Ask for "a coffee mug on a desk" and you get yours. This is language drift: the class noun has been captured.

Prior preservation fixes it by training on two losses at once:

In plain English: learn my mug from my photographs, and simultaneously keep producing your original idea of a generic mug when the rare token is absent. The second set of images is generated by the base model itself before training starts — typically 100–200 samples from the prompt "a photo of a mug". You are asking the model to stay consistent with its own past output. λ is usually 1.0.

1# generate the class images ONCE, from the unmodified base model2class_images = [base_pipe("a photo of a mug").images[0] for _ in range(200)]34# then each training step sees one of each5loss_subject = mse(unet(noisy_sub, t, emb("a photo of sks mug")), noise_sub)6loss_prior   = mse(unet(noisy_cls, t, emb("a photo of a mug")), noise_cls)7loss = loss_subject + 1.0 * loss_prior

The cost is roughly double the compute per step, plus the one-off generation. The benefit is a model that still knows what mugs are. Skip it and your adapter is unusable in any prompt that mentions the class.

Captioning: the counterintuitive rule

This is where most subject training quietly fails. The rule is: caption everything that varies across your images, and omit what stays constant. Whatever you do not name gets absorbed into the rare token.

If all 18 photographs were shot on the same oak table and every caption reads "a photo of sks mug", then sks learns "olive mug on an oak table". Every generation will drag the table along, and no prompt will remove it. Caption them "a photo of sks mug on an oak table", "a photo of sks mug on grass", "a photo of sks mug held in a hand", and the token keeps only the invariant — the mug itself.

Textual inversion: learning a word, not a model

Textual inversion is the extreme end of the spectrum. It changes no model weights at all. It adds one row to the token embedding table and trains only that row.

Concretely: add a new token <olive-mug> to the vocabulary, give it a 768-dimensional embedding vector, freeze the U-Net, the VAE and the entire text encoder, and run the ordinary diffusion loss — backpropagating all the way through the frozen model into that one vector. 768 trainable parameters. The resulting file is 3–5 KB.

pipe.load_textual_inversion("./olive-mug.bin", token="<olive-mug>")img = pipe("a <olive-mug> on a rustic table, morning light").images[0]

Two mechanics decide whether it works.

Initialisation matters enormously. Starting the new embedding from random noise means thousands of steps of aimless wandering in 768-dimensional space. Initialise it from the embedding of a semantically close existing word — "mug", "cup", "vessel" — and you start inside the right neighbourhood. This routinely cuts training time by more than half.

The learning rate is unusual: around 5e-4 to 5e-3, far higher than any other technique here. You are training 768 numbers, not millions, and there is nothing to destabilise — nothing downstream of the embedding can be damaged, because everything downstream is frozen.

The corresponding limitation is severe. A single 768-d vector can only express things the frozen model can already draw. It can find the direction in concept space that means "your mug", but only if that region exists. Genuinely novel structures — an object with no analogue in the training data — are beyond it. Textual inversion is excellent for styles and for subjects close to something the model knows, and hopeless for anything else.

Choosing between them

The decision, in one line each: a style that the model can nearly do already → textual inversion or a low-rank LoRA. A specific subject you need reliably and repeatedly → DreamBooth captioning trained with LoRA at rank 16–32 and prior preservation. Maximum fidelity with no other constraints and a subject the model finds genuinely alien → full DreamBooth, accepting the file size and the loss of composability.

Almost every production use is DreamBooth's data recipe trained through LoRA's parameterisation. They are answers to different questions, and combining them is the default, not a compromise.

Where people go wrong

That last row deserves emphasis. Diffusion training loss barely correlates with adapter quality, and there is no validation metric that reliably substitutes for looking. The standard workflow is to checkpoint every 200–500 steps, generate the same 6 prompts with the same 4 seeds from each checkpoint, and choose the grid you prefer. Overfitting is usually visible several hundred steps before it becomes irreversible: the subject sharpens, then poses stop varying, then backgrounds start repeating.

What this means when you build something

Think of these as an asset pipeline, not as model training. One base model sits in memory. Around it you keep a library of small files, and a request assembles the ones it needs.

1CATALOGUE = {2    "olive-mug":  ("./loras/olive_mug.safetensors", 0.9, "sks mug"),3    "watercolour":("./loras/watercolour.safetensors", 0.45, "wcstyle"),4}56def render(prompt, assets):7    names = []8    for key in assets:9        path, weight, trigger = CATALOGUE[key]10        pipe.load_lora_weights(path, adapter_name=key); names.append(key)11        prompt = f"{trigger}, {prompt}"           # triggers must be present12    pipe.set_adapters(names, [CATALOGUE[k][1] for k in assets])13    out = pipe(prompt, num_inference_steps=25).images[0]14    pipe.unload_lora_weights()                    # always clean up15    return out

Three operational rules follow directly from the mechanisms above. Version the base model alongside every adapter, because a LoRA is a diff and a diff is meaningless without the thing it applies to — store the base checkpoint hash in the adapter's metadata. Record the trigger token and the training α/r with the file, since neither is recoverable from the weights and both are required to use it correctly. And unload adapters between requests: adapter state persists on the pipeline object, so a forgotten unload_lora_weights() means request 200 is silently rendering with the style someone asked for in request 3.

The economics are what make this worth doing. Full DreamBooth for ten products is 34 GB of checkpoints, ten separate model loads, and no way to combine any two. Ten rank-16 LoRAs are 120 MB total, load in under a second each, and compose. That difference — not fidelity — is why low-rank adaptation became the default way to specialise these models.