ControlNet, LoRA & Conditioning

A prompt like "a woman in a red dress walking a dog on a busy street" gives the model no information about where the dog is, what pose the woman is in, or the perspective of the street. Text pins down about 10% of what you need to specify an image. The rest is visual and cannot be described efficiently in words.

Training a new conditional model from scratch for each signal (pose, depth, canny, segmentation) is prohibitive. The alternative is to keep the 2.6B-param SDXL backbone frozen, attach a small side-network that reads the conditioning, and let it nudge the backbone's intermediate features. That is ControlNet.

Teaching the model new concepts (your face, your product, your style) runs into the same cost problem, because retraining the full model is too expensive. You want a delta about 100x smaller. That is LoRA: low-rank adapters that plug into existing attention weights.

ControlNet, LoRA, and text together make up the 2026 practitioner's toolkit. Most production image pipelines layer 2-5 LoRAs, 1-3 ControlNets, and an IP-Adapter on top of an SDXL / SD3 / Flux base.

ControlNet (Zhang et al., 2023)

ControlNet is a trainable clone of a pretrained Stable Diffusion U-Net's encoder that accepts an extra conditioning input (edges, depth, pose) and feeds its features into the original decoder. To build one, take a pretrained SD, clone the encoder half of the U-Net, and freeze the original. Train the clone so it accepts the conditioning input. Connect it back to the decoder half of the original through zero-convolution skip connections: 1×1 convs initialized to zero, so they start as a no-op and learn a delta.

SD U-Net decoder:   ... ← orig_enc_features + zero_conv(controlnet_enc(condition))

Because of the zero-conv init, ControlNet starts as identity and does no harm even before training. Train it on about 1M (prompt, condition, image) triples with the standard diffusion loss.

Per-modality ControlNets ship as small side models (~360M for SDXL, ~70M for SD 1.5). You can compose them at inference:

features += weight_a * control_a(depth) + weight_b * control_b(pose)

LoRA (Hu et al., 2021)

LoRA (low-rank adaptation) freezes a pretrained weight matrix and learns a low-rank delta that is added on top of it. For any linear layer W ∈ R^{d×d} in the model, add:

W' = W + ΔW,  ΔW = B @ A,  A ∈ R^{r×d},  B ∈ R^{d×r}

with r << d. Initialize B to zeros (as main.py does) and ΔW starts at zero, so a fresh LoRA is also a no-op, the same trick as zero-conv. Rank 4-16 is standard for attention, rank 64-128 for heavy fine-tunes. The new parameter count is 2 · d · r instead of d². For SDXL attention with d=640 and r=16, that is 20k params per adapter instead of 410k, a 20x reduction. Across the whole model, a LoRA is usually 20-200MB against a 5GB base.

At inference you can scale the LoRA: W' = W + α · B @ A. α = 0.5-1.5 is normal. Multiple LoRAs stack additively, though they interact in non-linear ways.

IP-Adapter (Ye et al., 2023)

An IP-Adapter is a tiny adapter that accepts an image as conditioning alongside text. It uses the CLIP image encoder to produce image tokens and injects them into cross-attention next to the text tokens. It costs ~20MB per base model and lets you generate an image in the style of a reference without a LoRA.

ControlNet ≈ spatial, LoRA ≈ semantic. Use both.

code/main.py simulates the two mechanisms in pure Python:

1. LoRA. Take a pretrained linear layer W and freeze it. Train a low-rank B @ A so that W + BA matches a target layer whose difference from W is a rank-1 matrix. The script uses 6×6 matrices, tries r = 1, 2, 4, and prints the residual MSE for each. All three print 0.00000, so r = 1 is already enough for a rank-1 correction (the script prints the residual but does not assert it).

2. ControlNet-lite. A "frozen base" predictor plus a "side network" that reads an extra signal. A learnable scalar gate, initialized to zero (our version of zero-conv), scales the side network's output. This part is scalar. Train and watch the gate and the side weight move together toward the target.

Step 1: LoRA math

The snippet below is illustrative pseudocode. The real code is lora_forward, which computes (W + alpha * B @ A) @ x with a plain matrix-vector helper.

def lora(W, A, B, x, alpha=1.0):
    # W is frozen; A, B are the trainable low-rank factors.
    return [W[i][j] * x[j] for i, j in ...] + alpha * (B @ (A @ x))

Step 2: zero-init side network

side_out = control_net(x, condition)
gated = gate * side_out  # gate initialized to 0
h = base(x) + gated

At step 0 the output is identical to base, so the branch cannot disturb it before training starts. w_side starts small but nonzero, so gate gets a gradient immediately, while w_side gets none until gate has left zero. After that they climb together toward a product near the 0.7 in the target. The printout shows only steps 100, 300, 500, and 700, and by step 100 the gate is already near 0.8, so print the first few steps yourself to see the ramp.

• Over-scaling LoRAs. α = 2 or α = 3 is a common "make it stronger" hack, and it produces over-stylized or broken outputs. Keep α ≤ 1.5.
• ControlNet weight conflict. A Pose ControlNet at weight 1.0 plus a Depth ControlNet at weight 1.0 usually overshoots. A sum of weights around 1.0 is a safe default.
• LoRA on the wrong base. SDXL LoRAs silently no-op on SD 1.5 because the attention dimensions do not match. Diffusers warns about this in 0.30+.
• Textual Inversion drift. Tokens trained on one checkpoint drift badly on another. LoRA is more portable.
• LoRA weight-merging and storage. You can bake a LoRA into the base model weights for faster inference (no runtime addition), but then you lose the ability to scale α at runtime. Keep both versions.

Save outputs/skill-sd-toolkit-composer.md. The skill takes a task (input assets: prompt, optional reference image, optional pose, optional depth, optional scribble) and outputs the tool stack, weights, and a reproducible seed protocol.

1. Easy. In code/main.py, the target delta is rank-1 and the script tries r in [1, 2, 4]. Change the delta to a sum of two outer products (rank 2) and rerun. At what rank does the residual MSE reach about zero?
2. Medium. Train two separate LoRAs on two target transforms. Load them together and show their additive interaction. When does the interaction break linearity?
3. Hard. Use diffusers to stack: SDXL-base + Canny-ControlNet (weight 0.8) + a style LoRA (α 0.8) + IP-Adapter (weight 0.6). Measure FID-vs-prompt-adherence trade-off as the stack weights vary.

A real text-to-image SaaS serves hundreds of LoRAs and a dozen ControlNets over the same base checkpoint. The serving problem looks a lot like LLM multi-tenancy (the production literature covers the LLM case under continuous batching and LoRAX / S-LoRA):

• Hot-swap LoRAs, don't merge. Merging W' = W + α·B·A into the base gives ~3-5% faster per-step inference but freezes α and the base. Keep LoRAs hot in VRAM as rank-r deltas. Diffusers exposes pipe.load_lora_weights() plus pipe.set_adapters([...], adapter_weights=[...]) for per-request activation. A swap costs the 2 · d · r · num_layers weights: MB-scale, sub-second.
• ControlNet as a second attention lane. The cloned encoder runs in parallel with the base. Two ControlNets at weight 1.0 each mean two extra forward passes per step, not one merged pass, and batch-size headroom drops quadratically. Budget ~1.5× step cost per active ControlNet.
• Quantize the LoRAs too. If you quantized the base (see Lesson 07, Flux on 8GB), the LoRA delta also quantizes cleanly to 8-bit or 4-bit. QLoRA-style loading lets you stack 5-10 LoRAs on top of a 4-bit Flux base without blowing memory.

Flux-specific: Niels' Flux-on-8GB notebook quantizes the base to 4-bit, and stacking a style LoRA (pipe.load_lora_weights("user/style-lora")) on that quantized base at weight_name="pytorch_lora_weights.safetensors" still works. This is the recipe most SaaS agencies ship in 2026.

• Zhang, Rao, Agrawala (2023). Adding Conditional Control to Text-to-Image Diffusion Models: ControlNet.
• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models: LoRA (originally for LLMs; ports to diffusion).
• Ye et al. (2023). IP-Adapter: Text Compatible Image Prompt Adapter: IP-Adapter.
• Mou et al. (2023). T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability: lighter alternative to ControlNet.
• Ruiz et al. (2023). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation: DreamBooth.
• HuggingFace Diffusers: ControlNet / LoRA / IP-Adapter docs: reference pipelines.