BuildPython

Inpainting, Outpainting & Image Editing

Text-to-image makes new things. Inpainting fixes old ones. In production, 70% of billable image work is editing: swap a background, remove a logo, extend the canvas, regenerate a hand. Inpainting is where diffusion earns its keep.

TIME: ~75 minutesPREREQUISITES: Phase 8 · 07 (Latent Diffusion), Phase 8 · 08 (ControlNet & LoRA)

Learning Objectives

Tell inpainting, outpainting, and whole-image editing apart, and pick the pipeline from the Use It table that fits each task.
Explain why the naive mask-and-reinject approach leaves seams and how the 9-channel inpainting U-Net fixes it.
Compare SDEdit, InstructPix2Pix, and RePaint, including the fidelity versus freedom trade-off set by SDEdit's starting t/T.
Implement mask-aware reverse diffusion on 5-D toy data and reuse one function for both inpainting and outpainting.
Budget the latency of an edit pipeline stage by stage across SAM, VAE encode/decode, and the U-Net.

The Problem

A client sends a perfect product photo with a distracting sign in the background. You want to erase the sign and leave everything else pixel-identical. Running text-to-image from scratch won't work, because the result comes back with a different color, different lighting, and a different product angle. You want to regenerate only the masked region, and you want the regeneration to respect the surrounding context.

That is inpainting. Variants:

Inpainting. Regenerate inside a mask, keep outside pixels.
Outpainting. Regenerate outside a mask (or beyond the canvas), keep inside.
Image editing. Regenerate the whole image but keep semantic or structural fidelity to the original (SDEdit, InstructPix2Pix).

Most diffusion pipelines in 2026 ship an inpainting mode: Flux.1-Fill, Stable Diffusion Inpaint, SDXL-Inpaint, DALL-E 3 Edit. They share one principle.

Warm-up0/3

1. In latent diffusion, what does the VAE encoder do before the denoising U-Net runs?Warm-up

Four boxes in a row joined by arrows: Image, VAE encoder, a dashed box with a question mark, and Denoising U-Net. — Stage order only. The dashed box is what the question asks about.

2. What does ControlNet add to a frozen diffusion model?Warm-up

Schematic of the setup in the stem. The dashed box is what the question asks about.

3. During the forward process of a diffusion model, what happens to a training image?Warm-up

A training image x0 at step t equals 0 feeds a box named forward process running t from 0 up to T, which leads to a dashed box reading x_T equals question mark. — Setup only. The question is what the forward process does to the image.

Check: The Problem0/3

1. What is inpainting?Define

A schematic product photo (a bottle on a floor in front of a wall) with a red-bordered sign on the wall, and a dashed orange mask drawn around the sign. — Schematic drawing, not a model output.

2. What is outpainting?Define

A wide hatched canvas with a schematic product photo in its middle, outlined in orange and labelled original photo. The hatched areas on both sides are labelled new canvas, no pixels yet. — Schematic drawing, not a model output.

3. A client needs the sky replaced and every other pixel unchanged. A teammate proposes SDEdit at t/T = 0.9 on the whole image instead of SD-Inpaint with a sky mask. Why does that plan fail the brief?Analyze

Schematic drawing. Brief and plan are from the stem.

The Concept

Inpainting: mask-aware denoising with context-preserving reinjection

The naive approach (and why it falls short)

Run standard text-to-image with a mask. At each sampling step, replace the unmasked region of the noisy latent with the forward-diffused clean image. It runs, but the results are poor. Boundary artifacts bleed through because the model has no information about what is in the masked region.

The proper inpainting model

Train a modified U-Net that takes 9 input channels instead of 4:

input = concat([ noisy_latent (4ch), encoded_image (4ch), mask (1ch) ], dim=channel)

The extra channels are a copy of the VAE-encoded source image plus a single-channel mask. At training time, you randomly mask regions of the image and train the model to denoise only the masked region while the unmasked region is given as a clean conditioning signal. At inference, the model can "see" what surrounds the masked region and produces coherent completions.

SD-Inpaint, SDXL-Inpaint, Flux-Fill all use this 9-channel (or analog) input. In diffusers, look at StableDiffusionInpaintPipeline and FluxFillPipeline.

SDEdit (Meng et al., 2022): free editing

Add noise to the source image up to some intermediate t, then run the reverse chain from t down to 0 with a new prompt. No retraining. The choice of starting t trades fidelity for creative freedom:

t/T = 0.3 → nearly identical to source, small stylistic changes
t/T = 0.6 → moderate edits, preserves coarse structure
t/T = 0.9 → generated from near-noise, little of the source survives

InstructPix2Pix (Brooks et al., 2023)

Fine-tune a diffusion model on (input_image, instruction, output_image) triples. At inference, condition on both the input image and a text instruction ("make it sunset", "add a dragon"). Two CFG scales: image scale and text scale.

RePaint (Lugmayr et al., 2022)

Keep a standard unconditional diffusion model. At each reverse step, resample: occasionally jump back to a noisier state and regenerate. This reduces boundary artifacts, and it is what you reach for when you don't have a trained inpainting model.

Check: The Concept0/10

1. What does a proper inpainting U-Net receive beyond the 4-channel noisy latent?Define

Setup only. The dashed box is what the question asks about.

2. How does RePaint reduce boundary artifacts without a trained inpainting model?Define

A line chart of t against reverse step falling steadily from 39 to 0 over 40 steps with no interruptions, with notes that boundary artifacts appear along the mask edge and no trained inpainting model exists. — T = 40 as in the lesson's main.py. Plain reverse chain.

3. What is SDEdit?Define

A horizontal axis of t/T from 0 (source image) to 1 (pure noise) with coloured markers at 0.3, 0.6 and 0.9. — Start points as listed in the Concept section.

4. What is InstructPix2Pix?Define

A schematic product photo feeds a box named InstructPix2Pix, whose output is a dashed box reading edited photo question mark. — Schematic.

5. Why does running standard text-to-image with a mask, and re-injecting the unmasked region at each step, give poor results?Explain

A 6 by 6 latent grid with a dashed masked 2 by 2 block, next to three steps: noisy latent, replace unmasked cells with the forward-diffused clean image, one denoising step, repeated at every sampling step. — Schematic of the procedure described in the stem.

6. What does starting SDEdit at a high t/T such as 0.9 trade off?Explain

A schematic source photo on the left, and an axis of t/T from 0 (source) to 1 (pure noise) with an orange-red marker labelled start 0.9. — Setup only. Schematic photo.

7. How does InstructPix2Pix differ from SDEdit?Explain

Two cards side by side. SDEdit (Meng et al., 2022): start t/T of 0.3, 0.6 or 0.9. InstructPix2Pix (Brooks et al., 2023): examples make it sunset and add a dragon. — Examples quoted from the Concept section.

8. You want a small stylistic change that keeps the photo nearly identical to the source. Which SDEdit starting point fits?Apply

A schematic source photo and an axis from source at 0 to noise at 1 with the prompt to choose the starting t/T. — Setup only. Schematic photo.

9. An SDEdit edit at t/T = 0.9 looks nicely styled, but the subject, layout and colors no longer match the photo. What is the cause?Analyze

Schematic drawing showing the mismatch described in the stem, not a model output.

10. You have a standard unconditional diffusion model and no inpainting checkpoint, and you must remove an object. Which approach fits, and why?Design

A schematic product photo with a dashed mask around the sign, next to two cards: have a standard unconditional model, and have not an inpainting checkpoint. — Schematic photo. Cards are from the stem.

Build It

code/main.py implements a toy inpainting scheme on 5-dimensional data. We train a DDPM on 5-D mixture data where each sample is 5 floats from one of two clusters. At inference, we "mask" 2 of the 5 dimensions, inject the noisy-forward version of the unmasked three at each step, and regenerate only the masked dimensions.

Step 1: 5-D DDPM data

python

def sample_data(rng, d=5):
    cluster = rng.choice([0, 1])
    center = [-1.0 if cluster == 0 else 1.0] * d
    return [c + rng.gauss(0, 0.2) for c in center], cluster

Step 2: train denoiser over all 5 dims

Standard DDPM. The net outputs a 5-D noise prediction for a 5-D noisy input.

Step 3: at inference, mask-aware reverse

python

def inpaint(net, alphas, alpha_bars, T, t_dim, d, clean, mask, rng):
    x = [rng.gauss(0, 1) for _ in range(d)]
    for t in range(T - 1, -1, -1):
        a_bar = alpha_bars[t]
        for i in range(d):
            if not mask[i]:
                x[i] = math.sqrt(a_bar) * clean[i] + math.sqrt(1 - a_bar) * rng.gauss(0, 1)
        # ...then run the normal reverse step on x
    for i in range(d):
        if not mask[i]:
            x[i] = clean[i]
    return x

mask[i] == True means dim i is regenerated. Unmasked dims are overwritten with a freshly noised copy of the clean source before every network call, then reset to the exact clean values at the end. This is the naive approach, and it is enough for toy 5-D data. Real image inpainting uses the 9-channel input because texture coherence matters more.

Step 4: outpainting

Outpainting reuses the same inpaint function with a different mask. main.py regenerates dims 0-1 and pins dims 2-4, which stands in for new canvas next to known content. The training objective stays the same.

Check: Build It0/7

1. In the toy `inpaint` loop, why are the unmasked dims overwritten with a freshly noised copy of the clean source before every network call, rather than with the clean values?Explain

Eight lines of code from main.py's inpaint: the reverse loop, a_bar lookup, and for each unmasked dim x[i] set to sqrt(a_bar) times clean[i] plus sqrt(1 minus a_bar) times a Gaussian draw, followed by the network call. — Code copied from main.py with one expression wrapped across two lines.

2. Why does naive pinning work well enough on the 5-D toy data but leave seams on real images?Explain

Five lines from a real run of main.py showing pinned values and filled values for toy samples, and below them a schematic photo with a faint lighter rectangle where a sign was patched. — Top: values from a real run of main.py, reformatted. Bottom: schematic drawing, not a model output.

3. In the Build It toy scheme, how many dimensions does each sample have, and how many are regenerated at inference?Apply

A row of boxes x[0], x[1], an ellipsis and x[d-1] for one sample, with notes that a sample is d floats from one of two clusters, True means regenerated and False means pinned. — Generic row. The actual d and mask are what the question asks about.

4. In the toy `inpaint` function, `mask = [False, False, True, True, True]` and `mask[i] == True` means dim i is regenerated. Which dims are overwritten with a noised copy of `clean` before each network call?Apply

Five slots dim 0 to dim 4 with mask values False, False, True, True, True, and four lines of inpaint code that overwrite x[i] when not mask[i]. — Code copied from main.py with one expression wrapped across two lines.

5. After the reverse loop in `inpaint` finishes, what do the unmasked dims hold?Apply

Abridged code from the end of main.py's inpaint: the last reverse update of x, the else branch x equals mean, and a comment that the reverse loop ends, with the clean values of the three pinned dims of one sample. — Code abridged and wrapped. Clean values from a real run of main.py.

6. `main.py` outpaints by regenerating dims 0-1 and pinning dims 2-4. Which mask passed to `inpaint` does this?Apply

Five slots dim 0 to dim 4 with dims 0 and 1 coloured as regenerated and dims 2 to 4 as pinned, plus three lines of real main.py outpainting output showing pinned tail values and filled head values. — Output lines from a real run of main.py.

7. A developer sees clean results from naive mask-and-reinject on the 5-D toy and ships the same idea for photos, which show visible boundaries. What was the wrong assumption?Analyze

Left, a number line from -1.5 to 1.5 with five rows of dots from a real toy run: orange dots for pinned values and blue dots for filled values. Right, a schematic photo with a visible lighter rectangle around the patched area. — Left: values from a real run of main.py. Right: schematic drawing, not a model output.

Pitfalls

Seams. The naive approach leaves visible boundaries because the model has no signal about the masked region. Fix: dilate the mask by 8-16 pixels, or use a proper inpainting model.
Mask leakage. If the conditioning image's unmasked region is low-quality or noisy, it pollutes the generation inside the mask. Denoise or blur slightly.
CFG interacts with mask size. High CFG on a small mask = saturated patch. Reduce CFG for small edits.
SDEdit fidelity cliff. Going from t/T = 0.5 to t/T = 0.6 can lose the subject's identity. Sweep the range and save checkpoints.
Prompt mismatch. The prompt should describe the whole image, not just the new content. "A cat sitting on a chair" not "a cat".

Check: Pitfalls0/6

1. What causes visible seams with the naive approach, and what is one fix?Explain

A schematic product photo in which the sign has been removed, leaving a slightly lighter rectangle with a thin outline on the wall where the sign was. — Schematic drawing, not a model output.

2. What causes mask leakage, and how do you fix it?Explain

A schematic product photo with a dashed orange mask over the bottle, containing scattered coloured specks, with text noting stray specks appear inside the mask and that this is called mask leakage. — Schematic drawing, not a model output.

3. What happens when you use high CFG on a small mask?Explain

A schematic product photo with a small dashed orange mask on the bottle label, and a CFG scale slider set near the high end. — Schematic drawing. The mask and slider position are illustrative.

4. Which step in SDEdit can lose the subject's identity, and what should you do about it?Explain

A schematic source photo and an axis of starting t/T from 0 (source) to 1 (noise) with ticks every 0.1. — Setup only. Schematic photo.

5. How should the prompt be written for an inpainting edit that adds a cat to a chair?Apply

A schematic chair against a wall with a dashed orange mask above the seat, and a dashed box labelled prompt question mark. — Schematic drawing. The prompt is what the question asks about.

6. A client photo has a small sign on a wall. Your first SD-Inpaint result shows a faint rectangle where the sign was. Which change targets that seam most directly?Analyze

A schematic brick wall with the sign removed and a slightly lighter rectangle with a thin outline where it hung. — Schematic drawing, not a model output.

Use It

Task	Pipeline
Remove object, small mask	SD-Inpaint or Flux-Fill, standard prompt
Replace sky	SD-Inpaint + "blue sky at sunset"
Extend canvas	SDXL outpaint mode (8px feather) or Flux-Fill with outpaint mask
Regenerate hand / face	SD-Inpaint with prompt re-describing the subject + ControlNet-Openpose
Change style of one region	SDEdit at `t/T=0.5` on masked region
"Make it sunset"	InstructPix2Pix or Flux-Kontext
Background replacement	SAM mask → SD-Inpaint
Ultra-high-fidelity	Flux-Fill or GPT-Image (hosted) for hardest cases

SAM (Meta's Segment Anything, 2023) plus diffusion inpaint is the standard 2026 background-removal pipeline. SAM 2 (2024) works on video.

Check: Use It0/4

1. What does SAM contribute to an editing pipeline?Define

A schematic product photo feeds a box named SAM, then a box named SD-Inpaint, then the same bottle on a differently coloured wall labelled new background. — Schematic drawing, not a model output.

2. Which pipeline does the Use It table list for background replacement?Apply

Schematic drawing, not a model output.

3. A user types "make it sunset" on a photo and supplies no mask. Which pipeline from the Use It table fits?Apply

A schematic uploaded photo, a prompt bubble reading make it sunset, and a dashed box reading mask: none supplied. — Schematic photo.

4. A client wants a landscape photo widened into a banner while keeping the original pixels. Which design fits?Design

Schematic drawing.

Ship It

Save outputs/skill-editing-pipeline.md. Skill takes an original image + edit description + optional mask (or SAM prompt) and outputs: mask-generation approach, base model, CFG scales (image + text), SDEdit-t or inpainting mode, and QA checklist.

Author's Note

I'd run main.py and read the output before touching the 9-channel diagram. It trains a small 5-D DDPM on two clusters, pins dims 0-2 while regenerating dims 3-4, then masks dims 0-1 and pins dims 2-4 for the outpainting run. Each line prints the pinned values next to the filled ones, so you can check by eye that the filled dims land on the same side of zero as the pinned ones.

The part worth rereading is the pinning inside the inpaint loop. At every step, the unmasked dims are overwritten with sqrt(a_bar) * clean + sqrt(1 - a_bar) * noise before the network runs, so the net always sees a context at the right noise level. After the loop, the pinned dims are reset to the exact clean values. Outpainting needs no new code: it is the same function called with a different mask.

I'd keep the toy in proportion. Every dimension of a sample shares one cluster center, so the pinned dims tell the net a lot about the masked ones. Real images give weaker hints, which is why naive pinning shows seams there and the 9-channel input exists.

Exercises

Easy. In code/main.py, vary the fraction of dimensions masked from 0.2 to 0.8. At what fraction does the inpaint quality (residual in masked dims) equal unconditional generation?
Medium. Implement RePaint: at every 10th reverse step, jump back 5 steps (add noise) and re-denoise. Measure whether it reduces boundary residual at the mask edge.
Hard. Use Hugging Face diffusers to compare: SD 1.5 Inpaint + ControlNet-Openpose vs Flux.1-Fill on 20 face-regeneration tasks. Score pose adherence and identity preservation separately.

Key Terms

Term	What people say	What it actually means
Inpainting	"Fill the hole"	Regenerate inside a mask; keep outside pixels.
Outpainting	"Extend the canvas"	Regenerate outside the canvas; keep inside.
9-channel U-Net	"Proper inpainting model"	U-Net with `noisy \| encoded-source \| mask` as input.
SDEdit	"Img2img with noise level"	Noise to time `t`, denoise with new prompt.
InstructPix2Pix	"Text-only edits"	Fine-tuned diffusion on (image, instruction, output) triples.
RePaint	"No retraining"	Re-noise periodically during reverse to reduce seams.
SAM	"Segment Anything"	Mask generator by clicks or boxes; pairs with inpaint.
Flux-Kontext	"Edit with context"	Flux variant that accepts a reference image + instruction for edits.

Production note: edit pipelines are latency-sensitive

Users editing an image expect sub-5-second round trips. A 30-step SDXL-Inpaint at 1024² is 3-4 s on an L4, plus SAM mask generation (~200 ms) and VAE encode/decode (~500 ms combined). In production terms this is latency-bound rather than throughput-bound: batch 1, low concurrency, so trim every stage.

SAM-H is the slow one. SAM-H at 1024² is ~200 ms; SAM-ViT-B is ~40 ms with minor quality loss. SAM 2 (video) adds temporal overhead, so skip it for single-image edits.
Skip the encode when possible. pipe.image_processor.preprocess(img) encodes to latents. If you have the latents from the previous generation (typical in iterative-edit UIs), pass them directly via latents=... to skip one VAE encode.
Mask dilation matters for throughput too. A small mask means most of the U-Net forward pass is wasted (the unmasked pixels are clamped anyway). diffusers' StableDiffusionInpaintPipeline runs the full U-Net regardless.
Flux-Kontext is the 2025 answer. It makes a single forward pass over (source_image, instruction), with no separate mask and no SDEdit noise sweep. On an H100 it ships an edit in ~1.5 s. The takeaway is to collapse the stages.

Check: Production note: edit pipelines are latency-sensitive0/7

1. What is Flux-Kontext?Define

A table with columns Task and Pipeline and three rows: remove object with SD-Inpaint or Flux-Fill, replace sky with SD-Inpaint, and make it sunset with InstructPix2Pix or Flux-Kontext. — Rows quoted from the lesson's Use It table.

2. How do SAM-ViT-B and SAM-H compare at 1024²?Explain

A 1024 by 1024 image feeds two alternative mask generators, SAM-H and SAM-ViT-B, each producing a mask, with time per mask and quality marked question mark. — Setup only. Time and quality are what the question asks about.

3. How can an iterative-edit UI skip one VAE encode?Explain

Two identical rows of seven boxes: image, VAE encode, latents, U-Net, latents, VAE decode, image. The top row is edit 1 and the bottom row is edit 2, which repeats the same stages. — Schematic of the stages named in the Production note.

4. What is the main architectural advantage of Flux-Kontext over an SDEdit plus mask pipeline?Explain

A row of five boxes joined by arrows: SAM mask, VAE encode, SDEdit noise sweep, U-Net denoise, VAE decode. — Stages named in the Production note.

5. Why does shrinking the mask not make `StableDiffusionInpaintPipeline` much faster?Explain

Schematic. Mask sizes are illustrative.

6. A 30-step SDXL-Inpaint at 1024² takes 4 s, SAM-ViT-B takes about 40 ms, and VAE encode plus decode takes about 500 ms combined. About how long is one edit end to end?Apply

Three cards: SAM-ViT-B for the mask, about 40 milliseconds; VAE encode plus decode combined, about 500 milliseconds; SDXL-Inpaint 30 steps at 1024 by 1024, 4 seconds. — Times from the stem.

7. An iterative-edit UI must return each edit in under 5 s at batch 1. Which combination cuts latency best without changing the edit?Design

A timeline from 0 to 5 seconds with a stacked bar: SAM-H 0.2 s, VAE encode and decode 0.5 s, and the U-Net 3 s plus a dashed extra 1 s for the 3 to 4 s range, and a red 5 second limit line. — Stage times from the Production note.