Conditional GANs & Pix2Pix

A conditional GAN (Mirza & Osindero, 2014) is a GAN where both the generator G and the discriminator D take a condition c as an extra input. Pix2Pix (Isola et al., 2017) specialized this: the condition is a full input image, the generator is a U-Net, the discriminator is a patch-based classifier (PatchGAN), and the loss is adversarial + L1.

An unconditional GAN samples arbitrary faces. That is fine for a demo, but you get no say in what comes out. The tasks you actually want look like this: map a sketch to a photo, map a map to an aerial photo, map a daytime scene to nighttime, colorize a grayscale image. Each one gives you an input image x and asks for an output y with some semantic correspondence to it. Many plausible ys fit a single x, and mean-squared error flattens them into mush. An adversarial loss doesn't, because "looks real" is sharp.

The Pix2Pix recipe outperforms from-scratch text-to-image models on narrow image-to-image domains even in 2026, because it is trained on paired data. You have the signal you need.

Conditional G. G(x, z) → y. In Pix2Pix, z is dropout inside G. There is no input noise, because Isola found explicit noise got ignored.

Conditional D. D(x, y) → [0, 1]. D takes the pair (condition, output), and that is the key difference from an unconditional GAN: it has to judge whether y is consistent with x, not just whether y looks real.

U-Net generator. An encoder-decoder with skip connections across the bottleneck. The skips matter for tasks where input and output share low-level structure (edges, silhouette). Without them, high-frequency detail vanishes.

PatchGAN discriminator. Instead of a single real/fake score, D outputs an N×N grid where each cell judges a receptive field of ~70×70 pixels, and the cells are averaged. This is a Markov random field assumption: realism is local. It trains much faster, has fewer parameters, and gives sharper output.

Loss.

loss_G = -log D(x, G(x)) + λ · ||y - G(x)||_1
loss_D = -log D(x, y) - log (1 - D(x, G(x)))

The L1 term stabilizes training and pushes G toward the known target. L1 gives sharper edges than L2 (medians, not means). λ = 100 was the Pix2Pix default.

Everything above describes full Pix2Pix as published. The Build It demo implements only the conditioning half of it (see there for what it leaves out).

CycleGAN (Zhu et al., 2017) is the unpaired version of this idea. Pix2Pix needs paired (x, y) data, and CycleGAN drops that requirement at the cost of an extra loss: the cycle consistency loss. It uses two generators, G: X → Y and F: Y → X, trained so that F(G(x)) ≈ x and G(F(y)) ≈ y. That lets you translate horses to zebras or summer to winter without paired examples.

In 2026, unpaired image-to-image is mostly done with diffusion (ControlNet, IP-Adapter) rather than CycleGAN, but the cycle-consistency idea survives in many unpaired domain adaptation papers.

main.py (200 lines, pure Python, hand-written backprop) implements a tiny conditional GAN on 1-D data. The condition c is a class label (0 or 1), and the task is to produce a sample from the conditional distribution for the given class: real data is N(-2, 0.3) for class 0 and N(2, 0.3) for class 1. Each network is an MLP with one hidden layer of 16 leaky-ReLU units. The snippets below are illustrative and condensed, so names and signatures differ a little from the file.

This is the conditioning half of Pix2Pix only. The file has no U-Net, no PatchGAN, no dropout, and no CycleGAN. Its default run also trains without the L1 term: update_g accepts optional l1_w and targets arguments, but main() never passes them.

Step 1: append the condition to both G and D inputs

Both networks need to see c. Without it in G, nothing steers the output; without it in D, nothing punishes a G that ignores the condition. In the file, z and x are Python lists, so + concatenates them with the one-hot vector:

def g_forward(z, c, G, num_classes):
    inp = z + one_hot(c, num_classes)
    ...  # linear -> leaky ReLU -> linear
 
def d_forward(x, c, D, num_classes):
    inp = x + one_hot(c, num_classes)
    ...  # linear -> leaky ReLU -> linear -> sigmoid

One-hot encoding is the simplest way to attach the condition. Larger models use learned embeddings, FiLM modulation, or cross-attention.

Step 2: train conditional

The generator has to match the real distribution for the given condition, not the marginal. Each step updates D on real and fake pairs that share a class, then updates G with fresh noise and fresh random classes:

for step in range(1, 601):
    reals = sample_real_conditional(batch, num_classes, rng)  # [([x], c), ...]
    cs = [c for _, c in reals]
    fakes = [(g_forward(z, c, G, num_classes)[0], c) for z, c in zip(noise, cs)]
    update_d(reals, fakes, D, num_classes, d_lr)
 
    noise = [...]                                   # fresh noise for G's update
    cs = [rng.randrange(num_classes) for _ in range(batch)]
    update_g(noise, cs, G, D, num_classes, g_lr)

Step 3: check the output class by class

A marginal check cannot tell you the condition was used, so look at each class separately. main.py contains no assert. Every 150 steps it draws 300 probes, picks a class at random for each probe (so the count n differs per class), and prints the mean generated value per class:

probes = {c: [] for c in range(num_classes)}
for _ in range(300):
    c = rng.randrange(num_classes)
    z = [rng.gauss(0, 1) for _ in range(z_dim)]
    probes[c].append(g_forward(z, c, G, num_classes)[0][0])

Class 0 near -2 and class 1 near +2 means G uses the condition. Both near 0 means it ignores it. At the end the script prints six raw samples per class, with no mean and no check. With the shipped seed the final means are about -2.56 and +2.47, so an assert you add yourself should use a loose tolerance (0.75, say), not a tight one.

The L1 bullets below describe full Pix2Pix. The Build It demo trains without an L1 term by default.

• Condition ignored. G learns to marginalize, and D doesn't penalize it because the condition signal is weak. Fix: condition D more aggressively (early layer, not just late), or use a projection discriminator (Miyato & Koyama 2018).
• L1 weight too low. G drifts to arbitrary real-looking outputs instead of faithful ones. Start at λ≈100 for Pix2Pix-style tasks.
• L1 weight too high. G produces blurry outputs, because L1 is still an L_p norm. Anneal it down once training stabilizes.
• Ground-truth leakage in D. Concatenate (x, y) as D's input, not just y. Without the pair, D cannot check consistency.
• Mode collapse per class. Each class can collapse independently, so run class-conditional diversity checks.

Here is the 2026 state of image-to-image tasks:

Pix2Pix is still the right tool when you have thousands of paired examples, the task is narrow and repeatable, and you need fast inference. On generic open-domain tasks, diffusion wins.

Save outputs/skill-img2img-chooser.md. The skill takes a task description, data availability (paired vs unpaired, N samples), and a latency/quality budget. It outputs an approach (Pix2Pix, CycleGAN, ControlNet variant, SDXL + IP-Adapter), training data requirements, inference cost, and an eval protocol (LPIPS, FID, task-specific).

1. Easy. Modify main.py to add a third class. This is not drop-in: sample_real_conditional hardcodes c == 0 as N(-2, 0.3) and every other class as N(2, 0.3), and main() sets num_classes = 2. Give class 2 its own mode (say N(0, 0.3)) in the sampler and set num_classes = 3. The one-hot and probe code adapt on their own. Confirm each class's mean lands near its own mode.
2. Medium. Replace L1 with a perceptual-style loss in the 1-D setting (e.g. a small frozen D acting as feature extractor). Does it change sharpness of the conditional distribution?
3. Hard. Sketch a CycleGAN in the 1-D setting: two distributions, two generators, cycle loss. Show that it learns to map between them with no paired data.

When you have paired data and a narrow task (sketch → render, semantic map → photo, day → night), Pix2Pix's one-shot inference beats multi-step diffusion by 40x or more on latency. The production comparison usually looks like this:

On the table above, Pix2Pix is about 40x faster than the 20-step SD paths (~30 ms vs ~1.2 s), roughly 100x or more faster than ControlNet + SDXL (~30 ms vs ~3-5 s), and about 5-12x faster than the 1-4 step SDXL-Turbo path. Pix2Pix wins on throughput in static batches, where each request costs the same FLOPs. Diffusion wins on quality and generalization. The modern play is often to ship a Pix2Pix-style distilled model for the narrow task and a diffusion fallback for tail inputs.

• Mirza & Osindero (2014). Conditional Generative Adversarial Nets: the cGAN paper.
• Isola et al. (2017). Image-to-Image Translation with Conditional Adversarial Networks: Pix2Pix.
• Zhu et al. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks: CycleGAN.
• Wang et al. (2018). High-Resolution Image Synthesis with Conditional GANs: Pix2PixHD.
• Park et al. (2019). Semantic Image Synthesis with Spatially-Adaptive Normalization: SPADE / GauGAN.
• Miyato & Koyama (2018). cGANs with Projection Discriminator: the projection D.