BuildPython

Janus-Pro: Decoupled Encoders for Unified Multimodal Models

A single visual encoder can't be good at both halves of a unified model. Understanding wants semantic features, the concept-rich output vectors of SigLIP or DINOv2. Generation wants reconstruction-friendly codes, VQ tokens that compose back into crisp pixels. Janus (DeepSeek, October 2024) and Janus-Pro (DeepSeek, January 2025) stop trying to reconcile them: share the transformer body, but route understanding through SigLIP and generation through a VQ tokenizer. At 7B, Janus-Pro beats DALL-E 3 on GenEval while matching LLaVA on MMMU. This lesson covers why two encoders work where one struggles.

TIME: ~120 minutesPREREQUISITES: Phase 12 · 13 (Transfusion), Phase 12 · 14 (Show-o)

Learning Objectives

Explain why a single shared encoder compromises either understanding or generation quality.
Describe Janus-Pro's routing: SigLIP features on the input side for understanding, VQ tokens on both input and output for generation.
Trace the data-mix scaling that makes Janus-Pro succeed where Janus did not.
Compare decoupled (Janus-Pro), coupled-continuous (Transfusion), and coupled-discrete (Show-o) architectures.

The Problem

Unified models share one transformer body across understanding and generation. Earlier attempts (Chameleon, Show-o, Transfusion) also share one visual tokenizer across both directions, and that tokenizer is a compromise:

Optimized for reconstruction (generation): VQ-VAE captures fine-grained pixel detail but produces tokens with weak semantic coherence.
Optimized for semantics (understanding): SigLIP embeddings group "cat" images near "cat" tokens but do not permit good reconstruction.

Show-o and Transfusion pay for this with a visible quality tax on one direction. Janus-Pro asks why the two tasks should share a tokenizer at all when their needs differ.

Warm-up0/3

1. An autoencoder compresses an image into a small code. What is its decoder trained to do?Warm-up

An 8 by 8 grayscale-style image feeds an encoder that outputs a small code of four numbers, which a decoder turns into an unknown output marked with a question mark. — Illustrative pixel values and code numbers (typed, not from a trained model).

2. A decoder-only language model writes a reply. How does it produce the text?Warm-up

A prompt box asking for the capital of France goes into a decoder-only language model, which outputs a reply shown as separate tokens: The, capital, of, France, is, Paris, a period. — Typed example prompt and reply.

3. Show-o and Transfusion both handle images inside one transformer. How do they differ?Warm-up

Two side by side panels named Show-o and Transfusion. Each takes text tokens and an image whose form is marked with a question mark into one transformer that outputs text and images. — Both panels are drawn identically on purpose; the form of the image input is the unknown.

Check: The Problem0/3

1. In Show-o and Transfusion, which part of the model is shared between understanding and generation?Explain

A table with five rows, text tokenizer, visual tokenizer, transformer body, output head and loss function, and two columns, understanding and generation. Every cell holds a question mark. — The cells are the unknown; the rows are components of a unified model.

2. What tension makes a single shared visual encoder a compromise for unified models?Explain

A cat picture goes into one shared visual encoder, which feeds two jobs: understanding (answer questions about images) and generation (make images from text). A question asks what the encoder must give each job. — Schematic.

3. A unified model uses one VQ tokenizer for both tasks. Its images look fine, but captioning and question answering lag behind SigLIP-based models. What is the most likely cause?Analyze

A single VQ tokenizer feeds a unified model used for two tasks. Image generation looks fine, while captioning and question answering are behind SigLIP-based models. A question asks for the likely cause. — Schematic of the situation in the question.

The Concept

Decoupled visual encoding

Janus-Pro separates the two encoders:

Understanding path. Input image → SigLIP-SO400m → 2-layer MLP → transformer body.
Generation path. Input image (if conditioning on an existing image) → VQ tokenizer → token IDs → transformer body.
Output generation. Image tokens predicted by the transformer → VQ decoder → pixels.

The transformer body is shared. Everything upstream and downstream of it is task-specific.

Inputs are disambiguated by prompt format: a <understand> tag routes through SigLIP and <generate> routes through VQ. Alternatively the routing is implicit from the task.

Why this works

The understanding loss gets SigLIP features, which CLIP-style pretraining has tuned for semantic similarity. Perception benchmarks improve over Show-o and Transfusion because the input features suit the task.

The generation loss gets VQ tokens, which the tokenizer has tuned for reconstruction. Image quality improves over Show-o because VQ codes compose back to pixels cleanly.

The shared body sees two input distributions (SigLIP and VQ) and has to learn both. The claim is that with enough data and enough parameters, the body absorbs the switching.

Data scaling: Janus vs Janus-Pro

Janus (the original, arXiv 2410.13848) introduced the decoupling at small scale: 1.3B params and limited data. Janus-Pro (arXiv 2501.17811) scaled it up:

7B params (vs 1.3B).
90M image-text pairs for stage 1 (alignment), up from 72M.
72M for stage 2 (unified), up from 26M.
200k image-gen instruction samples added for stage 3.

The result is that Janus-Pro-7B matches LLaVA on MMMU (60.3 vs ~58) and beats DALL-E 3 on GenEval (0.80 vs 0.67). One open model, competitive on both sides of the unified spectrum.

JanusFlow, the rectified flow variant

JanusFlow (arXiv 2411.07975) swaps the VQ generation path for a continuous rectified-flow path. The split becomes SigLIP for understanding and rectified flow for generation, and the quality ceiling lifts further. The architecture is still decoupled encoders around a shared body.

The shared body's job

The transformer body processes a unified sequence drawn from two input distributions. Its job is:

For understanding: consume SigLIP features and text tokens, then emit text autoregressively.
For generation: consume text tokens and (optionally) image VQ tokens, then emit image VQ tokens autoregressively.

The body has no modality-specific weights per block. It is the text-style transformer you'd expect inside Qwen or Llama, plus the two input adapters.

That means the body can start from a pretrained LLM, and Janus-Pro does exactly that, initializing from DeepSeek-MoE-7B. The choice matters because the LLM contributes reasoning ability that pure-from-scratch unified models struggle to reach.

Compared to InternVL-U

InternVL-U (Lesson 12.10) is the 2026 follow-up. It combines:

Native multimodal pretraining (InternVL3 backbone).
Decoupled-encoder routing (SigLIP in, VQ + diffusion heads out).
Unified understanding, generation, and editing.

InternVL-U folds Janus-Pro's architectural choice into a larger framework. At scale, decoupled encoders are now the default for unified models.

Limitations

Decoupled encoders add complexity: two tokenizers to train, two input paths to maintain, two sets of failure modes. If your product doesn't need generation, Janus-Pro is over-engineered, so pick a LLaVA-family understanding model.

If your product doesn't need understanding, Janus-Pro is overqualified, so pick a Stable Diffusion 3 or Flux model.

If your product needs both, Janus-Pro is the reference open architecture.

Check: The Concept0/20

1. What does decoupled encoding mean in a unified vision-language model?Define

Two cards. Understanding: a cat picture and the question 'What breed is this cat?' with text as output. Generation: the prompt 'A cat asleep on a sofa' with an image as output. — Typed example tasks.

2. What is shared between the understanding and generation paths in Janus-Pro?Define

Two inputs, an image with a question and a text prompt, enter a dashed box labelled Janus-Pro with its internals hidden. Two outputs leave it, a text answer and a new image. — Schematic of the task endpoints only.

3. What role does SigLIP play in Janus-Pro?Define

An image labelled seed 42 goes into a box labelled SigLIP-like mock encoder, which outputs a list of 32 floats; the first four values are shown. A question mark asks about SigLIP's role. — Real output of SiglipStub in main.py (random numbers seeded by the image, not a real SigLIP model).

4. What does the VQ path give Janus-Pro for generation?Define

A text prompt 'Draw a lighthouse at dusk' enters a dashed box labelled Janus-Pro that says 'VQ path gives: ?' and an image comes out the other side. — Schematic of the request only.

5. What does JanusFlow change relative to Janus-Pro?Define

Three cards listing Janus (arXiv 2410.13848, Wu et al.), Janus-Pro (arXiv 2501.17811, Chen et al.) and JanusFlow (arXiv 2411.07975, Ma et al.). JanusFlow is outlined, with a question about what it changes. — Titles, ids and first authors come from the lesson's Further Reading list.

6. How does Janus-Pro decide which encoder handles an input?Define

Two requests, a photo with the question 'What is on this table?' and the prompt 'A watercolor fox', go into a decision marked with a question mark that leads to two boxes, encoder path 1 and encoder path 2. — Typed example requests; paths are unlabelled on purpose.

7. In Janus-Pro, what does the understanding path route through?Define

A cat photo with the question 'How many cats are in the photo?' passes through an unknown stage marked with a question mark, then into the rest of Janus-Pro, which produces a text answer. — Schematic of the understanding path.

8. What happens to image tokens the transformer predicts during generation?Define

A row of predicted image tokens (tok, tok, tok, and so on) goes through an unknown step marked with a question mark and ends as a final image. — Schematic; tokens are unnamed placeholders.

9. Janus-Pro is asked to generate an image while conditioned on an existing image. How does that input image enter the model?Explain

Schematic with typed example prompt.

10. How did Janus-Pro scale training relative to Janus?Explain

A table comparing Janus and Janus-Pro with rows model size, stage 1 alignment, stage 2 unified and stage 3 instruction. Every value cell holds a question mark. — Row names are the lesson's three training stages plus model size.

11. What did Janus-Pro keep from Janus, and what did it change?Explain

Janus details from the lesson's Data scaling section.

12. Why is Janus-Pro called decoupled while Show-o and Transfusion are called coupled?Explain

Three cards: Show-o and Transfusion are tagged coupled, Janus-Pro is tagged decoupled. A question asks why the labels differ. — Labels come from the lesson's learning objectives.

13. Why does Janus-Pro initialize its transformer body from DeepSeek-MoE-7B?Explain

Names come from the lesson's description of the shared body.

14. How does InternVL-U relate to Janus-Pro?Explain

Two boxes, Janus-Pro (decoupled-encoder model) and InternVL-U (Lesson 12.10), joined by a dashed line labelled 'how related?' with a question mark. — Schematic; the relationship is the unknown.

15. Janus-Pro-7B beats DALL-E 3 on GenEval (0.80 vs 0.67) but only matches LLaVA on MMMU (60.3 vs ~58). What is the best reading?Analyze

Two small bar charts. GenEval: Janus-Pro-7B 0.80 and DALL-E 3 0.67. MMMU: Janus-Pro-7B 60.3 and LLaVA about 58. Bars start from zero. — Numbers from the question; the LLaVA score is approximate.

16. Janus used 1.3B parameters with limited data. A colleague says it shows decoupled encoders are a dead end. What does Janus-Pro's result suggest?Analyze

A Janus card (1.3B parameters, limited data) beside a quoted colleague saying decoupled encoders are a dead end, with a question about what the Janus-Pro result suggests. — Details from the question.

17. When is Janus-Pro's decoupled architecture the wrong pick?Design

Three illustrative products: a photo captioning API, a poster generator, and a photo chat that also draws. Each row has 'Janus-Pro: ?'. — The products are made-up examples, not from the lesson.

18. You are building an app that must caption user photos and also create new images from text prompts, using open weights. A teammate suggests two separate models. What is the strongest argument for a Janus-Pro-style design instead?Design

Three requirements: caption user photos, create images from text, open weights. The teammate's plan is two models, a captioning model and an image generation model. Below is a dashed box asking for the strongest argument for a Janus-Pro-style design. — Schematic of the question's scenario.

19. You like Janus-Pro's SigLIP understanding but want a higher quality ceiling on generation than VQ codes allow. Which change fits?Design

The understanding path (image in, SigLIP) is marked keep. The generation path (text prompt, VQ codes, image) is marked with a quality ceiling that is too low. A question asks which change to the generation side fits. — Schematic of the question's goal.

20. A team needs understanding, generation and editing in one open model, with native multimodal pretraining at frontier scale. Which choice fits?Design

A checklist of five requirements for one open model: understanding, generation, editing, native multimodal pretraining and frontier scale, with a dashed box asking which choice fits. — Requirements are taken from the question.

Use It

main.py simulates Janus-Pro routing with stdlib Python:

Two mock encoders: a SigLIP-like one (32-dim semantic vectors) and a VQ-like one (integer codes).
A keyword router that picks the path from the prompt, and labels ties as ambiguous.
A shared body stand-in that handles token sequences whichever encoder produced them.
A table of the Janus to Janus-Pro scaling numbers across the three training stages.

It prints the routed path for five prompts, covering image QA, text-to-image, and a mixed prompt.

Check: Use It0/7

1. In main.py, what does route() return for "Explain how many cats are in the photo, then draw one"?Apply

The prompt 'Explain how many cats are in the photo, then draw one' above the two keyword lists from main.py, with the return value of route() shown as a question mark. — Keyword lists copied by the script from main.py.

2. What does run_pipeline do for the prompt "Sketch a cat and then describe its breed"?Apply

The prompt 'Sketch a cat and then describe its breed' above the two keyword lists from main.py and a trace with task, route and output all shown as question marks. — Keyword lists copied by the script from main.py.

3. For "Render a cyberpunk cityscape at night", what does the output field of the trace look like?Apply

The prompt 'Render a cyberpunk cityscape at night' above the two keyword lists from main.py and a trace with task, route and output shown as question marks. — Keyword lists copied by the script from main.py.

4. The scaling table lists Janus at 1.3B parameters and Janus-Pro at 7B. About how many times larger is Janus-Pro?Apply

Two bars drawn to scale: Janus at 1.3B parameters and Janus-Pro at 7B parameters, with the size ratio left as a question mark. — Bar lengths are proportional to 1.3 and 7.

5. Stage 2 data grew from 26M pairs in Janus to 72M in Janus-Pro. Roughly what multiple is that?Apply

Two bars drawn to scale: Janus stage 2 with 26M pairs and Janus-Pro stage 2 with 72M pairs, with the multiple left as a question mark. — Bar lengths are proportional to 26 and 72.

6. route("Draw whatever you want") returns ambiguous even though the user wants an image. Why?Analyze

The prompt 'Draw whatever you want' above the two keyword lists from main.py, with route() returning ambiguous. — Keyword lists copied by the script from main.py; the return value is given in the question.

7. A reader sees SharedBody.process ignore its input stream and concludes the real shared body is proven to absorb two input distributions. What is wrong with that conclusion?Analyze

The source of SharedBody.process from main.py (list values abridged), followed by a quoted conclusion: the real shared body is proven to absorb two input distributions. — Source copied by the script from main.py; the two list expressions are abridged to [...].

Ship It

This lesson produces outputs/skill-decoupled-encoder-picker.md. Given a product that wants unified generation and understanding at near-frontier quality, the skill picks Janus-Pro, JanusFlow, or InternVL-U and attaches a concrete data-scale recommendation.

Author's Note

I'd run main.py and read the five routing traces before anything else. The router is a keyword count, nothing learned. "Describe what's in this image" scores two understanding keywords and goes down the SigLIP path, while "Render a cyberpunk cityscape at night" goes down the VQ path. "Sketch a cat and then describe its breed" ties one to one, and the script labels it ambiguous and runs both encoders.

My bet is that the shared body is the part worth staring at. In the code, SharedBody.process ignores its input stream and branches only on whether the output is text or image. That's a stub, so don't read it as evidence that the real body absorbs two input distributions. It only shows where the seam sits: two encoders in, one body, two output heads. Whether the body copes is an empirical claim about data and scale, which is why the script prints a scaling table.

I'd also keep the toy in proportion. The encoders are seeded random-number stubs, and the outputs are placeholders. The image_seed comes from hash(p) on a string, so the printed tokens will differ between runs. Don't compare them across runs; compare the routes. For the real thing I'd start from the released Janus-Pro weights instead of rebuilding the encoders.

Exercises

Janus-Pro-7B beats DALL-E 3 on GenEval. Explain why a 7B open model can match a frontier proprietary model on generation but not on understanding.
Implement a router function: given prompt text, classify as understand or generate. How do you handle ambiguous prompts like "describe and then sketch"?
JanusFlow replaces the VQ path with rectified flow. What does the transformer body now output, and what changes in the loss?
Propose a fourth task the Janus-Pro architecture could handle with one more decoupled encoder. Examples: image segmentation (DINO-style), depth (MiDaS-style).
Read Janus-Pro Section 4.2 on data scaling. Which data stage contributes most to the T2I quality gain vs Janus?

Key Terms

Term	What people say	What it actually means
Decoupled encoding	"Two visual encoders"	Separate tokenizer or encoder per direction: semantic for understanding, reconstruction for generation
Shared body	"One transformer"	Single transformer processes either encoder's output; no modality-specific weights
SigLIP for understanding	"Semantic features"	CLIP-family vision tower providing rich conceptual features but poor reconstruction
VQ for generation	"Reconstruction codes"	Vector-quantized tokens that decode cleanly back to pixels
JanusFlow	"Rectified-flow variant"	Janus-Pro with a continuous flow-matching generation head instead of VQ
Routing tag	"Task tag"	Prompt marker (`<understand>` / `<generate>`) that picks the input encoder