Multimodal RAG and Cross-Modal Retrieval

Single-modality RAG is a settled pattern: embed the query, embed the chunks, retrieve, stuff the results into the LLM. Multimodal RAG adds four requirements:

1. Multiple retrieval heads, because each modality needs embeddings in a compatible space.
2. Fusion of retrieval results across modalities.
3. Generation grounding that cites sources across modalities.
4. Evaluation metrics that cover cross-modal signal.

The 2025 surveys arrive at the same taxonomy.

Cross-modal retrieval

Retrieve documents of modality B given a query of modality A. There are three patterns.

1. Shared embedding space. CLIP and CLAP produce text + image and text + audio embeddings in a shared space, so cosine similarity across modalities works directly. You are limited to what CLIP-style training pairs cover.

2. Per-modality encoder + translation. A text encoder, an image encoder, and a small translator module that maps between the spaces. Sen2Sen by Gupta et al. and other 2024 designs work this way. It is flexible but adds moving parts.

3. VLM as encoder. Use a VLM's hidden states as the retrieval representation, and any modality the VLM supports works. Quality is higher and so is the cost.

My pick: CLIP / SigLIP 2 for text+image, CLAP for text+audio, and VLM hidden states when you need frontier-quality cross-modal retrieval.

Fusion strategies

Say you retrieved 10 results: 5 images, 3 text passages, 2 audio clips. How do you merge them?

Score fusion is the cheapest. Each modality has its own retriever and returns its own scores. Normalize the scores within each modality, then sum. It is simple and often good enough.

Attention-based fusion concatenates all retrieved items and lets a small attention network weight them. It needs training.

MoE fusion uses a gating network to route to modality-specific experts. Different query types route differently, so a visual question weights images higher.

The production default is score fusion with a slight bias toward the query's dominant modality. Move to MoE if an A/B test shows clear wins on your domain.

Generation grounding

The LLM should cite which retrieved item drove each claim. With mixed modalities the formats look like this:

• Text source: standard citation [1].
• Image source: [img 3] with a short caption.
• Audio: [audio 2 at 0:34].

To get this behavior, train the generator on grounding-aware data where each claim in the training target is tagged with its source index. At inference the model then emits citations on its own.

The 2025 surveys

Abootorabi et al. (arXiv:2502.08826, "Ask in Any Modality") give a taxonomy for multimodal RAG covering retrieval, fusion, and generation. It has the broadest coverage of the three.

Mei et al. (arXiv:2504.08748, "A Survey of Multimodal RAG") focus on sub-task benchmarks and failure modes, which makes it the one to read when you design evaluation.

Zhao et al. (arXiv:2503.18016) wrote a vision-focused survey that is strong on ColPali-family work.

Read all three and you have the state of the art as of spring 2025. Most of the sub-problems are still open.

MuRAG, the foundational paper

MuRAG (Chen et al., 2022) was the first multimodal RAG. It retrieved images and text from a multimodal KB and generated answers from them, which showed the idea was feasible before the VLM wave. Modern systems like REACT, VisRAG, and M3DocRAG build on it.

A production trip-planner example

Query: "find me a quiet vegan brunch with natural light."

The pipeline:

1. Decompose the query. "quiet" maps to an audio or review keyword, "vegan brunch" to a menu item, and "natural light" to an image feature.
2. Retrieve per modality:
   • Text retrieval on reviews: "vegan brunch, quiet ambiance."
   • Image retrieval on restaurant photos: "natural light, airy."
   • Audio retrieval on ambient-sound clips: "low decibel, no music."
3. Fuse scores so each restaurant gets a composite score.
4. Send the top-k restaurants to a VLM generator along with the evidence, and return an answer with citations.

That goes well beyond text-RAG. Each modality adds signal that text alone misses.

Agentic multimodal RAG

Multi-hop retrieval handles the case where the first pass does not return a high-confidence answer: the LLM reformulates the query and retrieves again. The agentic RAG patterns from Phase 14 apply here. Two examples:

• Retrieve the initial top-10, then the LLM says "too noisy, filter for <40 dB" and re-retrieves.
• Retrieve images, notice that one shows a menu, then retrieve the menu text and answer.

This adds complexity, but it handles queries that single-shot retrieval cannot.

Evaluation

Cross-modal evaluation is still immature. The common proxies:

• Recall@k per modality.
• Fused top-k accuracy.
• Human-judged end-to-end satisfaction.
• Task-specific outcomes (bookings completed, purchases made).

No standard benchmark spans all modalities, so most papers evaluate on domain-specific tasks.

code/main.py is a stdlib-only toy over a five-restaurant corpus. It contains:

• Three mock retrievers (text, image, audio). The text one is keyword counting over review text, the image one matches tags like natural_light and airy, and the audio one scores restaurants by ambient decibel level when the query says "quiet" or "calm".
• Score fusion that combines the modality scores with configurable weights. It is a plain weighted sum, with no per-modality normalization.
• A generator stub that prints the ranked restaurants with their citation tags. No model is involved.
• A simple agentic loop that reformulates the query when the top fused score falls below a confidence floor.

This lesson produces outputs/skill-multimodal-rag-designer.md. Give it a product spec with a multimodal query flow and it designs the retrievers, fusion, generator, and evaluation.

1. Propose a medical-triage multimodal RAG: query = photo of injury + text symptoms. What modalities retrieve from what KB?

2. Score fusion is a simple weighted sum. What failure mode does it have that MoE fusion avoids?

3. Read Abootorabi et al.'s taxonomy (Section 3). What are the three canonical sub-problems and how do they map to your chosen product?

4. Design an eval spec for a trip-planner multimodal RAG. What metrics cover image recall, audio recall, and composite correctness?

5. Agentic multi-hop RAG has a latency tax per round-trip. At what query difficulty does the accuracy gain justify the latency?

• Abootorabi et al., Ask in Any Modality (arXiv:2502.08826)
• Mei et al., A Survey of Multimodal RAG (arXiv:2504.08748)
• Zhao et al., Vision RAG Survey (arXiv:2503.18016)
• Chen et al., MuRAG (arXiv:2210.02928)
• Liu et al., REACT (arXiv:2301.10382)