Multimodal Evaluation

Multimodal evaluation scores a model on held-out data along three surfaces: retrieval, visual question answering, and captioning. Training loss can't stand in for any of them. It measures fit on the training distribution, not whether the model can rank pairs in a held-out batch, answer a question, or write a caption a human would accept, and a plateaued loss curve is a poor signal that the model is finished.

The three surfaces:

• Retrieval (R@1, R@5, R@10). Build the joint embedding for a query caption, rank every image in the eval pool by cosine, and report whether the matching image lands in the top 1, top 5, or top 10. The symmetric (image-to-text) form runs the same way.
• Visual question answering (exact match). Given an (image, question) pair, the model outputs an answer token. Exact match is one bit per sample: the predicted answer either equals the reference answer or it doesn't. Average over the eval set.
• Captioning (BLEU-4). Generate a caption, then take the geometric mean of the 1-gram through 4-gram precisions against the reference captions, with a brevity penalty. Multi-reference is the standard form: one image, several reference captions.

Each metric is a thin function, and I build all three in code so the math stays concrete and the eval surface stays in your hands. Real benchmark suites (MS-COCO, VQA v2, GQA, OK-VQA) plug into the same function shapes.

Recall@K from a similarity matrix

Recall@K is the fraction of queries whose correct match lands in the top K results. To compute it, build the (N, N) cosine similarity matrix between image and caption embeddings and sort each row's columns by descending similarity. A row counts as a hit when its diagonal column index sits within the top K positions. Symmetric Recall@K (caption-to-image) comes from the transposed matrix, and both numbers get reported. On an N=100 eval, R@1 = 0.6 means 60 of the 100 captions retrieved their correct image as the top match.

VQA exact match

Exact match scores a VQA prediction as correct only when the predicted answer equals the reference answer. For each (image, question, answer), encode the image, embed the question, fuse the two via the decoder, and read out the next-token prediction at the last non-pad question position. Compare the predicted token id to the reference id, then average over the eval set. Real VQA datasets ship with multiple human-annotated answers per question and score with a soft-accuracy formula (1.0 if at least 3 of 10 annotators agree, scaled below that). Here I use single-answer exact match for clarity.

BLEU-4

BLEU-4 is the geometric mean of the modified 1-gram through 4-gram precisions, scaled by a brevity penalty:

BLEU-4 = BP * exp(mean(log p1, log p2, log p3, log p4))

Here p_n is the modified n-gram precision (the clipped count of generated n-grams that appear in any reference, divided by total generated n-grams), and BP is the brevity penalty:

BP = 1                if generated length > reference length
   = exp(1 - r/g)     otherwise, where r is the length of the reference closest
                      in length to the generated caption (ties go to the shorter) and g is generated

Small samples often leave some p_n at zero, which needs smoothing. The implementation uses Chen and Cherry "method 1" (add 1 to numerator and denominator for any zero count), the safest default for low-count regimes. Smoothing only rescues orders that have candidates but no matches. A generated caption shorter than 4 tokens has no 4-grams at all, so bleu4 returns 0.0 early even if the caption equals a reference exactly. The demo's captions are 8 tokens, so it never hits this.

Synthetic eval suite

The suite is a 50-sample eval built in memory from the same mock corpus pattern as lesson 62, using a held-out seed. It holds three lists:

• pairs: 50 (image, caption_ids) pairs for retrieval.
• vqa: 50 (image, question_ids, answer_id) triples.
• caps: 50 (image, [reference_caption_ids, ...]) entries with up to 3 references per image.

The suite is deterministic from the seed and held out from the training corpus, so the metrics are computed on data the model never saw. Persisting the suite to JSON is left for you to add.

A 50-step training run on synthetic data won't produce high metrics, and not every metric clears its baseline. In our run R@5, R@10, text-to-image R@1 and BLEU-4 rise above it, but VQA exact match prints 0.0000 after training (0.0200 before) and image-to-text R@1 does the same. VQA is stuck at chance because the question is random tokens, the answer is the first token of the caption, and the image carries only a 7-level brightness signal, so nothing in the input predicts the answer. At N=50 each retrieval and VQA metric also moves in steps of 0.02, so one sample flipping is noise.

code/main.py implements:

• recall_at_k(sim_matrix, k), returning a float in [0, 1] for both directions.
• vqa_exact_match(predictions, references), returning the mean over int equality.
• bleu4(generated, references, smoothing=True), with multi-reference support.
• build_eval_suite(seed, n_samples, vocab_size, max_len), returning three deterministic eval lists.
• evaluate(model, suite), which runs all three metrics and returns a dict of numbers.
• A demo that loads a freshly-initialized multimodal model from lesson 62, evaluates it, trains it for 50 steps, and evaluates again, printing the before/after metrics.

Run it:

python3 code/main.py

Output: the before/after metric table. Retrieval moves from near-random toward the model's learned signal (R@5 and R@10 rise, while image-to-text R@1 can print 0.0000), VQA stays at chance (0.0000 after training in our run), and BLEU-4 improves from about 0.12 to 0.17 (the synthetic structure is enough for a 4-gram precision lift).

Each metric maps onto a production benchmark:

• Retrieval. MS-COCO 5K val, Flickr30K, and ImageNet zero-shot are all R@K problems on the same similarity matrix. Swap in the real files and the function signature stays unchanged.
• VQA. VQA v2, GQA, and OK-VQA use the same exact-match shape, with soft accuracy instead of single-answer EM for VQA v2.
• BLEU-4. MS-COCO captioning, NoCaps, and Flickr30K captioning use BLEU-4 alongside CIDEr and METEOR. Adding CIDEr is one more function.

For real benchmarks, replace build_eval_suite with a real loader and keep the function bodies. The math doesn't depend on the benchmark.

code/test_main.py covers:

• recall@k returns 1.0 on a perfect identity similarity matrix and 0.0 on a flipped one for k < N
• recall@k respects k <= N upper bound
• bleu4 returns 1.0 when generated equals one of the references exactly
• bleu4 returns 0.0 on disjoint vocabulary
• vqa exact match equals the fraction of equal pairs
• build_eval_suite returns the expected number of pairs, vqa items, and caption entries

Run them:

python3 -m unittest code/test_main.py

1. Add CIDEr to the captioning metrics. CIDEr uses TF-IDF weighting on n-grams, which rewards informative tokens.

2. Implement soft-accuracy VQA: multiple human answers per question, accuracy is min(human_count / 3, 1) if any matches. Replicates VQA v2.

3. Add a NaN-safe variant of bleu4 that handles empty generated sequences without crashing.

4. Compute mean reciprocal rank (MRR) alongside R@K. MRR is sensitive to where the correct item lands beyond the top K; R@K is sensitive to whether it lands in the top K.

5. Run the eval on the model at six checkpoints during training (step 0, 10, 20, 30, 40, 50) and plot the learning curve. Confirm the metric trajectories track the loss trajectory.

• VQA v2 paper for the soft-accuracy formula and dataset statistics.
• CIDEr paper for TF-IDF-weighted n-gram captioning.
• BLEU original (Papineni et al., 2002) for the smoothing variants.
• MS-COCO captioning eval scripts for the canonical reference implementation.