Multimodal Hallucination and Evaluation

Reading: does it say only what is there?

A vision-language model's answer comes from a language model that is very good at plausible text. Li and colleagues found that large vision-language models mostly suffer from severe object hallucination, and that objects frequent in the visual instruction data, or that often co-occur with the objects actually in the image, are especially likely to be hallucinated. Established A kitchen photo invites "a refrigerator" whether or not one is visible.

Their POPE method asks yes/no questions about objects, with absent objects drawn three ways: at random, from the most frequent objects, and adversarially from objects that often co-occur with what is present. Established The adversarial setting is the informative one, because it asks exactly where the model's priors and the pixels disagree.

For harder questions, MMMU collected 11.5K college-level questions across 30 subjects and 30 image types (charts, diagrams, chemical structures, sheet music); at release GPT-4V scored 56% and Gemini Ultra 59%, and in 150 analysed GPT-4V errors, 35% were perceptual. Established Error analysis like this separates "did not see it" from "saw it but reasoned wrongly", which call for different fixes.

Making: is it good, varied, on-prompt and new?

A generated image set can fail in four separate ways, and metrics differ in which they catch:

FID, introduced by Heusel and colleagues, compares statistics of Inception-network features of generated and real images. Established It is computed over whole sets, so it cannot tell you whether any single image matches its prompt. CLIPScore rates image–text compatibility with CLIP and correlated well with human judgments of captions, but was weaker where captions need outside context, such as news. Established Because CLIP-style models often ignore word order (CLIP), a CLIP-based score can rate "a red cube on a blue sphere" highly for the swapped scene.

Carlini and colleagues extracted over a thousand training images from diffusion models, so originality needs checking too. Established

Tiny example (lab numbers)

The diffusion lab shows why one metric is not enough. At guidance 8 with the prompt "plus", every sample that lands on a shape is on the plus: perfect prompt adherence. Yet fewer than half the samples land on any shape, and only 17% of the plus is covered. A metric that only asks "is it on-prompt?" would call this the best setting in the lab.

A checklist

1. Which failure does each reported metric detect, and which does it miss?
2. Are reading results broken down into perception and reasoning errors?
3. For absent-object probes, are the negatives adversarial or random?
4. For generation, are quality, diversity, alignment and originality all reported?
5. Could the benchmark images, or close copies, be in the training data (contamination)?

Chapter 16 takes these questions further: evaluation, reliability and interpretability for every kind of model.

Mini experiment

In the diffusion lab, choose "plus" and compare guidance 3 and guidance 5. Their "On the plus" percentages are close; their coverage is not. Which would a prompt-alignment metric prefer, which would a diversity metric prefer, and which would you ship?