Preprint · arXiv:2603.28026v2

BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA

Taeyun Roh¹, Suhyeong Park^1,2, Dongyoung Lee¹, Eunyeong Jo³, Wonjune Jang^4,2, Junha Jung^1,2, Jaewoo Kang^1,2

¹Korea University · ²AIGEN Sciences · ³KAIST · ⁴Myongji University

BUZZY is a training-free decoding method that uses the text-only context as a natural control. By subtracting text-only candidate preferences from multimodal scores, it suppresses the “buzz” of linguistically plausible choices and brings visually grounded evidence forward.

arXiv Code

BUZZY scores answer candidates with multimodal and text-only inputs, then subtracts the text-prior score to promote visually grounded candidates.

Problem

Choices do more than define the answer space.

Multimodal MCQA places every candidate inside the text prompt. This can make a VLM favor an answer because it sounds plausible from the question and choices alone, even when the image supports another option.

Prior work mostly targets structural effects such as answer-label or position bias. BUZZY instead isolates semantic text priors and asks how much the image actually shifts the model away from them.

Motivating Observation

Correct answers depart further from text-only preferences.

Across scientific and general-purpose benchmarks, correct greedy predictions have higher Jensen–Shannon divergence and lower cosine similarity between multimodal and text-only answer distributions. Incorrect predictions stay closer to the textual baseline.

Multimodal versus text-only distributions with Qwen3.5-4B.

Distance between multimodal and text-only answer distributions.
Benchmark	JS Correct	JS Incorrect	Cosine Correct	Cosine Incorrect
MAC	0.248	0.145	0.556	0.722
SciFIBench	0.116	0.073	0.762	0.825
MMSci	0.087	0.054	0.801	0.879
MMBench	0.153	0.049	0.721	0.886
MMStar	0.106	0.082	0.774	0.819

Method

Use text-only preference as the contrastive signal.

BUZZY applies one fixed correction to every candidate: l_buzzy(c) = l_mm(c) − α l_txt(c). The paper uses α = 0.5 globally, without dataset- or backbone-specific tuning.

Multimodal score

Score each option using the image, question, and complete candidate set.

Text-only control

Remove only the image and score the same options to expose the preference induced by language alone.

Contrastive correction

Penalize candidates already favored by text and select the option with the strongest corrected visual support.

Main Results

Best average performance for every evaluated backbone.

BUZZY is evaluated on MAC, SciFIBench, MMSci, MMBench, and MMStar with five VLM families. It consistently outperforms greedy, prompt-enhanced, VCD, ICD, and ALC decoding in average accuracy and Macro-F1.

57.84 Average accuracy across 25 backbone–benchmark pairs

53.56 Average Macro-F1 across the full evaluation suite

+1.99 Accuracy points over the strongest aggregate baseline

Backbone-level averages. Values are ACC / Macro-F1.

Average results across five multimodal MCQA benchmarks.
Backbone	Greedy	VCD	BUZZY
Qwen3.5-4B	53.45 / 49.44	53.81 / 50.41	56.52 / 50.90
Qwen3.5-9B	60.09 / 56.43	61.22 / 57.74	63.05 / 59.25
Phi-3.5-Vision	52.75 / 49.45	55.36 / 51.59	57.09 / 52.43
Molmo-7B-D	44.10 / 37.99	46.45 / 41.68	46.93 / 43.21
GLM-4.6V-Flash	61.87 / 57.81	62.40 / 57.10	65.59 / 61.99

Efficiency

One visual pass, not two.

VCD and ICD construct their reference distributions with another full multimodal pass. BUZZY pairs one multimodal pass with a much cheaper text-only pass, avoiding repeated processing of visual tokens.

Average latency is 0.30 seconds, compared with 0.45 seconds for VCD and 0.42 seconds for ICD. This is more than a 28% reduction relative to prior multimodal contrastive methods.

Analysis

The remaining signal tracks image-conditioned evidence.

The paper decomposes multimodal candidate preferences into a text-aligned projection and a text-orthogonal residual. Replacing the matched image with a hard negative and then a random image progressively weakens the residual, showing that it contains visual evidence rather than another textual artifact.

Robust global weight

Performance remains stable for α values from 0.3 to 0.7. α = 0.5 reaches the best aggregate scores: 57.84 accuracy and 53.56 Macro-F1.

Beyond base models

On a ChartQA-fine-tuned Qwen3-VL-8B model, BUZZY achieves 70.53 accuracy and 67.51 Macro-F1, exceeding VCD by 1.10 and 1.08 points.

Takeaway

Measure visual support relative to the textual baseline.

High multimodal confidence alone does not prove that a VLM used the image. BUZZY rewards the support a choice gains from visual context, not the plausibility it already had from text.

The result is a simple, training-free correction that improves multimodal MCQA across scientific and general benchmarks while preserving an efficient inference path.

Resources

Citation

@article{roh2026buzzy,
  title={BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA},
  author={Roh, Taeyun and Park, Suhyeong and Lee, Dongyoung and Jo, Eunyeong and Jang, Wonjune and Jung, Junha and Kang, Jaewoo},
  journal={arXiv preprint arXiv:2603.28026},
  year={2026}
}