Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

This paper investigates a subtle but important distinction in collaborative dialogue: whether vision-language models can tell apart what could be shared (from shared perception) versus what has been shared (through grounding in interaction). Using 13,077 annotated reference expressions from HCRC MapTask dialogues, we evaluate VLMs under controlled manipulations of dialogue context and map-information access.

A key finding is that providing authentic map images improves overall VLM performance but introduces a systematic bias toward over-predicting alignment between participants — models tend to assume interlocutors share the same interpretation simply because they share the same visual input. Interestingly, textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, suggesting the bias is driven by task-relevant content rather than the visual modality itself.

This has implications for anyone working on dialogue systems, grounded language understanding, or VLM evaluation: current models conflate perceptual access with communicative grounding, which is precisely the kind of error that matters in real collaborative settings. We'd be curious to hear thoughts on how this bias might be mitigated — whether through training objectives that explicitly model asymmetric information states, or through architectural changes that separate perceptual and discourse-level representations.