Zeng et al. (2026)

LVLMs and Humans Ground Differently in Referential Communication

arXiv

Find It

DOI

Abstract

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. We present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We show that LVLMs cannot interactively generate and resolve referring expressions in a way that enables smooth communication, a crucial skill that underlies human language use. We release our corpus of 356 dialogues (89 pairs over 4 rounds each) along with the online pipeline for data collection and the tools for analyzing accuracy, efficiency, and lexical overlap.

Reference

Zeng, P., Li, W., Paige, A. J., Wang, Z., Kaliosis, P., Samaras, D., Zelinsky, G., Brennan, S. E., & Rambow, O. (2026). LVLMs and Humans Ground Differently in Referential Communication. arXiv preprint arXiv:2601.19792. https://doi.org/10.48550/arxiv.2601.19792

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Partner Model Representational Structures

Measured in AI

In a live experiment with 103 adults (32 human pairs, 39 human-AI pairs, plus 18 AI-AI pairs) matching objects over four rounds, human pairs rose from 80% to over 90% accuracy while using fewer words each round (slope -74.9 words); GPT-5.2 (2025), given the full dialogue history, fell in accuracy from 90% when paired with itself (slope -5.3 points per round) and from 85% when directing a human (slope -15.3), and its word counts did not fall