Maharana et al. (2024)

Evaluating Very Long-Term Conversational Memory of LLM Agents

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13851-13870

Find It

DOI

Abstract

Paraphrase, not the publisher's wording. Prior work on long-term open-domain dialogue evaluates model responses over no more than five chat sessions, leaving the effectiveness of long-context models and retrieval augmentation in genuinely long-term conversation untested. The authors introduce a machine-human pipeline that generates high-quality very long-term dialogues by grounding LLM agent conversations in personas and temporal event graphs, with agents able to share and react to images. Human annotators verify and edit the output for long-range consistency and grounding. The resulting dataset, LoCoMo, supports an evaluation benchmark covering question answering, event summarisation, and multi-modal dialogue generation. Results indicate that models struggle to understand lengthy conversations and to grasp long-range temporal and causal dynamics; long-context models and retrieval help but still lag well behind people.

Reference

Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 13851–13870). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.747

What Rests on This

4 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Autobiographical Memory Representational Structures

Measured in AI

On LoCoMo, question answering over conversations averaging about 600 turns between agents given personas and timelines of life events, the best model, GPT-4-turbo, scored 51.6 F1 (a 0-100 answer-overlap score) against 18.4 to 42.8 for other models and a reported human score of 87.9; the paper does not report how many people were tested or how, and the task is reading another's conversation rather than remembering one's own life

Integrative Inference Memory Processes

Measured in AI

For saved memory, a long-conversation benchmark asked questions that require combining information from different sessions: across five systems multi-hop scores ranged from 12.8 to 42.0 F1 (GPT-3.5-turbo-16K best) against a listed human score of 85.8, but the paper does not report how many people answered or whether they could reread the conversations

Source Monitoring Metacognition

Measured in AI

In a benchmark of 50 very long two-person conversations, long-context models were especially prone to assigning events to the wrong speaker, scoring as low as 2.1 F1 on trick questions versus 70.2 for GPT-4-turbo; a human score of 89.4 is reported without the number of people or procedure, so it cannot serve as a documented baseline

Metacognitive Control Metacognition

Measured in AI

For the current conversation and saved memory, withholding was tested with unanswerable trick questions about very long chats: scores (F1, out of 100) ranged from 2.5 (Claude 3 Sonnet) and 15.7 (GPT-4 Turbo) to 80.0 (Llama 3 70B, which could see only a short part of the chat), and from 9.9 to 32.5 when GPT-3.5 looked up stored chat excerpts; a human score of 89.4 is reported without saying how it was obtained, so it cannot be used