Wu et al. (2024)

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

arXiv

Find It

DOI

Abstract

Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval. Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at this https URL.

Reference

Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K. W., & Yu, D. (2024). Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.

What Rests on This

6 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Episodic Memory Memory Systems

Measured in AI

Saved memory in commercial assistants: across short three-to-six-session chat histories, ChatGPT's memory feature answered 57.7% (GPT-4o) and 71.1% (GPT-4o-mini) of 97 questions correctly and Coze 33.0% and 24.7%, versus 91.8% when GPT-4o read the full history directly; the benchmark reports no human baseline

Autobiographical Memory Representational Structures

Measured in AI

Commercial assistants with saved memory recalling a user's past chats (3 to 6 sessions) answered 24.7% to 71.1% of 97 questions correctly (best: ChatGPT with GPT-4o-mini), versus 91.8% when GPT-4o could read the full history in its context window, meaning the text it can see in the current conversation; no human baseline was run

Partner Model Representational Structures

Measured in AI

Across sessions, saved memory of a single user has been tested only without human baselines: answering questions about a user's earlier chats, ChatGPT's memory feature scored 57.7% and Coze 33.0% (both on GPT-4o, 2024) against 91.8% when the same model read the chat history directly; related benchmarks of a user's changing preferences (PersonaMem, PrefEval) likewise report no human scores, so none of this can be placed on a human scale

Recognition Memory Processes

Measured in AI

For saved memory, benchmarks test only something close: LongMemEval includes 30 false-premise questions requiring a chat assistant to recognize that information never appeared in past sessions, with no human baseline; with recognition measured in training and the conversation but not saved memory, and familiarity but not recollection, the AI level is indirect and no comparison with documented human results has yet been made

Temporal Placement Memory Processes

Measured in AI

For saved memory in chat assistants, a benchmark of questions about past conversations found that ChatGPT's memory feature answered 44% (GPT-4o) to 65% (GPT-4o-mini) of temporal questions and Coze 4% to 39%, and that tagging stored items with event dates improved retrieval for temporal questions by about 7% to 11%. The "human evaluation" in that study means people scored the systems; no human group took the test

Confidence Calibration Metacognition

Measured in AI

Neither study examined saved memory; the closest test, a benchmark of chat assistants' long-term memory of past sessions, scores whether they decline questions their memory cannot answer (abstention), not whether confidence tracks accuracy, and has no human comparison