Pink et al. (2024)

Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks

arXiv

Find It

DOI

Abstract

Current LLM benchmarks focus on evaluating models' memory of facts and semantic relations, primarily assessing semantic aspects of long-term memory. However, in humans, long-term memory also includes episodic memory, which links memories to their contexts, such as the time and place they occurred. The ability to contextualize memories is crucial for many cognitive tasks and everyday functions. This form of memory has not been evaluated in LLMs with existing benchmarks. To address the gap in evaluating memory in LLMs, we introduce Sequence Order Recall Tasks (SORT), which we adapt from tasks used to study episodic memory in cognitive psychology. SORT requires LLMs to recall the correct order of text segments, and provides a general framework that is both easily extendable and does not require any additional annotations. We present an initial evaluation dataset, Book-SORT, comprising 36k pairs of segments extracted from 9 books recently added to the public domain. Based on a human experiment with 155 participants, we show that humans can recall sequence order based on long-term memory of a book. We find that models can perform the task with high accuracy when relevant text is given in-context during the SORT evaluation. However, when presented with the book text only during training, LLMs' performance on SORT falls short. By allowing to evaluate more aspects of memory, we believe that SORT will aid in the emerging development of memory-augmented models.

Reference

Pink, M., Vo, V. A., Wu, Q., Mu, J., Turek, J. S., Hasson, U., Norman, K. A., Michelmann, S., Huth, A., & Toneva, M. (2024). Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks. arXiv preprint arXiv:2410.08133. https://doi.org/10.48550/arxiv.2410.08133

What Rests on This

2 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Episodic Memory Memory Systems

Measured in AI

Training, the fair condition where neither side could look back: a 2024 preprint by the same group found Llama3-8B and Mistral-7B trained on the books for five epochs no better than copies trained without them (about 51%, near chance), while 155 readers averaged 64% on 50-word passages (56% on 20-word ones), were significantly above chance beyond 4,000 words, and peaked at 76% beyond 25,000 words; this condition was dropped from the peer-reviewed version
Measured in AI

Saved memory and spread across systems in the same preprint: four models using retrieval (RAG, meaning the system searches stored passages and pastes matches into its input) scored 55–67% at distances up to 16,500 words, overlapping the human scores with no statistical comparison, while 11 models given book excerpts in view ranged from 56% (Llama2-7B) to 95% (Mixtral-8x22B), a range that does not count against people because the text was visible

Temporal Placement Memory Processes

Measured in humans

In a book-length order test (a 2024 preprint), 155 readers of The Murder of Roger Ackroyd, tested on average 7.5 days after finishing it, judged which of two passages came first (10 pairs each). Accuracy averaged 64% for 50-word and 56% for 20-word passages, reached 76% for passages more than 25,000 words apart, and was reliably above chance (50%) only beyond 4,000 words
Measured in AI

A 2024 preprint adapted recency-judgment tasks from cognitive psychology into a passage-order test for language models, using books unlikely to be in their training, and tested all three places AI holds information. Although framed as episodic memory, the task is a relative-order (recency) judgment, so it bears on this mechanism directly; it does not test dating or how-long-ago judgments. Before seeing the books, 11 models scored 49% to 57% (GPT-4 highest)
Measured in AI

Deciding comparison, the one where neither side could look at the book: Llama 3 8B and Mistral 7B (2023–2024 releases) further trained on the books and then tested without them scored about 50% to 51% across five rounds of training, no better than copies trained without the books (McNemar tests, p > .05), while readers averaged 56% to 64% depending on passage length. The tests differ (nine books versus one, repeated training versus one reading, different passage pairs), but the result is clear because the models stayed at chance while readers were well above it
Measured in AI

That match was on a test easier for the AI (the models could still see the book while readers had to remember it), so it counts only as a provisional Comparable and does not outrank the training result, which tests memory rather than search over visible text. Across systems the spread is wide: with short excerpts visible, 10 models ranged from 56% (Llama 2 7B) to 95% (Mixtral 8x22B); with retrieval-augmented generation, meaning a system that searches the stored book and pastes matching passages into the prompt, accuracy was 55–67%