Liu & Gabriel (2026)

PM-Bench: Evaluating Prospective Memory in LLM Agents

arXiv

Find It

DOI

Abstract

A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.

Reference

Liu, G., & Gabriel, S. (2026, July). PM-Bench: Evaluating Prospective Memory of LLM Agents. In Third Conference on Language Modeling.

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Prospective Initiation Memory Processes

Measured in AI

PM-Bench, a peer-reviewed 2026 benchmark, adapts the psychologists' Virtual Week paradigm for AI agents (language models that take actions with tools): over a simulated seven-day week with 81 scored event- and time-based tasks, the agent must keep up an ongoing activity and act on earlier intentions when the right cue or time arrives
Measured in AI

Across eight models, per-model scores (Set-F1, which penalizes both missed and unneeded actions) ranged from 42.0% (Qwen3-8B with no added support) to 79.1% (GPT-5.4 given optional periodic check-in prompts); averaged over models the best setup reached only 65.1%, and tasks needing clock-checking had a 54.7% hit rate
Measured in AI

Not yet compared, because none of these studies tested people, though a human group could be given the same tasks: human studies report the proportion of cues acted on over minutes to days, whereas the AI benchmarks use different tasks, delays measured in words of conversation, and scores such as Set-F1 or model-judged accuracy, so there is no common scale; the AI failures are notable because the intention often remained visible to the model