Feng et al. (2026)

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

arXiv

Find It

DOI

Abstract

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Reference

Feng, Z., Ma, J., & Chersoni, E. (2026). Know It, Act on It: Investigating Memory Utilization in LLM Personalization. arXiv preprint arXiv:2607.29433. https://doi.org/10.48550/arxiv.2607.29433

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Standing Constraints Memory Processes

Measured in AI

Saved memory and long histories (a 2026 preprint testing 16 systems on 1,000 preferences): systems that could state a preference when asked often failed to use it in a later scenario that did not mention it; GPT-4o-mini recalled explicit preferences 83.7% of the time but acted on them 19.8% of the time, and average act accuracy ranged from 11.2% (Zep/Graphiti memory) to 61.5% (Claude 4.6 Sonnet), with health-related preferences used least by memory systems