Zhao et al. (2025)

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

arXiv

Find It

DOI

Abstract

Large Language Models (LLMs) are increasingly used as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PrefEval, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PrefEval comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PrefEval contains user personalization or preference information in both explicit and implicit forms, and evaluates LLM performance using a generation and a classification task. With PrefEval, we evaluated the aforementioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in proactively following users' preferences during conversations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PrefEval significantly improves performance. We believe PrefEval serves as a valuable resource for measuring, understanding, and enhancing LLMs' preference following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https://prefeval.github.io/.

Reference

Zhao, S., Hong, M., Liu, Y., Hazarika, D., & Lin, K. (2025). Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs. arXiv preprint arXiv:2502.09597. https://doi.org/10.48550/arxiv.2502.09597

What Rests on This

2 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Involuntary Retrieval Memory Processes

Measured in AI

In the current conversation (the model's context window, meaning what it can see right now): on PrefEval, models had to bring up a preference stated earlier when a later request made it relevant, without being reminded; on one topic, zero-shot accuracy was 0.50 for the best model (o1-preview) at 10 turns, falling to 0.14 at 300 turns, versus 0.07 and 0.02 for Claude 3.5 Sonnet, and most of 10 models were below 10% at 10 turns

Standing Constraints Memory Processes

Measured in AI

Current conversation (what the model can see in its context window): in PrefEval, where a user states a preference, often an avoidance such as an allergy, phobia or refusal, and later asks for advice that a generic answer would violate, most of 10 models followed it less than 10% of the time after 10 turns; the best, o1-preview, managed 50% at 10 turns and 14% at 300 turns, yet a one-line reminder raised it to 98%, so the fact was still visible but did not surface on its own