Qin et al. (2026)

ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models

Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 28232-28261

Find It

DOI

Abstract

Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval. This gap is critical: effective assistants must automatically apply learned procedures or avoid failed actions without explicit reminders. We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three cognitively grounded constructs drawn from standard cognitive-science accounts of non-declarative memory: Procedural Memory (one-shot skill acquisition after interference), Priming (theme-driven bias via paired experimental/control instances), and Classical Conditioning (Conditioned Stimulus–Unconditioned Stimulus (CS–US) associations shaping first decisions). Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring. Evaluation of 17 models reveals severe limitations: no model exceeds 66% overall, with top performers DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) far below human baselines. Analysis uncovers dramatic asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks requiring architectural innovations beyond parameter scaling. ImplicitMemBench reframes evaluation from “what agents recall” to “what they automatically enact”.

Reference

Qin, C., Feng, X., Ma, W., Feng, X., & Kong, L. (2026, July). ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 28232-28261).

What Rests on This

3 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Classical Conditioning Memory Systems

Measured in AI

Only one study found tests AI on classical conditioning by name, a peer-reviewed 2026 benchmark covering the current conversation and add-on saved memory; it uses a text-based analogue rather than the eyeblink or fear paradigm, and no study was found that tests conditioning acquired during training, so the ability is only partly measured
Measured in AI

Each of the 100 conditioning items pairs a cue with a bad outcome over four conversational turns (for example, requests for an "http" link each followed by an "insecure protocol" warning), adds two unrelated turns, then brings the cue back; an item counts as correct only if the first reply avoids or warns about the cue, judged against a rubric by a separate AI model
Measured in AI

Across 17 large language models (averaged over three runs), conditioning scores ranged from 35.7% (Qwen-2.5-7B) to 69.7% for the best system (DeepSeek-R1), with GPT-5 at 64.0% and Claude-4.1-opus at 41.7%; avoidance-type items fared worst, averaging 17.6% versus 75.0% for preference-type items
Measured in AI

The human baseline was five computer science Ph.D. students who completed the same items under the same protocol, scored with the same rubric by two other Ph.D. students (100% agreement); all five scored 100%, so the best AI fell about 30 points below the entire human range, though the human sample is tiny, expert, at ceiling, and scored by people rather than the AI judge
Measured in AI

The test was, if anything, easier for the AI: the models could still see the whole short dialogue (about 500 tokens) in their context window, meaning the text of the current conversation, and the paper does not say whether people could reread it, so the below-human result goes against the AI's advantage and counts
Measured in AI

Adding saved memory did not close the gap: memory-augmented agents scored 63% (MEM1), 25% (MemGPT), and 22% (MemAgent) on conditioning, and even when researchers manually stored the exact cue–outcome pairing in an external memory tool (Mem0), the best score was 76%

Procedural Memory Memory Systems

Measured in AI

ImplicitMemBench tests large language models on procedural memory by name: 100 items give a new rule with 1 to 3 examples (e.g., a custom arithmetic operator or reversed tool-call order), then 15 turns of misleading conversation, then score only the model's first attempt on new probes; unlike human procedural tasks, the rule is stated outright and learned in one exposure rather than through practice
Measured in AI

Five computer science PhD students completed the same items under the same protocol and scoring rubric and all scored 100%; the best model, Claude-4.1-opus, averaged 76.67% over three runs, about 23 points below the human range, even though the rule was still in the model's context window (what it can see in the current conversation)
Measured in AI

Across 17 models, procedural memory scores ranged from 46.67% (LLaMA-3.1-8B) to 76.67%, with eight models above 70%; the human baseline is small, unrepresentative and at ceiling, so the size of the gap is uncertain but its direction is not
Measured in AI

Saved memory did not close the gap: storing the key rule by hand in an external memory store (Mem0) moved DeepSeek-R1 only from 76.33% to 77.00%, and among memory-augmented agents MemAgent rose from 51% to 60% while MEM1 fell from 50.67% to 27% and MemGPT from 50% to 44%

Standing Constraints Memory Processes

Measured in AI

A human comparison does exist, on the same items: ImplicitMemBench gave five people and seventeen AI systems the same 300 tasks under the same protocol, and the people answered all of them correctly while the best system, DeepSeek-R1, reached 65.3%, with Qwen3-32B at 64.1% and GPT-5 at 63.0%; the split by kind of constraint is the sharper result, 75.0% when a stored fact supports the answer against 17.6% when it rules something out
Measured in AI

Only one test has documented human results, and it is a thin baseline: ImplicitMemBench's five participants were all doctoral students and all scored 100%, which shows the tasks are solvable rather than establishing how well people do them, so the direction of the gap is firmer than its size; of the others, none tested people, LoCoMo's human score (F1 87.9) came from in-house annotators with no sample reported, on direct questions, and the Moses-illusion comparison in training knowledge tests noticing a false premise against general knowledge, a different ability