Human memory is more than just facts. What happens when you test AI on more than recall?

Key Points from the Study

1. 01

   Much of human memory has never been compared to AI.

   Of the 29 memory mechanisms examined, 9 have been tested in AI and in people but never against each other, and 6 cannot be compared at all. That means unmeasured, not inadequate.

2. 02

   AI does well on facts and poorly on people.

   It meets or exceeds human performance on factual systems such as semantic knowledge, and matches people on working memory. It falls below on the mechanisms that build a picture of a person over time: partner model, episodic memory, temporal placement.

3. 03

   The more relational the task, the more types of memory it takes.

   Advice and consulting draws on 13 of the 29 mechanisms; therapy draws on 23. What therapy adds lands almost entirely on mechanisms AI performs worse at, or that no one has compared.

Memory Mechanisms for Humans Against AI

1. 01

   Most of memory has not been compared at all.

   Of the 29 mechanisms, 15 have no comparison behind them: 9 could be compared and no study has done it, and 6 have no fair test that could put people and AI on one scale.

2. 02

   Where the comparisons exist, they are mostly about memory systems.

   7 of the 8 memory systems have been compared, against 3 of the 12 memory processes: the kinds of memory are studied and the operations on them are not.

3. 03

   AI performed one memory function better than humans.

   Semantic Knowledge is the only ability where AI did better than people. 8 of the 14 comparisons put it below human performance, and 5 found it within the human range.

4. 04

   Not one AI-to-human comparison is unqualified.

   Every one of the 14 comparisons carries a caveat: 9 rest on tests that were not matched between people and AI, 12 on a single study, a single system, or work that has not been refereed, and 7 on both. Not one is a clean, matched, replicated test.

See Table 1 in full

What Each Relationship Depends On

1. 01

   Therapy uses far more types of memory than consulting or advice does.

   Therapy draws on 23 of the 29 memory mechanisms; advice draws on 13, and every one of those is also used by therapy. The gap is not a product of therapy's longer list of elements: the average set of 9 of its 15 elements still draws on 19.5 mechanisms.

2. 02

   What therapy needs more of is mostly memory processes.

   Of the 10 mechanisms therapy uses and advice does not, 7 are memory processes. Therapy draws on all 6 retrieval and surfacing mechanisms, where advice draws on 1, and it is the only one of the two that needs metacognition. How AI performs on this gap is mostly unknown rather than worse: of those 10, 4 have not yet been compared with human memory and 2 cannot be, against 3 rated Below Human and 1 Comparable to Humans.

3. 03

   Much of what these relationships need has never been compared with human memory.

   Of the 29 mechanisms, 9 have been tested in AI and in people but never against each other, and 6 cannot be compared at all, because no fair test could put both sides on one scale. Of the 14 mechanisms either relationship needs, 4 are waiting on a comparison and 1 are beyond one. That is unmeasured against human memory, not inadequate.

See Table 2 in full

Advice and Consulting Relationships

1. 01

   How advice is delivered outranks whether it would work.

   Politeness is the strongest predictor of feeling helped (β = .531), more than twice efficacy (β = .201). Consulting is more relational than its framing as expertise suggests.

2. 02

   Knowledge and the person split the load.

   Semantic Knowledge is needed by 3 of the 9 elements and Partner Model by 3: knowledge for whether the advice is sound, the person for whether it fits.

3. 03

   Tailoring to the person cuts both ways.

   Feasibility, which needs a model of what this person can manage, predicts coping on its own (ρ = .375) but turns negative once the other predictors are controlled (β = −.132).

4. 04

   Consulting uses few memory mechanisms.

   24 of the 29 mechanisms are needed by no element, and likability and trustworthiness need no mechanism at all.

See Table 2a in full

Therapeutic Relationships

1. 01

   The relationship runs on a model of the person.

   Partner Model is needed by 11 of the 15 elements. The next most needed, Episodic Memory, is needed by 5.

2. 02

   Memory matters most where the relationship spans sessions.

   Alliance, goal consensus, collaboration, and rupture repair each need a record of this person or of particular sessions. At the other end, positive regard, congruence, and emotional expression need no mechanism at all.

3. 03

   The most memory-hungry elements have the thinnest evidence.

   Self-disclosure and immediacy each need 4 mechanisms, more than any other element, and both are rated promising but insufficient, resting on case counts rather than effect sizes.

4. 04

   Therapy draws on every kind of memory, metacognition included.

   Therapy needs mechanisms from all four categories: 3 memory systems, 3 representational structures, 3 memory processes, and 3 metacognitive functions. It draws on 9 of the 12 memory processes and 5 of the 5 metacognitive functions. Advice, by comparison, needs 1 of the 12 memory processes and 0 of the 5 metacognitive functions.

See Table 2b in full

What Follows From All This

1. 01

   Good AI memory is a match between a system and a task, not a score.

   There is no general answer to give. An advice relationship draws on 13 of the 29 memory mechanisms and a therapeutic one on 23, so a system that serves one of them well can still be missing most of what the other needs.

2. 02

   Ask what memory a job needs, not whether the memory is good.

   Of the 29 mechanisms, 9 have never been compared between people and AI and 6 cannot be, so whether an AI has good memory is not a question the evidence can answer. Whether it has been tested on the ability your task rests on is.

3. 03

   For Clinicians: Understand Your Tools Under the Hood.

   Knowing how a platform's memory is built is part of knowing the tool. What it keeps between sessions, what it quietly drops, and what it reconstructs rather than recalls all change what a patient is actually getting from it. That knowledge is teachable, too: a patient who understands what a system does with what they tell it is in a far better position to judge it. Particular attention is warranted for cross-linguistic and cross-cultural users, whose memory patterns may be poorly represented in training data.

4. 04

   For AI Designers: Understand Your Audience.

   A use case is not a feature list. Before building for one, work out which kinds of memory the relationship actually runs on: an advice relationship draws on 13 of the 29 mechanisms and a therapeutic one on 23, and most of what therapy adds is memory processes rather than places to put things. Designing from what is technically interesting rather than from what the function requires is how a system ends up impressive at the wrong thing.

5. 05

   For Users: AI Literacy.

   Understand, research, do the due diligence. Before trusting a platform with something that matters, find out how its memory actually works: what it retains, what it discards, and what it reconstructs rather than recalls. AI Basics explains how a model works, and the four layers a platform can hold memory in.

See the implications in full