AI Memory ≠ Human Memory
Menu

Scoring

Score Audit

Two checks on the scores behind Table 1. A search of the literature for every mechanism filed as Not Yet Compared, to try to prove that label wrong; and a comparison of each score against what its own write-up concluded.

Literature searched 21 September 2026. The disagreements below are read from the files each time this page loads, so a row leaves it by being settled, not by being edited out. Back to the scores.

Testing the Ten Uncompared Rows

Ten mechanisms are scored Not Yet Compared, meaning a comparison could be made and nobody has made one. Each was given to an independent search instructed to falsify that: three passes per mechanism, one on the construct's own name, one on the human paradigm's name without it, and one chasing citations forward and backward from the AI-only papers already held. The bar was a single paper reporting human and AI numbers on the same or a deliberately matched task, drawing the comparison itself. A study testing only AI, however faithfully it borrows a human paradigm, does not count.

4 of the 10 did not survive. In every one of those, the mechanism's own write-up had already reached the same conclusion the search did.

All five searches ran without general web search available, working instead through the arXiv, OpenAlex, Europe PMC, Crossref, Semantic Scholar and bioRxiv interfaces directly. Coverage is good but not identical to a search that included Google Scholar.

Where a Score and Its Write-Up Disagree

The scores in db/scoring.yml were researched blind to the construct write-ups, so the two are independent readings of the same literature. Where they land differently, the blinding has done its job and left something to settle. On 11 of 29 mechanisms the write-up's own VERDICT sentence opens with a judgement the badge above it contradicts.

Reading both sides, these turned out to be three different problems rather than 11 of one.

The score credits less than the evidence carries 2

Here the write-up names a comparison, and an independent search found the same one. The score is the outlier, and it is the only group where the poster currently understates what is known.

  • Schema Application
    Badge
    Not Yet Compared
    Write-up
    Below Human
    Its words
    “Below human. The best model reached roughly two thirds of the human score on the same items, most models fell far short of that, and the category is the one where supplying more retrieved context made performance worse rather than better.”
  • Integrative Inference
    Badge
    Not Yet Compared
    Write-up
    Below Human
    Its words
    “Below human, for the product-level mechanism. Measured against a human baseline on the same items and the same metric, the best model tested falls roughly 34 F1 points short on multi-hop questions.”

The two passes read different mechanisms 5

One row's name covers more than one mechanism, and the write-up and the scored findings each examined a different one. Both can be right about what they read. Semantic Knowledge is the case to know: the write-up analysed knowledge editing in the weights and found no human number, while the score rests on the WAIS-IV Information subtest, scored by clinical psychologists against norms from 2,200 adults, where nine of ten models reached the test's ceiling at the 99.9th percentile. The score stands; the row's name is what is ambiguous.

  • Working Memory
    Badge
    Comparable to Humans
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. The obstacle is not a missing human number but the absence of a common measure: the human capacity limit and the model capacity limit are both placed at n = 3, but by different metrics, in different modalities, in different studies.”
  • Semantic Knowledge
    Badge
    Exceeds Human
    Write-up
    Comparison Not Possible
    Its words
    “No comparison, for both mechanisms. All figures above come from full texts read in the original, so the measurements themselves are firm, but neither mechanism has been placed against a human number in the same paradigm.”
  • Schema Formation
    Badge
    Below Human
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. Both mechanisms have been measured in models, and both show what the human construct predicts: structure that accounts for more of the training set is acquired earlier, and material consistent with what the system already holds is acquired faster than material that is not.”
  • Retrieval Interference
    Badge
    Comparable to Humans
    Write-up
    Below Human
    Its words
    “Below human at the memory layer for standard dense retrieval, Comparable at the memory layer for the late-interaction and graph-indexed systems, and No comparison at weights and context. Dense passage retrieval loses discriminability substantially faster than people do as associations accumulate, and the intervals on the two slopes do not overlap.”
  • Metacognitive Control
    Badge
    Below Human
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. The operation is directly measured in models across a large and varied item set, and a quantified human paradigm exists that measures the same decision under the same pair of metrics, but the two have never been run over shared items, and the difficulty scales are not linked.”

The same words mean opposite things 4

These write-ups open “No comparison.” meaning nobody has made one. The code named no_comparison means one cannot be made. Source Monitoring settles its own case: it should be revisited if a benchmark adds a human condition, “which for M3-SLU would be straightforward”. That is Not Yet Compared, which is what the badge says. Nothing to change but the sentence.

  • Consolidation
    Badge
    Not Yet Compared
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. Both mechanisms are measured, and one is measured under the construct's own name and ablated against a no-consolidation control, but the model-side measures share no units with the human retention benefit and no study has tested both populations on one task.”
  • Context-Dependent Retrieval
    Badge
    Not Yet Compared
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. Cue-based retrieval from spatial and temporal cues is measured in models, and it degrades in a direction the human literature would recognise as cue overload, but the defining property of the human construct, that the surround is captured without intent and that reinstating it helps relative to a mismatched surround, is not manipulated in any assessed mechanism, and no commensurable human baseline exists.”
  • Prospective Initiation
    Badge
    Not Yet Compared
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. Prospective initiation is measured directly in models, by two benchmarks built explicitly on the human construct, and it is measured as clearly deficient against the benchmarks' own ceilings and against the same models' retrospective performance on identical inputs.”
  • Source Monitoring
    Badge
    Not Yet Compared
    Write-up
    Comparison Not Possible
    Its words
    “No comparison. Speaker attribution is now measured in models directly (M3-SLU's Speaker Attribution task), indirectly (Chang and Chen's hallucination rate on unanswerable queries), and qualitatively (LoCoMo's error taxonomy), and the three converge on the same result: models track content well and origin poorly.”

What Follows