Ji-An et al. (2024)

Linking In-context Learning in Transformers to Human Episodic Memory

arXiv

Find It

DOI

Abstract

Understanding connections between artificial and biological intelligent systems can reveal fundamental principles of general intelligence. While many artificial intelligence models have a neuroscience counterpart, such connections are largely missing in Transformer models and the self-attention mechanism. Here, we examine the relationship between interacting attention heads and human episodic memory. We focus on induction heads, which contribute to in-context learning in Transformer-based large language models (LLMs). We demonstrate that induction heads are behaviorally, functionally, and mechanistically similar to the contextual maintenance and retrieval (CMR) model of human episodic memory. Our analyses of LLMs pre-trained on extensive text data show that CMR-like heads often emerge in the intermediate and late layers, qualitatively mirroring human memory biases. The ablation of CMR-like heads suggests their causal role in in-context learning. Our findings uncover a parallel between the computational mechanisms of LLMs and human memory, offering valuable insights into both research fields.

Reference

Ji-An, L., Zhou, C. Y., Benna, M. K., & Mattar, M. G. (2024). Linking In-context Learning in Transformers to Human Episodic Memory. arXiv preprint arXiv:2405.14992. https://doi.org/10.48550/arxiv.2405.14992

What Rests on This

2 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Episodic Memory Memory Systems

Measured in AI

A peer-reviewed NeurIPS 2024 study found that attention heads in pretrained models favour the item that followed a repeated item, a temporal contiguity and forward bias resembling human free-recall curves (PEERS, N = 171); this is a mechanistic parallel to the temporal context model, not a comparison of how much either side remembers

Context-Dependent Retrieval Memory Processes

Measured in AI

Induction heads (attention components that find an earlier occurrence of the current word and retrieve what followed it) behave like the temporal-context model of human free recall across five model families from GPT-2 to Llama-3-8B, with fitted context-drift parameters in the range of human data (N = 171); this compares the shape of a recall bias on still-visible text, not retrieval success