Hu & Lewis (2025)

Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm

arXiv

Find It

DOI

Abstract

Cognitive tasks originally developed for humans are now increasingly used to study language models. While applying these tasks is often straightforward, interpreting their results can be challenging. In particular, when a model underperforms, it is often unclear whether this results from a limitation in the cognitive ability being tested or a failure to understand the task itself. A recent study argues that GPT 3.5's declining performance on 2-back and 3-back tasks reflects a working memory capacity limit similar to humans (Gong et al., 2024). By analyzing a range of open-source language models of varying performance levels on these tasks, we show that the poor performance is due at least in part to a limitation in task comprehension and task set maintenance. We challenge the best-performing model with progressively harder versions of the task (up to 10-back) and experiment with alternative prompting strategies, before analyzing model attentions. Our larger aim is to contribute to the ongoing conversation around refining methodologies for the cognitive evaluation of language models.

Reference

Hu, X., & Lewis, R. L. (2025). Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm. arXiv preprint arXiv:2412.18120. https://doi.org/10.48550/arxiv.2412.18120

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Working Memory Memory Systems

Measured in AI

Spread across systems is wide: on n-back up to 10-back, the best of eight models (Llama 3.1 70B) kept 93% retrieval accuracy at 3-back and 57% at 10-back while smaller open models and GPT-3.5 fell in lower performance tiers, and the authors traced many failures to misunderstanding or losing track of the task rather than to storage limits