Coda-Forno et al. (2024)

CogBench: a large language model walks into a psychology lab

arXiv

Find It

DOI

Abstract

Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study highlights the crucial role of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, we find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs' behavior. Finally, we explore the effects of prompt-engineering techniques. We discover that chain-of-thought prompting improves probabilistic reasoning, while take-a-step-back prompting fosters model-based behaviors.

Reference

Coda-Forno, J., Binz, M., Wang, J. X., & Schulz, E. (2024). CogBench: A large language model walks into a psychology lab. arXiv. https://arxiv.org/abs/2402.18225

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Instrumental Learning Memory Systems

Measured in AI

CogBench ran 35 language models in conversation on the probabilistic instrumental learning task from Lefebvre et al. (2017), four interleaved slot-machine pairs over 96 trials, scoring money earned on a scale where 0 is a random chooser and 1 is the average human participant; all seven established models in its main figure reached human level on this task, and GPT-4 and Claude-1 reached it on five of six tasks, while every one of the seven reached it on at least three
Measured in AI

The comparison is approximate and uneven: the task and money scale were shared, but every past choice and reward stayed written in the model's prompt while people had to learn and remember them, and CogBench reports only the human average, with no spread and no statistical test against people; GPT-4 therefore matched the human average on a test that favoured it, a tie that counts as Comparable and is provisional