Cai et al. (2024)

Do large language models resemble humans in language use?

Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, 37-56

Find It

DOI

Reference

Cai, Z., Duan, X., Haslett, D., Wang, S., & Pickering, M. (2024). Do large language models resemble humans in language use?. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, 37-56. https://doi.org/10.18653/v1/2024.cmcl-1.4

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Priming Memory Systems

Measured in AI

Run on a classic 1998 sentence-completion experiment (32 prime and target pairs, 1,000 runs each), ChatGPT and Vicuna reused the primed structure (ChatGPT 0.71 vs 0.58; Vicuna 0.81 vs 0.51), more so when prime and target shared a verb, as people do. Against the human means plotted in the paper (an effect of about 5 points with different verbs and 18 with the same verb), ChatGPT's effect (about 9 and 14) sits within the human values and Vicuna's (about 21 and 38) above them
Measured in AI

The comparison is approximate: materials and procedure differed, the human means come from the original study with no reported spread, and no test compared people with the models. The within-range reading still holds because the models could still see the prime in the conversation, so Vicuna's larger effect cannot count in its favour, and ChatGPT, the stronger system (it matched the human pattern in 10 of 12 experiments, Vicuna in 7), sits among the human values