Jumelet et al. (2024)

Do Language Models Exhibit Human-like Structural Priming Effects?

Findings of the Association for Computational Linguistics ACL 2024, 14727-14742

Find It

DOI

Abstract

We explore which linguistic factors—at the sentence and token level—play an important role in influencing language model predictions, and investigate whether these are reflective of results found in humans and human corpora (Gries and Kootstra, 2017). We make use of the structural priming paradigm—where recent exposure to a structure facilitates processing of the same structure—to investigate where priming effects manifest, and what factors predict them. We find these effects can be explained via the inverse frequency effect found in human priming, where rarer elements within a prime increase priming effects, as well as lexical dependence between prime and target. Our results provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models.

Reference

Jumelet, J., Zuidema, W., & Sinclair, A. (2024, August). Do language models exhibit human-like structural priming effects?. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 14727-14742).

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Priming Memory Systems

Measured in AI

Across GPT-2 Large and 7-billion-parameter Llama-2, Falcon, Mistral and Zephyr models (base and tuned versions), priming was far more lopsided than in people: sentences that strongly primed one dative structure tended to prime the other weakly or negatively (correlation -0.72 to -0.77), with only 26 to 38% priming in both directions, whereas people generally show positive priming for both. Like people, the models were primed more by rarer structures (the inverse frequency effect)