Sun et al. (2025)

How new data permeates LLM knowledge and how to dilute it

arXiv

Find It

DOI

Abstract

Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts. To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update pruning method. These approaches reduce undesirable priming effects by 50-95\% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/

Reference

Sun, C., Aksitov, R., Zhmoginov, A., Miller, N. A., Vladymyrov, M., Rueckert, U., Kim, B., & Sandler, M. (2025). How new data permeates LLM knowledge and how to dilute it. arXiv preprint arXiv:2504.09522. https://doi.org/10.48550/arxiv.2504.09522

What Rests on This

2 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Priming Memory Systems

Measured in AI

Beyond the conversation, a 2025 preprint found a training-level analogue it calls priming: after fine-tuning (further training that changes the model's stored weights) on one new fact, PaLM-2, Gemma and Llama models applied it in unrelated contexts, predictably from the fact's pre-training word probabilities, but with no human comparison. No study was found that tests priming from AI assistants' saved memory

Involuntary Retrieval Memory Processes

Measured in AI

From training: after a new fact is learned through training updates, models show a 'priming' effect in which the new keyword intrudes into unrelated contexts after as few as 3 spaced exposures, most strongly for words the model found improbable beforehand, across PaLM-2, Gemma and Llama models