Aerni et al. (2025)

Measuring Non-Adversarial Reproduction of Training Data in Large Language Models

arXiv

Find It

DOI

Abstract

Large language models memorize parts of their training data. Memorizing short snippets and facts is required to answer questions about the world and to be fluent in any language. But models have also been shown to reproduce long verbatim sequences of memorized text when prompted by a motivated adversary. In this work, we investigate an intermediate regime of memorization that we call non-adversarial reproduction, where we quantify the overlap between model responses and pretraining data when responding to natural and benign prompts. For a variety of innocuous prompt categories (e.g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet. In worst cases, we find generations where 100% of the content can be found exactly online. For the same tasks, we find that human-written text has far less overlap with Internet data. We further study whether prompting strategies can close this reproduction gap between models and humans. While appropriate prompting can reduce non-adversarial reproduction on average, we find that mitigating worst-case reproduction of training data requires stronger defenses -- even for benign interactions.

Reference

Aerni, M., Rando, J., Debenedetti, E., Carlini, N., Ippolito, D., & Tramèr, F. (2025). Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. arXiv preprint arXiv:2411.10242. https://doi.org/10.48550/arxiv.2411.10242

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Involuntary Retrieval Memory Processes

Measured in AI

No study was located that tests AI on involuntary retrieval by name or runs the vigilance-task, sound-cue or diary paradigms on an AI system; the closest evidence comes from studies of unrequested recall under other names (training-data reproduction, priming, proactive preference use, memory leakage), so the level is indirect
Measured in AI

From training: when given ordinary requests such as writing a letter or tutorial, chat models reproduced text they had learned from the internet without being asked to, with 50-plus-character matching snippets making up about 7% of output for Gemini 1.5 up to more than 15% for Claude 3 Opus across four model families
Measured in AI

Comparison is not possible because there is no common scale with documented human results: the only human baseline found (median zero overlap for human writing versus up to 7.5% median for models) used online posts with no count of people, no controlled procedure, possible internet access and some plagiarism, and it measures verbatim copying rather than memories coming to mind; none of the other AI studies included people