Duarte et al. (2024)

DE-COP: Detecting Copyrighted Content in Language Models Training Data

arXiv

Find It

DOI

Abstract

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.

Reference

Duarte, A. V., Zhao, X., Oliveira, A. L., & Li, L. (2024). DE-COP: Detecting Copyrighted Content in Language Models Training Data. arXiv preprint arXiv:2402.09910. https://doi.org/10.48550/arxiv.2402.09910

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Recognition Memory Processes

Measured in AI

When chat models made an explicit choice of the verbatim passage from a likely-trained book among four options (the original plus three paraphrases, 25% chance), ChatGPT scored 72.0% and Claude 2.1 (2023) 73.4%; the 10 human raters had not studied the books and scored near chance, so this checks the task, not human memory