Khalifa et al. (2024)

Source-Aware Training Enables Knowledge Attribution in Language Models

arXiv

Find It

DOI

Abstract

Large language models (LLMs) learn a vast amount of knowledge during pretraining, but they are often oblivious to the source(s) of such knowledge. We investigate the problem of intrinsic source citation, where LLMs are required to cite the pretraining source supporting a generated response. Intrinsic source citation can enhance LLM transparency, interpretability, and verifiability. To give LLMs such ability, we explore source-aware training -- a recipe that involves (i) training the LLM to associate unique source document identifiers with the knowledge in each document, followed by (ii) an instruction-tuning stage to teach the LLM to cite a supporting pretraining source when prompted. Source-aware training borrows from existing pretraining/fine-tuning frameworks and requires minimal changes to the model architecture or implementation. Through experiments on synthetic data, we demonstrate that our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's perplexity compared to standard pretraining. Our findings also highlight the importance of pretraining data augmentation in achieving attribution. Code and data available here: \url{https://github.com/mukhal/intrinsic-source-citation}

Reference

Khalifa, M., Wadden, D., Strubell, E., Lee, H., Wang, L., Beltagy, I., & Peng, H. (2024). Source-Aware Training Enables Knowledge Attribution in Language Models. arXiv preprint arXiv:2404.01019. https://doi.org/10.48550/arxiv.2404.01019

What Rests on This

2 mechanisms cite this source. Each finding below is quoted as it appears on that mechanism's own page.

Recognition Memory Processes

Measured in AI

No study located tests the recollection route in AI (remember/know judgments, an ROC recollection estimate, or retrieving the study context of a recognized item); the nearest is a 2024 study that trained a language model on synthetic documents to cite the document a fact came from, a source-memory task with no human baseline, so a familiarity-only AI signal is being set against a human score that can draw on both routes

Source Monitoring Metacognition

Measured in AI

For training, a 1.1-billion-parameter model trained on synthetic biographies with document labels answered 88.8% of questions on unseen documents correctly but named the correct source document only 47.3% of the time, a gap between content and source memory; the ability required special training, since standard models are not trained to track sources