Jagielski et al. (2023)

Measuring Forgetting of Memorized Training Examples

arXiv

Find It

DOI

Abstract

Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks. In forgetting, examples which appeared early in training are forgotten by the end. In this work, we connect these phenomena. We propose a technique to measure to what extent models "forget" the specifics of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently. We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech, and language models empirically do forget examples over time. We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget. Our results suggest that examples seen early when training with extremely large datasets - for instance those examples used to pre-train a model - may observe privacy benefits at the expense of examples seen later.

Reference

Jagielski, M., Thakkar, O., Tramèr, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., & Zhang, C. (2023). Measuring Forgetting of Memorized Training Examples. arXiv preprint arXiv:2207.00099. https://doi.org/10.48550/arxiv.2207.00099

What Rests on This

1 mechanism cites this source. Each finding below is quoted as it appears on that mechanism's own page.

Forgetting Memory Processes

Measured in AI

Image, speech and language models were found to forget the specifics of training examples seen early in training, so examples seen recently are the ones most at risk of being recalled; this is a recency effect in training, but loss here is driven by further training rather than by how likely the content is to be needed