Anil et al. (2024)

Many-shot Jailbreaking

Advances in Neural Information Processing Systems 37, 129696-129742

Find It

DOI

Abstract

We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.

Reference

Anil, C., Durmus, E., Panickssery, N., Sharma, M., Benton, J., Kundu, S., ... & Duvenaud, D. (2024). Many-shot jailbreaking. Advances in Neural Information Processing Systems, 37, 129696-129742.

What Rests on This

Cited in the argument rather than behind a score: this is one of the papers the taxonomy is built on, named where the site explains itself rather than in the findings under a mechanism.