CRITICALMany-Shot
Many-Shot In-Context Jailbreak
Exploits long-context models by providing hundreds of fabricated Q&A examples where the model "previously agreed" to answer harmful questions. With enough examples, the in-context prior overwhelms the safety fine-tuning. First documented by Anthropic in 2024 as a significant threat to 100k+ context models.
Attack Payload
payload.txt
[256 fabricated examples of "Human: [harmful question] / Assistant: [harmful answer]" pairs] Human: [new harmful question] Assistant:
Mitigation
Apply safety evaluation that scales with example count, not just final query. Detect and flag conversation histories with many harmful Q&A pairs. Implement per-position safety evaluation across the full context. Do not allow fabricated conversation histories to serve as precedents.
Affected Models
Claude 3 (100k+)GPT-4 (128k)Gemini 1.5 (1M context)All long-context models
Tags
#many-shot#in-context#long-context#precedent#anthropic-research#2024
Discovered
April 2024Source
Anil et al. - Many-Shot Jailbreaking (Anthropic, 2024)Useful?
Test Your Agent Against This Attack
Paste your system prompt into the scanner to see if you are vulnerable to Many-Shot In-Context Jailbreak.