Skip to content

"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers

Nov 2025 · arXiv.org · Vol abs/2511.01287 · 6 citations · ⚡ 1 influential · 31 references
Computer Science

TL;DR

This work proposes two classes of attacks: a static attack, which employs a fixed injection prompt, and an iterative attack, which optimizes the injection prompt against a simulated reviewer model to maximize its effectiveness.

Abstract

With the rapid advancement of AI models, their deployment across diverse tasks has become increasingly widespread. A notable emerging application is leveraging AI models to assist in reviewing scientific papers. However, recent reports have revealed that some papers contain hidden, injected prompts designed to manipulate AI reviewers into providing overly favorable evaluations. In this work, we present an early systematic investigation into this emerging threat. We propose two classes of attacks: (1) static attack, which employs a fixed injection prompt, and (2) iterative attack, which optimizes the injection prompt against a simulated reviewer model to maximize its effectiveness. Both attacks achieve striking performance, frequently inducing full evaluation scores when targeting frontier AI reviewers. Furthermore, we show that these attacks are robust across various settings. To counter this threat, we explore a simple detection-based defense. While it substantially reduces the attack success rate, we demonstrate that an adaptive attacker can partially circumvent this defense. Our findings underscore the need for greater attention and rigorous safeguards against prompt-injection threats in AI-assisted peer review.

View source

Similar papers

#natural language process... Preprint Oct 2026

Prevented is Not Protected: What Attack Success Rate Hides about RAG

Aligned RAG Evaluation is introduced, a matched-query protocol that jointly examines clean and poisoned responses, with and without defense, and exposes critical blind spots in ASR-based evaluation and motivates a more comprehensive assessment of RAG poisoning defenses.

Chahana Dahal, Zuo-Bin Xiong · 0 citations
Review Oct 2026

Adaptive Code Revision Attacks on AI Pull Request Reviewers

Pull-request review protects software before new code reaches users, helping prevent vulnerabilities that could expose users to attacks. AI agents increasingly perform these reviews and explain which problems need fixing. However, for an attacker submitting vulnerable code, this feedback also reveals what changes may s...

Jing-Zhi Gong, Jie M. Zhang, Gunel Jahangirova et al. · 1 citation
Sep 2026

pikit: A Composable Toolkit for Indirect Prompt Injection Research and Evaluation

Indirect prompt injection embeds malicious instructions within external content retrieved by LLM-based agents, altering target behavior without user authorization. We introduce pikit, a research toolkit designed to systematically evaluate these threats across three core dimensions: attacks (13 methods), channels (16 ca...

Zong-Hao Ying, Xiang-Fan Wu, Bo Yang et al. · 0 citations
Preprint Nov 2026

The Devil is in the Context: Nine Dimensions of Prompt Injection Attacks

Prompt injection is usually discussed and evaluated as a single, homogeneous type of attack. However, vulnerabilities reported in real-world systems show that prompt injections vary widely. Moreover, the same instruction can be a prompt injection in one context and a legitimate request in another, depending on the user...

Jia-Min Chang, Kristina Nikolic, Rishi Sharma et al. · 0 citations
#natural language process... Preprint Oct 2026

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit...

Farhan Zarif, Kazi Shahid Ahmed Galib, Synthia Akter Promi et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.