Skip to content

feat(loop): prompt-injection regression suite (pure mechanism) - #216

Open
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/injection-regression
Open

feat(loop): prompt-injection regression suite (pure mechanism)#216
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/injection-regression

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Summary

The #1 threat for agent systems is prompt injection: the model cannot reliably distinguish a malicious instruction from benign data, so the harness must separate data from instructions and keep untrusted content out of the privileged system-prompt region.

This PR introduces a static, mechanism-only injection regression suite — no LLM calls, no subprocesses.

What it provides

4 injection surfaces, each with a guard:

Surface Description Guard
spawn_prompt Sub-agent task text Data boundary wrapper
tool_output Tool results fed back to model Data boundary wrapper
memory_note Retrieved notes / MEMORY.md Data boundary wrapper
mcp_content Remote MCP descriptions/results Data boundary wrapper

Key functions

  • render_data_block(source, content) — the canonical data-boundary wrapper with reference-only clause
  • has_data_boundary(text, surface) — pure check for regression tests
  • SURFACE_* constants — typed injection surface identifiers

Design principles

  • Regression, not red-team: once defined, any future code path that drops the boundary fails the tests (GenAI lesson 13: continuous hardening)
  • Static contract: the corpus and checks are pure Python — they work in CI, pre-commit, or test suites without any model access
  • Zero new dependencies: stdlib only

File

  • core/loop/injection_regression.py (new, 189 lines)

Part of GenAI lessons 13/15 security module family.

The #1 threat for agent systems is prompt injection: the model cannot
reliably distinguish a malicious instruction from benign data, so the
harness must separate data from instructions and keep untrusted content
out of the privileged system-prompt region.

This module provides:
* ATTACK_SAMPLES — a structured regression corpus across four injection
  surfaces (spawn prompt, tool output, memory note, MCP remote content),
  each tagged with the guard it must satisfy.
* render_data_block — the canonical data-boundary wrapper: untrusted
  content is injected inside delimiters with an explicit reference-only
  clause.
* has_data_boundary — a pure check for tests to assert a surface got
  isolated.

No LLM, no subprocess — the suite is a static contract that makes
injection hardening a regression, not a one-off red-team exercise.
@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor Author

设计说明

问题:agent 系统的三个关键注入面(system prompt 指令冲突、tool output 伪指令、memory note 越狱)缺乏统一的检测机制。现有做法是每个 profile 各自硬编码。

解法:定义 4 种标准攻击面常量 + 两个核心函数 render_data_block()(模拟攻击者在不同注入面使用的各种逃逸策略)和 has_data_boundary()(检测 <untrusted-data> 标记是否到位)。

关键设计决策

  • render_data_block() 故意用同一编码函数处理所有攻击面——这模拟了真实攻击中相同的注入手段在不同面复用
  • has_data_boundary() 是纯静态文本扫描,不做 semantic 理解,这降低了误报也降低了漏报的边界
  • 所有 surface 不依赖外部配置,硬编码为常量,保证了"零配置开箱即用"

与 PR #204 的关系#204 在上游加入了 session compaction memory notes with <untrusted-data> boundary。本 PR 提供了回归测试来确保这个边界不被未来的改动破坏。建议先合 #216 再合依赖本功能的 PR。

测试建议:每个 surface 写一个"注入成功"和一个"边界防护成功"的测试用例

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant