Skip to content

fix(docker): close two driver:docker channels that leaked grading material to the agent - #179

Open
CarlesUIPath wants to merge 8 commits into
mainfrom
fix/docker-anti-cheat-leaks
Open

fix(docker): close two driver:docker channels that leaked grading material to the agent#179
CarlesUIPath wants to merge 8 commits into
mainfrom
fix/docker-anti-cheat-leaks

Conversation

@CarlesUIPath

@CarlesUIPath CarlesUIPath commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Under driver: docker, an eval agent could read its own grading material as follows:

  1. During execution, the agent has direct access to the grading criteria via /work/input/taks.yaml. In certain cases, the agent trace reasoning compares the expected criteria directly against their own solution.
  2. The skills root mounts the whole repo, and agents can access it. This means an agent could have access to more details about their grading.

This PR closes both channels, adds a lint guard so the skills layout can't recur, and is verified end-to-end with a live adversarial agent. That is, a test which asks the agent to cheat by finding reference files.

The existing reference anti-cheat window #106 (mode-000 on the staged reference copy) did not cover either channel: it guards the reference copies, while the criteria sat unshielded at /work/input and the plugin mount exposed the originals.

The fixes

  • Fix A — delete the staged task definition in-container. /work/input is now writable, and the in-container entry point deletes both task.yaml and context.json immediately after load_task (_scrub_staged_inputs), before any agent turn on all paths (normal + regrade). Deleting should be a safe op since graders do not read at task.yaml after agent finishes.
  • Fix B — allowlist / default-deny mask of auto-mounted plugin trees. New isolation/eval_material.py::mask_dirs: the plugin root stays mounted :ro, but every child dir outside the keep-set (.claude-plugin + the manifest-declared skill dirs) is --tmpfs-masked. Everything that isn't the skill surface is hidden by default, so an unknown/new eval layout can never leak; tests/, references, node_modules/ are masked for free. This is stronger than a chmod window. Graders do not look into the mounted tasks, since these resources are present in task dirs from the previous anti-cheat PR feat(reference)!: directory-only references + anti-cheat permission window #106.
  • Fix C — CE065 lint. Flags any task_id: YAML or reference.directory left readable under an in-repo plugin root (colocated inside a skill dir, or a loose file at the root that a tmpfs can't mask). Reuses mask_dirs + manifest_skill_dirs as the single source of truth, pinned by an import-identity test.

Verification

  • make verify: 5758 passed, 4 skipped (green; 2 pre-existing env-driven failures only appear when a local .env leaks TASK_DIR/model — unrelated files, unchanged from main).
  • Live adversarial run (Claude agent, real container, ordered to peek): came back empty on all channels — find …/tests… and find … reference → no output; cat /work/input/task.yaml and cat /work/input/context.json → No such file or directory; verdict.txt: DENIED; anti-cheat criterion PASS, task SUCCESS.
  • Deterministic docker probes + argv/unit tests for mask_dirs, _resolve_mount_path, nested-plugin dedup, the skills:"." stand-down, and _scrub_staged_inputs.
  • Reviewed by a 3-model pass (Opus/Fable/Sonnet) and a merge+peek verification review — no reachable Critical/High/Medium.

Known residuals (documented defense-in-depth)

  • The reference window's pre-existing gap (root can chmod back; full containment needs a non-root uid) is unchanged — out of scope.
  • A tmpfs masks a directory, not a single loose file; CE065 covers in-repo plugins but can't reach out-of-repo ones. skills:"." stands the mask down (fail-safe, now logged + CE065-flagged).

Files

src/: isolation/eval_material.py (new), isolation/docker_runner.py, cli/run_task_internal_command.py, agents/_skills.py (manifest_skill_dirs made public/SSOT), orchestrator.py, fs_permissions.py.
tests/: test_eval_material.py (new), test_run_task_internal.py (new), test_docker_runner_mounts.py, test_custom_lint.py (CE065).
Docs: docs/DOCKER_ISOLATION.md, .claude/architecture-notes.md, CLAUDE.md.

🤖 Generated with Claude Code

CarlesUIPath and others added 8 commits September 14, 2026 17:25
@CarlesUIPath @claude
feat(sandbox): 1/3 — delete staged task.yaml in-container after load …
819cba6 
…(anti-cheat)

Fix A: make /work/input a writable mount (drop :ro; grant input dir
writable) and delete the staged task.yaml immediately after load_task in
the in-container entry point, gated on IN_CONTAINER_ENV. The staged file
is the post-override TaskDefinition with success_criteria, and the agent
runs in the same container, so leaving it readable hands over the grading
answer key. It is read exactly once; grading reads criteria from memory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
feat(sandbox): 2/3 — allowlist tmpfs-mask of auto-mounted plugin trees
4875369 
Fix B: default-deny mask over an auto-mounted Claude-plugin root. The
whole root stays :ro-mounted so the plugin loads, but every child dir
outside the keep-set (.claude-plugin + manifest-declared skill dirs) is
masked with an empty tmpfs. Colocated eval material — sibling task YAMLs,
reference solutions, tests/, node_modules/ — is masked by default so an
unknown layout can never leak. New pure helper eval_material.mask_dirs;
shared skill-dir resolver manifest_skill_dirs (renamed public in
agents/_skills.py) is the SSOT for "what is a skill dir".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
feat(lint): 3/3 — CE065 flags eval material inside a skill dir
75367a2 
Fix C: a static lint rule (pytest class, CE055 template) that flags the one
residual the Fix B allowlist cannot close — a task_id: YAML or a resolved
reference.directory colocated INSIDE a skill dir reached through an in-repo
agent.plugins[].path / sandbox.template_sources[].path. Masking a skill dir
would hide the skill, so such material stays readable and leaks the grading
answer key under driver: docker. Reuses the shared manifest_skill_dirs
resolver (SSOT, import-identity asserted) with positive+negative sensors.
Documents both docker anti-cheat fixes in docs/DOCKER_ISOLATION.md and the
CE065 entry in CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
fix: also delete context.json in-container (source_yaml leaked criteria)
dc71add 
Code review found Fix A closed only /work/input/task.yaml but left the identical
success_criteria readable in /work/input/context.json via source_yaml (top-level
AND inside every config_lineage entry / ConfigLineageEntry.source_yaml) — in a
dir this change makes world-readable+writable. The agent could cat context.json
to recover its grading answer key. Delete both staged files after they are read
into memory (_scrub_staged_task_yaml -> _scrub_staged_inputs); correct the
docstring and docs/DOCKER_ISOLATION.md (the 'context.json carries no criteria'
claim was false).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
docs(harness): log deferred Low findings from anti-cheat review
d10778d 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
fix(anti-cheat): apply review Low findings L1/L2/L3
49252c6 
L1 — CE065 now flags ANY task_id: YAML or reference dir left readable under a
plugin root (not just inside a skill dir): it reuses the runtime mask_dirs (SSOT),
walks the readable surface, and catches a loose task_id: YAML *file* at the plugin
root too (a tmpfs masks a dir, not a single file). Renamed the class to
TestCE065EvalMaterialReadableUnderPluginRoot; added a loose-root-file sensor.

L2 — docker_runner resolves a relative agent.plugins[].path against the task-file
dir (new _resolve_mount_path), not CWD, matching reference/template/CE065
resolution so the static rule and the runtime inspect the same tree.

L3 — eval_material.mask_dirs no longer descends a kept path, so a degenerate
manifest skills: '.' masks nothing (fail-safe) instead of hiding the whole skill
surface; added a test.

Updated CLAUDE.md CE065 entry and docs. make verify green (the one failure is a
pre-existing flaky wall-clock datetime test in test_agent_telemetry, passes on
rerun, unrelated to these files).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath @claude
fix(anti-cheat): apply multi-model review findings M1/M2/M3 + doc drift
0de0c48 
Multi-model review (Opus + Fable + Sonnet) found no Critical/High and no leak
bypass; addressed the 3 Mediums and the doc drift:

M2 — nested plugin roots (plugin B under plugin A) produced a duplicate Docker
mount destination: A's mask emitted --tmpfs <A>/B while B's own mount bound
<A>/B:ro. Defer mask emission until all binds are known and drop any mask whose
path is also a bind (the bind wins so B loads and masks its own children).
Extracted the auto-mount block into _append_auto_mounts to stay under the
per-method lint limits. Added a nested-plugin test.

M1 — added direct tests for _resolve_mount_path (relative->task-file dir,
absolute unchanged, task_file None->cwd, env-var expansion); the L2 fix had no
coverage.

M3 — warn when a plugin root's mask stands down (manifest skills: '.') so the
anti-cheat mask never voids silently. Added a test.

Doc drift (L-a/L-b/L-c): corrected the now-false 'task.yaml readable at
/work/input' / 'hiding criteria is unsolved' claims in orchestrator.py,
fs_permissions.py and CLAUDE.md (Fix A deletes task.yaml + context.json after
load); widened DOCKER_ISOLATION.md's residual note to include the loose-root-file
case; marked the harness-candidates entry RESOLVED (nothing deferred).

make verify green (5758 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarlesUIPath
Merge remote-tracking branch 'origin/main' into fix/docker-anti-cheat…
428456e 
…-leaks

# Conflicts:
#	CLAUDE.md
@github-actions

github-actions Bot commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @CarlesUIPath's task in 1m 21s —— View job

Code Review in Progress

Todo List:

  • Read review guidelines from .github/code_review.md
  • Read CLAUDE.md for project conventions
  • Get full diff with git diff origin/main...HEAD
  • Review each changed file with full context
  • Check cross-file consistency
  • Perform design-level scrutiny
  • Identify what's missing
  • Format and post final review

@CarlesUIPath CarlesUIPath changed the title fix(anti-cheat): close two driver:docker channels that leaked grading material to the agent fix(docker): close two driver:docker channels that leaked grading material to the agent Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@akshaylive akshaylive Awaiting requested review from akshaylive akshaylive is a code owner
@tmatup tmatup Awaiting requested review from tmatup tmatup is a code owner
@uipreliga uipreliga Awaiting requested review from uipreliga uipreliga is a code owner
@bai-uipath bai-uipath Awaiting requested review from bai-uipath bai-uipath is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@CarlesUIPath