fix(docker): close two driver:docker channels that leaked grading material to the agent

.claude/architecture-notes.md

@@ -59,7 +59,7 @@ lint rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 
 ## Reference solutions and the anti-cheat window
 
-- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half.
+- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory copy IS shielded by the same window (`self.sandbox.task_dir` is in the `set_permissions` list); the post-override YAML is also staged at `/work/input`, but the in-container entry point DELETES `/work/input/task.yaml` **and** `context.json` (whose `source_yaml` carries the criteria top-level and per `config_lineage` entry) after load via `_scrub_staged_inputs`, so the agent cannot read its criteria from there either. **Auto-mounted plugin trees (`agent.plugins[].path`) are default-deny masked** under `driver: docker`: the plugin root is bind-mounted `:ro` so it loads, and every child dir outside the keep-set (`.claude-plugin` + the manifest-declared skill dirs) is `--tmpfs`-masked (`isolation/eval_material.py::mask_dirs`), so eval material colocated as a sibling of the skills dir (task YAMLs, reference solutions, `tests/`) can never be read; CE065 flags the residuals the mask cannot cover (eval material inside a skill dir, or a loose `task_id:` file at the plugin root). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half.
 
 
 ---

.claude/harness-candidates.md

@@ -873,3 +873,12 @@ re-derive from scratch.
   timing work (the function is zero lines of its diff) and not a guardrail
   candidate — a small real bug needing its own change. Caught in: the
   turn-timing consolidation final review (gpt-5.6-sol).
+
+## docker anti-cheat auto-mount allowlist (fix/docker-anti-cheat-leaks) — RESOLVED in-branch
+
+Surfaced by the code review of the Fix A/B/C branch. The Critical (context.json source_yaml
+leak) was fixed in commit dc71add6; the three Low follow-ups (L1 loose task_id file at plugin
+root, L2 CE065-vs-runtime path-resolution divergence, L3 manifest `skills: "."` collapsing the
+keep-set) and the multi-model review's M1/M2/M3 (test coverage for `_resolve_mount_path`, nested-
+plugin duplicate-mount crash, silent mask stand-down) were all fixed in the same branch. Nothing
+deferred. Left here only as a pointer to the branch history.

docs/DOCKER_ISOLATION.md

@@ -337,6 +337,45 @@ If the task declares a `reference:` block, a throwaway **copy** of its directory
 
 See [Reference Solutions](TASK_DEFINITION_GUIDE.md#reference-solutions).
 
+### Two more passive-read anti-cheat blocks
+
+These are `driver: docker` only. `driver: tempdir` shares the host uid and has no
+filesystem isolation, so neither applies there (nor can — there is nothing to
+mask). Both are defense-in-depth passive-read blocks, consistent with the
+reference window's posture above; neither contains an adversarial agent.
+
+- **The staged grading inputs are deleted after load.** The host stages the
+  post-override `TaskDefinition` (with `success_criteria`) at `/work/input/task.yaml`
+  **and** a `context.json` whose `source_yaml` is the raw task text — criteria
+  verbatim, both at the top level and inside every `config_lineage` entry — for the
+  in-container orchestrator to load once at startup. The agent runs in the same
+  container, so leaving *either* readable would hand it the grading answer key
+  (deleting only `task.yaml` leaves the identical criteria one file over in
+  `context.json`). The in-container entry point deletes **both** immediately after
+  they are consumed — `context.json` is parsed into memory in the command body and
+  `task.yaml` by `load_task`, both before the delete (gated on
+  `CODER_EVAL_IN_CONTAINER`). They are read exactly once — grading reads criteria
+  from the in-memory task, never from disk. The `/work/input` mount is therefore
+  read-write (a `:ro` mount rejects `rm` with EROFS). `prior.json` is kept: it is
+  read later on the regrade path, and a regrade runs no agent so it is not a leak.
+
+- **Auto-mounted plugin trees are default-deny masked.** An `agent.plugins[].path`
+  (or a `TemplateDirSource.path` that is itself a plugin root) is auto-mounted at
+  its host path `:ro` so the plugin loads. Eval material colocated under that tree
+  as siblings of the skills dir — sibling task YAMLs, reference solutions, test
+  fixtures — would otherwise be readable. So the runner keeps the whole root
+  mounted but layers an empty `--tmpfs` over every child dir OUTSIDE the keep-set
+  (`.claude-plugin` + the manifest-declared skill dirs). Everything that is not
+  the plugin surface is masked by default, so an unknown or new eval layout can
+  never leak; `tests/`, `node_modules/`, and reference solutions are masked for
+  free. A root agent cannot `umount` a tmpfs (`CAP_SYS_ADMIN` is not in Docker's
+  default set), so this mask is *stronger* than the mode-000 reference window. Two
+  residuals the mask cannot cover — an eval def or reference COLOCATED inside a
+  skill dir (masking it would hide the skill), and a `task_id:` YAML **file** loose
+  at the plugin root (a tmpfs masks a directory, not a single file) — are caught by
+  lint rule CE065 (keep eval material out of skill dirs and off the plugin root;
+  put it under a sibling `tests/`).
+
 Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
 
 A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.

src/coder_eval/agents/_skills.py

@@ -29,7 +29,7 @@
 _SKILL_FILE = "SKILL.md"
 
 
-def _manifest_skill_dirs(root: Path) -> list[Path]:
+def manifest_skill_dirs(root: Path) -> list[Path]:
     """Skill directories a Claude-plugin root declares, in manifest order.
 
     Reads the ``skills`` field of ``<root>/.claude-plugin/plugin.json`` (a string
@@ -89,7 +89,7 @@ def _plugin_skill_dirs(
                 hint,
             )
             continue
-        candidates = [directory for directory in _manifest_skill_dirs(root) if directory.is_dir()]
+        candidates = [directory for directory in manifest_skill_dirs(root) if directory.is_dir()]
         # A path that is ALREADY a bare skills directory (<root>/<name>/SKILL.md)
         # has no `skills/` subdir, so use it as-is. Deliberately not a fallback for
         # a root that HAS one: `skills.paths` is scanned recursively and a repo

src/coder_eval/cli/run_task_internal_command.py

@@ -18,6 +18,7 @@
 import contextlib
 import json
 import logging
+import os
 from pathlib import Path
 
 import typer
@@ -46,6 +47,37 @@
 logger = logging.getLogger(__name__)
 
 
+def _scrub_staged_inputs(task_yaml: Path, context_json: Path) -> None:
+    """Delete the staged grading inputs after they are loaded (anti-cheat).
+
+    ``/work/input`` holds BOTH copies of the grading answer key:
+
+    * ``task.yaml`` -- the post-override :class:`TaskDefinition`, ``success_criteria``
+      included.
+    * ``context.json`` -- whose ``source_yaml`` is the RAW task YAML *text*
+      (``success_criteria`` verbatim), present at the top level AND inside every
+      ``config_lineage`` entry (``ConfigLineageEntry.source_yaml``). Deleting only
+      ``task.yaml`` would leave the identical criteria one file over -- and this
+      driver makes ``/work/input`` readable+writable, so the agent in this same
+      container could ``cat /work/input/context.json`` to recover them.
+
+    Both are read exactly once at startup (``context.json`` into memory in the
+    command body before this call; ``task.yaml`` by ``load_task``) and never again
+    during the agent turn or grading -- grading reads criteria from the in-memory
+    task, never from disk -- so delete both now.
+
+    Gated on ``IN_CONTAINER_ENV``: docker-only by construction (a host/tempdir
+    invocation shares our uid and has no filesystem isolation, so there is nothing
+    to protect and nothing to delete). ``missing_ok`` so a re-entrant or host call
+    never crashes on an absent file. ``prior.json`` is deliberately left in place:
+    it is read after this point on the regrade path, and a regrade runs no agent,
+    so it is not a leak.
+    """
+    if os.environ.get(IN_CONTAINER_ENV) == "1":
+        task_yaml.unlink(missing_ok=True)
+        context_json.unlink(missing_ok=True)
+
+
 def heartbeat_is_alive(current: str, last_counter: str, current_mtime: float, last_mtime: float) -> bool:
     """True when the heartbeat shows a fresh signal of life.
 
@@ -248,6 +280,14 @@ def run_task_internal_command(
     # `TASK_DIR` env exposed to `run_command` criteria -- resolves to the
     # original host task directory rather than `/work/input/`.
     task, source_yaml = load_task(task_yaml)
+    # ANTI-CHEAT: both task.yaml and context.json have been fully consumed by this
+    # point -- context.json was parsed into memory above (source_yaml/config_lineage
+    # already extracted into `context`), and load_task is task.yaml's sole reader.
+    # Both the normal and the regrade paths reach here before their dispatch, and
+    # prior.json is read INSIDE _grade_recorded_run (strictly after this). Delete
+    # both staged files now so the agent in this same container cannot read its own
+    # grading criteria back -- from task.yaml OR from context.json's source_yaml.
+    _scrub_staged_inputs(task_yaml, context_json)
     if host_source_yaml is not None:
         source_yaml = host_source_yaml
     # The path below is never re-read; it only seeds Orchestrator's TASK_DIR.

src/coder_eval/fs_permissions.py

@@ -19,7 +19,10 @@
 
 This shields grading MATERIAL that happens to live in the task directory (a
 ``reference/`` subdirectory, fixtures), not the task DEFINITION: ``task.yaml``
-is separately staged at ``/work/input``, which the agent can still read.
+(and ``context.json``'s ``source_yaml``) are separately staged at ``/work/input``
+and DELETED after load by the in-container entry point
+(``run_task_internal_command._scrub_staged_inputs``), so the agent cannot read
+the criteria from there.
 
 Windows **stack**, which is what makes a mid-turn re-grant expressible: code
 that runs inside the turn but is not the agent can open a narrower window to