Skip to content

Isolate test runs, score fixes independently, correct README claims - #1

Merged
Tajaddin merged 21 commits into
mainfrom
engineering-2026-11
Nov 7, 2026
Merged

Tajaddin merged 21 commits into
mainfrom
engineering-2026-11

Conversation

@Tajaddin

@Tajaddin Tajaddin commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Model-written code ran with the caller's full environment, ANTHROPIC_API_KEY included; three workspace tools were unconfined; passed was the model's own run_tests verdict; and the prompt carried the bug id, which names the bug class. This branch fixes each and corrects the README.

Security

  • Tests run with only PATH, SYSTEMROOT and temp-dir variables, in their own process group, which a timeout kills.
  • --sandbox docker runs each test in a throwaway container with no network; auto (default) uses it when code-fix-agent:latest exists.
  • All five tools go through Workspace.resolve(); run_tests takes only a workspace file.

Correctness and reliability

  • Workspace.score() rebuilds the bug from the setup snapshot and runs the whole suite; every test collected at setup must report PASSED, so pytest.skip(), pytest.xfail() and os._exit(0) no longer count. Results also carry self_reported_pass, tests_modified, tool inputs and final_diff.
  • Failures show --tb=short; API calls time out after 60 s.
  • The mock benchmark exits 1 unless 12/12 pass and writes benchmark-mock.json, not the committed live record.

Docs and numbers

  • The README describes confinement, both runners and scoring; the 2026-05-20 12/12 stays, labelled as measured on the earlier harness. Durations, an invented CLI sample, the recruiter section and the stale "9/12 target" package description are gone.

CI

  • Actions pinned to commit SHAs, persist-credentials: false, token defaulted to contents: read. The docker job runs TestDockerSandbox and the mock benchmark with --sandbox docker, and inside the image as non-root.

Tests

Before: pytest -q, 22 passed (Windows, Python 3.12). After, at f0b1003: python -m pytest -q -p no:cacheprovider, 64 passed, 2 skipped (Docker test without the image, symlink test without admin). Coverage 88% (--cov=code_fix_agent) at de2c759, two doc-only commits earlier.

Not done

  • Live 12-bug benchmark not re-run on the new harness: needs an API key.
  • Docker runner never executed, Dockerfile not built (no daemon); the README says so.

Breaking

  • code-fix-agent run and run_benchmark --mode live refuse to start without the sandbox image unless --sandbox local is passed; the local runner cannot keep the API key from model-written code.

- Pin every third-party action to a full commit SHA (version kept in a
  comment), so a moved or compromised tag cannot change what runs.
- Check out with persist-credentials: false; no step needs the token
  left in .git/config afterwards.
- Default the workflow token to contents: read and grant write scopes
  only to the jobs that publish.
Only read and edit checked their path. list_dir("..") and absolute paths
listed anything on disk, search_code walked any tree and named outside
files that matched, and run_tests passed its argument straight to pytest,
so a value such as "--basetemp=<dir>" was taken as an option.

All five tools now go through Workspace.resolve(), which resolves
symlinks and rejects paths outside the root before any file access, so
the error does not reveal whether an outside path exists. run_tests only
accepts a file inside the workspace, optionally with a ::node suffix.
The model writes the code that run_tests executes, and that code
inherited the full environment, ANTHROPIC_API_KEY included: a snippet
added through edit_file could write the key to a file and read_file
returned it. A timeout also killed only pytest, so processes the tests
started kept running (and held the output pipes open).

Tests now run with only PATH, SYSTEMROOT and the temp-directory
variables, in a new session (POSIX) or process group (Windows); a
timeout kills the whole tree. -B and -p no:cacheprovider keep
__pycache__ and .pytest_cache out of the workspace listing.
--no-summary hid every failure detail: a failing run came back as
"FFF. 3 failed, 1 passed", while the system prompt tells the model to
read the traceback. Use --tb=short; output stays capped at the last
4,000 characters, which keeps the summary line.
The local runner keeps secrets out of the environment, but model-written
code still runs with the caller's file-system and network access. With
--sandbox docker each test run is a throwaway container: no network,
read-only root and workspace mount, all capabilities dropped,
no-new-privileges, 512 MB / 1 CPU / 128 processes, no host environment.
On timeout the container is killed by name, since killing the docker
client does not stop it.

--sandbox auto (the default for the CLI and the benchmark) uses Docker
when `docker image inspect code-fix-agent:latest` succeeds and falls
back to the local runner otherwise; --sandbox docker fails instead of
falling back. The runner used is recorded in every result. The image
now runs as a non-root user.
"passed" was whatever the model's own run_tests call reported, and the
model chooses the test target and can edit the test file. Running
test_buggy.py::test_zero on the unfixed off-by-one bug scored as a pass
while the full suite had 3 failures.

The workspace now snapshots its files at setup. After the loop,
Workspace.score() rebuilds the bug in a fresh directory (tests from the
snapshot, other original files from the workspace, nothing the run
created) and runs the whole suite with the same runner; that is the
verdict. AgentResult keeps the model's claim as self_reported_pass and
flags tests_modified when a test file no longer matches the snapshot's
SHA-256. final_summary is now the scoring run's status line.
The first user message included the bug id, and the ids name the bug
class (01_off_by_one, 04_mutable_default), which hands the benchmark's
answer category to the model. The temp directory was named after the id
too, and that path appears in tracebacks. Drop the id from the prompt
and use a plain "cfa-" prefix; the id is still recorded in results.
The tool trail kept only input key names, so a benchmark record could
not show which file the model edited or which tests it ran. Store the
full tool inputs, and add the unified diff of every file changed since
setup (Workspace.diff) to AgentResult, the CLI output and each
benchmark record.
main() ended with "0 if pass_rate >= 75 else 0", so the CI bench job
could never fail. Mock mode replays known fixes and should be 12/12;
exit 1 when it is not. Live mode still exits 0 and only reports.
The SDK default is 10 minutes per request, so one stalled call could hang a benchmark run. The timeout is a constructor argument; a timed-out call ends that attempt with the error recorded.
The README said the agent never sees the test assertions (the system
prompt tells it to read the test file), that only relative escapes were
possible and tests were "subprocess-isolated" (they ran with the full
environment and user rights), and that the model reads tracebacks (they
were suppressed). Describe the confinement, the two runners and their
limits, and the independent scoring as they now work.

The 2026-05-20 live 12/12 stays, labelled as measured on the earlier
harness (self-reported pass, bug id in the prompt, no sandbox) and not
yet re-run. Drop the mock and live durations, which scoring changes.
score() took pytest's exit code as the verdict, and the code under test
controls it. With only buggy.py edited, pytest.skip() or pytest.xfail()
in the function scored as passed ("4 skipped", "4 xfailed"), and
os._exit(0) at import scored as passed with no output at all.

score() now collects the node ids of the bug as it was at setup (cached
per file contents, since every attempt at a bug collects the same
tests), runs the suite with -rfEsxXp, and passes only when exactly
those ids are reported PASSED and the final counts hold nothing but
passes (warnings aside). Exit code 0 is still required.

The verdict is still read from the output of the pytest process the
code under test runs in, so deliberately forged output is not caught;
the README says so.
The local runner gives the tests a clean environment, but they run as
the same user next to the agent process, which holds ANTHROPIC_API_KEY.
On Linux a test reads it from /proc/<parent pid>/environ (shown in
WSL), and the runner has network access. So the clean environment only
stops accidental inheritance; the key is kept from model-written code
only under --sandbox docker.

Behaviour change: `code-fix-agent run` and `run_benchmark --mode live`
no longer let --sandbox auto fall back to the local runner when the
sandbox image is missing; they fail before calling the model and ask
for the image or an explicit --sandbox local. Mock runs, which replay
known fixes, still fall back. The compose fix service, which has no
Docker CLI inside, now passes --sandbox local.

The README and the sandbox docstring also stop saying the timeout kills
everything the tests started: a process that calls setsid(), or on
Windows one whose parent has exited, is not in the killed group/tree.
chown without -R changed only benchmarks/results itself. The
benchmark.json copied in by COPY stayed root-owned and 0644, so as
USER app a benchmark run writing that file failed with PermissionError
(owning the directory allows replacing a file, not writing into it).
Not built here: Docker was not available.
The Docker runner had never run anywhere: the CI docker job built the
image as code-fix-agent:ci, which the runner does not look for, and only
imported the package, so TestDockerSandbox was skipped in CI as well.

The job now tags the image code-fix-agent:latest, installs the package
on the runner, runs the Docker tests and the mock benchmark with
--sandbox docker (which fails outright if the image is not found), and
runs the mock benchmark inside the image as its non-root user over the
copied benchmark.json, which catches the ownership bug fixed in the
previous commit.

Until that job passes, the README marks the Docker runner as untested
instead of describing its guarantees as working fact.
The sample JSON for `run bugs/01_off_by_one` came from no run: the
earlier sample had been edited to add "sandbox": "docker" (Docker was
never run), self_reported_pass, tests_modified and final_diff, next to
the old elapsed time and pytest summary, which scoring now changes.
Replace it with a table of the fields and what each means, plus the
exit code, so the README states no values that were not measured.
Workspace.diff() walked only the files that existed at setup, so a
conftest.py or module planted by code the tests ran never appeared in
final_diff, though the audit trail is there to expose exactly that. It
now also walks the workspace, subdirectories included, and diffs new
files from /dev/null; a deleted file diffs to /dev/null instead of
showing as emptied.
`run_benchmark --mode mock`, the first command in the README, wrote by
default to benchmarks/results/benchmark.json and so replaced the
committed record of the 2026-05-20 live run. Without --out, mock runs
now write benchmark-mock.json (git-ignored) and live runs keep writing
benchmark.json.
The test ran the suite with a 6-second timeout, and pytest alone takes
3-4 seconds to start on Windows, so a slower runner could time out
before the test spawned its child and fail with "the test never
started its child process". The run now gets 20 seconds, and the child
writes its "late" file at a fixed time 2 seconds after the deadline
rather than 6 seconds after it started. Checked that it still fails
when the timeout kills only pytest and not its process tree.
"Why this matters for production" listed a "Recruiter signal" and named
companies and job titles, as the section removed from claude-agent-kit
did. Its one engineering point, how this compares with SWE-bench-Lite,
is already made under "What this is NOT", and its claim that a
SWE-bench-verified set drops in with no change to the agent loop was
never tried, so remove the whole section.
The description still advertised "Hero: pass-rate target 9/12 (75%) at
under 3 iterations per bug, under $0.05 total cost", the goal set
before the live run. The 2026-05-20 run measured 12/12, 3.58 mean
iterations and $0.10, on the harness that predates the current
sandbox and scoring, so neither set of numbers belongs in a one-line
summary; the README states the measured run with its caveats.
@Tajaddin
Tajaddin merged commit d88785e into main Nov 7, 2026
5 checks passed
@Tajaddin
Tajaddin deleted the engineering-2026-11 branch November 7, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant