Repository navigation
Isolate test runs, score fixes independently, correct README claims - #1
Merged
Merged
Conversation
- Pin every third-party action to a full commit SHA (version kept in a comment), so a moved or compromised tag cannot change what runs. - Check out with persist-credentials: false; no step needs the token left in .git/config afterwards. - Default the workflow token to contents: read and grant write scopes only to the jobs that publish.
Only read and edit checked their path. list_dir("..") and absolute paths
listed anything on disk, search_code walked any tree and named outside
files that matched, and run_tests passed its argument straight to pytest,
so a value such as "--basetemp=<dir>" was taken as an option.
All five tools now go through Workspace.resolve(), which resolves
symlinks and rejects paths outside the root before any file access, so
the error does not reveal whether an outside path exists. run_tests only
accepts a file inside the workspace, optionally with a ::node suffix.
The model writes the code that run_tests executes, and that code inherited the full environment, ANTHROPIC_API_KEY included: a snippet added through edit_file could write the key to a file and read_file returned it. A timeout also killed only pytest, so processes the tests started kept running (and held the output pipes open). Tests now run with only PATH, SYSTEMROOT and the temp-directory variables, in a new session (POSIX) or process group (Windows); a timeout kills the whole tree. -B and -p no:cacheprovider keep __pycache__ and .pytest_cache out of the workspace listing.
--no-summary hid every failure detail: a failing run came back as "FFF. 3 failed, 1 passed", while the system prompt tells the model to read the traceback. Use --tb=short; output stays capped at the last 4,000 characters, which keeps the summary line.
The local runner keeps secrets out of the environment, but model-written code still runs with the caller's file-system and network access. With --sandbox docker each test run is a throwaway container: no network, read-only root and workspace mount, all capabilities dropped, no-new-privileges, 512 MB / 1 CPU / 128 processes, no host environment. On timeout the container is killed by name, since killing the docker client does not stop it. --sandbox auto (the default for the CLI and the benchmark) uses Docker when `docker image inspect code-fix-agent:latest` succeeds and falls back to the local runner otherwise; --sandbox docker fails instead of falling back. The runner used is recorded in every result. The image now runs as a non-root user.
"passed" was whatever the model's own run_tests call reported, and the model chooses the test target and can edit the test file. Running test_buggy.py::test_zero on the unfixed off-by-one bug scored as a pass while the full suite had 3 failures. The workspace now snapshots its files at setup. After the loop, Workspace.score() rebuilds the bug in a fresh directory (tests from the snapshot, other original files from the workspace, nothing the run created) and runs the whole suite with the same runner; that is the verdict. AgentResult keeps the model's claim as self_reported_pass and flags tests_modified when a test file no longer matches the snapshot's SHA-256. final_summary is now the scoring run's status line.
The first user message included the bug id, and the ids name the bug class (01_off_by_one, 04_mutable_default), which hands the benchmark's answer category to the model. The temp directory was named after the id too, and that path appears in tracebacks. Drop the id from the prompt and use a plain "cfa-" prefix; the id is still recorded in results.
The tool trail kept only input key names, so a benchmark record could not show which file the model edited or which tests it ran. Store the full tool inputs, and add the unified diff of every file changed since setup (Workspace.diff) to AgentResult, the CLI output and each benchmark record.
main() ended with "0 if pass_rate >= 75 else 0", so the CI bench job could never fail. Mock mode replays known fixes and should be 12/12; exit 1 when it is not. Live mode still exits 0 and only reports.
The SDK default is 10 minutes per request, so one stalled call could hang a benchmark run. The timeout is a constructor argument; a timed-out call ends that attempt with the error recorded.
The README said the agent never sees the test assertions (the system prompt tells it to read the test file), that only relative escapes were possible and tests were "subprocess-isolated" (they ran with the full environment and user rights), and that the model reads tracebacks (they were suppressed). Describe the confinement, the two runners and their limits, and the independent scoring as they now work. The 2026-05-20 live 12/12 stays, labelled as measured on the earlier harness (self-reported pass, bug id in the prompt, no sandbox) and not yet re-run. Drop the mock and live durations, which scoring changes.
score() took pytest's exit code as the verdict, and the code under test
controls it. With only buggy.py edited, pytest.skip() or pytest.xfail()
in the function scored as passed ("4 skipped", "4 xfailed"), and
os._exit(0) at import scored as passed with no output at all.
score() now collects the node ids of the bug as it was at setup (cached
per file contents, since every attempt at a bug collects the same
tests), runs the suite with -rfEsxXp, and passes only when exactly
those ids are reported PASSED and the final counts hold nothing but
passes (warnings aside). Exit code 0 is still required.
The verdict is still read from the output of the pytest process the
code under test runs in, so deliberately forged output is not caught;
the README says so.
The local runner gives the tests a clean environment, but they run as the same user next to the agent process, which holds ANTHROPIC_API_KEY. On Linux a test reads it from /proc/<parent pid>/environ (shown in WSL), and the runner has network access. So the clean environment only stops accidental inheritance; the key is kept from model-written code only under --sandbox docker. Behaviour change: `code-fix-agent run` and `run_benchmark --mode live` no longer let --sandbox auto fall back to the local runner when the sandbox image is missing; they fail before calling the model and ask for the image or an explicit --sandbox local. Mock runs, which replay known fixes, still fall back. The compose fix service, which has no Docker CLI inside, now passes --sandbox local. The README and the sandbox docstring also stop saying the timeout kills everything the tests started: a process that calls setsid(), or on Windows one whose parent has exited, is not in the killed group/tree.
chown without -R changed only benchmarks/results itself. The benchmark.json copied in by COPY stayed root-owned and 0644, so as USER app a benchmark run writing that file failed with PermissionError (owning the directory allows replacing a file, not writing into it). Not built here: Docker was not available.
The Docker runner had never run anywhere: the CI docker job built the image as code-fix-agent:ci, which the runner does not look for, and only imported the package, so TestDockerSandbox was skipped in CI as well. The job now tags the image code-fix-agent:latest, installs the package on the runner, runs the Docker tests and the mock benchmark with --sandbox docker (which fails outright if the image is not found), and runs the mock benchmark inside the image as its non-root user over the copied benchmark.json, which catches the ownership bug fixed in the previous commit. Until that job passes, the README marks the Docker runner as untested instead of describing its guarantees as working fact.
The sample JSON for `run bugs/01_off_by_one` came from no run: the earlier sample had been edited to add "sandbox": "docker" (Docker was never run), self_reported_pass, tests_modified and final_diff, next to the old elapsed time and pytest summary, which scoring now changes. Replace it with a table of the fields and what each means, plus the exit code, so the README states no values that were not measured.
Workspace.diff() walked only the files that existed at setup, so a conftest.py or module planted by code the tests ran never appeared in final_diff, though the audit trail is there to expose exactly that. It now also walks the workspace, subdirectories included, and diffs new files from /dev/null; a deleted file diffs to /dev/null instead of showing as emptied.
`run_benchmark --mode mock`, the first command in the README, wrote by default to benchmarks/results/benchmark.json and so replaced the committed record of the 2026-05-20 live run. Without --out, mock runs now write benchmark-mock.json (git-ignored) and live runs keep writing benchmark.json.
The test ran the suite with a 6-second timeout, and pytest alone takes 3-4 seconds to start on Windows, so a slower runner could time out before the test spawned its child and fail with "the test never started its child process". The run now gets 20 seconds, and the child writes its "late" file at a fixed time 2 seconds after the deadline rather than 6 seconds after it started. Checked that it still fails when the timeout kills only pytest and not its process tree.
"Why this matters for production" listed a "Recruiter signal" and named companies and job titles, as the section removed from claude-agent-kit did. Its one engineering point, how this compares with SWE-bench-Lite, is already made under "What this is NOT", and its claim that a SWE-bench-verified set drops in with no change to the agent loop was never tried, so remove the whole section.
The description still advertised "Hero: pass-rate target 9/12 (75%) at under 3 iterations per bug, under $0.05 total cost", the goal set before the live run. The 2026-05-20 run measured 12/12, 3.58 mean iterations and $0.10, on the harness that predates the current sandbox and scoring, so neither set of numbers belongs in a one-line summary; the README states the measured run with its caveats.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Model-written code ran with the caller's full environment,
ANTHROPIC_API_KEYincluded; three workspace tools were unconfined;passedwas the model's ownrun_testsverdict; and the prompt carried the bug id, which names the bug class. This branch fixes each and corrects the README.Security
PATH,SYSTEMROOTand temp-dir variables, in their own process group, which a timeout kills.--sandbox dockerruns each test in a throwaway container with no network;auto(default) uses it whencode-fix-agent:latestexists.Workspace.resolve();run_teststakes only a workspace file.Correctness and reliability
Workspace.score()rebuilds the bug from the setup snapshot and runs the whole suite; every test collected at setup must reportPASSED, sopytest.skip(),pytest.xfail()andos._exit(0)no longer count. Results also carryself_reported_pass,tests_modified, tool inputs andfinal_diff.--tb=short; API calls time out after 60 s.benchmark-mock.json, not the committed live record.Docs and numbers
CI
persist-credentials: false, token defaulted tocontents: read. The docker job runsTestDockerSandboxand the mock benchmark with--sandbox docker, and inside the image as non-root.Tests
Before:
pytest -q, 22 passed (Windows, Python 3.12). After, at f0b1003:python -m pytest -q -p no:cacheprovider, 64 passed, 2 skipped (Docker test without the image, symlink test without admin). Coverage 88% (--cov=code_fix_agent) at de2c759, two doc-only commits earlier.Not done
Breaking
code-fix-agent runandrun_benchmark --mode liverefuse to start without the sandbox image unless--sandbox localis passed; the local runner cannot keep the API key from model-written code.