Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4173df7
Harden CI against supply-chain and token misuse
Tajaddin Nov 6, 2026
3464931
Confine every workspace tool to the workspace root
Tajaddin Nov 6, 2026
30d0a70
Run tests without the caller's environment, in their own process group
Tajaddin Nov 6, 2026
005c178
Show the model short failure tracebacks
Tajaddin Nov 6, 2026
bb68c55
Run tests in a network-less container when the image is available
Tajaddin Nov 6, 2026
f0afadb
Score each fix against a clean copy of the original tests
Tajaddin Nov 6, 2026
95974b0
Keep the bug id out of the model's view
Tajaddin Nov 6, 2026
4603436
Record tool inputs and the final diff for every run
Tajaddin Nov 6, 2026
ea0fc3d
Fail the mock benchmark when a scripted fix stops passing
Tajaddin Nov 6, 2026
e9f87eb
Give Anthropic API calls a 60-second timeout
Tajaddin Nov 6, 2026
181c581
Correct the README's isolation, traceback and scoring claims
Tajaddin Nov 6, 2026
d03319c
Require every original test to pass when scoring
Tajaddin Nov 6, 2026
4dcdd22
Refuse live runs on the local runner unless asked
Tajaddin Nov 6, 2026
45111dd
Give the image's user the whole results directory
Tajaddin Nov 6, 2026
89fcb8f
Run the Docker sandbox test and mock benchmark in CI
Tajaddin Nov 6, 2026
18f8940
Describe the CLI output by its fields, not an invented sample
Tajaddin Nov 6, 2026
5bc444b
Show files created during a run in the final diff
Tajaddin Nov 6, 2026
26254c8
Write mock benchmark results to their own file
Tajaddin Nov 6, 2026
de2c759
Make the timeout test independent of pytest start-up time
Tajaddin Nov 6, 2026
ea870f2
Drop the recruiter-facing section from the README
Tajaddin Nov 6, 2026
f0b1003
Drop the stale pass-rate target from the package description
Tajaddin Nov 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 31 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ on:
pull_request:
branches: [main]

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
Expand All @@ -14,8 +17,10 @@ jobs:
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: ${{ matrix.python-version }}
cache: pip
Expand All @@ -32,8 +37,10 @@ jobs:
runs-on: ubuntu-latest
needs: test
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.11"
cache: pip
Expand All @@ -48,6 +55,23 @@ jobs:
runs-on: ubuntu-latest
needs: test
steps:
- uses: actions/checkout@v4
- run: docker build -t code-fix-agent:ci .
- run: docker run --rm --entrypoint python code-fix-agent:ci -c "from code_fix_agent import Agent; print('ok')"
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.11"
cache: pip
# The tag the sandbox runner looks for.
- name: Build the sandbox image
run: docker build -t code-fix-agent:latest .
- name: Install
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"
- name: Sandbox tests
run: pytest tests/test_workspace.py -k Docker -rs -v
- name: Mock benchmark in the sandbox
run: python -m benchmarks.run_benchmark --mode mock --sandbox docker --out "$RUNNER_TEMP/bench-docker.json"
- name: Mock benchmark as the image's non-root user, over a copied results file
run: docker run --rm --entrypoint python code-fix-agent:latest -m benchmarks.run_benchmark --mode mock --out benchmarks/results/benchmark.json
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ venv/
*.swp

benchmarks/results/*.tmp.json
benchmarks/results/benchmark-mock.json
benchmarks/workspaces/

.DS_Store
Expand Down
6 changes: 5 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@ COPY bugs ./bugs
COPY benchmarks ./benchmarks
COPY README.md LICENSE ./

RUN pip install --no-cache-dir -e ".[dev]"
RUN pip install --no-cache-dir -e ".[dev]" \
&& useradd --create-home --uid 10001 app \
&& chown -R app benchmarks/results

USER app

HEALTHCHECK --interval=30s --timeout=15s --start-period=5s --retries=2 \
CMD python -c "from code_fix_agent import Agent; print('ok')" || exit 1
Expand Down
101 changes: 55 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# code-fix-agent

> Autonomous Python bug-fix agent on the Anthropic API. **12/12 pass on Claude Haiku 4.5 live, 3.58 mean iterations per bug, $0.10 total cost.** Real pytest-driven fix loop, isolated workspace per attempt, no jailbreaks. Reproduce live in 90 seconds.
> Autonomous Python bug-fix agent on the Anthropic API. **12/12 pass on Claude Haiku 4.5 live, 3.58 mean iterations per bug, $0.10 total cost** (May 2026 run, before the current sandbox and independent scoring; see below). Real pytest-driven fix loop, one temp workspace per attempt, a network-less Docker runner for the tests (not yet run; see [Isolation and scoring](#isolation-and-scoring)), and every fix scored against a clean copy of the original tests.

[![ci](https://github.com/Tajaddin/code-fix-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/Tajaddin/code-fix-agent/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
Expand All @@ -9,28 +9,33 @@
## Hero metrics

Reproducible:
- `python -m benchmarks.run_benchmark --mode mock` (11 seconds, no API key, pipeline smoke)
- `python -m benchmarks.run_benchmark --mode live` (90 seconds, needs ANTHROPIC_API_KEY, real Haiku 4.5)
- `python -m benchmarks.run_benchmark --mode mock` (no API key, pipeline smoke; writes `benchmarks/results/benchmark-mock.json`, not committed)
- `python -m benchmarks.run_benchmark --mode live` (needs ANTHROPIC_API_KEY, real Haiku 4.5; writes `benchmarks/results/benchmark.json`)

| Mode | Pass rate | Mean iterations / bug | Mean tool calls / bug | Total tokens | Realized cost (Haiku 4.5) |
|---|---:|---:|---:|---:|---:|
| **mock (pipeline smoke)** | **12 / 12** | 3.0 | 3.0 | 540 | $0.0013 |
| **live (Haiku 4.5)** | **12 / 12** | **3.58** | **5.58** | 77,457 | **$0.1012** |
| **live (Haiku 4.5), 2026-05-20** | **12 / 12** | **3.58** | **5.58** | 77,457 | **$0.1012** |

Live run logged 2026-05-20: all 12 bugs fixed, no skips, no errors. The agent average 3.58 LLM rounds and 5.58 tool calls per bug. Total spend across the 12-bug sweep was ~10 cents. Per-bug records are in `benchmarks/results/benchmark.json`.

That run predates the current harness. Its pass was the agent's own last `run_tests` call, not the independent scoring described under [Isolation and scoring](#isolation-and-scoring), its prompt included the bug id (which names the bug kind), its tests ran without a sandbox, and its records do not include tool inputs or diffs. Each record's summary shows the bug's full test count passing, so no bug passed on a subset of its tests. It has not yet been re-run on the current harness.

The mock mode scripts the canonical (read -> edit -> run_tests) for each of the 12 bugs. A perfect 12/12 in mock means the workspace, tool dispatch, and agent loop are healthy; CI runs this on every push.

Live mode hands the same 12 bugs to Claude Haiku 4.5 via the Anthropic API, with no scripted fixes. To reproduce:

```bash
export ANTHROPIC_API_KEY=sk-ant-...
docker build -t code-fix-agent:latest . # the test sandbox; see below
python -m benchmarks.run_benchmark --mode live --max-iterations 8
```

Live runs refuse to start without the sandbox image unless you pass `--sandbox local`, because the local runner cannot keep the API key from model-written code (see [Isolation and scoring](#isolation-and-scoring)).

## The dataset

12 hand-crafted Python bugs in `bugs/`. Each subdirectory contains `buggy.py` plus `test_buggy.py`. The fix is a 1-3 line surgical edit. The test file is held back from the agent in its system prompt instructions (the agent is told not to modify the test).
12 hand-crafted Python bugs in `bugs/`. Each subdirectory contains `buggy.py` plus `test_buggy.py`. The fix is a 1-3 line surgical edit. The agent may read the test file but is told not to change it; scoring ignores any change it makes.

| ID | Bug kind | Expected iters |
|---|---|---:|
Expand All @@ -53,49 +58,38 @@ Each bug has 3-4 pytest assertions; the agent needs to satisfy all of them.

| Path | Purpose |
|---|---|
| `src/code_fix_agent/workspace.py` | Isolated tempdir per attempt. Read, edit (str_replace with uniqueness check), search, run_tests via subprocess. |
| `src/code_fix_agent/workspace.py` | Tempdir per attempt. Read, edit (str_replace with uniqueness check), search and run_tests, all confined to the workspace root; `score()` and `diff()` for the harness. |
| `src/code_fix_agent/sandbox.py` | Runs pytest in a network-less Docker container, or in a subprocess with a minimal environment when Docker is unavailable. |
| `src/code_fix_agent/tools.py` | Five Anthropic tool definitions and their dispatcher: list_dir, read_file, edit_file, search_code, run_tests. |
| `src/code_fix_agent/agent.py` | The fix loop. Reads response content, dispatches tool_use blocks, sends tool_result back, stops when run_tests reports PASS or iteration budget is exhausted. |
| `src/code_fix_agent/llm.py` | `AnthropicLLM` + `MockLLM` (scripted tool-call sequence). Both return the same `(content, stop_reason, usage)` shape. |
| `benchmarks/run_benchmark.py` | Iterates the dataset, captures pass / iteration / token / cost stats per bug, writes JSON. |

## Why this matters for production

Recruiter signal this maps to:
- **SWE-agent / coding-agent JDs** (Mango Languages, Sekai, M3 USA, Cohere Agent Infrastructure)
- **Tool use + multi-turn loops** (any "Senior AI Engineer, Agentic Systems" role in 2026)
- **Real-pytest-driven evaluation** (not LLM-as-judge, not vibe-graded)
- **Bounded iteration budgets + token budgets** (the diff between a demo agent and a billable agent)

This is the small-bench cousin of SWE-bench-Lite. It runs in seconds on a laptop and costs cents per full sweep. Drop SWE-bench-verified in next to scale; the agent loop does not change.

## Quick start

```bash
pip install -e ".[dev]"
export ANTHROPIC_API_KEY=sk-ant-...

# Fix one bug, print JSON result
# Fix one bug, print JSON result (needs the sandbox image, or --sandbox local)
code-fix-agent run bugs/01_off_by_one

# Or run the full 12-bug benchmark
python -m benchmarks.run_benchmark --mode live
```

Sample output (`run bugs/01_off_by_one`):

```json
{
"bug_id": "01_off_by_one",
"passed": true,
"iterations": 3,
"tool_calls": 3,
"elapsed_seconds": 4.21,
"total_input_tokens": 2843,
"total_output_tokens": 142,
"final_summary": "PASS (exit 0) summary='4 passed in 0.03s'"
}
```
`run` prints one JSON object and exits 0 when the fix passes scoring, 1 otherwise. Its fields:

| Field | Meaning |
|---|---|
| `bug_id`, `sandbox` | The bug folder's name and the runner the tests used (`docker` or `local`). |
| `passed` | The verdict: the original tests on a clean copy (see [Scoring](#isolation-and-scoring)). |
| `self_reported_pass`, `tests_modified` | The agent's own last `run_tests` said PASS; a test file differs from its setup SHA-256. |
| `iterations`, `tool_calls`, `elapsed_seconds` | Model rounds, tool calls and wall time, scoring included. |
| `total_input_tokens`, `total_output_tokens` | Token usage summed over the rounds. |
| `final_summary` | Status line of the scoring run, e.g. `PASS (exit 0) summary='4 passed in ...'`. |
| `error` | The model call's error, when one ended the loop; otherwise `null`. |
| `final_diff` | Unified diff of the workspace since setup. |

## How the loop works

Expand Down Expand Up @@ -124,40 +118,55 @@ Sample output (`run bugs/01_off_by_one`):
|
yes
v
+-----+-----+
| PASS |
+-----------+
+-----------+-----------+
| score: full original | (also runs when the budget
| suite on a clean copy | runs out)
+-----------------------+
```

The agent does NOT see the test file's assertions in its prompt. It learns the bug shape from `read_file("buggy.py")` and the failure trace from the first `run_tests` call. This mirrors what a junior engineer sees on the first day.
The prompt names neither the bug nor its kind. The agent reads `buggy.py` and `test_buggy.py` with `read_file` (the system prompt tells it to) and gets a short traceback (`--tb=short`, last 4,000 characters) from every failing `run_tests` call.

## Isolation and scoring

**Files.** Each attempt runs in a fresh `tempfile.mkdtemp(prefix="cfa-")`. Every tool resolves its path, symlinks included, and refuses anything outside the workspace root with the same error whether or not the outside path exists. `run_tests` accepts only a test file inside the workspace (optionally `file::test`), never pytest options.

## Workspace isolation
**Test runs.** The model writes the code that `run_tests` executes, so it is treated as untrusted. `--sandbox` on the CLI and the benchmark picks the runner:

Each bug fix runs in a fresh `tempfile.mkdtemp(prefix="cfa-<bug_id>-")`. The agent cannot escape via relative paths (`..`); `edit_file` checks `path.resolve()` falls under the workspace root. Test execution is subprocess-isolated with a 30-second timeout.
| `--sandbox` | Runner |
|---|---|
| `docker` | Throwaway container from `code-fix-agent:latest` (override with `CFA_SANDBOX_IMAGE`): `--network none`, read-only root and workspace mount, all capabilities dropped, `no-new-privileges`, 512 MB / 1 CPU / 128 processes, non-root, no host environment. Fails if the image is missing. |
| `local` | Subprocess with only `PATH`, `SYSTEMROOT` and the temp-dir variables in its environment, in its own session (POSIX) or process group (Windows). The clean environment stops the API key from being inherited, not from being read: the tests run as your user, next to the agent process that holds the key, and can read it from there (on Linux, `/proc/<parent pid>/environ`). They also have your file-system and network access. The 30-second timeout kills the test process's group or tree; a process that calls `setsid()`, or on Windows one whose parent already exited, survives it. A fallback, not a sandbox. |
| `auto` (default) | `docker` when `docker image inspect` finds the image. Otherwise `local` for mock runs, and an error for live runs (`code-fix-agent run`, `--mode live`): pass `--sandbox local` to run those without the image. |

Build the image once with `docker build -t code-fix-agent:latest .`. Every result records the runner used.

This means a single failed agent run cannot corrupt the bug fixtures or any other repo state.
**The Docker runner is untested so far:** it has not run on any machine yet. The CI `docker` job builds the image, runs the container test (`TestDockerSandbox`) and the mock benchmark with `--sandbox docker`, and runs the benchmark inside the image as its non-root user; until that job passes, treat the `docker` row above as a design, not a result.

**Scoring.** The agent's own `run_tests` result is not the verdict: it chooses which tests to run and can edit the test file. After the loop, the harness rebuilds the bug in a fresh directory from the files present at setup (tests from the setup snapshot, other files from the workspace, nothing the run created) and runs the whole suite with the same runner. That is `passed`, and exit code 0 is not enough: every test collected from the bug as it was at setup must be reported `PASSED` and the counts must hold nothing else, so a `pytest.skip()`, a `pytest.xfail()` or an early `os._exit(0)` in the source fails. The verdict is read from pytest's output, and under either runner the code under test runs inside that pytest process: code written to forge the output (fake result lines, patched reporting) is not caught. `self_reported_pass` keeps the agent's claim, and `tests_modified` flags test files that no longer match their SHA-256 from setup. Benchmark records also keep every tool input and the final diff.

## Testing

```bash
pytest --cov=code_fix_agent
```

22 tests:
- `test_workspace.py` — copy, list, read, edit (success + missing-substring error), run_tests (fails on unfixed, passes on canonical fix).
- `test_tools.py` — schema completeness, dispatch, error surfacing.
- `test_agent.py` — full read-edit-test loop with `MockLLM` on bugs 1 and 2, max-iteration safety, LLM exception handling, token accounting.
- `test_bench_smoke.py` — runs the full mock benchmark (12 bugs) and asserts 12/12 pass.
66 tests (the Docker one is skipped when the image is not built, the symlink one where creating symlinks needs admin rights):
- `test_workspace.py` — copy, list, read, edit (success + missing-substring error), run_tests (fails on unfixed, passes on canonical fix), no inherited secrets, timeout kills child processes, sandbox selection (live runs do not fall back to the local runner) and the Docker command line.
- `test_tools.py` — schema completeness, dispatch, error surfacing, tracebacks in failing output, path confinement for every tool (`..`, absolute paths, symlinks).
- `test_agent.py` — full read-edit-test loop with `MockLLM` on bugs 1 and 2, max-iteration safety, LLM exception handling, token accounting, scoring against subset runs, edited tests, a planted `conftest.py` and source that skips, xfails or exits early; no bug id in the prompt, tool inputs and the diff (created and deleted files included) in the result, API timeout.
- `test_bench_smoke.py` — runs the full mock benchmark (12 bugs) and asserts 12/12 pass; the mock run exits non-zero on any failure and does not overwrite the live record.

The bench smoke test is the canary: if it ever drops below 12/12 in CI, either the dataset or the workspace layer regressed.

## Docker

```bash
docker compose run --rm bench # full mock benchmark
docker compose run --rm fix run bugs/01_off_by_one # one bug with live API
docker compose run --rm bench # full mock benchmark
docker compose run --rm fix run --sandbox local bugs/01_off_by_one # one bug with live API
```

The image runs as a non-root user. It is also the test sandbox image (`code-fix-agent:latest`). Inside a compose container there is no Docker CLI, so tests there use the local runner within that container, which holds the API key: the `fix` service passes `--sandbox local` to accept that.

## Project layout

```
Expand All @@ -178,7 +187,7 @@ src/code_fix_agent/
benchmarks/
run_benchmark.py # iterates all 12, captures stats
results/
benchmark.json
benchmark.json # the live run's record
```

## What this is NOT
Expand Down
Loading
Loading