Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
8bfe187
Coding factory runs bounded tasks with offline replay
zlareb1 Nov 19, 2026
24b6e78
Qualify historical factory tasks and retain failed attempt accounting
zlareb1 Nov 19, 2026
e2b6df9
Persist cohort reservations before provider dispatch
zlareb1 Nov 19, 2026
cf32ac4
Retain request IDs in factory transport events
zlareb1 Nov 19, 2026
de4bbc3
Analyze factory pilot attempts by task and repeat
zlareb1 Nov 20, 2026
74ea1d5
Refuse candidate-controlled grades and supervise factory attempts
zlareb1 Nov 20, 2026
dbfdeb0
Grade historical API behavior outside candidate execution
zlareb1 Nov 20, 2026
6715218
Describe the implemented host oracle in the pilot protocol
zlareb1 Nov 20, 2026
26269f4
Compose the coding factory with current main
zlareb1 Nov 20, 2026
54056f0
Compose the factory with current test diagnostics
zlareb1 Nov 20, 2026
5f0517e
Halt cohorts after visible grader infrastructure failures
zlareb1 Nov 20, 2026
adae29c
Keep visible grading budgets scoped to one attempt
zlareb1 Nov 20, 2026
35a37bb
Compose the factory with current source contracts
zlareb1 Nov 20, 2026
5ecca85
Remove unreachable hidden grading from historical diagnostics
zlareb1 Nov 20, 2026
a45c648
Describe historical test runs as visible diagnostics
zlareb1 Nov 21, 2026
76a669a
Merge commit '9af0d68f50b0a4b4ea927f03076fe07d737d55c8' into feat/fla…
zlareb1 Nov 21, 2026
b08ef6f
Treat observation transport deadlines as invalid grading
zlareb1 Nov 21, 2026
5b1bac0
Merge commit '89d7b5ea085416482dc7c5728fa9cb47a3bee540' into feat/fla…
zlareb1 Nov 21, 2026
f63e92d
Repository evaluation runs in isolated Harbor trials
zlareb1 Nov 21, 2026
00037ee
Merge remote-tracking branch 'origin/main' into feat/software-factory…
zlareb1 Nov 21, 2026
543fa0a
Merge remote-tracking branch 'origin/main' into feat/software-factory…
zlareb1 Nov 21, 2026
983bd81
Harbor reports retain final export and audit status
zlareb1 Nov 21, 2026
367a5f7
Public evaluations verify source and billing identities
zlareb1 Nov 21, 2026
d2bd6f6
Automate repository evaluations with code-linked project history
zlareb1 Nov 21, 2026
4d8225c
Merge remote-tracking branch 'origin/main' into feat/software-factory…
zlareb1 Nov 21, 2026
a8f7361
Use Haiku 5.5 by default for Anthropic inference
zlareb1 Nov 21, 2026
b52ac10
Meter Haiku 5.5 for native Loom evaluation requests
zlareb1 Nov 21, 2026
930cfac
Run evaluations without a dollar cap by default
zlareb1 Nov 21, 2026
8a6a45b
Preserve native prompt caching and pin the evaluation toolchain image
zlareb1 Nov 21, 2026
9408dbb
Keep Docker evaluation downloads on the host transport
zlareb1 Nov 21, 2026
318501a
Resolve native build dependencies through verified local image tags
zlareb1 Nov 21, 2026
5e06c0d
Validate the owned inference gateway before preparing scenarios
zlareb1 Nov 21, 2026
8d1248c
Wait for ClickHouse queries before starting the evaluation server
zlareb1 Nov 21, 2026
7408242
Fetch pinned source commits into isolated index mirrors
zlareb1 Nov 21, 2026
ecfa045
Load repository configuration through the supported environment source
zlareb1 Nov 21, 2026
7865d1d
Separate namespace installer credentials from scoped attempt tokens
zlareb1 Nov 21, 2026
26567cc
Serialize index setup within the evaluation memory limit
zlareb1 Nov 21, 2026
026b211
Read index audit rows through the typed JSON representation
zlareb1 Nov 21, 2026
7406542
Audit conditional declarations using native identity selection
zlareb1 Nov 21, 2026
6b98124
Separate historical issue comparisons from authored memory experiments
zlareb1 Nov 21, 2026
1f180f7
Resolve Python package names in memory code links
zlareb1 Nov 21, 2026
9e1ce10
Compare historical memory on the same Flask issues
zlareb1 Nov 21, 2026
a893d1b
Record setup failure locations without exception payloads
zlareb1 Nov 21, 2026
b753eb2
Keep Python SCIP symbols and calls within their declared scope
zlareb1 Nov 21, 2026
f772c30
Reconcile native edges after compiler ingestion
zlareb1 Nov 21, 2026
8ae5bb2
Qualify tool requests and baseline bindings before paid trials
zlareb1 Nov 21, 2026
4c77be5
Bring the frozen factory implementation into the reporting checkout
zlareb1 Nov 21, 2026
8311855
Preserve complete factory reporting evidence
zlareb1 Nov 21, 2026
8e8b312
Compare journal tool results by their JSON values
zlareb1 Nov 21, 2026
ed4af47
Bring the reviewed factory implementation into the native retrieval c…
zlareb1 Nov 21, 2026
9922063
Use native code retrieval defaults in the factory
zlareb1 Nov 21, 2026
13d4904
Use the common request allowance for native Loom calls
zlareb1 Nov 21, 2026
099c3fb
Inherit the common allowance for memory and index checks
zlareb1 Nov 21, 2026
92fbc0d
Bring the native factory implementation into the historical study che…
zlareb1 Nov 21, 2026
0c13c4a
Prepare historical issues and matched cold memory controls
zlareb1 Nov 21, 2026
7087092
Qualify memory lifecycle against each native anchor fingerprint
zlareb1 Nov 21, 2026
f5f1170
Use the memory declaration oracle for lifecycle qualification
zlareb1 Nov 22, 2026
224834e
Bring the reviewed factory implementation into the registry initializ…
zlareb1 Nov 22, 2026
6983f47
Load the repository registry before capturing native anchors
zlareb1 Nov 22, 2026
406546f
Include the declaration fingerprint documentation
zlareb1 Nov 22, 2026
79a1c50
Keep the factory example within its lint rules
zlareb1 Nov 22, 2026
2792b69
Keep the reason for persisting setup failures
zlareb1 Nov 22, 2026
631fac9
Bring the reviewed factory into the audit diagnostics checkout
zlareb1 Nov 22, 2026
f9bff87
Include the grep lesson namespace census declaration
zlareb1 Nov 22, 2026
dbd4a60
Retain native index qualification diagnostics
zlareb1 Nov 22, 2026
7e9f5b2
Bring the reviewed factory into the signature owner checkout
zlareb1 Nov 22, 2026
f215cf1
Use the merged owning symbol context fix in the factory
zlareb1 Nov 22, 2026
e5225a4
Preserve the reviewed factory for compiler parity qualification
zlareb1 Nov 22, 2026
6209c2f
Verify compiler contents against pinned snapshots and preserve row or…
zlareb1 Nov 22, 2026
3479c03
Use the merged SCIP metadata reconciliation fix
zlareb1 Nov 22, 2026
dca662b
Merge commit '3479c03a6494f59e9056f3666830a0caf3859b4f' into fix/fact…
zlareb1 Nov 22, 2026
f11795a
Factory agents can read public base-revision repository evidence
zlareb1 Nov 22, 2026
542bc8a
Agent runtime fixtures declare their read-only evidence scope
zlareb1 Nov 22, 2026
e5b852b
Merge remote-tracking branch 'origin/main' into fix/factory-public-ch…
zlareb1 Nov 22, 2026
5c53ac7
Publish the complete implementation-scope factory cohort
zlareb1 Nov 22, 2026
6951acc
Factory attempts include a common final contract review
zlareb1 Nov 22, 2026
9f9db33
Merge commit '6951accfcbfe9f238960b411591101cd7a992676' into fix/fact…
zlareb1 Nov 22, 2026
7bd9be5
Merge remote-tracking branch 'origin/main' into fix/factory-retrieval…
zlareb1 Nov 22, 2026
d67bf18
Factory code reads retain enclosing lessons in their bound scope
zlareb1 Nov 22, 2026
501c691
Every factory arm can search with regular expressions
zlareb1 Nov 22, 2026
009d5e0
Published result JSON uses repository formatting
zlareb1 Nov 22, 2026
8ec7e97
Memory tools preserve native retrieval risk and provenance
zlareb1 Nov 22, 2026
e96aeea
Link historical cleanup knowledge to its teardown definition
zlareb1 Nov 22, 2026
bafd6f8
Factory retains elided definition cards and exact symbol lookups
zlareb1 Nov 22, 2026
3bc2321
Symbol lookup guidance distinguishes definition and chunk IDs
zlareb1 Nov 22, 2026
715e4e8
Publish complete public-evidence cohort with verified zero-dispatch c…
zlareb1 Nov 22, 2026
9641c12
Factory image builds separate host authentication and container proxy…
zlareb1 Nov 22, 2026
90c6c18
Factory baseline instructions and review contracts match the shared w…
zlareb1 Nov 22, 2026
360f371
Merge commit '90c6c1804b9b80ae2617f1a794bbf53f241127c9' into fix/fact…
zlareb1 Nov 22, 2026
8dadd73
Merge remote-tracking branch 'origin/main' into fix/factory-regex-errors
zlareb1 Nov 22, 2026
da7f794
Factory regex errors remain recoverable across ordinary and indexed s…
zlareb1 Nov 22, 2026
0e7a411
Retain native navigation and package the grader dependencies
zlareb1 Nov 22, 2026
3e4b0a9
Merge unlimited retrieval defaults into the factory evaluation
zlareb1 Nov 22, 2026
9071813
Merge commit '3e4b0a9aec28a7ce5acc0130ff59e79769fbd384' into fix/fact…
zlareb1 Nov 22, 2026
aaed0f1
Merge remote-tracking branch 'origin/main' into fix/factory-search-so…
zlareb1 Nov 22, 2026
bf424da
Separate code-only arms and preserve native retrieval context
zlareb1 Nov 22, 2026
4861d7a
Keep evaluation status pending until billing and judging finish
zlareb1 Nov 22, 2026
6d4534f
Label incomplete reports partial without a verified run status
zlareb1 Nov 22, 2026
459f3e0
Align factory test fixtures with memory-disabled code arms
zlareb1 Nov 22, 2026
a1c9adc
Restrict report navigation to HTTP and HTTPS links
zlareb1 Nov 22, 2026
ee0e2fb
Compare identical completed issue and repeat blocks in live reports
zlareb1 Nov 22, 2026
188c67e
Render unattempted comparison filters without a reference error
zlareb1 Nov 22, 2026
fe2141d
Preserve grep scope ranges and allow precise source reads
zlareb1 Nov 22, 2026
ed99414
Merge remote-tracking branch 'origin/main' into fix/factory-grep-navi…
zlareb1 Nov 22, 2026
dedc6ee
Return input errors for unadvertised factory tools
zlareb1 Nov 22, 2026
2f6cc39
Return native input validation to the factory model
zlareb1 Nov 22, 2026
260cc90
Recover selected memory input errors after scope checks
zlareb1 Nov 22, 2026
c801cbf
Run coding comparisons with native agent environments
zlareb1 Nov 22, 2026
479338e
Merge remote-tracking branch 'origin/main' into fix/factory-grep-navi…
zlareb1 Nov 22, 2026
e3977ff
Keep native execution network and accounting explicit
zlareb1 Nov 22, 2026
cc2fdfc
Account for native connection initialization separately
zlareb1 Nov 22, 2026
003cd6f
Use native coding tool registration and agent environment
zlareb1 Nov 22, 2026
85e62fa
Verify native connections and isolate scenario dependencies
zlareb1 Nov 22, 2026
c25eb4d
Remove unused custom coding agent machinery
zlareb1 Nov 22, 2026
01eb34d
Keep structured judging independent of coding tools
zlareb1 Nov 22, 2026
d7ef070
Use plain task images and batch repository source exports
zlareb1 Nov 22, 2026
07c7c58
Check the native agent sandbox policy in its contract
zlareb1 Nov 22, 2026
5232ddb
Keep Oracle control timeouts finite
zlareb1 Nov 22, 2026
aa4539a
Remove evaluation execution deadlines
zlareb1 Nov 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:
uv sync --locked --python "${{ steps.python.outputs.python-path }}" --extra dev --extra code
echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
- name: ruff
run: ruff check server extractor sdk/python tests scripts .claude
run: ruff check server extractor sdk/python tests scripts .claude examples/software_factory
- name: ruff format
run: ruff format --check .
# The Python SDK's op table, TypedDicts and per-op client are generated
Expand Down Expand Up @@ -294,6 +294,11 @@ jobs:
# The published benchmark harnesses' scoring and dataset readers.
- name: pytest bench harnesses
run: pytest bench -q -p randomly --randomly-seed="${{ github.run_id }}${{ github.run_attempt }}"
- name: Offline software factory contracts
run: |
uv venv --python "${{ steps.python.outputs.python-path }}" .factory-venv
uv pip install --python .factory-venv/bin/python --require-hashes -r examples/software_factory/harbor/requirements.txt
PYTHONPATH="$PWD/sdk/python:$PWD" .factory-venv/bin/python -m unittest discover -s examples/software_factory/tests -t . -v
# The generic ingest script's dry run goes through this checkout's CLI.
- name: generic integration sample
run: PYTHON=python bash integrations/generic/test.sh
Expand Down
95 changes: 95 additions & 0 deletions docs/architecture/software-factory-evaluation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Software factory evaluation

The evaluator asks a coding agent to fix a public issue in its ordinary checkout.
Harbor runs its built-in Claude Code agent, collects its native trajectory and
passes the resulting implementation to a separate hidden verifier. The evaluator
provides no custom coding loop, source tools, hooks, forced retrieval or completion
review. Claude chooses its reads, edits, commands, retrieval and stopping point.

## Shared execution and project profiles

`examples/software_factory/harbor/` owns environment setup, frozen schedules,
measurement and reports. `evals/<id>/` supplies public issues, repository revisions,
source roots, dependency locks, historical facts, index recipes and grading. The
implemented profile is a Python E2E evaluation of Flask. Language and source-root
lists allow a future profile to describe a repository with several languages.

The public command builds pinned dependencies and isolated services, prepares the
indexes and historical memory, qualifies the grading contract and starts Harbor.
It records exact owned resources for cleanup. Native coding runs are uncapped;
capped launches are refused. Provider credentials and raw sessions stay outside
public result bundles.

## Treatments

Five arms receive the same original issue, model, pinned agent version, base
checkout image, resources and hidden grading contract:

- No Loom connection.
- Loom with a tree-sitter code index.
- Loom with a tree-sitter code index and historical memory.
- Loom with a SCIP code index.
- Loom with a SCIP code index and historical memory.

Loom connections use the native coding MCP roster. A connection header selects
its code scope. Native namespace permissions and caller-home resolution select
its optional memory scope. Code-only arms have no historical memory namespace or
memory write permission; they see the same coding roster as memory arms. The
model decides whether to call any advertised tool. No result-count or response
budget override is supplied by the evaluation.

Every attempt has separate credentials. Each memory attempt starts with its own
writable history, preventing earlier attempts from supplying solutions to later
ones. The historical facts have public sources and cutoffs before each issue's
base. Some facts link to native code symbols. Setup replays an older source head,
advances to the buggy base, checks changed anchors and revises one independently
sourced fact. This verifies a replayed memory lifecycle, rather than measuring
months of autonomous learning. Historical knowledge is extra information, so the
contrast does not isolate linking from access to that knowledge.

## Grading and accounting

Base and reference controls use Harbor's built-in oracle without model calls.
Unchanged source must fail the hidden checks; the published fix must pass. These
controls qualify the same frozen grading contract before coding begins.

The coding environment has the public base checkout and its tests. Hidden tests
and reference fixes remain outside it. Harbor collects implementation files as a
native artifact and restores them into the separate verifier. The verifier checks
source paths and sizes, reconstructs its own workspace and runs the fixed hidden
tests. Agent-authored tests do not replace the hidden tests. Full-suite regressions
are measured against the unchanged base. A posthoc blind judge sees the original
issue, public historical facts and candidate source differences; it cannot send
feedback to the coding agent.

Harbor's trajectory supplies total tool calls, Loom calls, input/output/cache
tokens, generation cost and agent time, including recorded subagent activity.
Tokens already include cache usage in native prompt totals; reporting does not
add cache tokens twice. Explicit test commands are observable in shell calls;
tests hidden inside scripts are not classified as test runs.

Authenticated attempt subjects attribute Loom provider events and persisted
memory versions. Initialization and refusal events retain their native scope
semantics and cost. Initialization events do not stand in for selected tool
calls. Missing usage remains unknown. Source index fingerprints before and after
each attempt detect mutation. Setup and judging costs remain separate from agent
cost; infrastructure and invoice reconciliation are outside that measure.

## Comparison and reproducibility

The profile contains three issues, three repeats and five arms: 45 primary
attempts. Six baseline A/A attempts measure variability separately. The schedule
balances arm positions and runs serially without Harbor retries. Every failure
remains included. Repeats do not create independent issues, and three issues do
not establish a general result across repositories.

Native results stay immutable. Derived Harbor views align matching issue/repeat
rows across arms and record the original identifiers and source hashes. Reports
contain per-arm, per-scenario and per-attempt measurements, with matched partial
comparisons and explicit pending values. Public export derives allowlisted
measurements from the original artifacts and publishes source fingerprints.
No cost, time or quality benefit is established before qualified measurements.

See the [example](../../examples/software_factory/README.md),
[reproduction instructions](../../examples/software_factory/evals/flask/REPRODUCE.md)
and [report layout](../../examples/software_factory/harbor/README.md).
67 changes: 67 additions & 0 deletions examples/software_factory/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Software factory evaluations

Run the Python E2E example with a local Docker daemon:

```bash
python3 -m examples.software_factory plan
ANTHROPIC_API_KEY=... python3 -m examples.software_factory run --output ./factory-run
```

Harbor runs its built-in Claude Code agent in a normal checkout. Claude chooses
its tools, edits, test commands and stopping point. The evaluator supplies the
original issue and environment, then grades the submitted implementation in a
separate container. It adds no coding loop, custom source tools, hooks, forced
retrieval or completion review.

The comparison has five arms:

- Claude Code without Loom.
- Claude Code with Loom's tree-sitter code index.
- Claude Code with Loom's tree-sitter code index and historical memory.
- Claude Code with Loom's SCIP code index.
- Claude Code with Loom's SCIP code index and historical memory.

All arms use the same model, pinned Claude Code version, issue, base checkout,
resources and hidden verifier. Loom-enabled arms use the native MCP server and
scoped credentials. Code-only arms have no memory namespace. Each memory attempt
has its own writable namespace seeded with the same 30 historical facts, 14
linked to code. Facts come from public sources before the issue's base revision;
none are inserted into the issue prompt. The memory contrast measures both access
to historical knowledge and native memory capability, rather than linking alone.

There are three public issues, three repeats and five arms: 45 primary attempts.
Six A/A attempts measure baseline variation separately. Six base/reference controls
qualify the grader without model calls. Twelve blind judgments assess submitted
fixes after coding; judgments do not provide feedback to the solver.

Harbor's native trajectory supplies total tool calls, Loom tool calls, input,
output and cache tokens, reported generation cost and agent time, including
recorded subagent activity. Loom's server ledger supplies attributed provider
usage and persisted memory writes. Missing measurements stay unknown. Provider
cost combines solver and Loom usage; setup and posthoc judging stay separate.
Infrastructure and invoice reconciliation are outside that cost measure.

The launcher automates dependencies, isolated services, indexes, historical
memory and native Harbor jobs. Coding, control solutions, indexing, builds and
grading have no execution deadline. Native coding runs have no dollar cap; capped
launches are refused. Harbor does not retry failed trials. Explicit resume preserves frozen
inputs and refuses interrupted paid requests that need reconciliation:

```bash
ANTHROPIC_API_KEY=... python3 -m examples.software_factory resume --output ./factory-run
python3 -m examples.software_factory cleanup --output ./factory-run
```

Cleanup removes recorded owned resources and retains results. It does not prune
unrelated Docker data. Reports and charts are generated before coding, after each
finished attempt and at completion. See [Harbor reports](harbor/README.md).

[Flask](evals/flask/README.md) is the implemented Python profile; its
[reproduction instructions](evals/flask/REPRODUCE.md) describe the public issues,
source pins and historical memory. `--eval flask` selects it explicitly.

Shared execution, accounting and reports live in `harbor/`. Project-specific source,
dependencies, index recipes, historical facts and grading live in `evals/<id>/`.
A suite declares `languages` and `source_roots`, allowing a future profile to cover
several languages in one checkout. `factory/` supplies data types and evaluator
utilities; it does not execute the benchmark's coding agent.
1 change: 1 addition & 0 deletions examples/software_factory/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""A bounded coding loop with application-owned context policy."""
6 changes: 6 additions & 0 deletions examples/software_factory/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Host entry point; the launcher imports only the Python standard library."""

from .harbor.launch import main

if __name__ == "__main__":
raise SystemExit(main())
Empty file.
49 changes: 49 additions & 0 deletions examples/software_factory/evals/flask/PILOT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Factory comparison protocol

Compare five arms on the same historical Flask issue and base revision:
`baseline`, `loom-treesitter`, `loom-treesitter-memory`, `loom-scip`, and
`loom-scip-memory`. Claude chooses every Loom call. Baseline has no Loom client or
schemas. Every arm uses the same native coding agent, issue, model and test capabilities.

Three issues and three repeats produce 45 primary attempts. Add one baseline A/A
pair per issue for 51 attempts. Freeze seed 741 and retain the interleaved arm
blocks and balanced positions. Retain nonuse, failures, invalid rewards and
unknown costs. Analyze paired task clusters and report A/A controls separately.

Task selection preceded solver outcomes on these three harder issues. Earlier
development outcomes informed the harness. The report discloses both facts.
Historical notes must have independent public provenance and precede the task's
cutoff. Target discussions, fixing PRs, reference patches, hidden requirements and
later information cannot supply memory. Freeze the corpus before inference and
never revise it based on that run's results.

The memory treatment adds 30 short historical notes, 14 linked to code. Baseline and code-only arms receive no historical memory. Every Loom arm sees
the same native coding roster; only memory arms have a historical memory namespace
and write permission. Their code tools may return linked lessons.
The treatment therefore measures historical knowledge plus retrieval. A claim
about linking alone requires another control with identical facts.

Each memory-enabled attempt has a private mutable memory namespace. Code-only and memory arms
share their certified read-only source index for that case and tier. Parser and
compiler tiers index the same source allowlist. Native memory writes, exact reads,
current code links, source hashes and serving build must qualify before inference.
The public source advances from an earlier real revision to the task base.
Native workers must flag changed anchors before a source-backed memory revision
relinks fresh. Code-only arms receive no memory namespace. This replays past knowledge
and code evolution, rather than elapsed autonomous learning.

Freeze source, tasks, evaluator inputs, model, prices, images, schema exposure,
prompts, provider limits and all namespace identities. Require base controls to
fail and reference controls to pass the hidden verifier. Harbor retries are disabled. Claude Code retains its native provider behavior
and ordinary checkout tools.

Report hidden success, fixed-only blind judgments, full-suite regressions, total
agent and Loom provider cost, agent time, full end-to-end time, tokens, agent tests,
Loom requests and persisted memory writes. Judge and setup costs are separate.
Setup writes are not agent writes. Missing billing evidence is unknown. All chart
bars retain the scheduled denominator and distinguish pending cells.

The targets are all nine primary attempts solved per arm and at least 20% lower
provider cost and agent time than paired baseline. Three task clusters cannot
establish a general win. Historical results and the current frozen run remain
separate. Follow [REPRODUCE.md](REPRODUCE.md) for the executable route.
74 changes: 74 additions & 0 deletions examples/software_factory/evals/flask/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Python E2E evaluation: Flask

The factory solves the same three public historical Flask issues across five
arms: no Loom, Tree-sitter, tree-sitter with historical memory, SCIP, and SCIP with historical memory.
Claude chooses every Loom call. Source revisions, issue text, model, ordinary
source tools, resource limits and independent tests match across the arms.

These harder issues were selected before measuring their solver outcomes. Earlier
development results informed harness changes. Two tasks come from the pinned
SWE-bench dataset; callback-order task 4230 uses the original upstream issue and
regression test. The source and test patches remain unchanged.
Three task clusters and repeated attempts cannot establish a general Flask win.

## Run

Follow [REPRODUCE.md](REPRODUCE.md). From the repository root with Python 3.12,
a local Docker daemon and an Anthropic API credential:

```bash
python3 -m examples.software_factory plan --eval flask
ANTHROPIC_API_KEY=... python3 -m examples.software_factory run --eval flask --output ./factory-run
```

The launcher prepares isolated services, source indexes, dependency images and
verifiers. Three repeats across five arms produce 45 primary attempts. Six
separate baseline A/A controls produce 51 total attempts. Twelve blind fixed-only
judgments use the same Sonnet 5.5 model. Harbor retries are disabled. Each attempt
uses a fresh native Claude Code session. Claude chooses when to stop; no extra
review turn is supplied by the evaluator.

## Historical knowledge

[historical_memories.json](memory/historical_memories.json) contains 30 short
project notes, including 14 code-linked notes. Public source URLs, content hashes,
publication and edit times, per-task cutoffs and excluded target discussions are
recorded beside the notes. Setup verifies source content before ingestion.
The facts concern prior architecture, compatibility and known pitfalls. The
corpus contains no reference fixes or hidden test requirements.

The two memory arms have optional native memory tools and private writable
namespaces containing historical facts. Code-only arms have no historical memory namespace or memory write permission. Facts are retrieved through
memory tools and code-linked lessons, with none inserted into prompts. Ordinary
project-note tools are disabled. This comparison measures the
combined contribution of historical knowledge and its native retrieval. It does
not isolate linking from the extra information; that needs a matched-facts ablation.

The history is replayed at the public 2.0.0rc2 revision, then the index advances
to each task's real buggy base. Native workers flag changed anchors. One memory
is revised from an independently published later document using a checked
version, and its link must become fresh again. This demonstrates a replayed
lifecycle; it does not measure autonomous learning over elapsed months. Each arm uses the same
current source revision as its paired code-only arm. No authored bug or synthetic
source revision is added to the task set.

## Verification and reporting

The model sees the public issue and can list, read and search the public
base-revision implementation, docs, examples and test sources. Claude can edit and test its ordinary checkout. Only implementation changes
are submitted for hidden grading. Reference patches, future Git history and
evaluator inputs stay outside its environment.
The independent hidden verifier decides success. Full-suite regressions and the
blind judge are separate measurements; authored tests do not establish success.

[Harbor reports](../../harbor/README.md) retain hidden success, blind fixed judgments,
full-suite regressions, agent and end-to-end time, agent plus Loom provider cost,
tokens, test runs, Loom selections and persisted memory writes. Setup, judging and
unpriced infrastructure are reported separately. Failed attempts remain included;
unknown values remain unknown. Zero optional Loom calls is valid.

The targets are 9/9 hidden successes per arm and at least 20% lower agent time and
provider cost than the paired baseline. They are hypotheses, not measured claims.
Read [PILOT.md](PILOT.md) for the comparison policy and
[the architecture](../../../../docs/architecture/software-factory-evaluation.md)
for the sandbox and source boundaries.
Loading
Loading