Repository navigation
Conversation
The application owns context lookup, source checks and attempt accounting. Scripted fixtures exercise the workflow without live provider calls. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep full source authority behind bounded agent reads. Preserve request counts, usage and failed result manifests when transport, output limits or graders fail. Add exact-base source export, fixed focused container grading, immutable tests, owned cleanup receipts, scope verification, quotas and controller deadlines. Live cohorts remain disabled pending integration and protocol qualification. Validation: 36 contracts, 9 historical controls, 18 isolation probes, 270 focused documentation checks and 9 replay attempts. No live model benchmark. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain interrupted attempts and unknown billing across restarts. Keep generation prices separate from total cost until server and transport billing is available. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Join dispatch and response receipts to usage evidence without retaining other headers or response content. Keep missing and malformed IDs unknown. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain failures, missing attempts and unknown billing in paired reports. Reuse the task bootstrap helper and separate A/A references from primary comparisons. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
zlareb1
marked this pull request as draft
November 20, 2026 14:18
4 of 5 tasks
Reject untrusted historical grades before clients or hidden evaluation. Check physical baseline exposure, collect offline contracts in CI, regenerate the example reach manifest, and retain cold-worker reservations. A replacement oracle and live eligibility remain required before a paid benchmark. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep authored assertions and final grades on the host. Pin exact source, image, probe seed and control receipts. Candidate crashes and output limits remain failed checks that allow revision; transport and cleanup errors halt. The narrow probes do not claim official test coverage or live model results. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use source-only behavioral preparation and distinguish authored host checks from candidate-controlled pytest diagnostics. Keep actual index, token, billing and exact protocol approval requirements explicit. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep grader faults distinct from candidate failures and agent tool errors, even when later hidden checks pass. Remove unreachable report-based qualification work while retaining its explicit refusal. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Distinguish exhausted candidate checks from grader infrastructure failures so bounded trajectories do not halt later cold attempts. Infrastructure timeouts still invalidate grading. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep only bounded visible diagnostics and refuse hidden grading before filesystem or container work. The diagnostic adapter no longer retains private hidden patches or selectors. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
6 of 8 tasks
Remove the hidden-test application claim from the diagnostic adapter description. Hidden grading uses the separate behavioral probes. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
An attach deadline establishes no candidate execution outcome. Retain the transport timeout and stop evaluation when the outcome is unknown. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Separate the shared factory and reporting harness from the Python repository profile. Preserve native trial receipts, blinded verification, provider accounting and public reproduction inputs. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…-harbor-pr Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…-harbor-pr Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Persist the reporting and public-export receipt in the run summary, and include the formatted assets and generated operation reach metadata. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Retain separate controller and server revisions, bind exported backend costs to native transport IDs, and classify exhausted visible-test quotas as refusals before execution. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Review the consolidated candidate diff with the same model and tools in every arm before completion. Retain its cost and trace, require the review in fairness checks, and let ordinary out-of-scope reads return a content-free refusal so the agent can correct the path. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…ory-retrieval-contract
Qualify and audit the memory scope for every selected code operation. State the implementation-only index boundary in the common prompt and direct public docs and tests to ordinary read tools. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Expose bounded RE2 matching through the ordinary repository search tool and pass the same optional flag through native grep. Preserve literal defaults, source boundaries and paginated results. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Normalize presentation bytes without changing parsed records or measurements, refresh the manifest and link the original export. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
The projection dropped abstention reasons and hit scores from actual search responses. Keep the native epistemic fields so an empty search and a verified fact cannot be confused, while retaining the existing scope checks. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
The existing teardown fact is invisible to code-linked recall without a source anchor. Bind it to the historical request-cleanup definition while retaining its dated source and excluding target requirements. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Body-budget elision must not discard a valid navigation pointer. Keep cards separate from grounded bodies and accept the native identifier needed to follow them. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Search cards can name chunks whose identifiers are not symbol lookup keys. Direct those cards through their native follow-up or name. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…osts Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
… addresses Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…orkflow Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…earch Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Keep scoped member outlines and attached lessons visible, retain authorized raw context for analysis, and reject broken grader packages during image builds. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
…ory-search-source-prefix
…urce-prefix Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Pin the next coding cohort to Sonnet with independent native pricing and a concise comparison report. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Check disabled memory access explicitly in the treatment contract. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Keep archived measurements and provenance unchanged while updating presentation link handling. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Retain unmatched attempts and spend as progress until every arm has a graded counterpart. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Expose the same optional line bounds to every arm and describe verified definition bodies accurately. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
Run repository coding evaluations in Harbor with isolated verifiers, optional code and memory tools, provider accounting and automatic comparison reports. The Python example compares five treatments on pinned historical Flask issues.
Why
Measure whether optional code retrieval lowers the cost and time of a correct issue fix, and whether historical memory improves correctness beyond code retrieval alone. The final patch is checked independently. Failures, nonuse, incomplete attempts and provider charges remain in the evidence.
This PR remains open until the requested evaluation goal is met. Existing complete results do not establish that goal.
Scope and risk
The shared factory contains the agent loop; the shared Harbor package owns scheduling, setup, accounting, verification and reporting. Repository profiles provide source revisions, dependencies, historical facts and correctness checks. The layout supports repositories with one or several languages.
The current protocol has five arms:
Every arm receives the same public issue, checkout revision, ordinary source tools, model, resources and visible test capabilities. Source listing, reading and RE2 search cover implementation, public docs, examples and base tests. Only implementation is editable. Claude chooses every retrieval call; no hook or forced lookup supplies context. A common final contract-review turn has the same tools and model, and its time and cost count.
Three issues cover blueprint reuse, nested callback ordering and nested CLI application ownership. Three repeats produce 45 primary attempts, plus six baseline A/A controls. The solver and twelve blind fixed-only judgments use
claude-sonnet-5-5; native model-backed operations inherit the merged product default. All arms share model and inference settings. Sonnet results form a separate cohort from the retained Haiku results. The ledger freezes standard prices by model and includes cache reads and writes. Unknown models refuse dispatch.The two memory arms receive 30 short historical facts, 14 linked to code, in 18 private writable namespaces. Both tiers index the same allowed source. The source advances from a real earlier revision to each buggy base; native receipts prove writes, exact read-back, changed links and a source-backed revision that relinks fresh. Historical facts must predate each task cutoff and have independent public provenance. Target discussions, fixing patches and hidden assertions cannot supply memory. This measures historical knowledge plus native memory retrieval; it does not isolate linking from possession of those facts or measure autonomous learning over months.
Code-only arms have neither memory tools nor memory namespaces, and explicitly disable code-linked lessons. The earlier retained Haiku cohorts exposed empty writable memory namespaces in the code-only arms. Those results retain their original protocol and do not serve as controls for this changed treatment.
Loom responses retain native navigation, outline children, memory confidence and source provenance after authorization and source checks. Search results may prefix source with retrieval context. The adapter now separates that prefix when the remaining body exactly matches the authorized checkout span. Unchanged files retain the context separately; edited files discard stale context. Other body validation remains strict. A recorded five-result response previously lost four valid bodies and now preserves all five. This offline replay establishes the adapter correction, not a coding-quality gain.
Visible checks run in the agent sandbox. Hidden checks and the full suite run in a separate verifier. Buggy-base controls must fail and reference controls must pass before inference. Trusted grader packaging includes its transitive source helpers and checks importability when the image is built. Provider retries are disabled, retrieval defaults are inherited, and all usage is retained. Setup and judgment spend are reported separately from per-attempt solver and Loom provider costs. Infrastructure monetary cost remains unknown.
Dynamic report links accept HTTP and HTTPS URLs and reject embedded credentials. The two archived chart presentations have the same correction; their recorded data, protocol and provenance retain their original bytes, and presentation manifest hashes are updated.
Live comparisons use only identical issue/repeat blocks graded in all five arms. Hidden-test failures remain included. Unmatched attempts and their spend remain in progress totals and individual records, and final target claims require the complete cohort.
The next controller retains validated grep scope ranges and offers optional inclusive line bounds on ordinary reads in every arm. It also states that definition tools return verified bodies. A replay of three captured responses preserves all 43 scope ranges that the earlier adapter dropped, with the same 52 source matches and no extra API calls. Source-range replay checks exact bounded reads on the pinned repository; it does not establish agent adoption or an end-to-end gain. These changes are outside the active Sonnet execution, whose controller remains frozen.
Reports show one five-arm summary table and one per-issue grid. Charts, tokens, Loom calls, memory writes, test counts, regressions, individual attempts, A/A controls and method details are expandable. Pending and ungraded attempts remain distinct from failures. Partial, unmatched comparisons do not display a winner or a cost/time target. Raw provider and scoped native ledgers support accounting replay; public exports contain validated measurements and provenance hashes.
Retained measured evidence
The latest completed Haiku cohort has 51 coding attempts and twelve blind comparisons. All twelve source proofs, 36 then-required memory certificates and six native base/reference controls passed. These results predate the strict memory-disabled arms and Sonnet selection.
Every Loom arm cost more and took longer than baseline. Full-suite regressions and agent memory writes were zero. A/A controls had substantial cost and time variation. The sanitized evidence is in
examples/software_factory/evals/flask/results/historical-issues-public-evidence-review/. Reproduce with the frozen controller commit recorded in that bundle. The earlier implementation-only evidence remains a separate bundle.The subsequent Haiku cohort was interrupted after a verified material adapter defect: valid search bodies with retrieval prefixes were discarded. All twelve attempt records remain, including the interrupted attempt. It is an incomplete comparison and supplies no winner claim. Owned resources were removed.
These three issues have been used repeatedly during harness development. Any improvement on them is development evidence; general claims require a separately frozen evaluation on fresh tasks. Original issue and verifier inputs remain unchanged.
Compatibility, rollout and rollback
python3 -m examples.software_factory plan --eval flask --json, thenpython3 -m examples.software_factory run --eval flask --output ./factory-run.Tests
LOOM_ENV_FILE= make format,make lint,make typecheckand the example Ruff checks pass. Generated contracts remain current.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated: Codex implemented the factory and reporting. Fresh agents review each pushed head; measurements come from retained native artifacts.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist