feat(harbor): feedback levers: failure transcripts, multi-fidelity screening, per-attempt detail

[shehabyasser-scale]

Adds three opt-in feedback levers to Mode B, answering the question this stack kept circling: the optimizer only gets one score per paid eval, which is very thin signal. Each lever is a build.yaml switch, default off, byte-identical behavior when off.

• feedback_transcripts (+ feedback_max_bytes, default 3000): every FAILED sample of an eval carries the tail of its rollout transcript in the per-sample feedback field. Rides the per-sample result files the sidecar writes ONLY for viewable splits, so it can never surface for non_viewable / no_access tiers.
• instruct_multifidelity: the compiled instruction teaches subset-eval screening (triage rough ideas cheaply, confirm survivors on the full split). Renders only when a viewable split exists (see leak fix below).
• expose_attempt_detail: per-sample attempts list with {reward, exception} per attempt, same viewable-only exposure.

The final commit closes what adversarial review found: a multi-fidelity hidden-split leak (1-sample subset evals could recover per-sample labels on non_viewable splits; the instruction is now gated to viewable splits), subset evals polluting the auto_best shortlist (ranking now prefers full-split evals), feedback_max_bytes <= 0 returning the whole transcript (now means "no feedback"), symlink/path-escape containment on transcript reads, typo-safe config keys (extra="forbid"), and attempt-ordering.

Live experimental evidence (2026-07-06). Three arms on the same substrate (the VeRO paper's minimal Pawn agent on gpt-4.1-mini, baseline 0.20, ceiling ~0.43), same optimizer (opus), same 10-eval budget, only the levers differ:

Arm	Levers	Hidden-split final	Optimizer behavior
A	multifidelity + attempt detail	0.139	prompt rewording only, flat
B	transcripts only	0.194	rewrote agent CODE: file parsing, image support, fixed a binary-read crash it saw in the transcripts
C	all three	0.167	code fixes too; subset screening diluted the budget

Transcripts changed the optimizer from prompt-tweaker (which never moves the score; see the campaign doc's 6-model flatness result) into code-fixer, which is where all measured gains come from. n=1 per arm; a replication and a 25-eval run are in flight.

Stacked on the harbor stack (base: harbor-all-fixes, the integration union of the open PRs; will re-target as the stack merges).

🤖 Generated with Claude Code

Greptile Summary

Adds three opt-in feedback levers to Mode B (feedback_transcripts, instruct_multifidelity, expose_attempt_detail), all off by default and byte-identical when disabled. The PR also closes several adversarial-review findings: a hidden-split leak through 1-sample subset evals, subset evals inflating the auto_best shortlist, transcript symlink/path-escape exfiltration, feedback_max_bytes <= 0 emitting the whole transcript, and typo-safe config keys via extra="forbid".

• Lever 1 (feedback_transcripts): Attaches the tail of a failed trial's transcript (terminus_2.pane → trajectory.json fallback) to SampleResult.feedback for samples scoring exactly 0.0. Includes byte-cap, symlink rejection, path-confinement, and multibyte-boundary safety. Exposure is gated upstream by the sidecar writing per-sample files only for viewable splits.
• Lever 2 (instruct_multifidelity): Renders a subset-eval screening section in the compiled instruction.md, gated on both a viewable split existing and EvalRequest actually accepting sample_ids/num_samples in the current tree — prevents teaching a knob the sidecar would 400 on.
• Lever 3 (expose_attempt_detail) + verifier fix: Populates output["attempts"] with per-attempt {reward, exception} entries; auto_best ranking now prefers full-split evals over subset evals when any exist, preventing a lucky 1-sample eval from displacing a genuinely better candidate.

Confidence Score: 4/5

Safe to merge. All three levers are default-off and gated behind explicit flags, preserving byte-identical behavior for existing deployments.

The implementation is thorough: symlink/path-escape containment, feedback_max_bytes<=0 guard, attempt sort stability, tier-gate tests, and the multi-fidelity hidden-split leak fix are all present and well-tested. Two edge cases in runner.py are worth a follow-up: an empty transcript file (b"") is decoded to "" and returned immediately rather than falling through to trajectory.json, and the error-path SampleResult (agent died before the verifier ran) never calls _failure_feedback, so feedback_transcripts=true provides no transcript for the most common crash scenario.

vero/src/vero/harbor/runner.py — _read_transcript_tail (empty-file early-return) and _sample_result's error branch (feedback not populated)

Important Files Changed

Filename	Overview
vero/src/vero/harbor/runner.py	Adds feedback-lever collation to HarborRunner: transcript tails (Lever 1) and per-attempt {reward, exception} detail (Lever 3). Core logic is sound; two P2 edge cases: empty transcript files return "" instead of being skipped, and the error path (no verifier rewards) never populates the feedback field despite the lever being on.
vero/src/vero/harbor/build/config.py	Adds four new BuildConfig fields and extra="forbid" to catch typo'd lever keys at load time. Clean and well-guarded.
vero/src/vero/harbor/build/compiler.py	Passes new lever fields through to serve.json and gates instruct_multifidelity on both EvalRequest having subset-eval fields and at least one viewable split existing. Mode A warning correct. No issues.
vero/src/vero/harbor/serve.py	Adds lever fields to ServeConfig and wires them into HarborRunner at build_components time. Clean plumbing.
vero/src/vero/harbor/verifier.py	Adds full-split-only ranking for auto_best shortlist; backward-compatible. Logic correct.
vero/tests/test_harbor_runner.py	Comprehensive coverage: byte-cap, symlink/path-escape rejection, multibyte boundary, attempt sort order, zero/negative cap, fallback to next attempt, end-to-end best-mode collation.
vero/tests/test_harbor_build.py	Tests for serve.json plumbing, multifidelity rendering/suppression, typo-rejection, and Mode A warnings.
vero/tests/test_harbor_server.py	Tier-gate tests: feedback and attempt detail written only for viewable splits, never for non_viewable or no_access.
vero/tests/test_harbor_verifier.py	Tests for subset-eval shortlist filtering: lucky-subset vs full-split and all-subset-evals-remain-rankable.

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
vero/src/vero/harbor/runner.py:480
**Empty transcript file silently produces `""` feedback**

`data[-self.feedback_max_bytes :].decode(...)` on an empty file returns `""`. Because the check in `_failure_feedback` is `if tail is not None`, an empty string passes and the function immediately returns `""` as feedback — rather than continuing to try `trajectory.json` (or skipping the attempt altogether). A caller checking `if result.feedback` will treat it as absent, but the JSON serialization will emit `"feedback": ""` instead of `"feedback": null`, which could confuse an optimizer. Adding `if not data: continue` before the decode would skip empty files and let the fallback candidate run.

### Issue 2 of 2
vero/src/vero/harbor/runner.py:368-374
**Error-path SampleResults never receive a feedback transcript**

The error branch (trial exists but verifier produced no rewards — agent died before scoring) calls `_out` (so `expose_attempt_detail` still populates `output["attempts"]`), but never calls `_failure_feedback`, leaving `feedback` at its implicit `None`. Since `_failure_feedback` only iterates attempts whose rewards are 0.0, the crash transcript of an attempt that died before the verifier ran is also unreachable through the scored-failure path. Users who set `feedback_transcripts: true` expressly to understand agent crashes will get no transcript for the most common failure mode (process exit / timeout before the verifier runs).

_{Reviews (1): Last reviewed commit: "fix(harbor): close multi-fidelity hidden..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[greptile-apps]

Empty transcript file silently produces "" feedback

data[-self.feedback_max_bytes :].decode(...) on an empty file returns "". Because the check in _failure_feedback is if tail is not None, an empty string passes and the function immediately returns "" as feedback — rather than continuing to try trajectory.json (or skipping the attempt altogether). A caller checking if result.feedback will treat it as absent, but the JSON serialization will emit "feedback": "" instead of "feedback": null, which could confuse an optimizer. Adding if not data: continue before the decode would skip empty files and let the fallback candidate run.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/runner.py
Line: 480

Comment:
**Empty transcript file silently produces `""` feedback**

`data[-self.feedback_max_bytes :].decode(...)` on an empty file returns `""`. Because the check in `_failure_feedback` is `if tail is not None`, an empty string passes and the function immediately returns `""` as feedback — rather than continuing to try `trajectory.json` (or skipping the attempt altogether). A caller checking `if result.feedback` will treat it as absent, but the JSON serialization will emit `"feedback": ""` instead of `"feedback": null`, which could confuse an optimizer. Adding `if not data: continue` before the decode would skip empty files and let the fallback candidate run.

How can I resolve this? If you propose a fix, please make it concise.

[shehabyasser-scale]

Fixed at the stack tip in bd1d1fe (harbor-9-ops, PR #36 branch): _read_transcript_tail now skips empty transcript files, so an empty pane falls through to the trajectory, and if every candidate is empty it returns None and the search moves to the next failed attempt instead of short-circuiting on "". Regression tests: test_empty_pane_falls_through_to_trajectory, test_all_empty_transcripts_yield_no_feedback.

[greptile-apps]

Error-path SampleResults never receive a feedback transcript

The error branch (trial exists but verifier produced no rewards — agent died before scoring) calls _out (so expose_attempt_detail still populates output["attempts"]), but never calls _failure_feedback, leaving feedback at its implicit None. Since _failure_feedback only iterates attempts whose rewards are 0.0, the crash transcript of an attempt that died before the verifier ran is also unreachable through the scored-failure path. Users who set feedback_transcripts: true expressly to understand agent crashes will get no transcript for the most common failure mode (process exit / timeout before the verifier runs).

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/runner.py
Line: 368-374

Comment:
**Error-path SampleResults never receive a feedback transcript**

The error branch (trial exists but verifier produced no rewards — agent died before scoring) calls `_out` (so `expose_attempt_detail` still populates `output["attempts"]`), but never calls `_failure_feedback`, leaving `feedback` at its implicit `None`. Since `_failure_feedback` only iterates attempts whose rewards are 0.0, the crash transcript of an attempt that died before the verifier ran is also unreachable through the scored-failure path. Users who set `feedback_transcripts: true` expressly to understand agent crashes will get no transcript for the most common failure mode (process exit / timeout before the verifier runs).

How can I resolve this? If you propose a fix, please make it concise.

[shehabyasser-scale]

Fixed at the stack tip in bd1d1fe (harbor-9-ops, PR #36 branch): the no-verifier-rewards error branch now calls _failure_feedback(0.0, attempts), so a sample whose agent died before scoring carries the crash transcript. You are right that this branch matters: a candidate edit that crashes the agent lands exactly here, and without the transcript the optimizer has no way to see the crash it caused. Regression test: test_no_rewards_error_sample_carries_crash_transcript.