feat: [no-ticket] harbor - infra resilience (dead-attempt classification, opt-in outage retry, key-budget alarm)

[shehabyasser-scale]

Stacked on #38. Fixes the measurement layer for the three infrastructure failure modes that hit the live transfer-matrix runs. The guiding rule throughout: infra classification may inform diagnostics, alarms, and (opt-in) retries, but it must never move a booked score, because every input to the classifier is ultimately raised inside candidate code.

The three live incidents

1. A 65-second host DNS blip killed 44 of 72 attempts of one eval with ConnectionError, and with no pause between attempts the whole burst fit inside the blip. Nothing in the record distinguished the blip from a bad candidate; the cell had to be diagnosed by hand from attempt timestamps.
2. An LLM key crossed its spend cap mid-matrix. litellm surfaces that as BadRequestError with "Budget has been exceeded" in the message, so two cells of budget-exceeded zeros were nearly booked as a portability finding.
3. Dead-attempt causes were visible (from fix: [no-ticket] harbor - floored rewards name their cause (candidate crash vs infra outage) #37) but not machine-classified, so invalidate-and-re-run decisions stayed manual forensics.

What this adds

• Classification of every dead attempt: a conservative exception-type allowlist (ConnectionError, timeouts, RateLimitError, 5xx, ...) marks transient infra as Type[infra]; the key-budget message signature marks BadRequestError[infra:llm-key-budget] (persistent infra: waiting cannot refill a key). Anything unlisted counts against the candidate. New n_dead_infra metric alongside n_dead; the no-rewards error path now also persists its structured dead_exception_types. Zero-fill is unchanged in every path.
• Opt-in within-eval retry (infra_retry_rounds, default 0) for samples whose every attempt died of a transient infra cause: bounded rounds, linear backoff, fresh sibling jobs dir per round, and recovered samples carry an infra_retry audit marker naming the discarded dead attempts.
• ERROR-level alarm on the key-budget signature, with hedged wording (the signature is read from candidate-process exceptions; corroborate against the key's spend records before invalidating results).

Why the retry is off by default

An adversarial-review pass on this diff (3 lenses, findings adversarially verified) established that with retries on, the qualifying predicate is a re-roll lever: agents are stochastic, and a candidate that raises an allowlisted builtin exception whenever an attempt is going badly loses nothing on partially-good samples (its fakes zero-fill exactly like honest failures) but converts every all-bad sample from a booked 0.0 into a fresh draw (worked example: per-attempt pass rate 0.3, one retry round lifts the expected booked score to 0.51). "A retry re-measures the same code" is only a defense for deterministic candidates. So the mechanism ships default-off, documented as trusted-candidate-only (frozen agents, operator-run matrices), which is exactly the context of the incidents it fixes.

The same review drove: sibling round dirs (nested dirs let a later resume rglob pool a discarded round's dead attempts into a fresh mean; reproduced booking 0.333 where 1.0 was correct), globally fresh dir ordinals (no stale-trial leakage across eval calls of one result dir), bracket neutralization in labels (a candidate exception class literally named XError[infra] must not walk in pre-suffixed), the audit marker (a discarded round must stay visible in the durable record), and two mutation-killing tests (all -> any in the retry gate and linear -> constant backoff both now fail the suite; both previously survived it).

Config

Both knobs (infra_retry_rounds, infra_retry_delay_s) are plain HarborConfig fields and ride the existing harbor: passthrough in serve configs; no schema plumbing needed.

Upstream note: the attempt-burst problem (no inter-attempt spacing inside one harbor run) belongs to harbor itself; this layer classifies and, where trusted, re-measures.

Tests

tests/test_harbor_runner.py: 11-test TestInfraResilience class covering label classification and zero-fill invariance, forged-suffix neutralization, key-budget labeling + alarm, default-off, single- and multi-round retries (backoff sequence, per-round task narrowing, sibling dir names), mixed-cause and partially-scored non-retry, persistent-outage error preservation, mean non-dilution, and the audit marker. Full harbor-family + engine suite: 203 passed, 1 skipped.

🤖 Generated with Claude Code

Greptile Summary

This PR adds infrastructure-failure resilience to the harbor evaluation runner: dead-attempt classification (transient vs. persistent infra vs. candidate-caused), an opt-in bounded retry for samples whose every attempt was killed by transient infra, and an ERROR-level alarm for the LLM key-budget exhaustion signature. All three address live incidents described in the PR.

• Classification (_dead_attempt_label, _INFRA_EXCEPTION_TYPES): bracket-neutralizes exception type names to prevent suffix forgery, adds n_dead_infra metric alongside existing n_dead, and persists structured dead_exception_types on the no-rewards error path so the retry gate can operate on structured data rather than string parsing.
• Retry (_retry_transient_infra, infra_retry_rounds=0): runs only on samples whose every attempt carries a transient-infra label, in a fresh sibling jobs dir per round with linear backoff, accumulating a full-history discard_history across rounds and stamping an infra_retry audit marker on recovery.
• Key-budget alarm (_alarm_key_budget): fires ERROR-level on the BadRequestError[infra:llm-key-budget] suffix with hedged wording; never triggers a retry since waiting cannot refill a key.

Confidence Score: 5/5

Safe to merge; the retry path is off by default, zero-fill invariant is unchanged across all code paths, and prior P1 validation findings are addressed.

The classification and alarm paths are additive and do not touch score computation. The retry path is disabled by default and guarded by a conservative allowlist. Both previously flagged issues are addressed in this revision. The one remaining finding only affects the audit marker field, not scores or retry eligibility, and requires an edge-case total runner crash to trigger.

The _retry_transient_infra method in runner.py is the most complex new path and the only one with a non-trivial edge case around discard_history accumulation when a round produces no output.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/config.py	Adds infra_retry_rounds (default 0) and infra_retry_delay_s (default 30.0) with full validation: negative rounds raise, and zero/negative delay raises when retries are enabled.
vero/src/vero/harbor/runner.py	Core of the PR: adds classification helpers, retry loop with sibling-dir isolation and linear backoff, audit-marker stamping, and _transient_infra_dead gate. The discard_history accumulation re-appends a stale entry when a retry round produces zero output for a task.
vero/tests/test_harbor_runner.py	Adds 11-test TestInfraResilience class covering classification, zero-fill invariance, suffix forgery neutralization, key-budget alarm, default-off gate, single/multi-round retry, backoff linearity, sibling-dir naming, persistent-outage preservation, mean non-dilution, and the audit marker.

Comments Outside Diff (1)

1. vero/src/vero/harbor/config.py, line 71-78 (link)

   A zero or negative infra_retry_delay_s makes every retry round fire with delay = 0, silently disabling the linear backoff. The PR's own incident data shows that immediate retries burned 6 of 8 attempts inside a single DNS blip. Mirroring the existing validation for aggregate_attempts prevents a misconfigured value from nullifying the protection.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: vero/src/vero/harbor/config.py
   Line: 71-78
   
   Comment:
   A zero or negative `infra_retry_delay_s` makes every retry round fire with `delay = 0`, silently disabling the linear backoff. The PR's own incident data shows that immediate retries burned 6 of 8 attempts inside a single DNS blip. Mirroring the existing validation for `aggregate_attempts` prevents a misconfigured value from nullifying the protection.
   
   
   
   How can I resolve this? If you propose a fix, please make it concise.

   

_{Reviews (2): Last reviewed commit: "fix(harbor): review follow-ups: validate..." | Re-trigger Greptile}