fix: [no-ticket] harbor - access-tier integrity (free-baseline privilege, k-anonymity floor, trusted nested CLI) by shehabyasser-scale · Pull Request #34 · scaleapi/vero

shehabyasser-scale · 2026-07-08T23:21:14Z

Stacked on #33 (harbor-6-selection-integrity). Part of the pre-paper hardening series; closes the access-tier group.

What leaks this closes

1. Free baseline eval was a privilege escalation (server.py). The agent's one budget-free baseline eval called engine.evaluate(admin=True). Admin does not just skip the ledger; it also bypasses the no_access tier gate, so the free eval could target the held-out test split and read its aggregate score straight off the EvalSummary response. Fix: engine.evaluate gains a distinct free parameter that waives only the budget debit and nothing else; admin and free are separate authorities. The freebie is also consumed only after the eval succeeds, so an infra failure no longer burns it.

2. The engine's no_access gate was dormant in the deployed sidecar (serve.py). build_components never passed split_accesses into the EvaluationEngine, so the engine-side hard rejection (added in an earlier PR) never armed. The de-facto gate was the budget ledger (no_access splits are unbudgeted), which is precisely what every unmetered path (admin, the free baseline) skips. Fix: wire split_accesses through; the gate fails closed on unlisted splits.

3. Per-sample label reconstruction on non_viewable splits (server.py, protocol.py). EvalRequest.sample_ids / num_samples let the agent evaluate arbitrary subsets, and the aggregate mean_score in the response over a singleton subset is that sample's score verbatim; n singleton evals reconstruct the hidden split's labels wholesale. Fix: k-anonymity floor (default 5, k_anonymity_floor in build.yaml/serve.json) on agent-chosen proper subsets of non_viewable splits, rejected up front (no commit transfer, no eval, no budget debit) and advertised in status() as min_subset_samples. Full-split evals always pass, so a split smaller than the floor degrades to full-split-only rather than becoming unevaluable. Scope note: the floor multiplies reconstruction cost by ~k (overlapping k-subsets still admit elimination); the binding constraint remains the budget. It is a cost lever, not a proof.

4. The nested scorer CLI resolved from the candidate's own lockfile (runner.py, config.py). Mode B shells out to uv run --project <candidate> harbor run ..., so the orchestrator that writes the trial result.json files came from whatever the agent's pyproject.toml/uv.lock pinned; one edited line could point it at a fork that fabricates perfect rewards without running anything. Fix: HarborConfig.harbor_requirement; when set, the runner layers --with <spec> over the candidate env, and uv's ephemeral overlay takes precedence over a conflicting project pin for both the console script and sys.path (verified empirically: project pinning six==1.16.0, --with six==1.17.0 resolves 1.17.0). Honest residual: agent code still imports into the nested harbor process via --agent-import-path, so in-process tampering remains possible; a full boundary needs out-of-process verification and is tracked for a later PR.

Tests

test_engine.py: free runs without debiting budget; free does NOT bypass the no_access gate.
test_harbor_server.py: free baseline passes free=True, admin=False; failed free eval keeps the freebie; floor rejects sub-k subsets before any work, allows at-floor subsets, full splits, viewable splits, admin, and floor<=1; status advertises the floor.
test_harbor_protocol.py: min_subset_samples advertised for non_viewable only.
test_harbor_runner.py: --with placement before the harbor executable; absent when unset.
158 passed, 2 skipped (pre-existing) across the affected files.

🤖 Generated with Claude Code

Greptile Summary

This PR closes four access-tier integrity gaps in the Harbor eval sidecar: the free baseline eval was riding admin=True (which bypasses the no_access tier gate), the engine-side no_access gate was never wired with split_accesses so every unmetered path walked past it, agents could reconstruct non-viewable split labels via singleton subset evals, and the nested scorer CLI resolved from the candidate's own lockfile.

admin/free separation (engine.py, server.py): engine.evaluate gains a distinct free parameter; free baseline no longer rides admin=True, so tier gates continue to apply.
Engine gate wired (serve.py): split_accesses is now passed to EvaluationEngine so the no_access rejection is fail-closed on all paths including unmetered ones.
k-anonymity floor (server.py, protocol.py, configs): subset evals on non_viewable splits below the configured floor (default 5) are rejected before commit transfer, budget debit, or eval.
Trusted harbor CLI (runner.py, config.py): harbor_requirement layers --with <spec> over the candidate env so the nested orchestrator resolves from a trusted pinned spec.

Confidence Score: 3/5

The core access-control fixes are correct and well-tested, but the free-baseline guard regresses: moving the consumed flag to after the async eval creates a window two concurrent requests can both slip through.

Three of the four fixes are clean. The fourth moves _free_baseline_used = True to after await engine.evaluate, creating a race window that didn't exist in the old code. The old code set the flag synchronously before the eval await, which was atomic in asyncio's cooperative scheduler. Now two concurrent baseline requests can both finish _transfer_commit, both read the flag as False, and both call the eval with free=True.

vero/src/vero/harbor/server.py (free baseline flag placement) and vero/src/vero/harbor/protocol.py (build_status default floor parameter)

Important Files Changed

Filename	Overview
vero/src/vero/evaluation/engine.py	Adds `free` parameter to `evaluate` — correctly separated from `admin` so a free eval still runs through the no_access tier gate.
vero/src/vero/harbor/server.py	Core sidecar handler. k-anonymity floor and free-eval separation are correct, but moving `_free_baseline_used = True` to after `await engine.evaluate` opens a concurrent-request race window.
vero/src/vero/harbor/serve.py	Wires `split_accesses` into `EvaluationEngine` (dormant no_access gate now armed) and threads `k_anonymity_floor` into the sidecar.
vero/src/vero/harbor/protocol.py	Adds `min_subset_samples` per split. Default `k_anonymity_floor=1` in the signature is inconsistent with the sidecar's enforced default of 5.
vero/src/vero/harbor/runner.py	Adds `--with harbor_requirement` overlay before the `harbor` executable. Clean and well-tested.
vero/src/vero/harbor/config.py	Adds `harbor_requirement: str
vero/src/vero/harbor/build/config.py	Adds `k_anonymity_floor: int = 5` build config field.
vero/src/vero/harbor/build/compiler.py	Threads `k_anonymity_floor` from build config into serve.json.
vero/src/vero/harbor/app.py	Registers `KAnonymityError` → HTTP 400 exception handler.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine
    participant Budget as BudgetLedger

    Agent->>Sidecar: evaluate(req)
    Note over Sidecar: [NEW] k-anonymity floor check
    Sidecar->>Sidecar: _transfer_commit(req.commit)
    Note over Sidecar: detect free_baseline
    Sidecar->>Engine: "evaluate(req, admin=admin, free=free_baseline)"
    Note over Engine: [NEW] no_access gate armed
    alt not admin AND not free
        Engine->>Budget: reserve(dataset_id, split, n)
    end
    Engine-->>Sidecar: Experiment
    Note over Sidecar: [CHANGED] _free_baseline_used=True after eval
    Sidecar-->>Agent: EvalSummary

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine
    participant Budget as BudgetLedger

    Agent->>Sidecar: evaluate(req)
    Note over Sidecar: [NEW] k-anonymity floor check
    Sidecar->>Sidecar: _transfer_commit(req.commit)
    Note over Sidecar: detect free_baseline
    Sidecar->>Engine: "evaluate(req, admin=admin, free=free_baseline)"
    Note over Engine: [NEW] no_access gate armed
    alt not admin AND not free
        Engine->>Budget: reserve(dataset_id, split, n)
    end
    Engine-->>Sidecar: Experiment
    Note over Sidecar: [CHANGED] _free_baseline_used=True after eval
    Sidecar-->>Agent: EvalSummary

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
vero/src/vero/harbor/server.py:120-133
**Race window on `_free_baseline_used` flag**

Moving `_free_baseline_used = True` to after the `await engine.evaluate` call reintroduces a race window. In asyncio, execution is synchronous between `await` points, so the old code (which set the flag **before** the eval await) was effectively atomic: once A resumed from `_transfer_commit` and set the flag, any concurrent B that resumed later would see `_free_baseline_used = True`. Now the flag isn't set until after the full eval completes, so two concurrent requests that both finish `_transfer_commit` will both read `_free_baseline_used = False`, both set `free_baseline = True`, and both call `engine.evaluate(..., free=True)` — giving the agent two free baseline evals.

The safer fix is to optimistically set the flag before the eval and restore it on failure:

```python
if free_baseline:
    self._free_baseline_used = True  # claim before any await
try:
    exp = await self.engine.evaluate(
        replace(req, commit=sha), admin=admin, free=free_baseline
    )
except Exception:
    if free_baseline:
        self._free_baseline_used = False  # restore so a retry is still free
    raise
```

### Issue 2 of 2
vero/src/vero/harbor/protocol.py:107-108
`build_status` defaults `k_anonymity_floor` to `1` (the "disabled" sentinel), while `EvaluationSidecar` defaults to `5`. Any caller that forgets to pass the floor will advertise `min_subset_samples = 1` for `non_viewable` splits, causing agents to send sub-floor requests that the sidecar then rejects with `KAnonymityError`. Defaulting to `5` — the same as `EvaluationSidecar.__init__` — keeps the function's own default consistent with the enforcement default.

```suggestion
    k_anonymity_floor: int = 5,
) -> StatusSummary:
```

_{Reviews (1): Last reviewed commit: "fix(harbor): access-tier integrity (free..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

…ity floor, trusted nested CLI) Four related fixes to keep hidden-split information and scoring authority where they belong: 1. The free baseline eval no longer rides the admin flag. engine.evaluate gains a distinct `free` parameter that waives only the budget debit; admin=True also bypassed the no_access tier gate, so the agent's one free eval could target the held-out test split and read its aggregate score off the response. The freebie is also now consumed only after a successful eval, so an infra failure no longer burns it. 2. serve.py now passes split_accesses into the EvaluationEngine. Without it the engine-side no_access gate was dormant and the budget ledger (no_access splits are unbudgeted) was the only gate, which is exactly what every unmetered path skipped. 3. k-anonymity floor on subset evals of non_viewable splits (default 5, configurable via build.yaml / serve.json). EvalSummary.mean_score over an agent-chosen singleton subset is that sample's label-derived score verbatim, so n singleton evals reconstructed a hidden split's labels wholesale. Full-split evals always pass (their aggregate is the intended surface), so splits smaller than the floor stay evaluable. The floor is advertised in status() as min_subset_samples. 4. HarborConfig.harbor_requirement: when set, the nested `harbor run` is layered over the candidate env with `uv run --with <spec>`, so the orchestrator that produces trial result.json resolves from the trusted spec, not from the candidate's own pyproject/uv.lock (one edited line there could point at a fork that fabricates results). Verified that uv's ephemeral overlay takes precedence over a conflicting project pin for both the console script and sys.path. This raises the bar, not a full boundary: agent code still imports into the nested harbor process; out-of-process verification is tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

greptile-apps · 2026-07-08T23:27:26Z

        free_baseline = (
            not admin
            and self.base_commit is not None
            and sha == self.base_commit
            and not self._free_baseline_used
        )
-        if free_baseline:
-            self._free_baseline_used = True
        exp = await self.engine.evaluate(
-            replace(req, commit=sha), admin=admin or free_baseline
+            replace(req, commit=sha), admin=admin, free=free_baseline
        )
+        # Consume the freebie only after the eval succeeded: an eval that
+        # raised (invalid split, infra failure) has given the agent nothing,
+        # so it must not burn the one free reference measurement.
+        if free_baseline:
+            self._free_baseline_used = True


Race window on _free_baseline_used flag

Moving _free_baseline_used = True to after the await engine.evaluate call reintroduces a race window. In asyncio, execution is synchronous between await points, so the old code (which set the flag before the eval await) was effectively atomic: once A resumed from _transfer_commit and set the flag, any concurrent B that resumed later would see _free_baseline_used = True. Now the flag isn't set until after the full eval completes, so two concurrent requests that both finish _transfer_commit will both read _free_baseline_used = False, both set free_baseline = True, and both call engine.evaluate(..., free=True) — giving the agent two free baseline evals.

The safer fix is to optimistically set the flag before the eval and restore it on failure:

if free_baseline: self._free_baseline_used = True # claim before any await try: exp = await self.engine.evaluate( replace(req, commit=sha), admin=admin, free=free_baseline ) except Exception: if free_baseline: self._free_baseline_used = False # restore so a retry is still free raise

Prompt To Fix With AI

This is a comment left during a code review. Path: vero/src/vero/harbor/server.py Line: 120-133 Comment: **Race window on `_free_baseline_used` flag** Moving `_free_baseline_used = True` to after the `await engine.evaluate` call reintroduces a race window. In asyncio, execution is synchronous between `await` points, so the old code (which set the flag **before** the eval await) was effectively atomic: once A resumed from `_transfer_commit` and set the flag, any concurrent B that resumed later would see `_free_baseline_used = True`. Now the flag isn't set until after the full eval completes, so two concurrent requests that both finish `_transfer_commit` will both read `_free_baseline_used = False`, both set `free_baseline = True`, and both call `engine.evaluate(..., free=True)` — giving the agent two free baseline evals. The safer fix is to optimistically set the flag before the eval and restore it on failure: ```python if free_baseline: self._free_baseline_used = True # claim before any await try: exp = await self.engine.evaluate( replace(req, commit=sha), admin=admin, free=free_baseline ) except Exception: if free_baseline: self._free_baseline_used = False # restore so a retry is still free raise ``` How can I resolve this? If you propose a fix, please make it concise.

Good catch, and you are right about the asyncio interleaving. Fixed in e1b1d6b (stack tip, harbor-9-ops): the flag is claimed BEFORE the eval await (so a concurrent second baseline eval sees the claim and pays) and refunded in an except block if the eval raises, which keeps the property this PR wanted (a failed eval does not burn the freebie). Regression test: test_freebie_claimed_before_eval_await asserts the flag is visible during the engine call.

greptile-apps · 2026-07-08T23:27:27Z

+    k_anonymity_floor: int = 1,
 ) -> StatusSummary:


build_status defaults k_anonymity_floor to 1 (the "disabled" sentinel), while EvaluationSidecar defaults to 5. Any caller that forgets to pass the floor will advertise min_subset_samples = 1 for non_viewable splits, causing agents to send sub-floor requests that the sidecar then rejects with KAnonymityError. Defaulting to 5 — the same as EvaluationSidecar.__init__ — keeps the function's own default consistent with the enforcement default.

Suggested change

k_anonymity_floor: int = 1,

) -> StatusSummary:

k_anonymity_floor: int = 5,

) -> StatusSummary:

Prompt To Fix With AI

This is a comment left during a code review. Path: vero/src/vero/harbor/protocol.py Line: 107-108 Comment: `build_status` defaults `k_anonymity_floor` to `1` (the "disabled" sentinel), while `EvaluationSidecar` defaults to `5`. Any caller that forgets to pass the floor will advertise `min_subset_samples = 1` for `non_viewable` splits, causing agents to send sub-floor requests that the sidecar then rejects with `KAnonymityError`. Defaulting to `5` — the same as `EvaluationSidecar.__init__` — keeps the function's own default consistent with the enforcement default. ```suggestion k_anonymity_floor: int = 5, ) -> StatusSummary: ``` How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Agreed; fixed in e1b1d6b (stack tip): build_status now defaults k_anonymity_floor=5, matching the sidecar's enforcement default, with a comment explaining why the two defaults must not drift. Test: test_default_floor_matches_sidecar_enforcement_default.

… floor default, ordinal resume, SE naming) - Free-baseline flag is claimed BEFORE the eval await and refunded on failure: setting it only after success reopened a window where two concurrent baseline evals both resolved free (asyncio interleaves at await points). Claim-then-refund keeps both properties: concurrent callers see the claim, and a failed eval does not burn the freebie. - build_status defaults k_anonymity_floor to 5, matching the sidecar's enforcement default: a caller that forgets to pass the floor must not advertise a laxer one than gets enforced. - _route_results resumes the eval ordinal past surviving __eN dirs on a reused volume: a restarted sidecar started back at e1 and silently wiped the prior session's evidence, the exact erasure the versioned dirs exist to prevent. - score_se renamed to mean_score_se and documented: it is the SE of the zero-filled mean_score over n_samples, not of the n_scored subset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

shehabyasser-scale mentioned this pull request Jul 8, 2026

fix: [no-ticket] harbor - honest measurement signals (summary qualifiers, versioned re-evals, dead-attempt causes) #35

Open

greptile-apps Bot reviewed Jul 8, 2026

View reviewed changes

shehabyasser-scale mentioned this pull request Jul 8, 2026

fix: [no-ticket] harbor - ops integrity (exhaust-budget instruction lever, fail-closed ledger restore) #36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: [no-ticket] harbor - access-tier integrity (free-baseline privilege, k-anonymity floor, trusted nested CLI)#34

fix: [no-ticket] harbor - access-tier integrity (free-baseline privilege, k-anonymity floor, trusted nested CLI)#34
shehabyasser-scale wants to merge 1 commit into
harbor-6-selection-integrityfrom
harbor-7-access-tiers

shehabyasser-scale commented Jul 8, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

greptile-apps Bot Jul 8, 2026

Uh oh!

shehabyasser-scale Jul 8, 2026

Uh oh!

greptile-apps Bot Jul 8, 2026

Uh oh!

shehabyasser-scale Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shehabyasser-scale commented Jul 8, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What leaks this closes

Tests

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

shehabyasser-scale Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

shehabyasser-scale Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shehabyasser-scale commented Jul 8, 2026 •

edited by greptile-apps Bot

Loading