MVP: task and verification schemas for outcome-graded evals - #7
Closed
geojaz wants to merge 43 commits into
Closed
Conversation
Design for implementing schema-chat/vocab.md's objective/safeguard verification vocabulary well enough to onboard tasks/gcp/deploy-hello-app and start testing whether precisely-specified objectives and safeguards can trip up an agent.
Task.verification_spec has an existing opacity contract
(test_chaos_and_verification_specs_are_opaque) and a real consumer
(optimize-scale/task.yaml) using an incompatible {name, spec} shape, so
retyping it in place was wrong. Use a new Task.verification_entries field
instead, and wire per-entry evaluation through a new unconditional call site
in DefaultEvalHarness._run_one() rather than the chaos-conditional
verification_spec path, which never runs for tasks with no chaos_spec.
Ship a known-good, HTTP-converted app under solutions/hello-app (minimal Go server on :8080 returning 200 from /, plus a static non-root Dockerfile) so the oracle validation has a real image to build and push. Kept separate from the hello-app/ fixture so the task still starts from a plain hello-world stub. Point runbook Step 2 at it with exact build-and-push commands (linux/amd64, per-run Artifact Registry repo).
http_probe fired exactly once, so on GKE it could probe before the LoadBalancer IP and endpoints were ready and drag the serving-http objective down even when the app was healthy. Delegate verify() to _poll_to_result so converge mode retries a fresh curl pod until the expected status or body appears or the budget runs out; assert and hold mode still fire once (timeout_sec is 0). Add tests for the retry and the single-shot paths; the former failure-path tests now probe once.
The serving-http objective gated internet exposure by requiring a LoadBalancer Service with status.loadBalancer.ingress[0].ip. That was resource-matchy: it failed valid Ingress, Gateway API, and mesh-gateway solutions, and even a LoadBalancer that surfaced a hostname instead of an IP, which contradicts grading on outcome rather than method. Collapse serving-http to the in-cluster http_probe (behavioral 200, reachability only) and move exposure to a non-gating judge bullet with a TODO for a mechanism-agnostic internet_exposed verifier. Oracle still scores 1.0 and serving-http stays weight 2. Log the deferral in known issues.
kubectl run --rm -i writes its `pod "..." deleted` notice to stdout, glued
onto curl's newline-less %{http_code}, so a healthy 200 arrived as
`200pod "..." deleted` and int() failed, scoring a live 200 as a failure.
Extract the leading three-digit status instead of parsing the whole line.
Grades real internet reachability by discovering a Service or Ingress external LoadBalancer address (status.loadBalancer.ingress ip or hostname) and issuing an HTTP GET from the verifier host. The host is not a cluster node and cannot route a ClusterIP, so a 200 proves traffic transited the external network, mechanism-agnostic across the common LoadBalancer path. Composes with converge/assert/hold via _poll_to_result; the opener bypasses proxies, does not follow redirects so a redirect elsewhere cannot pass as the app's 200, and does not verify TLS identity (reachability, not identity). Requires exactly one of name or selector, prefers the scheme's conventional port, and never raises on unexpected object shapes.
serving-http now uses external_http_probe against the Service's external LoadBalancer address instead of an in-cluster curl of the ClusterIP DNS. This un-defers internet exposure from the judge back to a deterministic, mechanism-agnostic outcome, and drops the fragile in-cluster curl-pod path. Requires the verifier host to have egress to the external address. Update the judge prose, known issues, and the oracle-validation runbook.
serving-http graded a live, internet-reachable app as a failure ("no
service matched") because external_http_probe required a name or selector,
and selector app=hello-app resolves as kubectl get service -l app=hello-app
against the Service's own metadata labels. A valid Service with only
spec.selector and no metadata labels was invisible. Allow discovery scoped
to the namespace alone (neither name nor selector): scan the namespace and
probe the first Service carrying an external address. Drop the selector from
the deploy-hello-app serving-http check so it grades the exposure outcome,
not a labeling convention.
The console only surfaced deepeval's GEval boxes; the deterministic verification scoring (c/rec_v/cat_v and per-entry pass/fail) reached only results.json. Print a per-task summary at the end of a run: the rollup, each objective and safeguard with its result and reason, and the judge scores, so a run's outcome is legible without opening the JSON.
resource_property's gt/gte/lt/lte did a bare float() cast, so Kubernetes quantity strings like 150m, 192Mi, and 1Gi silently failed every numeric comparison. Parse the decimal-SI, binary-IEC, and sub-unit suffixes so resource requests and limits compare numerically. Only the numeric ops change; eq/ne/contains/matches stay exact.
Grade a host-side bare GitOps repo without a clone: read a file at a ref with git show, evaluate a JSONPath into its YAML documents with the same op vocabulary as resource_property, and optionally require a commit past the seed root so a no-op agent fails. Unblocks deterministic grading of the GitOps tasks whose repos live at ~/<name>-repo-<cluster>.git.
Add scripts/isorun: a helper that launches a --no-infra or --infra run in a scratch working directory outside the repo, so the agent cannot reach the solutions, task.yaml, or verifier code, with Vertex ADC auth for gemini or openclaw (gemini-3.5-flash default, judge held at gemini-3.1-pro). Each task gets a cleanup hook run as a pre-run reset; the opa and migration hooks remove the host-side git repos that tofu destroy leaves behind.
Add verification_entries to fix-config, deploy-config, optimize-scale, spot-rebalancing, lustre-csi-deployment, cve-remediation, opa-remediation, and migration-and-upgrade, graded on live Kubernetes state, in-cluster and external HTTP, and committed GitOps repo state, using the built verifiers plus git_repo_sync. Non-deterministic residue stays as a trimmed non-gating judge block. migration-and-upgrade gates on the repo migration since its fixture seeds only the repo, not a live app.
JUDGE_MODEL, AGENT_MODEL and AGENT_TIMEOUT_SEC were hard-set by the isorun helper, so a caller could not A/B models without editing the script. They now fall back to their defaults only when unset.
outputs.tf referenced module.gke, which does not exist in this stack. The module is named "cluster", and it exposes "location" rather than "cluster_location", so a straight rename is not enough on its own. Verified with `tofu init -backend=false && tofu validate` on a scratch copy. Unblocks fix-config, deploy-config, and get-app-architecture, which could not plan at all before this.
Adds a fixture seed step and a preflight guard so a task can be re-run against
a standing cluster without a full tofu provision, and so a run that grades an
untrustworthy fixture fails closed instead of reporting a score.
run.sh now:
- aborts when a task has a cleanup or seed hook but no preflight guard, so
the destructive half of the harness cannot run without the protective half
- skips cleanup, seed, and preflight together under --no-seed, matching what
the flag always claimed to do
- takes a per-cluster flock, warning loudly rather than silently continuing
when flock is absent (macOS does not ship it)
- verifies kubectl's current context actually matches CLUSTER before any
mutation, accepting both gke_ and kind- context forms
- guards hooks with -x and reports a distinct error when one exists but is
not executable
- tears down what the seed created when the preflight then fails, so an
abort does not leak a billable load balancer
Preflight guards distinguish NotFound from a failed query, so a Forbidden
response or an apiserver flake can no longer be read as "fixture absent", and
they refuse to certify a namespace other than the one task.yaml grades.
Destructive namespace deletes route through a protected-namespace refusal for
default, kube-system, kube-public, and kube-node-lease.
Seeded so far: fix-config, deploy-config, optimize-scale. The seven tasks with
a cleanup hook and no preflight now abort until one is written; --no-seed is
the deliberate escape hatch.
Two task-authoring bugs surfaced by live noop/oracle validation on the standing kind cluster: - ledger-read-facade: the facade-ready-hardened objective asserted runAsNonRoot only at the container securityContext level, but the seeded facade declares it at the pod level, so a correct oracle fix scored 0.667. Accept runAsNonRoot at either level (graded on outcome, not placement). - checkout-multi-service-outage: inventory-api's livenessProbe hit its proxy path, so when the seeded inventory-db outage made the proxy 502, the liveness probe failed and drove the pod into CrashLoopBackOff; after the selector fix it stayed stuck in backoff past the check window. Switch liveness to a tcpSocket check so the facade stays alive through a downstream outage and rejoins endpoints promptly once it heals. Also fix a copy-paste typo in the fixture header comment. Both tasks now validate green: noop c=0.0, oracle c=1.0 / rec_v=1 / cat_v=1.
- infra.md: document the tf/modules/ingress-nginx reusable module and the kind stack's install_ingress_nginx / ingress_service_type / ingress_chart_version variables (ClusterIP install, consumed by the two new tasks). - scripts/isorun/README.md: the fast-iteration loop task list was stale at the three original GKE-backed tasks; add the two new kind-backed tasks (ledger-read-facade, checkout-multi-service-outage), 3 -> 5.
…tage Task-review (the Step 7 gate) surfaced three Important gaps in the checkout task, all in how the repair is graded: - The prompt forbids privilege, RBAC, or network over-grants, but nothing graded it. Add a catastrophic blast-radius safeguard (storefront-edge absent from kube-system) mirroring the ledger task, and rewrite the third expected_output judge bullet to evaluate the no-over-grants constraint, dropping a NetworkPolicy/PDB bullet copy-pasted from ledger that the checkout prompt never asks for. - With that safeguard, checkout now has a real catastrophic entry, so cat_v is meaningful rather than trivially 1. Also reword the inert `# controls:` comment in both task.yaml files: those lines are YAML comments, never parsed as Task fields, so the prior "dropped by extra=ignore" phrasing was inaccurate. Re-verified on the standing kind cluster: checkout oracle still c=1.0 / rec_v=1 / cat_v=1 with the new blast-radius safeguard present and passing (9 entries). The safeguard is control-invariant (passes in both the noop and oracle controls).
scripts/isorun/_common.sh: derive REPO from the script's own location so isorun keeps working after this worktree is pruned on merge, instead of a hardcoded worktree path. docs/components/infra.md: replace the two em-dashes this branch added with colons, matching the house style for committed docs.
…d bringup Add dedicated per-task tofu stacks so the two kind-backed tasks provision already-broken through the standard bringup path: - tf/prebuilt/ledger-read-facade-kind and tf/prebuilt/checkout-multi-service-outage-kind stand up kind + ingress-nginx (ClusterIP) and seed the broken fixture during tofu apply, via a null_resource running scripts/setup.sh. - Repoint both task.yaml files at their dedicated stacks and drop the now-unused infrastructure.variables blocks. - Move each fixture under tf/prebuilt/<task>-kind/manifests/ so the per-run tofu isolation copy includes it; the --no-infra seed hooks read the same file, keeping a single source of truth. Flip isorun's default from --no-infra to standard provisioning: with no flag, run.sh provisions during bringup and tears down at the end. --no-infra is now the opt-in fast loop (cleanup/seed/preflight and the context assertion run only in that mode); --infra is accepted as a no-op alias. Fix an empty-array expansion that crashed on macOS system bash 3.2 on the new default path.
geojaz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The thesis behind this branch: can precisely-specified objectives and safeguards trip up an agent that a subjective rubric would let slide? To test it, this PR adds a deterministic verification layer to devops-bench and uses it to author tasks graded on the outcome, not the narration.
Until now, tasks were scored almost entirely by an LLM judge. This PR adds a typed
verification_entriesschema so a task can assert exact, machine-checked outcomes against live cluster state, in-cluster and off-cluster HTTP, and committed GitOps repo state, with the judge demoted to the non-gating subjective residue. On top of that it shipsscripts/isorun, an isolated fast-iteration harness, and two new agent-stumping tasks.What's in it
Verification schema v1 (the core)
VerificationEntryandTask.verification_entriesfield, wired through the eval harness as an unconditional (non-chaos-gated) evaluation step.c(objective coverage),rec_v(recoverable safeguards),cat_v(catastrophic safeguards).results.json.Verifiers
resource_property: JSONPath over live Kubernetes objects, with quantity-aware numeric comparisons (150m, 192Mi, 1Gi, etc.).http_probe: in-cluster HTTP via a one-shot curl pod, polled until success or deadline.external_http_probe: off-cluster reachability. Discovers a Service/Ingress external LoadBalancer address and GETs it from the verifier host, proving traffic transited the external network (mechanism-agnostic across LoadBalancer/Ingress).git_repo_sync: grades a host-side bare GitOps repo at a ref without cloning.Tasks
deploy-hello-apponboarded to objective/safeguard verification, with an HTTP-server oracle image.verification_entries: fix-config, deploy-config, optimize-scale, spot-rebalancing, lustre-csi-deployment, cve-remediation, opa-remediation, migration-and-upgrade.isorun harness (
scripts/isorun/)solutions/,task.yaml, or the verifier code. Vertex ADC auth (no API keys), gemini or openclaw.python -m devops_benchrun. Under--no-infrait instead attaches to a standing cluster and runs a cleanup + seed + preflight fast-iteration loop: seed applies the broken fixture with kubectl, preflight asserts it's still in the exact broken pre-state the checks grade, and the run fails closed if it isn't.Terraform
tf/modules/ingress-nginxmodule. Two dedicated per-task kind stacks (prebuilt/ledger-read-facade-kind,prebuilt/checkout-multi-service-outage-kind) stand up kind + ingress-nginx (ClusterIP) and seed the broken fixture duringtofu apply, so the two tasks provision already-broken with no manual setup.prebuilt/kindalso gains a reusableinstall_ingress_nginxgate. Plus a hypercomputer-d1 output fix.Tests + docs
Starter commands
All three run through isorun (
scripts/isorun/run.sh <task.yaml> [gemini|oc]), which wraps the same provision → agent → verify → teardown pipeline as a standardpython -m devops_benchrun. The difference is convenience: it runs the agent from an isolated scratch workspace, so it can't reachsolutions/,task.yaml, or the verifier code, and it handles the environment for you: Vertex ADC auth, project/cluster/region from yourgcloudconfig or env, model and timeout defaults, and per-cluster locking so parallel runs don't collide. One command, zero manual setup.Common setup
gemini/gemini-3.5-flash; judge togemini-3.1-pro-preview; per-agent timeout 1800s. Override withAGENT_MODEL,JUDGE_MODEL,AGENT_TIMEOUT_SEC. Passocas the second arg to run openclaw instead.ISORUN_DRYRUN=1to print the exact command without touching a cluster.brew install flockfor cross-run locking (optional; isorun warns loudly if it's absent).ledger-read-facade (id 23)
Kind-backed. Prereqs: Docker running (kind needs it), plus
tofuandkubectlon PATH.Cold-provisions a kind cluster + ingress-nginx (ClusterIP), seeds the broken fixture via
tofu, runs the agent against the already-brokenledgernamespace, grades theverification_entries, and tears the cluster down. No standing cluster and no kubectl context juggling. Results land inresults/iso-ledger-read-facade/; read the deterministic + judge rollup printed at the end.checkout-multi-service-outage (id 24)
Kind-backed. Same prereqs: Docker running, plus
tofuandkubectlon PATH.Cold-provisions a kind cluster + ingress-nginx (ClusterIP), seeds the broken fixture via
tofu, runs the agent against the already-brokencheckoutnamespace, grades theverification_entries, and tears the cluster down. Results land inresults/iso-checkout-multi-service-outage/; read the deterministic + judge rollup printed at the end.deploy-hello-app (id 6)
GKE-backed. This task has no local fixture: the agent productionizes and deploys a hello-world app onto a real cluster, and part of the score is an off-cluster LoadBalancer probe, so kind won't do (it has no external LoadBalancer). Cold-provision a fresh cluster the same one-liner way as the kind tasks (it builds a real GKE cluster via the
prebuilt/minimumstack, then tears it down):The run also creates the Artifact Registry repo
hello-app-<cluster>(the agent builds and pushes its image there) and sweeps it at teardown, so neither the cluster nor the registry has to pre-exist. The project just needs the GKE, Artifact Registry, and Compute APIs enabled, IAM and quota to create those resources, Vertex access, and network egress from the verifier host to the LoadBalancer's external address for the off-cluster probe.