MVP: task and verification schemas for outcome-graded evals - #8
Draft
geojaz wants to merge 8 commits into
Draft
Conversation
…ispatch, and rollup Add the core deterministic verification framework: - VerificationEntry type and Task.verification_entries on the task schema - resource_property verifier with JSONPath (jsonpath-ng) and quantity-aware comparisons - http_probe verifier over one-shot curl pod with polling and robust status parsing - run_pod ephemeral pod helper in devops_bench.k8s - all/any/none combinators and quantifier dispatch - per-entry c/rec_v/cat_v rollup - per-entry mode dispatch (converge/assert/hold) via run_entry
Wire deterministic verification into the eval harness: - Evaluate verification_entries and emit c/rec_v/cat_v rollup metrics - Print a combined deterministic + judge run summary in the CLI
…d verification entries - Onboard deploy-hello-app to objective/safeguard verification entries - Add HTTP-server oracle image (Go) and oracle.yaml manifest - Add oracle validation runbook in docs/appendix - Add external_http_probe verifier for off-cluster probe grading - Fix namespace-scoped discovery for external_http_probe
Add a verifier that checks whether a Git repository has been synced to the expected state (branch, commit, or tag).
Add verification_entries to: cve-remediation, migration-and-upgrade, opa-remediation, optimize-scale, spot-rebalancing, deploy-config, fix-config, lustre-csi-deployment
… with kind stacks - Add reusable ingress-nginx Terraform module - Install ingress-nginx in prebuilt/kind behind a gate variable - Add ledger-read-facade-kind and checkout-multi-service-outage-kind prebuilt stacks with fixture seeding via setup.sh - Add ledger-read-facade task (id 23) and checkout-multi-service-outage task (id 24) with deterministic verification entries
- Update docs/components/infra.md for ingress-nginx module and new stacks - Link deploy-hello-app oracle validation runbook from docs index - Update known_issues.md - Fix hypercomputer-d1 cluster outputs
Autoformat deployers/, pkg/, scripts/, and tests/ with ruff to fix whitespace, trailing-newline, and indentation style.
geojaz
force-pushed
the
ehole/task-verification-schemas-mvp
branch
from
July 28, 2026 05:25
2a50c18 to
ce61479
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds a deterministic verification layer to devops-bench so tasks can assert exact, machine-checked outcomes against live cluster state — not just LLM-judge narration.
Until now, tasks were scored almost entirely by an LLM judge. This PR adds a typed
verification_entriesschema so a task can specify precise pass/fail conditions checked programmatically against Kubernetes resources, in-cluster and off-cluster HTTP endpoints, and GitOps repo state, with the judge demoted to non-gating subjective residue.Verification framework
VerificationEntryandTask.verification_entriesfield, wired through the eval harness as an unconditional evaluation step.c(objective coverage),rec_v(recoverable safeguards),cat_v(catastrophic safeguards).Verifiers
resource_property: JSONPath over live Kubernetes objects with quantity-aware numeric comparisons (150m, 192Mi, 1Gi, etc.).http_probe: in-cluster HTTP via a one-shot curl pod, polled until success or deadline.external_http_probe: off-cluster reachability — discovers a Service/Ingress external address and GETs it from the verifier host.git_repo_sync: grades a host-side bare GitOps repo at a ref without cloning.Tasks
verification_entries: fix-config, deploy-config, optimize-scale, spot-rebalancing, lustre-csi-deployment, cve-remediation, opa-remediation, migration-and-upgrade.Terraform
tf/modules/ingress-nginxmodule.prebuilt/ledger-read-facade-kind,prebuilt/checkout-multi-service-outage-kind) that seed the broken fixture duringtofu apply.prebuilt/kindgains a reusableinstall_ingress_nginxgate.Tests + formatting