feat(verification): typed verification_spec with roles, modes, and score rollup

[geojaz]

feat: verification_spec schema with roles, modes, and rollup

What this renovates and what is preserved unchanged

Renovates: Task.verification_spec moves from Any to list[VerificationEntry] | None.
Each entry carries a name, role (correctness, safety, catastrophic), and a raw spec
dict. BaseVerifier gains weight, mode, and hold-window fields. VerificationResult gains
weight. run_entry() returns a typed EvaluatedEntry that bundles role, name, and result
so there is no second parallel sequence to misalign; rollup() accepts list[EvaluatedEntry]
and produces (c, rec_v, cat_v) scores. Two new verifiers land: ResourcePropertyVerifier
and HttpProbeVerifier.

Preserved unchanged:

• BaseVerifier.verify() signature is untouched.
• pod_healthy.py and scaling_complete.py are untouched.
• optimize-scale/task.yaml parses without any authoring change (regression test included).
• The legacy raw-dict path in _build_verification_mapping is still the fallback for tests
  and any task that predates this schema.
• No existing verifier behavior changed.

The mode insight

Three of the four modes are scheduling strategies layered over the existing verify() contract,
not changes to it:

• converge: poll verify(N) until success or timeout (existing behavior, now named).
• assert: call verify(0) exactly once; poll_until evaluates the predicate before checking
  the deadline so timeout_sec=0 means one shot, no sleep.
• hold: sample verify(0) repeatedly across a window; any failing sample fails the check.

unchanged is designed but intentionally not built. It needs a baseline snapshot and a
diff strategy that depends on verifier type. A ValidationError is raised at task-load
time when any spec contains mode: unchanged (the VerificationEntry validator walks
the raw spec dict recursively); the NotImplementedError in _run_leaf is kept as
defense-in-depth.

What is real in this PR vs. proposed next

Real (in this PR):

• VerificationEntry model with Role literal
• Task.verification_spec: list[VerificationEntry] | None
• BaseVerifier.mode, weight, hold_window_sec, hold_interval_sec
• VerificationResult.weight
• Mode dispatch in VerificationRunner.run_entry(): converge, assert, hold
• run_entry() returns EvaluatedEntry (role + name + result in one object)
• rollup(list[EvaluatedEntry]) with RollupScores; ordering-safe by construction
• ResourcePropertyVerifier: kubectl get -o json, dotted path eval, quantifiers
• HttpProbeVerifier: kubectl run --rm curl pod, status + body matching
• Full unit test coverage for all of the above

Proposed next (not in this PR):

• Wire rollup() output into the judge/metrics layer.
• run_entry() called from the harness (today the harness still uses wait_for_condition).
• unchanged mode implementation.
• Additional probe verifiers (see vocabulary roadmap below).

Vocabulary roadmap

Probes that would be cheap to add under the same BaseVerifier contract:

• dns_probe: resolve a hostname from inside the cluster
• tcp_probe: TCP connect to host:port
• env_probe: check env var value inside a running container (kubectl exec)
• file_probe: check file contents inside a container
• log_probe: grep recent pod logs for a pattern
• can_i: kubectl auth can-i for RBAC verification
• cert_probe: TLS cert expiry / SAN check
• unchanged_outside: a scoped unchanged mode targeting resources outside a namespace
• cloud_resource_property: same as resource_property but for GCP/AWS/Azure resources
  via their respective APIs
• policy_compliance: OPA/Gatekeeper policy evaluation result
• forbidden_action: assert a named action was NOT taken (audit log or webhook check)

exec_probe (arbitrary shell inside a container) is a last resort. It works but it couples
verification logic to image contents and makes tasks harder to port.

[gemini-code-assist]

Code Review

This pull request refactors the verification harness by introducing a typed schema with VerificationEntry, execution modes (converge, assert, hold), and a scoring rollup mechanism. It also adds two new verifiers: HttpProbeVerifier and ResourcePropertyVerifier. The review feedback highlights several critical issues: HttpProbeVerifier needs the -i flag in kubectl run to capture stdout; the rollup scoring requires a guard against division by zero; the hold-mode logic in VerifierAgent needs to correctly handle 0.0 values for windows and intervals; and the regex matching in ResourcePropertyVerifier needs to handle None values safely to avoid false positives.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[geojaz]

Superseded by the split into two smaller, reviewable PRs: #3 (schema foundation, behavior-neutral) and #4 (verifier vocabulary + rollup, stacked on the foundation). Same code, reviewed in two chunks. Closing this combined PR in favor of those.