ci: add harness integration CI for the four agent CLIs

[omkargaikwad23]

Summary

Adds .ci/ — a Cloud Build pipeline running integration smoke tests for all four agent CLIs (agy, Claude Code, Codex, Gemini) against the real Cloud SQL Admin MCP endpoint. Existing coverage relies on unit tests with mocked subprocesses, which cannot catch CLI flag changes, self-updates, or output format shifts.

Key Design Choices

• Shared Evalset & Run Config: Standardizes on a single evalset canonicalized to <server>__<tool> via tool_naming.py, and a parameterized run config keyed by CI_HARNESS so results remain comparable across harnesses.
• Two Read-Only Scenarios: A list-then-get query, and a get query against a non-existent instance (zero provisioning and no leaked test infrastructure on failure).
• Parallel Execution: Harness build steps execute concurrently with allowFailure so failures in one CLI do not cancel the others.
• Authentication: agy uses a service-account key fetched from Secret Manager to carry quota_project_id; the other three CLIs authenticate via build credentials.
• Unpinned CLI Versions: The three npm-installed CLIs track @latest and agy self-updates, so the build surfaces version drift as it ships rather than when a pin is bumped.
• Scope: Real MCP servers only (skill/plugin installation smoke tests will follow in a separate PR).

Verification Gate (verify_harness.py)

Because evalbench.eval() exits 0 on run completion, verify_harness.py enforces a two-tier evaluation gate:

Tier	Scorers	Rule
Non-zero	7 telemetry scorers + trajectory_matcher	> 0
Liveness	3 LLM judges	row exists, no error, numeric

• Why non-zero for telemetry/trajectory? Telemetry scorers swallow parse errors and return 0.0 inside comparison_logs instead of raising an error. Gating trajectory_matcher > 0 verifies that expected MCP tools were actually invoked rather than falling back to shell execution.
• Why liveness only for judges? Judges are checked for execution health only; non-deterministic judge verdicts are never hard-gated to avoid pipeline flakiness.

---

Test Plan

• Tested verify_harness.py against synthetic results (healthy passes, Tier 1 at 0 fails, judges at 0 pass).
• Verified evalset placeholder expansion (unset variables raise errors; no literal ${...} reaches agent).
• Verified shared run config resolution across all four harnesses (correct model config, generator, and output paths).
• Triggered real Cloud Build run (verify-harnesses) and confirmed execution across all harnesses.

[omkargaikwad23]

/gcbrun

[omkargaikwad23]

/gcbrun

[omkargaikwad23]

/gcbrun

[omkargaikwad23]

/gcbrun

[prernakakkar-google]

/gcbrun

[prernakakkar-google]

Lets write a design document on this first.
We should use different GCP Project as its a security risk since data is visble publically

[prernakakkar-google]

check against latest

[prernakakkar-google]

we should test for skills as well

[omkargaikwad23]

That would be covered in followup CL (I have already added a note in PR description)

[omkargaikwad23]

Lets write a design document on this first. We should use different GCP Project as its a security risk since data is visble publically

Thanks for reviewing the changes!
External contributors won't be able to view the build logs or access or run the trigger. Same GCP project is used for current test trigger and for crema evals as well. To keep triggers in same gcp project, I decided to use the same project. Will share the doc soon.

[omkargaikwad23]

/gcbrun