Skip to content

Commit d3e6a1b

Browse files
feat(check): build inline simulation-run payloads and add a dry-run CLI
`npm run check -- <check>|--all --dry-run` reads vapi-checks.yml and, for each check target, builds the inline POST /eval/simulation/run body from the files on disk: the assistant or squad with its tools, handoffs and structured outputs, plus the scenarios, judges and personalities the check's suites and simulations name. Nothing is looked up on the platform and no key is needed; --print-payload writes the JSON. The builder follows push and the runtime so the check tests what push deploys: - tools in runtime order (model.tools, then toolIds, then toolRefs); knowledgeBase tools stay in toolIds by run-org UUID; - `##` comments stripped; UUID references resolved through state; - squad members inlined, handoffs to members by name; - hook toolIds, artifactPlan structuredOutputIds and linked structured outputs inlined; judges' structuredOutputId inlined; - credentials bound by name to the run org (promotion's bindings). Anything it can't place fails the build with the field named: missing files, handoffs leaving the target, legacy assistantDestinations by ID, tools by ID inside overrides (strict mocks), duplicate tool names, audio judges/hooks/no required judge over chat, leftover non-UUID references, and payloads over 4.5 MB. The tool mock policy and live runs follow in later changes; without --dry-run the command exits 2. Refs TEST-141 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 58e2d36 commit d3e6a1b

31 files changed

Lines changed: 2132 additions & 0 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ If setup reports the org is "already set up locally", do not delete anything to
9090
| Push with new resources | `npm run push -- <org> --allow-new-files` — bypass orphan-YAML gate. **AI agents**: do NOT auto-pass this flag; confirm with the human first (see push section below) |
9191
| Test a call | `npm run call -- <org> -a <assistant-name>` or `-s <squad-name>` |
9292
| Run a simulation suite | `npm run sim -- <org> --suite <name> --target <assistant-name>` |
93+
| Build PR-check payloads offline | `npm run check -- <check> --dry-run` (or `--all`); `--print-payload` writes the JSON |
9394
| Migrate a legacy state file | `npm run migrate` — one-shot, all orgs; required once after upgrading to the hash-store engine |
9495

9596
---
@@ -903,6 +904,7 @@ npm run validate -- <org> # Lint resources locally (fai
903904
npm run audit -- <org> # Read-only drift detector: orphan YAML, state ghosts, content-identical clusters, sibling base-slugs, dashboard orphans, inline model.tools. Exit 1 on findings.
904905
npm run audit -- <org> --type assistants # Scope audit to a single resource type
905906
npm run sim -- <org> --suite <name> --target <name> # Run a simulation suite against an assistant/squad (exit 0 pass, 1 fail, 3 incomplete; --timeout <min>)
907+
npm run check -- <check> --dry-run # Build a vapi-checks.yml check's inline run payload offline (exit 0 built, 2 config/build error; --print-payload [dir])
906908
npm run rollback -- <org> --to <ISO-timestamp> # Re-apply a snapshot taken before a push
907909
npm run rollback -- <org> --list # List available snapshots
908910

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ Every command works in two modes:
104104
| `npm run rollback` | — | `npm run rollback -- <org> --list` or `--to <ISO>` | Restore from a snapshot in `.vapi-state.<org>.snapshots/` (one is written before every push/apply). |
105105
| `npm run call` | ✅ | `npm run call -- <org> -a <name>` or `-s <squad>` | Start an interactive WebSocket call against an assistant or squad. |
106106
| `npm run sim` | — | `npm run sim -- <org> --suite <name> --target <name> [--timeout <min>]` | Run a simulation suite (or specific simulations) against a deployed assistant/squad. Prints the run link; exits 0 passed, 1 failed, 3 incomplete (timeout, Ctrl-C, missing results). |
107+
| `npm run check` | — | `npm run check -- <check>\|--all --dry-run [--print-payload [dir]]` | Build each `vapi-checks.yml` check target's inline simulation-run payload from the files on disk, with its tools, handoffs, judges and personalities. Offline: no API key, nothing sent. `--print-payload` writes the JSON (default `tmp/check-payloads/`). Exits 0 when every payload builds, 2 on a config or build error. |
107108
| `npm run migrate` | — | `npm run migrate` | One-time, all orgs at once: slim legacy state files to pure `name → uuid` and seed the per-developer `.vapi-state-hash/` baseline store from the old hashes. Required once after upgrading to the hash-store engine — `pull`/`push`/`apply` refuse legacy-shaped state until it runs. Idempotent. |
108109
| `npm run build` | — | — | Type-check the codebase (`tsc --noEmit`). |
109110
| `npm test` | — | — | Run regression tests (`node:test`). |

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
"validate": "tsx src/validate-cmd.ts",
2020
"audit": "tsx src/audit-cmd.ts",
2121
"sim": "tsx src/sim-cmd.ts",
22+
"check": "tsx src/check-cmd.ts",
2223
"rollback": "tsx src/rollback-cmd.ts",
2324
"promote": "tsx src/promote-cmd.ts",
2425
"build": "tsc --noEmit",

‎src/check-cmd.ts‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
// CLI entry: `npm run check -- <check>|--all --dry-run [--print-payload [dir]]`
2+
//
3+
// Builds each check target's inline simulation-run payload from the files
4+
// on disk (see check-payload.ts) and reports what would run. Config-free:
5+
// it needs no API key and makes no network calls in --dry-run.
6+
//
7+
// Exit codes: 0 every payload built, 2 usage, config or build error.
8+
9+
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
10+
import { join, relative, resolve } from "node:path";
11+
import { fileURLToPath } from "node:url";
12+
import type { CheckDefinition } from "./check-config.ts";
13+
import { CHECKS_CONFIG_FILE, checksConfigLoad } from "./check-config.ts";
14+
import type { CheckPayloadResult } from "./check-payload.ts";
15+
import { checkPayloadBuild } from "./check-payload.ts";
16+
import { promotionBindingsResolve, promotionStateParse } from "./promotion.ts";
17+
import { orgResourcesRead } from "./resource-parse.ts";
18+
import type { StateFile } from "./types.ts";
19+
20+
const USAGE = [
21+
"Usage:",
22+
" npm run check -- <check> --dry-run [--print-payload [dir]]",
23+
" npm run check -- --all --dry-run [--print-payload [dir]]",
24+
"",
25+
"Options:",
26+
` <check> A check name from ${CHECKS_CONFIG_FILE}`,
27+
" --all Every check",
28+
" --dry-run Build the payloads offline; nothing is sent",
29+
" --print-payload [dir] Write each payload as JSON (default: tmp/check-payloads)",
30+
"",
31+
"Exit codes: 0 payloads built, 2 usage, config or build error",
32+
].join("\n");
33+
34+
const DEFAULT_PAYLOAD_DIR = "tmp/check-payloads";
35+
36+
interface CheckArgs {
37+
check?: string;
38+
all: boolean;
39+
dryRun: boolean;
40+
payloadDir?: string;
41+
help: boolean;
42+
}
43+
44+
class UsageError extends Error {}
45+
46+
function argsParse(args: string[]): CheckArgs {
47+
const parsed: CheckArgs = { all: false, dryRun: false, help: false };
48+
for (let index = 0; index < args.length; index++) {
49+
const arg = args[index]!;
50+
if (arg === "--help" || arg === "-h") parsed.help = true;
51+
else if (arg === "--all") parsed.all = true;
52+
else if (arg === "--dry-run") parsed.dryRun = true;
53+
else if (arg === "--print-payload") {
54+
const next = args[index + 1];
55+
parsed.payloadDir =
56+
next && !next.startsWith("--") ? args[++index] : DEFAULT_PAYLOAD_DIR;
57+
} else if (arg.startsWith("-"))
58+
throw new UsageError(`Unknown option: ${arg}`);
59+
else if (parsed.check) throw new UsageError(`Unexpected argument: ${arg}`);
60+
else parsed.check = arg;
61+
}
62+
if (parsed.help) return parsed;
63+
if (parsed.all === Boolean(parsed.check))
64+
throw new UsageError("Name one check, or pass --all");
65+
return parsed;
66+
}
67+
68+
function emptyState(): StateFile {
69+
return promotionStateParse("{}");
70+
}
71+
72+
function stateRead(root: string, org: string, warnings: string[]): StateFile {
73+
const path = join(root, `.vapi-state.${org}.json`);
74+
if (!existsSync(path)) {
75+
warnings.push(
76+
`no .vapi-state.${org}.json: references by UUID and credential bindings can't resolve`,
77+
);
78+
return emptyState();
79+
}
80+
return promotionStateParse(readFileSync(path, "utf8"));
81+
}
82+
83+
function payloadFileName(check: string, target: string): string {
84+
return `${check}--${target.replace(/\//g, "--")}.json`;
85+
}
86+
87+
function resultPrint(
88+
label: string,
89+
result: CheckPayloadResult,
90+
check: CheckDefinition,
91+
): void {
92+
console.log(`\n${label}`);
93+
if (result.body) {
94+
const count = result.body.simulations.length;
95+
console.log(
96+
` ✅ ${count} simulation${count === 1 ? "" : "s"} × ${check.iterations} iteration${check.iterations === 1 ? "" : "s"} over ${check.transport}, ${(result.bytes / 1024).toFixed(1)} KB`,
97+
);
98+
} else {
99+
console.log(
100+
` ❌ ${result.errors.length} problem${result.errors.length === 1 ? "" : "s"}`,
101+
);
102+
}
103+
for (const warning of result.warnings) console.log(` ⚠️ ${warning}`);
104+
for (const error of result.errors) console.log(` ❌ ${error}`);
105+
}
106+
107+
async function checkDryRun(
108+
root: string,
109+
check: CheckDefinition,
110+
payloadDir: string | undefined,
111+
): Promise<boolean> {
112+
if (!existsSync(join(root, "resources", check.org))) {
113+
console.log(`\n${check.name}\n ❌ resources/${check.org}/ does not exist`);
114+
return false;
115+
}
116+
const warnings: string[] = [];
117+
const resources = await orgResourcesRead(root, check.org);
118+
const sourceState = stateRead(root, check.org, warnings);
119+
const runState =
120+
check.runOrg === check.org
121+
? sourceState
122+
: stateRead(root, check.runOrg, warnings);
123+
const bindingsResolved = await promotionBindingsResolve(
124+
root,
125+
check.org,
126+
check.runOrg,
127+
sourceState,
128+
);
129+
let ok = true;
130+
for (const target of check.targets) {
131+
const targetLabel = `${target.type}/${target.id}`;
132+
const result = checkPayloadBuild({
133+
check,
134+
target,
135+
resources,
136+
sourceState,
137+
runState,
138+
bindingsResolved,
139+
});
140+
result.warnings.unshift(...warnings);
141+
resultPrint(`${check.name} / ${targetLabel}`, result, check);
142+
if (!result.body) {
143+
ok = false;
144+
continue;
145+
}
146+
if (payloadDir) {
147+
const dir = resolve(root, payloadDir);
148+
mkdirSync(dir, { recursive: true });
149+
const file = join(dir, payloadFileName(check.name, targetLabel));
150+
writeFileSync(file, `${JSON.stringify(result.body, null, 2)}\n`);
151+
console.log(` 📝 ${relative(root, file)}`);
152+
}
153+
}
154+
return ok;
155+
}
156+
157+
export async function checkCommandRun(
158+
args = process.argv.slice(2),
159+
root = resolve(
160+
process.env.VAPI_GITOPS_ROOT ??
161+
fileURLToPath(new URL("..", import.meta.url)),
162+
),
163+
): Promise<number> {
164+
let parsed: CheckArgs;
165+
try {
166+
parsed = argsParse(args);
167+
} catch (error) {
168+
if (!(error instanceof UsageError)) throw error;
169+
console.error(`❌ ${error.message}\n\n${USAGE}`);
170+
return 2;
171+
}
172+
if (parsed.help) {
173+
console.log(USAGE);
174+
return 0;
175+
}
176+
let config;
177+
try {
178+
config = checksConfigLoad(root);
179+
} catch (error) {
180+
console.error(`❌ ${CHECKS_CONFIG_FILE}: ${(error as Error).message}`);
181+
return 2;
182+
}
183+
if (!config) {
184+
console.log(
185+
`No ${CHECKS_CONFIG_FILE} at the repository root; nothing to check.`,
186+
);
187+
return 0;
188+
}
189+
const names = parsed.all ? Object.keys(config.checks) : [parsed.check!];
190+
const missing = names.filter((name) => !config.checks[name]);
191+
if (missing.length > 0) {
192+
console.error(
193+
`❌ No check named ${missing.join(", ")} in ${CHECKS_CONFIG_FILE} (checks: ${Object.keys(config.checks).join(", ")})`,
194+
);
195+
return 2;
196+
}
197+
if (!parsed.dryRun) {
198+
console.error(
199+
"❌ Live runs aren't available yet: pass --dry-run to build the payloads offline.",
200+
);
201+
return 2;
202+
}
203+
let ok = true;
204+
for (const name of names) {
205+
if (!(await checkDryRun(root, config.checks[name]!, parsed.payloadDir)))
206+
ok = false;
207+
}
208+
console.log(
209+
ok ? "\n✅ Every payload built." : "\n❌ Some payloads could not be built.",
210+
);
211+
return ok ? 0 : 2;
212+
}
213+
214+
const isMainModule =
215+
resolve(process.argv[1] ?? "") === resolve(fileURLToPath(import.meta.url));
216+
if (isMainModule) {
217+
checkCommandRun().then(
218+
(code) => process.exit(code),
219+
(error: unknown) => {
220+
console.error(
221+
`❌ Check failed: ${error instanceof Error ? error.message : String(error)}`,
222+
);
223+
process.exit(2);
224+
},
225+
);
226+
}

0 commit comments

Comments
 (0)