You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8a438bc
Browse filesBrowse the repository at this point in the historyBrowse files
fix(sim): report failed, canceled and incomplete runs instead of a false pass
`npm run sim` read a `results` field the simulation-run API doesn't
return and counted `status === "pass"` (items are `passed`/`failed`), so
every watched run summarised as 0/0 and exited 0, including failing
runs.
- src/sim-result.ts: strict, pure verdict. Passed only when the run
ended, every expected item exists and passed, and each had a required
evaluation that was actually scored; otherwise failed or incomplete
with a reason.
- src/sim.ts: read run items (paginated or bare array, deduped by id),
wait for late item results, print the run link from the create
response and the failing judges, cancel the run on --timeout (default
20 min) or Ctrl-C.
- src/vapi-client.ts: config-free client; run creation is never retried
on a 5xx, since the run may already be queued.
- src/sim-cmd.ts: exported simCommandRun; exit 0 passed, 1 failed,
2 usage, 3 incomplete; --no-watch prints the link and exits 0.
- Docs: run/item shapes and the verdict in docs/learnings/simulations.md,
sim rows in README and AGENTS.md, improvements.md #33.
Refs TEST-141
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: AGENTS.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -902,7 +902,7 @@ npm run promote -- --pipeline <name> --from <source> --to <target> --apply # For
902
902
npm run validate -- <org> # Lint resources locally (fails fast on schema drift)
903
903
npm run audit -- <org> # Read-only drift detector: orphan YAML, state ghosts, content-identical clusters, sibling base-slugs, dashboard orphans, inline model.tools. Exit 1 on findings.
904
904
npm run audit -- <org> --type assistants # Scope audit to a single resource type
905
-
npm run sim -- <org> --suite <name> --target <name> # Run a simulation suite against an assistant/squad
905
+
npm run sim -- <org> --suite <name> --target <name> # Run a simulation suite against an assistant/squad (exit 0 pass, 1 fail, 3 incomplete; --timeout <min>)
906
906
npm run rollback -- <org> --to <ISO-timestamp> # Re-apply a snapshot taken before a push
907
907
npm run rollback -- <org> --list # List available snapshots
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -103,7 +103,7 @@ Every command works in two modes:
103
103
|`npm run cleanup`| ✅ |`npm run cleanup -- <org> [--force --confirm <org>]`| Inspect (default) or delete orphaned remote resources. Destructive run requires `--confirm <org>`. |
104
104
|`npm run rollback`| — |`npm run rollback -- <org> --list` or `--to <ISO>`| Restore from a snapshot in `.vapi-state.<org>.snapshots/` (one is written before every push/apply). |
105
105
|`npm run call`| ✅ |`npm run call -- <org> -a <name>` or `-s <squad>`| Start an interactive WebSocket call against an assistant or squad. |
106
-
|`npm run sim`| — |`npm run sim -- <org> --suite <name> --target <name>`| Run a simulation suite (or specific simulations) against a deployed assistant/squad. |
106
+
|`npm run sim`| — |`npm run sim -- <org> --suite <name> --target <name> [--timeout <min>]`| Run a simulation suite (or specific simulations) against a deployed assistant/squad. Prints the run link; exits 0 passed, 1 failed, 3 incomplete (timeout, Ctrl-C, missing results). |
107
107
|`npm run migrate`| — |`npm run migrate`| One-time, all orgs at once: slim legacy state files to pure `name → uuid` and seed the per-developer `.vapi-state-hash/` baseline store from the old hashes. Required once after upgrading to the hash-store engine — `pull`/`push`/`apply` refuse legacy-shaped state until it runs. Idempotent. |
108
108
|`npm run build`| — | — | Type-check the codebase (`tsc --noEmit`). |
Copy file name to clipboardExpand all lines: docs/learnings/simulations.md
+20-1Lines changed: 20 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -471,6 +471,23 @@ This is the unified executor. A "run" is a batch — it expands into many `runIt
471
471
}
472
472
```
473
473
474
+
The **create** response (`CreateSimulationRunResponse`) also carries two fields nothing else returns:
475
+
476
+
-`url` — the dashboard link to this run. `GET /eval/simulation/run/:id` does **not** return it, so read it from the create response.
477
+
-`simulationRunItemIds` — one id per item queued (simulations × iterations), i.e. how many items the run should end with.
478
+
479
+
A run has **no `results` field**. Pass/fail lives in `itemCounts` and on the items themselves (`GET …/item`, below). Creating a run queues paid work before the response returns, so a 5xx on create may still have started it — don't blindly retry the POST.
480
+
481
+
### How `npm run sim` decides pass/fail
482
+
483
+
`src/sim-result.ts` (`simRunVerdict`) reports **passed** only when all of these hold; otherwise **failed** (any failed item) or **incomplete**:
484
+
485
+
- the run `ended` and has `itemCounts`, with `total` equal to the number of items created (`simulationRunItemIds`) and greater than 0;
486
+
- nothing is `queued`, `running` or `canceled`, and `passed === total`;
487
+
- every item was fetched, is `passed`, and has at least one **required** evaluation that wasn't skipped (the all-skipped trap from the chat-mode gotcha above).
488
+
489
+
Exit codes: 0 passed, 1 failed, 2 usage error, 3 incomplete (timeout, Ctrl-C, canceled items, results that never arrived). On timeout or Ctrl-C the run is canceled.
490
+
474
491
### List runs — `GET /eval/simulation/run`
475
492
476
493
Query params:
@@ -486,7 +503,7 @@ Query params:
486
503
### Get / Cancel run
487
504
488
505
-`GET /eval/simulation/run/:id` → `SimulationRun`
489
-
-`PATCH /eval/simulation/run/:id` → cancels the run **and** all its queued items. No body required.
506
+
-`PATCH /eval/simulation/run/:id` → cancels the run **and** all its queued items. No body required. Returns 400 `Run has already ended` for an ended run and 409 when a concurrent cancel wins; both mean "nothing left to cancel".
490
507
491
508
---
492
509
@@ -498,6 +515,8 @@ Run items are system-managed — there's no create/update API for users; they're
Response shape depends on the query: **with**`limit` or `page` it's paginated (`{ results, metadata: { totalItems, itemsPerPage, currentPage } }`, `limit` up to 1000); **without** them it's a bare array. Pages are ordered only by creation time and a run's items share it, so OFFSET pages can overlap — dedupe by `id`. Items can also lag the run: a run can be `ended` while an item is `passed` but its `results` aren't written yet.
519
+
501
520
### Get a run item — `GET /eval/simulation/run/:id/item/:itemId`
502
521
503
522
**Response** — `SimulationRunItem` is rich; the highlights:
Copy file name to clipboardExpand all lines: improvements.md
+52Lines changed: 52 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -84,6 +84,7 @@ you which stack PR closes the row.**
84
84
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
85
85
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Open |
86
86
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |
87
+
| 33 |`npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
87
88
88
89
**Active backlog after cleanup:**`#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
89
90
@@ -1721,6 +1722,57 @@ fields.
1721
1722
1722
1723
---
1723
1724
1725
+
## 33. `npm run sim` reported every run as passed
1726
+
1727
+
**[RESOLVED 2026-10-01]**
1728
+
1729
+
**Discovered:** 2026-10-01, while designing simulation PR checks (TEST-141).
1730
+
1731
+
### Problem
1732
+
1733
+
`npm run sim` scored a run by reading `results[]` on the run and counting
1734
+
`status === "pass"`. The simulation-run API has no `results` field and
1735
+
item statuses are `passed` / `failed`, so every watched run summarised
1736
+
as 0 pass / 0 fail and the command exited 0 — including runs whose
1737
+
simulations failed.
1738
+
1739
+
### Current behavior (Verified, before the fix)
1740
+
1741
+
-`src/sim.ts``runSimulation` computed pass/fail from `last.results`,
1742
+
which `GET /eval/simulation/run/:id` never returns; pass/fail is in
1743
+
`itemCounts` and on the run items (`GET /eval/simulation/run/:id/item`).
1744
+
-`src/sim-cmd.ts` exited 1 only when `fail > 0`, which could never happen.
1745
+
- Polling also stopped on statuses that don't exist (`failed`,
1746
+
`completed`), never canceled a timed-out run, and didn't print the run
1747
+
link (only the create response carries `url`).
1748
+
1749
+
### Risk
1750
+
1751
+
Anyone gating on `npm run sim` (locally or in CI) got a green result for a
1752
+
failing suite.
1753
+
1754
+
### Current mitigation
1755
+
1756
+
None needed once the fix below lands.
1757
+
1758
+
### Possible fix (landed)
1759
+
1760
+
-`src/sim-result.ts``simRunVerdict`: passed only when the run ended,
1761
+
every expected item exists, passed, and had a required evaluation that was
1762
+
actually scored; otherwise failed or incomplete with a reason.
1763
+
-`src/sim.ts` reads items (paginated or bare-array, deduped by id),
1764
+
waits for late item results, prints the run link and failing judges, and
1765
+
cancels the run on `--timeout` (default 20 min) or Ctrl-C.
1766
+
-`src/vapi-client.ts`: a config-free client that never retries run
1767
+
creation on a 5xx (the run may already be queued).
0 commit comments