Compare Workshop evals on pull requests - #476
AshishKumar4 wants to merge 1 commit into
Conversation
Preview:
|
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
79348b1 to
f3b1ef6
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
f3b1ef6 to
2cf0dcb
Compare
2cf0dcb to
3c8d621
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
|
[P2] |
3c8d621 to
cb4d71b
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
31fd93b to
1a9a280
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
cad4717 to
108e487
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
2735269 to
607dd32
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 2 actionable inline findings. |
|
607dd32 to
1c4f2fb
Compare
|
1c4f2fb to
4311728
Compare
|
4311728 to
512e2b8
Compare
512e2b8 to
9c4ede7
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
9c4ede7 to
a0db36e
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
|
LGTM! |
a0db36e to
6931b20
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Posted 1 actionable inline finding. |
6931b20 to
611ae47
Compare
611ae47 to
b20cf5d
Compare
| duration: z.number().nonnegative(), | ||
| meta: z.object({ | ||
| harness: z.object({ | ||
| run: z.object({ |
There was a problem hiding this comment.
[P2] Handle the library’s generic failed-run shape
createHarness records failures thrown outside the custom partial-run attachment with createFailedHarnessRun, whose run contains session.events, usage: {}, and errors: [{ type, message }], but no Workshop metadata, model, or output. A valid results file containing such an infrastructure failure is therefore rejected by this schema, so comparison exits without the intended neutral non-comparable report. Parse generic failed runs as infrastructure failures at the report/cohort level instead of treating the whole report as malformed.
| export function validateEvalResults(text: string, expectedTrials: number): void { | ||
| const files = parseResults("baseline", text); | ||
| for (const file of files) { | ||
| if (file.assertionResults.length === 0) { |
There was a problem hiding this comment.
[P2] Reject suite-level failures after completed trials
Vitest records file-level errors such as an afterAll failure in testResults[i].message while retaining all completed assertionResults. This loop checks message only when there are zero assertions, so a three-trial report with a teardown failure passes validation, is cached, and is later compared as infrastructure-clean. Reject any nonempty file-level error before caching.
|
Posted 2 actionable inline findings. |
|
LGTM! |
1 participant
Run the existing Workshop trajectory suite on pull requests that change the agent's code, prompts, tool surface, or the eval machinery itself. Each run evaluates the PR's base commit and head commit in parallel, three trials per task, on the eval default model from
packages/workshop-evals/src/config.ts, and keeps the raw trajectories.Baselines are cached per base commit: the baseline arm reuses
workshop-evals-baseline-<base sha>when a trusted PR run has already produced one, and otherwise measures the base itself and caches the result ifvalidate-results.tsfinds every cohort complete and free of infrastructure failures. Keying on the commit is what keeps a reused baseline honest; a broken run is compared against but never cached.A deterministic reducer compares matching task/model cohorts and publishes one neutral Check. Infrastructure failures, unequal trial counts, and eval-definition changes between the two commits are reported as non-comparable; agent failures count as failed trials.
Inference goes through the AI Gateway and token the manual eval workflow and Bonk already use (
CF_AI_GATEWAY_NAME,CF_AI_GATEWAY_ACCOUNT_ID,CF_AI_GATEWAY_TOKEN), over HTTPS, so per-trial cost is reported and no new repository secret is needed. The existing manual eval workflow remains unchanged. This adds no gating or eval tasks.Merge #495 first so the eval default is GLM 5.3 Flash when this lane starts producing baselines.