Repository navigation
fix(hub): read check verdicts from the tool result's detail - #163
TheGreatAxios merged 2 commits into
Conversation
|
Triage: In progress Triage is waiting for code review before it decides. This comment shows the verdict once it has. |
007aca8 to
2fb01a7
Compare
![corbits-triage[bot]](https://avatars.githubusercontent.com/u/275831056?s=60&v=4){kind=small}
corbits-triage
Bot
added
Triage: Waiting for CI
Triage: Ready
and removed
Triage: Ready
Triage: Waiting for CI
labels
572dbcf to
0e74e31
Compare
0e74e31 to
ff8ac32
Compare
| return undefined; | ||
| const output = String(result.content); | ||
| if (!CHECK_APP_VERDICT.test(output)) return undefined; | ||
| return { checkPassed: succeeded(result) && APP_CHECK_PASSED.test(output) }; |
There was a problem hiding this comment.
P2 · A passing mockup check-app run is recorded as a failed check and blocks the done gate
Any run_shell output with a verdict line gets a CheckDetail. A passing mockup check (check passed: served .builder/designs ...) therefore gets {checkPassed:false}. The director's unchecked gate stops at the newest call that has a check detail and returns !checkPassed. So a passing mockup check that only reads files now counts as a newest failing check-app. It hides an earlier passing app check and holds the reply. The old text gate did not treat a passing mockup run as a failure.
A coder passes check-app, then runs check-app .builder/designs to look at the mockup and replies done. Or a designer checks the mockup with check-app from the shell. The reply is held even though nothing changed and both checks passed.
Resolve this comment. Triage will not finish while it is open.
| return call.isError || !says(call, /check passed/); | ||
| if (call.name === CHECK_DESIGN_TOOL) | ||
| return call.isError || !says(call, /check passed/); | ||
| if (call.check) return call.isError || !call.check.checkPassed; |
There was a problem hiding this comment.
P2 · A check-app run that prints no verdict no longer counts as a failed check, so the gate falls back to an older passing check
The old gate stopped at any run_shell command that named check-app and treated it as failed unless its output said it passed. appCheckDetail only records a detail when the output has a verdict line. When check-app is killed, times out or crashes, the call has no check, so unchecked skips it and keeps walking back. If it then finds an older passing check, it reports the work as checked. Any changes made by that same command or by its pre-steps that SHELL_WRITE doesn't match go unnoticed.
Newest first: bun run format && check-app times out with no verdict line, and before it there is a passing check-app. The reply is let through even though format rewrote files and no check passed after that.
Resolve this comment. Triage will not finish while it is open.
Closes CL-11908
The done gates decided whether check-app or check_design passed, and whether a design option was filed, by matching "check passed" and "Filed design_option:" in the tool's output text. History trimming replaces older tool results over 600 characters with a note, so a passing check lost those words and the agent was held for extra turns.
The check tools now record their verdict in the tool result's
detail, which the model never sees and trimming keeps: check-app runs recordcheckPassed(only when check-app is run as a command and the whole command succeeded), and check_design recordscheckPassedandfiled. The gates read that detail and no longer match output text.Verified with a director test that trims a passing check-app result with the real trimmer and then asks the gate; it failed 3 of 3 runs on the old gate and passes now. A unit test covers which shell commands count as a check-app run. Lint, format, typecheck, unit tests (252 pass), pins, vendor checks and
bun run test:e2e(9 pass, including the recorded project replay through check-app and check_design) pass.Runs already in flight when this deploys have check results without the detail, so their next reply can be held up to three times before it goes out.