fix: grade raw product responses in storyboards - #2549
Conversation
There was a problem hiding this comment.
Ladon verdict: Request changes
Request changes — 1 blocking finding.
The PR's intent is sound (witness-not-translator, DR-0001): routing storyboard get_products through the SDK's unprojected getProductsLegacy so conformance grades the raw seller response, with correct tests and an accurate patch changeset. However, the implementation flips the shared TASK_TO_METHOD default rather than the runner's per-step responseProjection, over-reaching into the fixture-seeding path.
Blocking findings
- src/lib/testing/storyboard/task-map.ts:28 — Global
get_productsremap breaks canonical-shape fixture seeding/matching for pre-3.2 (legacy) sellers.
The high-severity finding lands on a load-bearing path and always blocks (row 1). Recommend scoping the change to the runner's per-step projection instead of the shared task-map default.
Blocking findings
- src/lib/testing/storyboard/task-map.ts:28 — Global get_products remap breaks canonical fixture seeding for legacy sellers
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — prior HIGH resolved, no current findings.
This is a subsequent review. The prior run flagged one blocking HIGH (src/lib/testing/storyboard/task-map.ts:28 — global get_products remap breaking canonical fixture seeding). The delta resolves it: get_products now routes to getProductsLegacy (a raw executeTaskUnprojected pass-through) across all versions, the authored request flows through verbatim (legacy-wire hint injection is dropped for get_products), and canonical fixture seeding (creative_wire: 'canonical') is now graded raw. This correctly applies witness-not-translator by bypassing the projecting getProducts so SDK projection cannot rewrite conformance evidence. Tests and the patch changeset are coherent.
No current findings. Decision table: rows 1–8 do not fire (no critical/high/medium findings, gated_paths false, high_risk false, no team gate, prior escalate row 6 N/A since prior was request-changes with no fresh findings). Falls through to row 9 → approve.
Note: review_decision is CHANGES_REQUESTED, but that only forces escalate/approve-block under row 2 when gated_paths is true — here gated_paths is false, so it does not block. Net improvement over the prior run: the sole blocker is gone.
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean pass, no blocking or medium findings.
Checked:
- New export
defaultStoryboardResponseProjection('get_products') => 'raw'is wired into the runner viaeffectiveStep.response_projection ?? default, so authoredresponse_projectionstill wins and wire selection stays independent — consistent with witness-not-translator (grades raw seller evidence throughgetProductsLegacyrather than SDK convenience projections; no fabrication or re-shaping at the seam). - New export is directly unit-tested.
- Mock relabels are cosmetic on canonical-path branches guarded by call assertions.
- Patch changeset (
.changeset/grade-raw-product-responses.md) is present and accurate; type matches the additive behavior change (no removed/renamed export, no required-param flip, no response-shape break).
Gates: no no-auto-approve team match; gated_paths false; high_risk false. No findings at any severity. Falls through to row 9.
There was a problem hiding this comment.
Ladon verdict: Approve
Approve.
Delta narrows defaultStoryboardResponseProjection to grade raw get_products evidence only when comply_scenario === 'schema_compliance', preserving the canonical SDK projection for flow storyboards. Changeset prose, doc comment, function signature, and test move coherently; the four-cell test pins the new branch plus its flow-storyboard regression guards.
Checked:
- Witness-not-translator: the change tightens (not loosens) when raw evidence is graded — no new fabrication/normalization seam.
- Changeset present (
.changeset/grade-raw-product-responses.md) covering thesrc/lib/**change; type coherent with a response-shape refinement. - Author team gate: no match.
- Gated paths: false; high_risk: false.
No critical/high/medium findings. Prior decision was approve; this is a net-clean pass — falls through to row 9.
Note: review_decision is REVIEW_REQUIRED, but gated_paths is false so row 2 does not fire.
Summary
get_productscalls through the SDK unprojected response path by defaultWhy
Conformance storyboards must grade the seller response, not the SDK convenience projection. Requiring each storyboard step to opt into
response_projection: rawis error-prone and allows SDK-added or removed fields to alter compliance evidence.Validation
npm run buildnpx vitest run src/lib/testing/storyboard/task-map.test.tsnode --test --test-timeout=60000 test/lib/storyboard-task-map-error.test.jsnpm run lintnpm run typecheck