Skip to content

fix: grade raw product responses in storyboards - #2549

Merged
bokelley merged 4 commits into
mainfrom
codex/storyboard-raw-protocol-projection
Aug 16, 2026
Merged

fix: grade raw product responses in storyboards#2549
bokelley merged 4 commits into
mainfrom
codex/storyboard-raw-protocol-projection

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

  • route storyboard get_products calls through the SDK unprojected response path by default
  • keep creative-wire version negotiation independent from response projection
  • cover legacy 3.1, explicit canonical 3.1, and canonical 3.2 requests in both source and built-output tests

Why

Conformance storyboards must grade the seller response, not the SDK convenience projection. Requiring each storyboard step to opt into response_projection: raw is error-prone and allows SDK-added or removed fields to alter compliance evidence.

Validation

  • npm run build
  • npx vitest run src/lib/testing/storyboard/task-map.test.ts
  • node --test --test-timeout=60000 test/lib/storyboard-task-map-error.test.js
  • npm run lint
  • npm run typecheck
  • full pre-push validation

@bokelley
bokelley enabled auto-merge (squash) August 16, 2026 03:55
Comment thread src/lib/testing/storyboard/task-map.ts Outdated

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Request changes

Request changes — 1 blocking finding.

The PR's intent is sound (witness-not-translator, DR-0001): routing storyboard get_products through the SDK's unprojected getProductsLegacy so conformance grades the raw seller response, with correct tests and an accurate patch changeset. However, the implementation flips the shared TASK_TO_METHOD default rather than the runner's per-step responseProjection, over-reaching into the fixture-seeding path.

Blocking findings

  • src/lib/testing/storyboard/task-map.ts:28 — Global get_products remap breaks canonical-shape fixture seeding/matching for pre-3.2 (legacy) sellers.

The high-severity finding lands on a load-bearing path and always blocks (row 1). Recommend scoping the change to the runner's per-step projection instead of the shared task-map default.

Blocking findings

  • src/lib/testing/storyboard/task-map.ts:28 — Global get_products remap breaks canonical fixture seeding for legacy sellers

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Aug 16, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — prior HIGH resolved, no current findings.

This is a subsequent review. The prior run flagged one blocking HIGH (src/lib/testing/storyboard/task-map.ts:28 — global get_products remap breaking canonical fixture seeding). The delta resolves it: get_products now routes to getProductsLegacy (a raw executeTaskUnprojected pass-through) across all versions, the authored request flows through verbatim (legacy-wire hint injection is dropped for get_products), and canonical fixture seeding (creative_wire: 'canonical') is now graded raw. This correctly applies witness-not-translator by bypassing the projecting getProducts so SDK projection cannot rewrite conformance evidence. Tests and the patch changeset are coherent.

No current findings. Decision table: rows 1–8 do not fire (no critical/high/medium findings, gated_paths false, high_risk false, no team gate, prior escalate row 6 N/A since prior was request-changes with no fresh findings). Falls through to row 9 → approve.

Note: review_decision is CHANGES_REQUESTED, but that only forces escalate/approve-block under row 2 when gated_paths is true — here gated_paths is false, so it does not block. Net improvement over the prior run: the sole blocker is gone.

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Aug 16, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean pass, no blocking or medium findings.

Checked:

  • New export defaultStoryboardResponseProjection('get_products') => 'raw' is wired into the runner via effectiveStep.response_projection ?? default, so authored response_projection still wins and wire selection stays independent — consistent with witness-not-translator (grades raw seller evidence through getProductsLegacy rather than SDK convenience projections; no fabrication or re-shaping at the seam).
  • New export is directly unit-tested.
  • Mock relabels are cosmetic on canonical-path branches guarded by call assertions.
  • Patch changeset (.changeset/grade-raw-product-responses.md) is present and accurate; type matches the additive behavior change (no removed/renamed export, no required-param flip, no response-shape break).

Gates: no no-auto-approve team match; gated_paths false; high_risk false. No findings at any severity. Falls through to row 9.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Delta narrows defaultStoryboardResponseProjection to grade raw get_products evidence only when comply_scenario === 'schema_compliance', preserving the canonical SDK projection for flow storyboards. Changeset prose, doc comment, function signature, and test move coherently; the four-cell test pins the new branch plus its flow-storyboard regression guards.

Checked:

  • Witness-not-translator: the change tightens (not loosens) when raw evidence is graded — no new fabrication/normalization seam.
  • Changeset present (.changeset/grade-raw-product-responses.md) covering the src/lib/** change; type coherent with a response-shape refinement.
  • Author team gate: no match.
  • Gated paths: false; high_risk: false.

No critical/high/medium findings. Prior decision was approve; this is a net-clean pass — falls through to row 9.

Note: review_decision is REVIEW_REQUIRED, but gated_paths is false so row 2 does not fire.

@bokelley
bokelley merged commit b226ab8 into main Aug 16, 2026
32 checks passed
@bokelley
bokelley deleted the codex/storyboard-raw-protocol-projection branch August 16, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant