Skip to content

Declare planner acceptance kind ahead of its payload fields - #2103

Draft
ppXD wants to merge 1 commit into
mainfrom
fix/order-planner-acceptance-kind-first
Draft

ppXD wants to merge 1 commit into
mainfrom
fix/order-planner-acceptance-kind-first

Conversation

@ppXD

@ppXD ppXD commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Declare the planner acceptance kind ahead of argv, artifactPaths, rubric, schema and oraclePaths (order is now formatVersion, kind, argv, artifactPaths, rubric, schema, oraclePaths, description). Every planner-authored acceptance on main's real-model runs was dropped with Planner acceptance requires a non-empty argv|artifactPaths array while formatVersion was present (32 of 32 drops across the 16 main runs since ac484f2). The wire schema is compiled into a decoding grammar, and a decoder that writes properties in declared order has to commit to an optional payload before it has written the kind that says which one is owed. The decoder's ordering is a hypothesis, not something the repo proves; the live planner arm is the measurement.
  • Property definitions are byte-identical; only their order changed. WireSchema inherits the order through JsonSchemaCombinators.Strip, which round-trips through JsonNode and keeps property order (the new wire-side test pins that).
  • Fix the live planner arm, which could not pass as written: a planner-authored ArtifactPresent is promoted to the content oracle it is paired with or dropped as self-certifying (PlannerAcceptanceDraft.ReconcileArtifactPresent), so Single(Kind == ArtifactPresent) could never hold. The arm now asks for a JSON deliverable paired with a schema, asserts the promoted ArtifactSchema, and reports "payload omitted" separately from "dropped by the self-certifying policy". Stale class doc ("no formatVersion") corrected.

Ordering side effects checked:

  • The cassette key (RecordReplayStructuredLLMClient.CassetteKey) canonicalizes the schema with ordinal-sorted object keys, and the planner schema digest (QualificationRuntimeManifestObserver, via ToolCallKey.Canonicalize) does the same, so neither moves. Receipts: PlannerCassetteDriftTests (2 tests) pass unchanged, and the digest is 5b8f21da89fcba01def8151fcf44dc4bdeefdf92cbfc8531e1e9b8509227a688 both before and after while the raw schema text differs. Neither is re-pinned.
  • Other model-facing objects with a discriminator: the three supervisor acceptance objects (SupervisorDecisionSchema: plan subtask, stop, amend) also list command before kind, but command is required there, so the decoder cannot omit it; unchanged. The supervisor, arbiter and coordinator roots already lead with their discriminator. Only the planner acceptance changed.

Real-model runs

gh workflow run real-model.yml --ref fix/order-planner-acceptance-kind-first; counts are occurrences in the full run logs. Live_planner_keeps… is the [realmodel] census line of RealModelPlannerAcceptanceE2ETests.

Run Head requires a non-empty plan.author dropped Live_planner_keeps…
main 41016354131 (10-30) main 1 line (2 acceptances) 0 INFORMATIONAL-FAIL, 2 acceptances could not bind (argv omitted)
main 39344534922 (10-19) main 1 line (2 acceptances) 0 INFORMATIONAL-FAIL, same reason
main 39218614929 (10-19) main 1 line (2 acceptances) 0 INFORMATIONAL-FAIL, same reason
branch 38189233716 pre-rebase 0 0 no INFORMATIONAL-FAIL; plan persisted with 4 items
branch 42384156774 2dea61a 0 0 INFORMATIONAL-FAIL: planner returned no structured output (empty), 4 times
branch 42402963014 2dea61a 0 0 INFORMATIONAL-FAIL: planner returned no structured output (empty), 4 times

Reading: only branch run 1 shows a planner-authored plan on this branch. In runs 2 and 3 the planner call returned an empty body on every attempt (structured completion produced no JSON ... Content preview: (empty)), so no acceptance was emitted at all; that is a gateway or model output failure upstream of the acceptance schema and says nothing for or against the ordering hypothesis. Neither run dropped a payload. The evidence for the fix is therefore one clean run out of three; the other two are inconclusive. Those runs also failed unrelated jobs (delivery gate, supervisor whole-loop, trajectory), which are untouched by this change.

Test plan

  • Unit: On_the_wire_acceptance_kind_precedes_every_kind_dependent_property on both the validation schema and WireSchema, with a fixture check that all 8 properties are declared. Red on main's order (2 failed), green after (40/40 in TypedModelSchemaBranchTests).
  • Full unit suite after rebase onto main: 14971 passed, 1 skipped, 0 failed.
  • PlannerCassetteDriftTests unchanged and green.
  • Real model: 0 requires a non-empty and 0 plan.author dropped in all three branch runs; only one of them reached a planner-authored plan (see table).

The planner's acceptance object listed argv, oraclePaths and artifactPaths
before kind. The gateway compiles the wire schema into a decoding grammar,
and a grammar-constrained decoder is expected to write properties in
declared order: it has to decide on a payload before it has chosen the
oracle that says which one is owed. Both payloads are optional, so it skips
them, then writes kind and cannot go back. Every acceptance on every main
real-model run since the combinator-free wire schema landed was dropped
with "requires a non-empty argv|artifactPaths array" while formatVersion
was present, so the subtasks ran ungraded.

Reorder to formatVersion, kind, argv, artifactPaths, rubric, schema,
oraclePaths, description. Each property definition is byte-identical and
the wire schema inherits the order through Strip. The cassette key and the
planner schema digest both canonicalize with sorted keys, so neither moves.

The decoder behaviour is a hypothesis; the live planner arm is what
measures it. That arm could not pass as written: a planner-authored
ArtifactPresent is promoted to the content oracle it is paired with, or
dropped as self-certifying, so a bare ArtifactPresent never survives to be
asserted. It now asks for a JSON deliverable paired with a schema, asserts
the promoted ArtifactSchema, and reports a payload omitted by the model
separately from a drop by the self-certifying policy.
@ppXD
ppXD force-pushed the fix/order-planner-acceptance-kind-first branch from 6bfed17 to 2dea61a Compare November 7, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant