fix: restore core clarify boundary

README.md

@@ -6,17 +6,16 @@
 
 Context Compiler is a deterministic conversational state authority for LLM applications.
 It handles canonical directive execution, semantic validation, deterministic
-clarification and confirmation flows, checkpoint restore, and structured
-authoritative state for the host.
+clarify decisions, checkpoint restore, and structured authoritative state for
+the host.
 
 ## What Context Compiler provides
 
 Context Compiler gives hosts fixed state rules:
 
 - handle canonical explicit state changes with deterministic rules
 - clarification instead of silent overwrite for blocked/ambiguous changes
-- pending confirmation flows that must resolve before anything else changes
-- export and import checkpoints to restore saved state and pending confirmation flow
+- export and import checkpoints to restore saved state in a versioned engine snapshot
 - produce structured authoritative state for downstream host decisions
 
 The model generates responses. The compiler owns state.
@@ -178,12 +177,12 @@ pip install context-compiler
 context-compiler
 ```
 
-Preload options keep saved rules separate from confirmation state in progress:
+Preload options keep authoritative state transport separate from checkpoint session snapshots:
 
 - `--initial-state-json` / `--initial-state-file` load saved state
   (via exported state JSON).
-- `--initial-checkpoint-json` / `--initial-checkpoint-file` restore full
-  continuation checkpoint (saved state + pending confirmation state).
+- `--initial-checkpoint-json` / `--initial-checkpoint-file` restore the full
+  checkpoint envelope (saved state plus reserved continuation field).
 
 REPL commands (controller layer, not engine directives):
 
@@ -202,12 +201,12 @@ for non-interactive usage.
 context-compiler --json < input.txt
 ```
 
-Preload options keep saved rules separate from confirmation state in progress:
+Preload options keep authoritative state transport separate from checkpoint session snapshots:
 
 - `--initial-state-json` / `--initial-state-file` load saved state
   (via exported state JSON).
-- `--initial-checkpoint-json` / `--initial-checkpoint-file` restore full
-  continuation checkpoint (saved state + pending confirmation state).
+- `--initial-checkpoint-json` / `--initial-checkpoint-file` restore the full
+  checkpoint envelope (saved state plus reserved continuation field).
 
 ## Installation
 
@@ -359,13 +358,12 @@ the compiler.
 `export_json()` / `import_json()` and the checkpoint APIs serve different boundaries:
 
 - `export_json()` / `import_json()` transport **authoritative state only**
-- checkpoint APIs transport **serialized continuation**:
+- checkpoint APIs transport a **versioned engine snapshot**:
   - authoritative state
-  - pending confirmation flow state
+  - reserved `pending` field for canonical continuation compatibility
 
 Use state JSON when you only need authoritative state. Use checkpoint APIs when
-you also need resumable continuation state across process or request
-boundaries.
+you want the stable checkpoint envelope across process or request boundaries.
 
 For the checkpoint object shape, API-level usage notes, and serialization
 details, see [docs/api-reference.md](docs/api-reference.md#checkpoint-apis).
@@ -405,7 +403,7 @@ User: clear state
 Grammar invariant: one input may contain at most one canonical directive.
 If another canonical directive start appears later in the same input, the
 input is invalid and Context Compiler returns `clarify` without mutating
-authoritative state or creating pending confirmation state.
+authoritative state or creating pending continuation state.
 
 Examples:
 

demos/09_llm_pending_clarification.py

@@ -1,8 +1,8 @@
-"""Demo 9: pending clarification requires confirmation-only continuation."""
+"""Demo 9: invalid replacement does not create core pending continuation."""
 
 from context_compiler import (
     DECISION_CLARIFY,
-    DECISION_UPDATE,
+    DECISION_PASSTHROUGH,
     POLICY_USE,
     State,
     create_engine,
@@ -22,7 +22,7 @@
 )
 from demos.llm_client import complete_messages
 
-DEMO_NAME = "09_pending_clarification_continuation — confirmation-only state transition"
+DEMO_NAME = "09_pending_clarification_boundary — invalid replacement stays non-pending"
 TURN_1 = "use podman instead of docker"
 TURN_2 = "maybe"
 TURN_3 = "yes"
@@ -46,6 +46,7 @@ def main() -> None:
     print_decision("turn 1", first, engine.state)
     pending_after_first = engine.has_pending_clarification()
     state_unchanged_after_first = _is_initial_authoritative_state(engine.state)
+    checkpoint_after_first = engine.export_checkpoint()
 
     second = engine.step(TURN_2)
     print_decision("turn 2", second, engine.state)
@@ -95,27 +96,31 @@ def main() -> None:
         print_model_output("Compiler-mediated + compact", compact_output)
     else:
         print_messages("compiler-mediated + compact", [])
-        compact_output = "[no call] compact replay resolved pending flow"
+        compact_output = "[no call] compact replay did not create pending continuation"
         print_model_output("Compiler-mediated + compact", compact_output)
 
     blocked_initial_mutation = first["kind"] == DECISION_CLARIFY and state_unchanged_after_first
-    pending_preserved_on_nonconfirm = (
-        second["kind"] == DECISION_CLARIFY
-        and pending_after_first
-        and pending_after_second
-        and state_unchanged_after_first
+    no_pending_after_invalid_replacement = (
+        not pending_after_first and checkpoint_after_first["pending"] is None
+    )
+    unrelated_followup_passthrough = (
+        second["kind"] == DECISION_PASSTHROUGH
+        and not pending_after_second
         and state_unchanged_after_second
     )
-    confirmation_only_resolution = third["kind"] == DECISION_UPDATE and not pending_after_third
-    deterministic_final_state = _has_podman_use(engine.state)
+    confirmation_token_not_consumed = (
+        third["kind"] == DECISION_PASSTHROUGH and not pending_after_third
+    )
+    deterministic_final_state = not _has_podman_use(engine.state)
 
     baseline_has_pending_state_machine = False
     reinjected_has_pending_state_machine = False
 
     compiler_pass = (
         blocked_initial_mutation
-        and pending_preserved_on_nonconfirm
-        and confirmation_only_resolution
+        and no_pending_after_invalid_replacement
+        and unrelated_followup_passthrough
+        and confirmation_token_not_consumed
         and deterministic_final_state
     )
 
@@ -131,17 +136,22 @@ def main() -> None:
         context="compiler-mediated",
     )
     print_host_check(
-        "PENDING_PRESERVED_ON_NONCONFIRM",
-        yes_no(pending_preserved_on_nonconfirm),
+        "NO_PENDING_AFTER_INVALID_REPLACEMENT",
+        yes_no(no_pending_after_invalid_replacement),
+        context="compiler-mediated",
+    )
+    print_host_check(
+        "UNRELATED_FOLLOWUP_PASSTHROUGH",
+        yes_no(unrelated_followup_passthrough),
         context="compiler-mediated",
     )
     print_host_check(
-        "CONFIRMATION_ONLY_RESOLUTION",
-        yes_no(confirmation_only_resolution),
+        "CONFIRMATION_TOKEN_NOT_CONSUMED",
+        yes_no(confirmation_token_not_consumed),
         context="compiler-mediated",
     )
     print_host_check(
-        "FINAL_POLICY_PODMAN_USE",
+        "FINAL_POLICY_PODMAN_ABSENT",
         yes_no(deterministic_final_state),
         context="compiler-mediated",
     )
@@ -153,18 +163,18 @@ def main() -> None:
         compiler_pass=compiler_pass,
         compiler_compact_pass=compact_pass,
         expected=(
-            "replacement flow should require pending clarification, preserve pending on "
-            "non-confirmation, and apply only on confirmation"
+            "invalid replacement should clarify without creating pending continuation, "
+            "and later yes/no-style input should not resolve a synthesized operation"
         ),
         actual=(
-            "compiler enforced blocked mutation, pending continuation, and confirmation-only "
-            "resolution to final podman policy"
+            "compiler blocked mutation, kept checkpoint pending null, and treated later "
+            "inputs as ordinary passthrough"
             if compiler_pass and compact_pass
-            else "compiler did not consistently enforce pending clarification continuation"
+            else "compiler did not consistently preserve the non-pending replacement boundary"
         ),
         passed=compiler_pass and compact_pass,
-        result_pass="pending clarification continuation enforced deterministically",
-        result_fail="pending clarification continuation not enforced deterministically",
+        result_pass="invalid replacement stayed outside core pending continuation",
+        result_fail="invalid replacement still behaved like core pending continuation",
     )
 
 

demos/README.md

@@ -36,7 +36,7 @@ Runnable application-layer enforcement-point integrations live in
 | [06](./06_llm_context_compaction.py) | Context compaction | saved compiler state replacing transcript context | small or local models |
 | [07](./07_llm_prompt_vs_state.py) | Prompt engineering comparison | prompting vs saved compiler state | any model with long transcript sensitivity |
 | [08](./08_llm_replacement_precondition.py) | Replacement precondition | invalid replacement blocked without state mutation | any model |
-| [09](./09_llm_pending_clarification.py) | Pending clarification continuation | confirmation-only resolution of suspended mutation | any model |
+| [09](./09_llm_pending_clarification.py) | Pending boundary | invalid replacement does not become pending continuation | any model |
 
 Stronger frontier models may show these behaviors less often, but the same
 patterns still appear in real applications.
@@ -171,7 +171,7 @@ Notes:
 - Anthropic runs in this repo are executed through the `openai_compatible` provider path.
 - `PASS` means the demo-specific expected-behavior check for that path succeeded; `FAIL` means it did not.
 - `reinjected-state` can be enough for some persistence cases; in this demo set it is intentionally used as a prompt-only comparison baseline.
-- Scored checks focus on app-side authority rules (for example blocked mutation and confirmation-only resolution), not model prose quality. `reinjected-state` remains plain text injection only.
+- Scored checks focus on app-side authority rules (for example blocked mutation and non-pending invalid replacement handling), not model prose quality. `reinjected-state` remains plain text injection only.
 - Interpretation:
 
 - Demos `01`-`05` and `07` mostly test persistence and policy-following behavior across turns.

docs/DirectiveGrammarSpec.md

@@ -314,6 +314,8 @@ When `Decision.kind = "clarify"`, prompt text is deterministic only for the case
   `Replacement requires an active 'use' policy.`
 - Pending clarification unmatched input (Section 9 case 11):
   reuse the existing pending prompt unchanged.
+  In the current engine contract, no canonical operation produces pending
+  clarification state, so this case is reserved but unreachable.
 - `remove policy ITEM` with empty/whitespace-only payload (Section 9 case 12):
   `Policy item cannot be empty.`
   `Use 'remove policy <item>' with a non-empty value.`
@@ -375,6 +377,14 @@ Resolution:
 - Negative token: discard pending event, clear pending, apply no mutation, return `update` with unchanged state
 - Any other input: remain in `clarify`, no mutation, and keep the existing pending prompt
 
+Current engine contract note:
+
+- The current engine establishes no pending clarification state for any
+  canonical directive family.
+- Section 10 remains the reserved core boundary for future canonical
+  continuation semantics and checkpoint compatibility.
+- Non-null `pending` checkpoint payloads are not currently produced by core.
+
 ## 11. Context Serialization Contract
 
 The compiler outputs structured state only; the host formats prompts.
@@ -430,13 +440,14 @@ Checkpoint semantics:
 - `pending` captures confirmation-required continuation for supported canonical
   operations only.
 - `pending` is `null` when there is no outstanding continuation.
+- In the current engine contract, `pending` is always `null`.
 - Restore is all-or-nothing: invalid checkpoint payloads raise and no partial state restore occurs.
 - Continuation behavior is deterministic after restore (same confirmation token handling and resolution outcomes as live pending state).
 - `checkpoint_version` is independent of authoritative state `version`; it must be bumped when checkpoint contract shape changes (especially `pending`).
 
 ## 12. Invariants
 
-1. State changes only from valid canonical directive transitions or pending-confirmation acceptance for canonical operations.
+1. State changes only from valid canonical directive transitions or pending-confirmation acceptance for canonical operations when such operations are supported by the active engine contract.
 2. Same input sequence yields identical state and decisions.
 3. LLM output never mutates state.
 4. No mutation occurs when returning `clarify`.

docs/api-reference.md

@@ -86,7 +86,7 @@ Boundary notes:
 - validation returns `None` for any non-canonical input
 - rendering is syntax-only and performs no state interpretation
 - `engine.step(...)` remains the authority for clarification, state
-  transitions, pending confirmation, and mutation behavior
+  transitions, reserved canonical continuation semantics, and mutation behavior
 - `engine.step(...)` is not a general natural-language repair surface; host
   code should send canonical directives when it wants deterministic mutation
 - pending confirmation is limited to deterministic continuation of supported
@@ -106,6 +106,9 @@ traversal unless you are working at an explicit serialization boundary.
 
 Return whether a confirmation-required clarification is currently pending.
 
+In the current engine contract, this returns `False` because no canonical
+operation currently produces pending clarification state.
+
 Typical use:
 
 ```python
@@ -213,10 +216,9 @@ Export a checkpoint as canonical JSON text.
 
 Validate and restore a checkpoint from JSON text.
 
-Use checkpoint APIs when you need both:
-
-- authoritative state
-- pending confirmation/continuation state
+Use checkpoint APIs when you need a versioned engine-session snapshot. Today,
+that means authoritative state plus a reserved `pending` field for canonical
+continuation compatibility.
 
 Checkpoint object shape:
 
@@ -241,6 +243,7 @@ API-level contract notes:
 - `pending` is `null` when no continuation is waiting for confirmation
 - `pending` captures confirmation-required continuation for canonical
   operations supported by the active engine contract
+- in the current engine contract, `pending` is always `null`
 - non-canonical repair state is not part of the intended core contract
 - imported policy keys are normalized during `import_json(...)` and checkpoint
   authoritative-state restore
@@ -255,8 +258,8 @@ API-level contract notes:
 Typical use cases:
 
 - stateless host or integration boundaries where engine instances are short-lived
-- resume after interruption without losing pending clarification flow
-- preserve pending confirmation flow state across process or request boundaries
+- authoritative-state persistence with a stable checkpoint envelope
+- future-compatible storage where hosts want one artifact for engine session snapshots
 
 ## Controller APIs
 

docs/architecture.md

@@ -8,13 +8,13 @@ authority inside a larger host application stack.
 Responsibilities:
 
 - apply deterministic state transitions
-- enforce clarification and confirmation gates for core-owned state semantics
+- enforce deterministic clarification gates for core-owned state semantics
 - export/import authoritative state and checkpoints
 
 Examples:
 
 - Context Compiler core engine
-- checkpoint continuation behavior
+- checkpoint/session snapshot behavior
 
 Repository:
 
@@ -29,8 +29,9 @@ Boundary:
 - core does not own human-facing normalization, malformed-input recovery, or
   intent inference as a general responsibility
 - core does not convert failed canonical operations into different directives
-- pending yes/no confirmation is limited to deterministic continuation of
+- pending yes/no confirmation is reserved for deterministic continuation of
   canonical operations already established by core
+- the current engine contract produces no live pending continuation state
 
 ## Acquisition Layer
 
@@ -108,8 +109,8 @@ Policy independence is a major contributor to:
 - cross-language conformance
 
 Because policies are independent flat assertions, directive semantics stay
-simple, exported state stays portable, replay remains exact, and checkpoint
-continuation does not depend on hidden relational logic.
+simple, exported state stays portable, replay remains exact, and any future
+checkpoint continuation does not depend on hidden relational logic.
 
 Relationship-heavy semantics may still be useful, but they generally belong in
 drafting, orchestration, or domain-specific layers rather than in