Skip to content

Preserve observed tool results during transcript repair - #446

Merged
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
fix/preserve-observed-tool-results
Oct 20, 2026
Merged

Brian Krabach (bkrabach) merged 1 commit into
mainfrom
fix/preserve-observed-tool-results

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

What this fixes

Transcript repair used to discard a genuine tool result if an ordinary user
message arrived between the assistant's tool call and that result. It then
substituted "no result was captured." A host persisting the repaired list could
therefore replace recorded evidence with an incorrect synthetic account.

This change retains the complete original result message and places it beside
its unambiguous declaring call. It gathers parallel results as a complete batch
before inserting any derived closing notice. Only actually missing results get
an explicitly unknown-outcome placeholder; a missing receipt is not proof that
an action failed, was cancelled, or may safely be repeated.

Scope and trade-offs

  • Pure diagnose/repair API remains unchanged; diagnosis can additionally report
    unresolved_tool_ids. Top-level compatibility calls and canonical tool_call
    blocks are indexed, with equivalent same-message mirrors counted once.
  • Reused/conflicting IDs, duplicate results, results before their call, unbound
    results, and unsafe associations retain their evidence and remain broken.
    Mirrored argument comparison preserves JSON types (including true versus 1).
  • Whole original messages retain content, signatures, metadata and extra fields.
    Non-result messages keep their relative order.
  • Repair CAN change a result's placement relative to an interruption. This is
    not an immutable-journal or chronology-preservation change. Hosts that reset
    their own position metadata may reindex it after installing the repaired list.
  • No new provider/Core dependency, storage schema, tool execution or model call.
    This does not add a universal host dispatch-refusal policy.

Verification

Actual isolated DTU, Python 3.13.11, public Rust Core 2.0.2:

  • Foundation tests/: 2,483 passed, 4 existing skips. No case exclusion.
    Skips: two optional CLI-documentation checks, tiktoken-dependent measurement,
    and one Windows-only process test. Bundled tool-delegate tests are not included
    in that tests/ count.
  • Four affected repair/history/script test files: 173 passed.
  • Independent installed-package JSON-type ambiguity falsifiers: 3 passed;
    these failed against the first candidate before normalization was corrected.
  • Cross-repo installed-package receipt/save/converter falsifiers: 12 passed;
    the corrected original baseline had 8 failures and 4 successful controls.
  • Separate actual CLI helper/save/store/resume boundary checks: 64 passed,
    including repeated save/load/resume and backup rotation. Not a complete REPL
    launch or shared-root-lock repair acceptance.

The standalone Foundation environment excludes the optional CLI resolver. The
earlier combined environment's unmodified resolver test expected the
Foundation-only implementation and failed with the CLI implementation present;
that same case ran and passed in the standalone suite. Ten other initial broad
suite failures came from an overbroad TEST audit denying local Unix IPC; the
corrected test guard permits only AF_UNIX IPC and still denies all IP/DNS.
Positive Unix IPC and negative IPv4/DNS controls passed before the rerun.

Ruff 0.16.1 format check passed for six selected files across the change set.
The broad default lint invocation was not clean: it reported baseline findings
also present in the original source (28 originally; 22 after the final narrow,
documented trusted-helper test exception). No all-lint-green claim is made.
Initial failures and raw receipts are retained; none were relabelled.

Python 3.11/3.12 and hosted CI are not claimed as locally executed. No live
vendor/model/desktop tests ran. The cross-repo environment used Anthropic SDK
1.15.0/OpenAI SDK 3.35.0, not exact normal-environment OpenAI 3.35.1 parity.

Review and integration

Astra (openai/gpt-6-astra) and Fable (fable/claude-fable-5-1) independently reviewed
the same supplied evidence/design packet and recommended this bounded repair
with the explicit placement trade-off. Those are bounded design opinions, not
full-diff audits or GitHub approvals. A separate source review identified the
JSON-type counterexample; executed falsifiers, a correction and a follow-up
source review closed it.

Provider survey covered all 11 accessible Microsoft provider-prefix repos.
Providers cannot recover a receipt already erased upstream. Separate local
changes cover Managed recovery save-failure rollback, a confirmed Chat
Completions receipt-conversion defect, and host regression tests. Downstream
publication waits for the Foundation dependency in accordance with integration
order. Legacy retirement/migration remains explicitly deferred, not fulfilled.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants