Skip to content

Add OrcaReplay to coding (record and replay provider exchanges) - #64

Merged
autogame-17 merged 3 commits into
EvoMap:mainfrom
xizhuomengcontin:add-orcareplay
Oct 20, 2026
Merged

autogame-17 merged 3 commits into
EvoMap:mainfrom
xizhuomengcontin:add-orcareplay

Conversation

@xizhuomengcontin

@xizhuomengcontin xizhuomengcontin commented Sep 17, 2026 •

Copy link
Copy Markdown

Adds OrcaReplay to coding, following the four steps in CONTRIBUTING.md.

Disclosure: I maintain OrcaReplay. Apache-2.0, Node 20+.

Why it belongs in coding rather than a new category

The coding category already holds the agents themselves — Claude Code, Codex, Cline, goose, Pi — plus tooling around them (agent-skills, Taste-Skill). This is tooling of the second kind: a way to run a past session again.

It records the agent's exchange with its model provider at the HTTP boundary, so the recording is the bytes the agent actually sent and received rather than a summary it wrote about itself. orca replay last runs that exchange again with no provider contacted and nothing spent.

orca replay last --from <n> --model <other> replays the prefix from the recording and then continues on a different model, so the model is the only variable. This is bounded by checkpoints, not by arbitrary events: deriveCheckpoints only produces a checkpoint where there is both a complete conversation prefix and a filesystem snapshot, --fork refuses a run that has none, and a --from that does not land on a checkpoint is snapped back to the nearest prior one with fork.snapped. So a reader comparing models from a particular event starts from an earlier state, or finds the run cannot fork.

Against the inclusion criteria

  • Relevant — agent coding / agent infrastructure. Several coding entries (Claude Code, Codex, Cline, goose) are agents it records directly; goose and Claude Code have adapters and Claude Code is validated against a real bug fix.
  • Active — commits this week.
  • Open source — Apache-2.0. The on-disk trace format is specified separately under CC BY 4.0 (spec/orca-trace-v0.md), so a recording is readable without the CLI.
  • Documented — README with install and usage; orca quickstart writes a small buggy project plus a recording and replays it, so the claim is checkable in one command.
  • Traction — published on npm as orcareplay. (Star count left to update-stars.js rather than written into the PR text.)

Not a cross-list entry — it is not a multi-agent framework.

Steps followed

node scripts/generate-readme.js   -> regenerated
node scripts/validate.js          -> Validated 121 projects across 10 categories / Validation passed.
git diff --check                  -> clean
node scripts/check-links.js       -> Results: 121 ok, 0 failures, 0 warnings

Run on Node 22.23.2. The branch is rebuilt on current main, so the earlier conflict is gone; the diff is +13 −0 in data/projects.json and +1 −0 in the generated README.

What I am not claiming

  • A matching replay is not a determinism result. It proves the recorded exchange reproduces, not that the agent is deterministic or that a fresh run would fail the same way.
  • Replay withholds the model provider only. It is not a sandbox, and it is not offline — recorded tool calls still execute for real, and a tool that opens its own connection still reaches the network.
  • Fork is bounded by available checkpoints, as described above.
  • The default adapter does not capture every call; embedding calls, for one, are not captured.

🤖 Generated with Claude Code

@autogame-17 autogame-17 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head f0808deb762d2f8bf4cc0eeaffbf5b9adae4d797, including the canonical repository, license and replay implementation. The coding-tool scope is appropriate; please tighten one factual claim before inclusion.

data/projects.json:1543 and generated README.md:156: “replay can fork from any step” should say that it forks from available derived checkpoints. At pinned upstream 6af80889:

  • deriveCheckpoints requires both a complete conversation prefix and a filesystem snapshot.
  • The fork command rejects runs without checkpoints and snaps a non-checkpoint --from sequence back to the nearest prior checkpoint, emitting fork.snapped.

A reader comparing models from a particular event can therefore start from an earlier state, or find that the run cannot fork. “Replay can fork from available checkpoints onto a different model” preserves the useful capability without promising arbitrary-event boundaries. Please update the JSON description, regenerate README, and qualify the equivalent “any step” wording in the PR body.

Actual Node 24.19.0 checks on this head: 120 projects / 10 categories validated, two generator passes were byte-identical to the committed README, the reviewed diff passed whitespace checking, and the initial full link pass was 120 ok / 0 failures / 0 warnings. A later full check of current upstream main found 118 ok / 1 failure / 0 warnings due to HTTP 404 on existing Fzkuji/OpenProgram; that separate catalog availability issue also prevents using the earlier green result as a current all-links-clear result. OrcaReplay's URL passed, its Apache-2.0 license was present, and the normal weekly star snapshot drift is not a requested change.

After updating the description:

node scripts/generate-readme.js
node scripts/validate.js
git diff --check
node scripts/check-links.js

Both Actions runs on this PR are currently action_required with 0 jobs (validate, links); Cursor Bugbot success is not a replacement for these checks. Maintainer workflow approval and successful execution are still required. No merge has been performed.

Review was right. deriveCheckpoints requires both a complete conversation
prefix and a filesystem snapshot, and `replay --fork` rejects a run with no
checkpoints and snaps a non-checkpoint --from back to the nearest prior one
(fork.snapped). The entry now says it forks from available checkpoints.

Also dropped "offline" from the same sentence -- replay withholds the model
provider, it does not take the run off the network -- and said so explicitly.

Regenerated README; validate, diff --check and check-links all clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 10, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@xizhuomengcontin

Copy link
Copy Markdown
Author

Thank you for reading the implementation rather than the description — the finding is correct and I have changed it.

data/projects.json description now reads:

Records a coding agent's exchange with its model provider and replays the run without calling the provider; replay can fork from available checkpoints onto a different model. Recorded tool calls still execute on replay.

Two things changed, not one:

  1. "fork from any step" → "fork from available checkpoints." Your reading of deriveCheckpoints and the fork command matches what the code does: a checkpoint needs both a complete conversation prefix and a filesystem snapshot, --fork refuses a run with none, and a non-checkpoint --from is snapped back with fork.snapped. "Any step" promised a boundary the tool does not offer.
  2. Dropped "offline" from the same sentence. Replay withholds the model provider; it does not take the run off the network, because recorded tool calls still execute. That was a second overclaim in the same line and it should not have survived into the entry either.

I also qualified the equivalent wording in the PR body, including the --from example, and removed the hardcoded star count from the body so it does not go stale.

Checks on this head, Node 22.23.2:

node scripts/generate-readme.js   -> regenerated
node scripts/validate.js          -> Validated 121 projects across 10 categories / Validation passed.
git diff --check                  -> clean
node scripts/check-links.js       -> Results: 121 ok, 0 failures, 0 warnings

Fzkuji/OpenProgram resolves again, so the full link pass is green rather than 118 ok / 1 failure.

The branch is rebuilt on current main — it had drifted into a conflict since September, and the PR is now a clean +13 −0 in the JSON plus the one generated README line. The Actions runs still need maintainer approval to execute.

Disclosure: I maintain OrcaReplay, so this is a self-submission.

@autogame-17

Copy link
Copy Markdown
Member

Rechecked the revised head 2eca16c61d6965ba13b028305192afb2e493d126 on 2026-10-12. The previously requested description fix is addressed. data/projects.json:1569 and the generated README.md:157 now say “available checkpoints” and explicitly disclose that recorded tool calls execute during replay. This matches current upstream a9178ca26af95cfd3231b8eb2da4ae11dbbd0a84: deriveCheckpoints and the fork path enforce the checkpoint boundary.

Actual local verification using the current trusted main scripts on this head's data and README, with Node 24:

  • Validation: 121 projects / 10 categories; passed.
  • Two generator passes: byte-identical to the committed README.
  • Whitespace check: passed.
  • Merge simulation against current main 596c20ba06738d5d7fcb10afd9934e507b4b81bd: clean.
  • The canonical repository is accessible, active, Apache-2.0 licensed, and neither archived nor disabled (285 stars at this check).

The remaining blocker is CI execution, not the old wording: Validate curated data and Check Links are both action_required, with 0 jobs. A maintainer must review the workflow execution boundary and authorize those runs through the normal GitHub process; both checks must actually succeed before approval. The PR's locally reported link pass and Bugbot's billing-paused state are not replacements for that evidence.

No further description edit is requested by this follow-up. Keeping the PR open/blocked; no approval, merge, or workflow execution was issued by this maintenance run.

@autogame-17 autogame-17 changed the title Add OrcaReplay to coding (record a run, replay it offline) Add OrcaReplay to coding (record and replay provider exchanges) Oct 20, 2026
autogame-17 and others added 2 commits October 20, 2026 22:14
Refresh the canonical star snapshot while preserving the reviewed replay description.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolve against current main and regenerate the catalog README.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@autogame-17 autogame-17 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the refreshed catalog-only diff against current main. The previously corrected available-checkpoint boundary is retained without additional wording changes; the title now accurately describes provider-exchange replay. Generated README, validation, and link checks passed locally and in the two current GitHub Actions jobs.

@autogame-17
autogame-17 merged commit 399c5a5 into EvoMap:main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants