Skip to content

feat: add privacy-safe tracing and deterministic replay evals - #54

Open
mianbao4917 wants to merge 2 commits into
LiuMengxuan04:mainfrom
mianbao4917:codex/feat-runtime-tracing
Open

mianbao4917 wants to merge 2 commits into
LiuMengxuan04:mainfrom
mianbao4917:codex/feat-runtime-tracing

Conversation

@mianbao4917

Copy link
Copy Markdown

Summary

  • add opt-in, privacy-safe JSONL runtime tracing for model requests, tool execution, failures, token usage, and context-compression events
  • exclude prompt text, model responses, tool inputs, and tool outputs from trace records
  • add a /trace status command and trace recording for stdio, TUI, and sub-agent runs
  • add deterministic model/tool replay evaluation against the real agent loop, with behavior and budget assertions plus machine-readable reports
  • document both features in English and Chinese and include a smoke evaluation fixture
  • integrate the tracing hooks with the latest Plan, Goal, and Loop runtime changes

Why

This provides lightweight observability and reproducible regression checks without adding dependencies or requiring live-model calls. It keeps the agent loop directly inspectable while making latency, tool failures, termination behavior, and token budgets easier to diagnose.

Verification

  • npm run check — passed
  • npm run lint — passed
  • git diff --check — passed
  • focused trace/eval/CLI tests were attempted, but Node.js 24 on this Windows environment failed before loading the test files with uv_os_get_passwd / ENOMEM; GitHub CI should provide the authoritative test run

Reproduce

  • enable tracing with MINI_CODE_TRACE=1 npm run dev, then run /trace
  • run the replay smoke suite with npm run dev -- eval examples/eval-smoke.json --output eval-report.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant