Skip to content

feat(codex): add tool-excluded active Tok/s metrics#805

Open
ihearttokyo wants to merge 1 commit into
getagentseal:mainfrom
ihearttokyo:agent/fix-daily-history-scan
Open

feat(codex): add tool-excluded active Tok/s metrics#805
ihearttokyo wants to merge 1 commit into
getagentseal:mainfrom
ihearttokyo:agent/fix-daily-history-scan

Conversation

@ihearttokyo

Copy link
Copy Markdown
Contributor

Summary

This contribution adds a low-CPU, retrospective throughput view for Codex rollouts. CodeBurn now reports generated tokens per second (Tok/s) from persisted JSONL timing while excluding recorded tool wait, so the estimate reflects active model work more closely than wall-clock turn duration.

What changed

  • Added codeburn codex-tps with one-shot text output, --json, bounded --limit, and incremental --watch mode.
  • Added bounded 64 KiB JSONL reading so large MCP arguments/results do not require loading an entire rollout into memory.
  • Preserved native MCP, function/custom-tool, fork-replay, and task timing semantics; nested argument/result duration fields are not mistaken for the payload duration.
  • Propagated active duration, generated-token, and tool-wait fields through parsing, caching, session summaries, and model breakdowns.
  • Added Tok/s immediately after 1-shot in the TUI’s By Model panel, with a concise explanation of the active-time denominator and excluded tool wait.
  • Stacked the report panels below 130 columns so the new metric remains readable in ordinary terminals.

Presentation

The repository’s generic dashboard image shows the existing CodeBurn visual language and is included here as a neutral reference:

Generic CodeBurn dashboard

This is an offline JSONL estimate—not live decoder speed. Server, prompt, and other unrecorded latency can still be included when the rollout does not expose a more precise interval.

Validation

  • 121 focused tests pass, including 29 Codex-provider cases, 6 throughput cases, 3 CLI validation cases, 21 dashboard cases, and the existing cache/session suites.
  • npx tsc --noEmit passes.
  • npm run build:cli passes.
  • npm run build passes for the CLI and dashboard; Vite reports its existing large-chunk advisory only.
  • CLI/TUI smoke checks pass for one-shot throughput, invalid limits, missing rollouts, and 120/80-column layouts.
  • Final find-bugs and code-reviewer passes at GPT-5.6 Sol High approved the staged tree, including 5 MiB MCP argument/result collision fixtures for object and string durations.

The unrelated full repository renderer harness remains environment-limited because its current setup lacks the Testing Library Chai matchers and has teardown noise; the feature-focused suite is green. npm audit --omit=dev --audit-level=high still reports the repository’s existing transitive advisories; npm audit fix --force would require a breaking MCP SDK change and was not applied.

Thank you for maintaining CodeBurn and for making room for focused contributions like this—your project made it possible to diagnose the gap from real rollout data.

@ihearttokyo

Copy link
Copy Markdown
Contributor Author

During code review, model found more and more discrepancies. This might be too big a merge but these models get really out of hand and I'm not expert enough to know what I should or should not include. Perhaps just have a model review the changes. The tok/s column is really useful and if you dont want to incorporate these bugfixes then feel free to just reimplement the tok/s column and dump my PR. I worked about an hour on it manually and the code review / bug bash lasted an hour and 8 minutes haha

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant