refactor(codex): count native child tokens in run metrics by dhshah13 · Pull Request #8569 · fullsend-ai/fullsend

dhshah13 · 2026-10-19T14:33:31Z

Summary

Adds Codex sub-agent token usage to the run totals and per_model_usage, read from each child's rollout inside the sandbox after codex exits. A failed or partial read prints a warning, keeps the children it read, and never fails the run. No agent on main can spawn yet (#7829 turns codex's multi-agent tools off), so at merge the only change is a parent-only per_model_usage for a Codex agent with registered roles.

Related Issue

Part of #6970.

Changes

Child ids come from completed spawn_agent items, read off a tee of the --json stream.
The fold also runs after a non-zero codex exit: codex writes a child's token_usage_record as each response completes.
Usage source: each child's last token_usage_record (the thread's cumulative total), not the token_count ADR 0126 names, because codex zeroes token_count's counters when a request overflows the context window.
docs/cli/run.md documents codex per_model_usage. docs/runtimes/codex.md and docs/contributing/runtime-implementation.md change in the codex runtime: run the review/retro sub-agent rosters through codex's native spawn_agent roles #6970 PR that turns delegation on.

Testing

make lint passes
Tests added in internal/runtime/codex_subagents_test.go

Checklist

PR title follows Conventional Commits (correct type, ! for breaking changes)
Commits are signed off (DCO) — human and human-directed agent sessions only
I wrote this contribution myself and can explain all changes in it

🤖 Generated with Claude Code

A Codex child is a separate thread: its tokens never reach the parent's exec --json stream and live only in the child's rollout. This adds the read and the fold, called by nothing yet: codexChildUsageReadCommand (one bounded in-sandbox command, ids validated and shell-quoted), parseCodexChildUsage (the first turn_context, the last token_count), codexChildIDs (the completed spawn_agent receivers off a tee of the stream) and foldCodexChildUsage (the parent's entry first and always, one request per child thread with usage, child reasoning to the run total only), plus delegating() on the runner-held digest set. Fixtures: four child rollouts modelled on the 0.157.0 paid probe and one stream with completed and failed spawns. Signed-off-by: Dharit Shah <dhshah@redhat.com>

When a Codex agent has registered sub-agent roles or its exec --json stream reports a completed spawn, Run reads the children's rollouts with one bounded sandbox command after codex exits and folds each child's last cumulative usage into the totals and per_model_usage. The ids come off a tee of the stream; parseCodexStream's signature is unchanged. A failed read is a printer warning, never a failed run, and a killed, cancelled or timed-out iteration reports the parent's tokens only. Inert until the multi-agent tools are turned on: no agent on main can spawn, so the one visible change is that a delegating agent's metrics.json gains per_model_usage with the parent's entry. Signed-off-by: Dharit Shah <dhshah@redhat.com>

qodo-code-review · 2026-10-19T14:34:23Z

PR Summary by Qodo

Count Codex sub-agent tokens in run metrics

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Include Codex child-thread tokens in run totals and per-model usage after successful runs.
• Preserve parent-only metrics when rollout reads fail or Codex exits unsuccessfully.
• Document the accounting rules and test stream collection, rollout parsing, and run behavior.

Diagram

graph TD
  RUN["Codex Run"] --> STREAM["Stream collector"] --> GATE{"Clean exit and delegation?"} -->|yes| READ["Bounded rollout read"] --> FOLD["Usage fold"] --> METRICS["Run metrics"]
  ROLES["Registered roles"] --> GATE
  GATE -->|no| METRICS
  READ -->|read failure| FOLD

High-Level Assessment

The single bounded post-run read is appropriate because the parent stream excludes child tokens and the rollouts hold their cumulative usage. Per-child sandbox commands would add overhead; failing the run on an accounting read would unnecessarily change its outcome.

Files changed (11) +978 / -8

Enhancement (3) +310 / -0

codex_integrity.goIdentify delegation from runner-held roles +4/-0

Identify delegation from runner-held roles
• Adds a helper that determines whether Codex registered sub-agent roles using the runner-held digest set rather than the agent-writable manifest.
internal/runtime/codex_integrity.go

codex_run.goFold child usage after a successful Codex run +20/-0

Fold child usage after a successful Codex run

• Tees the parent stream to collect child IDs without changing the event parser. After a clean exit, folds child usage when roles or spawns indicate delegation; read failures produce a warning without failing the run.

internal/runtime/codex_run.go

codex_subagents.goRead and account for native child-thread usage +286/-0

Read and account for native child-thread usage

• Adds bounded rollout retrieval, completed-spawn ID collection, and parsing of each child's model and last cumulative token snapshot. Folds available usage into run totals and per-model entries while retaining the parent entry on read failure.

internal/runtime/codex_subagents.go

Tests (6) +631 / -0

codex_subagents_test.goTest child usage collection and run accounting +597/-0

Test child usage collection and run accounting

• Covers shell command execution, ID validation, stream scanning, token arithmetic, retries, and read failures. End-to-end tests verify successful folds, parent-only breakdowns, warnings, and nonzero exits.

internal/runtime/codex_subagents_test.go

child_usage_failed.jsonlFixture for a child without token usage +5/-0

Fixture for a child without token usage
• Represents a child whose model request fails before a token-count event, so it contributes neither tokens nor a request.
internal/runtime/testdata/codex/child_usage_failed.jsonl

child_usage_no_model.jsonlFixture for usage without a model +2/-0

Fixture for usage without a model
• Provides a token-count event without turn context to verify that usage is attributed to the unknown-model bucket.
internal/runtime/testdata/codex/child_usage_no_model.jsonl

child_usage_other_model.jsonlFixture for a child on a different model +6/-0

Fixture for a child on a different model
• Supplies nonzero token categories for a child whose model differs from the parent's, exercising a separate per-model entry.
internal/runtime/testdata/codex/child_usage_other_model.jsonl

child_usage_same_model.jsonlFixture for cumulative same-model usage +7/-0

Fixture for cumulative same-model usage
• Contains two cumulative token snapshots for a child on the parent's model, verifying that only the last snapshot is folded.
internal/runtime/testdata/codex/child_usage_same_model.jsonl

collab_spawn.ndjsonFixture for completed and failed collaboration calls +14/-0

Fixture for completed and failed collaboration calls
• Provides completed and failed spawns, wait and close calls, and parent usage to test child-ID selection and run-level behavior.
internal/runtime/testdata/codex/collab_spawn.ndjson

Documentation (2) +37 / -8

run.mdDocument Codex per-model accounting +16/-8

Document Codex per-model accounting

• Adds Codex to the runtimes that emit per-model usage and explains its model keys, zero dollar cost, child request counts, unknown-model bucket, and parent-only metrics on cancellation or timeout.

docs/cli/run.md

README.mdDescribe Codex child-accounting fixtures +21/-0

Describe Codex child-accounting fixtures
• Documents the spawn stream and rollout fixtures, their provenance, and the usage cases each represents.
internal/runtime/testdata/codex/README.md

fullsend-ai-review · 2026-10-19T14:35:36Z

🤖 Finished Review · ✅ Success · Started 2:35 PM UTC · Completed 2:51 PM UTC

Commit: fc09465 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.17

qodo-code-review · 2026-10-19T14:36:23Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. ~~Regressed child counters undercount tokens~~ ✓ Resolved

Description

codexChildUsageReadCommand selects only the last cumulative token_count snapshot for each child,
unlike the parent stream processor, which retains per-field high-water values. If a child's later
snapshot reports a lower counter, its previously recorded usage is lost from the run totals and
per-model entry.

Code

internal/runtime/codex_subagents.go[86]

+		`for id in %s; do command -p find %s -type f -name "rollout-*-$id.jsonl" 2>/dev/null | while IFS= read -r f; do printf '%%s %%s\n' %s "$id"; command -p grep -a -e '"type":"turn_context"' -- "$f" 2>/dev/null | command -p head -n 1; command -p grep -a -e '"type":"token_count"' -- "$f" 2>/dev/null | command -p tail -n 1; done; done | command -p head -c %d`,

Evidence
The existing parent processor explicitly guards against cumulative snapshots dropping, but the new
child command discards every snapshot except the last before its parser can apply the same
protection.
internal/runtime/codex_progress.go[116-133]
internal/runtime/codex_progress.go[590-599]
internal/runtime/codex_subagents.go[85-87]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Selecting the last child snapshot can discard usage when a cumulative counter regresses.
## Fix Focus Areas
- internal/runtime/codex_subagents.go[85-87]
- internal/runtime/codex_subagents.go[125-133]
## Recommended Fix
Compute a per-field high-water snapshot across each child's token-count events before folding it, consistent with the parent stream's cumulative-counter handling, while keeping the runner-side read bounded.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~A sandbox read error can expose credentials~~ ✗ Dismissed

Description

foldCodexChildUsage places captured stderr into an error after applying only sanitizeOutput,
and Run forwards that error to StepWarn. When the sandbox command emits credential-bearing text,
the warning path can publish it because sanitizeOutput removes formatting characters but does not
perform secret redaction.

Code

internal/runtime/codex_subagents.go[R252-257]

+	stdout, stderr, exitCode, err := sandbox.Exec(sandboxName, cmd, 10*time.Second)
+	if err != nil {
+		return 0, fmt.Errorf("reading the child rollouts: %w", err)
+	}
+	if exitCode != 0 {
+		return 0, fmt.Errorf("reading the child rollouts: exited %d: %s", exitCode, strings.TrimSpace(sanitizeOutput(stderr)))

Evidence
The changed code captures sandbox stderr, embeds it in the returned error, and emits that error
through the user-facing warning printer. The documented Go credential-redaction guidance requires
secret replacement and redactor scanning, while sanitizeOutput only removes formatting/control
sequences.
Rule 3092695: Consult Go credential redaction guide for external-facing code changes
internal/runtime/codex_subagents.go[252-257]
internal/runtime/codex_run.go[667-671]
internal/runtime/sanitize.go[9-16]
docs/contributing/go-code.md[465-493]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The child-rollout read error includes captured sandbox `stderr`, and the warning path only applies formatting sanitization rather than the credential-redaction process required for external output. Credential-bearing subprocess output could therefore appear in run warnings.

## Fix Focus Areas
- internal/runtime/codex_subagents.go[252-257]
- internal/runtime/codex_run.go[667-671]
- docs/contributing/go-code.md[465-493]

## Recommended Fix
Apply the repository's prescribed secret replacement and `security.SecretRedactor` scanning to captured `stderr` and the final warning text before sending it to `StepWarn`. Preserve useful non-sensitive error context while ensuring runner, provider, and registered runtime secrets are removed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. ~~Missing child rollouts go unwarned~~ ✗ Dismissed

Description

codexChildUsageReadCommand suppresses find and grep errors, and the pipeline reports the final
head command's success even when no rollout was read. If a completed spawn's rollout is missing or
unreadable, foldCodexChildUsage silently omits that child's tokens instead of reaching the warning
path.

Code

internal/runtime/codex_subagents.go[86]

+		`for id in %s; do command -p find %s -type f -name "rollout-*-$id.jsonl" 2>/dev/null | while IFS= read -r f; do printf '%%s %%s\n' %s "$id"; command -p grep -a -e '"type":"turn_context"' -- "$f" 2>/dev/null | command -p head -n 1; command -p grep -a -e '"type":"token_count"' -- "$f" 2>/dev/null | command -p tail -n 1; done; done | command -p head -c %d`,

Evidence
The shell command hides lookup and read errors behind the final head; the fold warns only on a
nonzero command result and otherwise skips records without usage. The new test also demonstrates
that a requested ID with no rollout yields no record.
internal/runtime/codex_subagents.go[85-87]
internal/runtime/codex_subagents.go[252-263]
internal/runtime/codex_subagents_test.go[114-140]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A completed spawn can have a missing or unreadable rollout, but the shell pipeline succeeds and no warning is emitted.
## Fix Focus Areas
- internal/runtime/codex_subagents.go[85-87]
- internal/runtime/codex_subagents.go[248-263]
## Recommended Fix
Preserve read failures and check that each requested child ID has a successfully read rollout; return an error for missing or unreadable rollouts so Run can warn while retaining the parent metrics.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

View medium (1)

4. ~~Truncated reads silently lose child tokens~~ ✓ Resolved

Description

codexChildUsageReadCommand caps the combined output at 1 MiB, while parseCodexChildUsage returns
parsed records without indicating that the output was cut short. If selected rollout lines exceed
that cap, later children—or a cut-off usage line—are omitted from both token totals and the
breakdown without a warning.

Code

internal/runtime/codex_subagents.go[R85-87]

+	return fmt.Sprintf(
+		`for id in %s; do command -p find %s -type f -name "rollout-*-$id.jsonl" 2>/dev/null | while IFS= read -r f; do printf '%%s %%s\n' %s "$id"; command -p grep -a -e '"type":"turn_context"' -- "$f" 2>/dev/null | command -p head -n 1; command -p grep -a -e '"type":"token_count"' -- "$f" 2>/dev/null | command -p tail -n 1; done; done | command -p head -c %d`,
+		strings.Join(quoted, " "), shellQuote(sessionsDir), shellQuote(codexChildUsageMarker), codexChildUsageMaxBytes), nil

Evidence
The cap applies to all children together. The parser skips malformed or cut-off lines and has no
error return, and the fold counts only the records it received.
internal/runtime/codex_subagents.go[40-44]
internal/runtime/codex_subagents.go[85-87]
internal/runtime/codex_subagents.go[94-135]
internal/runtime/codex_subagents.go[259-285]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The output cap can discard child usage while the fold treats the bounded read as complete.
## Fix Focus Areas
- internal/runtime/codex_subagents.go[85-87]
- internal/runtime/codex_subagents.go[94-135]
- internal/runtime/codex_subagents.go[252-285]
## Recommended Fix
Make truncation detectable, check scanner errors, and return a read error when the bounded output is incomplete. Keep the existing successful-run behavior while surfacing the loss through Run's warning.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Context sources

❌ Tickets: failed to load — Ticket collection could not establish complete evidence

✅ Compliance rules (platform): 88 rules

✅ Skills: 5 invoked

writing-user-docs

writing-adrs

check-patch-coverage

cutting-releases

adr-corner

✅ Cross-repo context — repo relationships

Explored: repo: fullsend-ai/agents (sha: 2da538b8) — View relationship

Review mode: Auto: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 14/18, lines 986/200; both must reach the floor). Router rationale: Complex runtime accounting spans parsing, sandbox execution, metrics, and failure paths.

Tip of the day

💡 Did you know, you can open the rule or relationship behind a finding in one click

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

github-actions · 2026-10-19T14:38:36Z

Site preview

Preview: https://16a5ab64-site.fullsend-ai.workers.dev

Commit: 949bd95b55b5d0f94c391f5f67e594a844e65343

codecov · 2026-10-19T14:42:29Z