[weireweire]

Summary

Tag the AgentX (AGENTIC_REPLAY) warmup-start server-metrics baseline scrape as warmup instead of profiling, so profiling counter deltas no longer absorb the whole warmup.

Root cause

_build_agentic_warmup_config synthesizes the agentic warmup CreditPhaseConfig without a phase_kind. ServerMetricsManager.collect_baseline mapped every kind other than "warmup" to PROFILING, so the scrape taken at warmup start was tagged PROFILING.

The phase-scoped profiling summary selects samples by tag. For any counter series that already existed at that scrape, the profiling series therefore started at warmup start, and its first delta covered the entire warmup. Series that first appeared during warmup were unaffected, so ratios between counters were skewed.

Observed on a GB200 DeepSeek-V4 Dynamo-SGLang disaggregated AgentX run at concurrency 1280:

• sglang:prompt_tokens on each prefill endpoint started its profiling timeslices 1674 s before profiling start, with a single jump of 1,258,951,023 tokens. That is exactly the warmup total for the same series in warmup_metrics.
• sglang:cached_tokens and the decode endpoint's counters started at profiling start.
• The prefix cache hit rate computed from these counters was 89.6%. Over the profiling window it is 96.3%, matching the 96.4% summed from every prefill rank's batch logs.

Fix

• Set phase_kind="warmup" on the synthesized agentic warmup phase.
• Carry the credit phase on PhaseBaselineRequestMessage (filled in by the phase publisher). collect_baseline uses it when phase_kind is unset, instead of defaulting to PROFILING.

Validation

• New tests:
  • The agentic warmup config has phase_kind == "warmup".
  • Baseline scrapes are tagged with the boundary phase for phase_kind warmup/profiling/unset.
• pytest tests/unit/server_metrics tests/unit/timing tests/unit/records: 2359 passed, 1 failed. The failure is test_records_manager.py::TestRecordsManagerAnalyzerMetrics::test_completed_excludes_analyzer_metrics, which fails identically on the base commit (scenario lookup on a MagicMock).
• ruff check and ruff format --check pass on the changed files.
• Not yet verified end to end. On the next AgentX run, check that prefill prompt_tokens profiling timeslices start at profiling start.

---

Note

Low Risk
Localized server-metrics phase tagging and message fields; behavior change is limited to baseline scrape labels with new unit tests.

Overview
Fixes agentic (AGENTIC_REPLAY) warmup-start server-metrics baseline scrapes being labeled profiling, which made phase-scoped counter deltas include the whole warmup (e.g. skewed prefix-cache hit rates).

Synthesized agentic warmup now sets phase_kind="warmup" on CreditPhaseConfig. Phase boundary baseline requests carry optional phase (CreditPhase); the phase publisher fills it from config. ServerMetricsManager.collect_baseline maps phase_kind to warmup/profiling explicitly and, when kind is missing, uses message.phase instead of defaulting non-warmup to profiling.

Unit tests cover agentic warmup phase_kind and parametrized boundary scrape tagging.

^{Reviewed by Cursor Bugbot for commit 5899651. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@5899651b7d3ed8a61a86ae571eedd2f0fb8234b9

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@5899651b7d3ed8a61a86ae571eedd2f0fb8234b9

---

Last updated for commit: 5899651 • Browse code

[weireweire]

@cquil11 hi could you help to review this?

[chunfangamd]

Independent check of this PR against current master (104c7c8), in case it helps the review:

• The diff applies cleanly on 104c7c8, and the two test files it changes pass there: tests/unit/server_metrics/test_server_metrics_manager.py (46 passed) and tests/unit/timing/test_phase_config_agentic_replay.py (13 passed). This was a local run with a pinned subset of the test dependencies, not a full .[dev] install and not CI.
• We had written separate tests for the same defect that drive the real path end to end: the delivered agentic phase config through TimingConfig.from_run, START/END baseline requests from a real PhaseRunner + PhasePublisher, ServerMetricsManager.collect_baseline, storage, the accumulator and the export. Only the Prometheus HTTP fetch and the message bus are replaced, and the observations are synthetic. On master the profiling counter total, histogram sum and histogram count come out as 700 / 700 / 7; with this PR they are 300 / 300 / 3, as expected. The chain cases for a START scrape in flight across the warmup start, named multi-warmup, late-created series, counter reset and a missing profiling START baseline also pass with this PR.
• One design difference, for the record: a baseline message that carries neither phase_kind nor phase is still tagged PROFILING here. With this PR's publisher change that only happens if some sender does not fill phase, for example a component from an older build. Our version skipped such a scrape with a warning instead; either seems reasonable to us.

Happy to contribute the end-to-end test file (about 530 lines) to this branch or as a follow-up PR if that would be useful.

[chunfangamd]

Post-merge measurement of this fix in live AgentX runs

Thanks for merging this. We measured its effect in paired live runs and are recording the results here; no action is needed on this PR.

Setup. Each pair runs one frozen serving recipe (image, model, trace, durations) twice: H0 with AgentX 1.0.11 (104c7c80), H1 with 1.0.11 plus exactly this PR's change (the added and removed lines match this PR's diff; only diff context differs because of the base). The v1.0.12 release also contains #58, which H1 did not include.

Measure. Within each run, we compare the prefix-cache hit rate that the current InferenceX result adapter derives from the AIPerf server-metrics export with a client-side reference computed from the same run's per-request cached_tokens over the profiling requests. Both come from the same run, so run-to-run throughput variation does not enter. Values are the adapter rate minus the reference, in percentage points.

Serving framework (GPU)	Models	Pairs	H0: adapter − reference	H1: adapter − reference
ATOM (MI355X)	Kimi-K3, DeepSeek-V4-Pro, DeepSeek-V4.1-Flash, GLM-5.2, MiniMax-M3	12	−0.90 to −1.92	0.00 to +0.01
vLLM (B300)	Kimi-K3	1	−3.10	0.00
vLLM (B200)	MiniMax-M3	1	−1.08	0.00
Dynamo + vLLM (GB200)	Kimi-K3	1	−2.90	−0.02
SGLang (MI355X, B200, GB200)	Muse Spark, DeepSeek-V4.1-Flash	3	−0.02 to 0.00	−0.03 to 0.00
TRT-LLM (B200)	MiniMax-M3	1	−5.81	−5.76

Four more vLLM B200 pairs (DeepSeek-V4-Pro, DeepSeek-V4.1-Flash) have only an interval reference for H0, and that interval contains the adapter value; with H1 their reference is exact and the difference is 0.00.

Reading.

• With 1.0.11, the warmup START baseline opened the profiling window early, so the cold-cache warmup increment entered the profiling totals and the adapters understated the hit rate by about 1 to 3 points. With this fix, the ATOM, vLLM and Dynamo rows match the reference to within 0.02 points.
• In money terms, a lower hit rate overstates cached-input revenue. At a diagnostic price ratio (cached input at 10% of the input price), the H0 gap overstated token revenue by 6 to 9% on the non-Kimi ATOM rows and the B200 vLLM MiniMax-M3 row. At Kimi-K3 list prices, where cached input costs about 69% of uncached input, it was 0.5 to 0.9%.
• TRT-LLM is not changed by this fix: its gap comes from the adapter's gauge average, which Fix TRT reuse counters and P/D prompt totals / 修正 TRT 复用计数与 P/D 输入总量 InferenceX#4163 addresses. The SGLang rows were already within 0.03 points in both arms.

Limits. This isolates the collector effect. It does not show that every producer adapter is correct, and an H1 value is not by itself exact or eligible for pricing; that is the scope of SemiAnalysisAI/InferenceX#3964, #4163, #4164 and SemiAnalysisAI/InferenceX-app#1377. Most configurations have one pair (ATOM: Kimi-K3 4, DeepSeek-V4-Pro 3, GLM-5.2 and MiniMax-M3 2 each), all at concurrency 10 to 16. AMD runs used an internal MI355X cluster. NVIDIA runs are InferenceX E2E runs: B300 Kimi-K3 H0 and H1, B200 MiniMax-M3 vLLM H0 and H1, GB200 Kimi-K3 H0 and H1.

Prepared by Claude Opus 5.5 (Cursor agent) for Chun Fang, who approved publishing verified results; the values are computed from the immutable per-run summaries.