From aed763cdab8bdac11843fb45808185e1a423a4ff Mon Sep 17 00:00:00 2001 From: Francesco Di Natale <3429989+FrankD412@users.noreply.github.com> Date: Mon, 19 Oct 2026 21:42:47 -0700 Subject: [PATCH] feat(agentx): run AgentX on upstream AIPerf MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the agentx-harness 1.0.12 client with ai-dynamo/aiperf at 8aa7039f872cd029f05a4c313fb8759e9a482cd3. The replay runs `aiperf profile --scenario inferencex-agentx-mvp` and passes the fork's agentx preset as explicit flags and AIPERF_* defaults, which callers can still override. Job events record the pinned AIPerf commit. 中文： 将 AgentX 客户端从 agentx-harness 1.0.12 切换为 ai-dynamo/aiperf 8aa7039f872cd029f05a4c313fb8759e9a482cd3。回放改为运行 `aiperf profile --scenario inferencex-agentx-mvp`，并以显式参数和 AIPERF_* 默认值传入 fork 中 agentx 预设的设置，调用方仍可覆盖。 作业事件记录所固定的 AIPerf 提交。 --- README.md | 4 +- README_zh.md | 4 +- inferencex-e2e/docs/agentx-standalone.md | 40 +++++++++++----- inferencex-e2e/infx/bench/agentic/replay.py | 34 ++++++++++---- .../infx/bench/agentic/requirements.txt | 2 +- inferencex-e2e/infx/bench/agentic/run.py | 7 +-- inferencex-e2e/infx/bench/agentic/venv.py | 12 ++--- inferencex-e2e/infx/datasets/proxy_to_weka.py | 2 +- inferencex-e2e/infx/launch/event.py | 11 +++-- .../infx/tests/bench/test_agentic_command.py | 12 +++-- .../infx/tests/bench/test_agentic_replay.py | 47 ++++++++++++++----- .../infx/tests/bench/test_agentic_runtime.py | 6 +-- .../infx/tests/launch/test_job_event.py | 9 ++-- inferencex-e2e/perf-changelog.yaml | 10 ++++ 14 files changed, 137 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 2968379bba..ed022ee219 100644 --- a/README.md +++ b/README.md @@ -94,8 +94,8 @@ PRs are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for more details on the For maintainer and agent instructions, start with [`AGENTS.md`](AGENTS.md). It links the configuration, workflow, eval, and recovery references. To benchmark an existing server without CI or Slurm, see -[Run AgentX-Harness Standalone](inferencex-e2e/docs/agentx-standalone.md) -for client installation and a direct `agentx profile` command. +[Run AgentX with AIPerf Standalone](inferencex-e2e/docs/agentx-standalone.md) +for client installation and a direct `aiperf profile` command. ## Acknowledgements & Supporters Thank you to Lisa Su and Anush Elangovan for providing the MI355X and CDNA3 GPUs for this free and open-source project. We want to recognize the many AMD contributors for their responsiveness and for debugging, optimizing, and validating performance across AMD GPUs. diff --git a/README_zh.md b/README_zh.md index 43b69a882e..f2232e0b6c 100644 --- a/README_zh.md +++ b/README_zh.md @@ -93,8 +93,8 @@ SGLang、vLLM、TensorRT-LLM、CUDA、ROCm 等 AI 软件通过核函式優化、 维护者与 Agent 请从 [`AGENTS.md`](AGENTS.md) 开始阅读，其中链接了配置、Workflow、评估与恢复操作参考。 无需 CI 或 Slurm 即可对已有服务进行基准测试，请参阅 -[独立运行 AgentX-Harness](inferencex-e2e/docs/agentx-standalone.md)， -其中包含客户端安装步骤和直接执行的 `agentx profile` 命令。 +[使用 AIPerf 独立运行 AgentX](inferencex-e2e/docs/agentx-standalone.md)， +其中包含客户端安装步骤和直接执行的 `aiperf profile` 命令。 ## 致谢与支持者 感谢 Lisa Su 与 Anush Elangovan 为这一免费开源项目提供 MI355X 与 CDNA3 GPU。我们也要感谢众多 AMD 贡献者的积极响应，以及他们在各类 AMD GPU 上进行调试、优化与性能验证所付出的努力。 diff --git a/inferencex-e2e/docs/agentx-standalone.md b/inferencex-e2e/docs/agentx-standalone.md index ae78978aab..b0131273cc 100644 --- a/inferencex-e2e/docs/agentx-standalone.md +++ b/inferencex-e2e/docs/agentx-standalone.md @@ -1,19 +1,18 @@ -# Run AgentX-Harness Standalone +# Run AgentX with AIPerf Standalone Run the AgentX client against an already-running OpenAI-compatible server. ## Install -Install the pinned PyPI release in a Python 3.11–3.13 virtual environment: +Install the pinned upstream AIPerf commit in a Python 3.11–3.13 virtual environment: ```bash python3 -m venv .venv-agentx source .venv-agentx/bin/activate -python -m pip install 'agentx-harness==1.0.12' +python -m pip install 'aiperf @ https://github.com/ai-dynamo/aiperf/archive/8aa7039f872cd029f05a4c313fb8759e9a482cd3.tar.gz' ``` -The package version matches the `agentx-v1.0.12` GitHub release. The `agentx` -command selects the `agentx` scenario by default. +The commit matches `inferencex-e2e/infx/bench/agentic/requirements.txt`. ## Run @@ -26,20 +25,32 @@ DATASET="semianalysis_cc_traces_weka_062126" CONC=8 OUTPUT_DIR="$PWD/results/agentx-c${CONC}-$(date -u +%Y%m%dT%H%M%SZ)" +export AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800 +export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800 +export AIPERF_UI_REALTIME_METRICS_ENABLED=true +export AIPERF_HTTP_TCP_USER_TIMEOUT=900000 + mkdir -p "$OUTPUT_DIR" -agentx profile \ +aiperf profile --scenario inferencex-agentx-mvp \ --url "$SERVER_URL" --endpoint /v1/chat/completions \ --model "$SERVED_MODEL_NAME" --tokenizer "$TOKENIZER" \ --tokenizer-trust-remote-code \ - --concurrency "$CONC" \ + --concurrency "$CONC" --benchmark-duration 3600 \ + --endpoint-type chat --streaming --stats-interval 30 --random-seed 42 \ + --trajectory-start-min-ratio 0.25 --trajectory-start-max-ratio 0.75 \ + --trace-idle-gap-cap-seconds 300 --use-server-token-count --no-gpu-telemetry \ + --num-dataset-entries 393 --slice-duration 1.0 \ + --failed-request-threshold 0.10 --warmup-requests-per-lane 10 --warmup-grace-period 1800 \ --public-dataset "$DATASET" \ --output-artifact-dir "$OUTPUT_DIR/aiperf_artifacts" \ 2>&1 | tee "$OUTPUT_DIR/aiperf.log" ``` -The `agentx` preset supplies a one-hour profile, ten warmup requests per lane, -and shared reporting and runtime settings. Explicit CLI arguments and `AIPERF_*` -environment variables override preset defaults. Model, tokenizer, dataset, +The `inferencex-agentx-mvp` scenario locks the replay invariants: agentic replay +timing, streaming, `ignore_eos`, no input truncation, first-turn cache busting, +the accepted trace loaders and a 900-second minimum profile. The remaining flags +and `AIPERF_*` variables are the AgentX run settings InferenceX passes for every +point; keep them when reproducing a result. Model, tokenizer, dataset, concurrency, and output remain caller inputs; `--tokenizer-trust-remote-code` is needed only for tokenizers with custom code. @@ -49,9 +60,9 @@ with `dsr1`, `qwen3.5`, or `qwen3.8next` explicitly select the 256k-capped corpus (`semianalysis_cc_traces_weka_062126_256k`). New model families use the full corpus unless added to `CAPPED_FAMILIES` in `infx/bench/agentic/traces.py`. `WEKA_LOADER_OVERRIDE` can pin either loader -regardless of model family; it does not configure the standalone `agentx` CLI. +regardless of model family; it does not configure the standalone `aiperf` CLI. -The default random seed is **42**. Keep this seed when reproducing AgentX results, +The random seed is **42**. Keep this seed when reproducing AgentX results, or pass `--random-seed 123` to use a different seed. Allow additional time for dataset preparation, warmup, and draining requests. @@ -64,7 +75,7 @@ Add `--server-metrics "${SERVER_URL}/metrics"` if the server exposes metrics. ## Optional session-affinity flags For DP-attention runs requiring router affinity, export these before -`agentx profile`. They add headers without replacing `X-Correlation-ID`. +`aiperf profile`. They add headers without replacing `X-Correlation-ID`. ```bash # Adds X-Dynamo-Session-ID; subagents also get X-Dynamo-Parent-Session-ID. @@ -73,3 +84,6 @@ export AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true # Also sends X-Session-ID, if a front-end router needs it. export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=true ``` + +AIPerf sends `X-Session-Affinity` with the correlation ID by default; set +`AIPERF_HTTP_X_SESSION_AFFINITY_FROM_CORRELATION_ID=false` to omit it. diff --git a/inferencex-e2e/infx/bench/agentic/replay.py b/inferencex-e2e/infx/bench/agentic/replay.py index 221677330d..fec69524ef 100644 --- a/inferencex-e2e/infx/bench/agentic/replay.py +++ b/inferencex-e2e/infx/bench/agentic/replay.py @@ -21,6 +21,21 @@ "AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING", ) +SCENARIO = "inferencex-agentx-mvp" +# AIPerf's scenario locks invariants but leaves these to the caller. +PRESET_FLAGS = ( + "--endpoint-type", "chat", "--streaming", "--stats-interval", "30", "--random-seed", "42", + "--trajectory-start-min-ratio", "0.25", "--trajectory-start-max-ratio", "0.75", + "--trace-idle-gap-cap-seconds", "300", "--use-server-token-count", "--no-gpu-telemetry", + "--num-dataset-entries", "393", "--slice-duration", "1.0", +) # fmt: skip +PRESET_ENVIRONMENT = { + "AIPERF_DATASET_CONFIGURATION_TIMEOUT": "1800", + "AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT": "1800", + "AIPERF_UI_REALTIME_METRICS_ENABLED": "true", + "AIPERF_HTTP_TCP_USER_TIMEOUT": "900000", +} + MIN_SCENARIO_DURATION_S = 900 FAST_DURATION_S = 1200 FAST_WARMUP_REQUESTS_PER_LANE = "1" @@ -135,17 +150,20 @@ def _server_metrics_urls(env: Mapping[str, str]) -> tuple[str, ...]: return tuple(parts) +def replay_environ(environ: Mapping[str, str]) -> dict[str, str]: + """``environ`` over the preset's AIPerf runtime settings; the caller's values win.""" + return {**PRESET_ENVIRONMENT, **environ} + + def replay_argv(cfg: ReplayConfig, cli: str | Path) -> list[str]: - """The ``agentx profile`` argv for ``cfg``; ``cli`` is the ``agentx`` executable.""" - argv = [str(cli), "profile", "--url", cfg.url] + """The ``aiperf profile`` argv for ``cfg``; ``cli`` is the ``aiperf`` executable.""" + argv = [str(cli), "profile", "--scenario", SCENARIO, "--url", cfg.url] argv += ["--model", cfg.model, "--tokenizer", cfg.tokenizer] argv += ["--concurrency", str(cfg.concurrency), "--benchmark-duration", str(cfg.duration)] - if cfg.live_failed_request_threshold is not None: - argv += ["--failed-request-threshold", cfg.live_failed_request_threshold] - if cfg.warmup_requests_per_lane is not None: - argv += ["--warmup-requests-per-lane", cfg.warmup_requests_per_lane] - if cfg.warmup_grace_period is not None: - argv += ["--warmup-grace-period", cfg.warmup_grace_period] + argv += PRESET_FLAGS + argv += ["--failed-request-threshold", cfg.live_failed_request_threshold or "0.10"] + argv += ["--warmup-requests-per-lane", cfg.warmup_requests_per_lane or "10"] + argv += ["--warmup-grace-period", cfg.warmup_grace_period or "1800"] if cfg.extra_inputs: argv += ["--extra-inputs", *cfg.extra_inputs] if cfg.dynamo_session_timeout is not None: diff --git a/inferencex-e2e/infx/bench/agentic/requirements.txt b/inferencex-e2e/infx/bench/agentic/requirements.txt index 0425bee6db..f91668089f 100644 --- a/inferencex-e2e/infx/bench/agentic/requirements.txt +++ b/inferencex-e2e/infx/bench/agentic/requirements.txt @@ -1,2 +1,2 @@ -agentx-harness==1.0.12 +aiperf @ https://github.com/ai-dynamo/aiperf/archive/8aa7039f872cd029f05a4c313fb8759e9a482cd3.tar.gz datasets>=4.7.0 diff --git a/inferencex-e2e/infx/bench/agentic/run.py b/inferencex-e2e/infx/bench/agentic/run.py index e4fda327af..4350a83494 100644 --- a/inferencex-e2e/infx/bench/agentic/run.py +++ b/inferencex-e2e/infx/bench/agentic/run.py @@ -17,6 +17,7 @@ REQUIRED as REPLAY_REQUIRED, ReplayConfig, replay_argv, + replay_environ, ) from infx.bench.agentic.venv import Runtime, bootstrap @@ -133,11 +134,11 @@ def execute(plan: Plan, runtime: Runtime, environ: Mapping[str, str]) -> int: rc = _open_power_window(plan, python, env) if rc: return rc - argv = replay_argv(cfg, runtime.agentx) + argv = replay_argv(cfg, runtime.aiperf) (plan.result_dir / "benchmark_command.txt").write_text(f"{shlex.join(argv)}\n") log = plan.result_dir / "benchmark.log" with proc.DeferSignals() as signals: - replay_rc = 0 if signals.received else proc.tee(argv, log, env) + replay_rc = 0 if signals.received else proc.tee(argv, log, replay_environ(env)) if signals.received: return 128 + signals.received return _score(plan, python, env, replay_rc) @@ -247,7 +248,7 @@ def main(argv: list[str]) -> int: plan = Plan.from_env(os.environ) runtime = Runtime.for_job(os.environ) if not runtime.active(): - print(f"Preparing the AgentX runtime in {runtime.venv}", flush=True) + print(f"Preparing the AIPerf runtime in {runtime.venv}", flush=True) rc = bootstrap(runtime) if rc: return rc diff --git a/inferencex-e2e/infx/bench/agentic/venv.py b/inferencex-e2e/infx/bench/agentic/venv.py index 779df64348..442eb0a803 100644 --- a/inferencex-e2e/infx/bench/agentic/venv.py +++ b/inferencex-e2e/infx/bench/agentic/venv.py @@ -1,4 +1,4 @@ -"""The isolated AgentX runtime: a per-job uv venv with the pinned PyPI release.""" +"""The isolated AIPerf runtime: a per-job uv venv with the pinned upstream commit.""" from __future__ import annotations @@ -20,7 +20,7 @@ @dataclass(frozen=True) class Runtime: - """Paths of one job's AgentX runtime directory.""" + """Paths of one job's AIPerf runtime directory.""" root: Path @@ -42,8 +42,8 @@ def python(self) -> Path: return self.venv / "bin" / "python" @property - def agentx(self) -> Path: - return self.venv / "bin" / "agentx" + def aiperf(self) -> Path: + return self.venv / "bin" / "aiperf" @property def hf(self) -> Path: @@ -77,8 +77,8 @@ def bootstrap(runtime: Runtime) -> int: "benchmark client dependency bootstrap failed; inspect network/package " "resolution before recipe repairs" ) - if not (_executable(runtime.agentx) and _executable(runtime.hf)): - raise BenchError(f"isolated AgentX environment is incomplete at {runtime.venv}") + if not (_executable(runtime.aiperf) and _executable(runtime.hf)): + raise BenchError(f"isolated AIPerf environment is incomplete at {runtime.venv}") return 0 diff --git a/inferencex-e2e/infx/datasets/proxy_to_weka.py b/inferencex-e2e/infx/datasets/proxy_to_weka.py index 1e5436d78e..2f6edb8323 100644 --- a/inferencex-e2e/infx/datasets/proxy_to_weka.py +++ b/inferencex-e2e/infx/datasets/proxy_to_weka.py @@ -3,7 +3,7 @@ Reads /.jsonl produced by `sample_proxy_traces.py` and writes /..//.json in the v1 weka trace format consumed by the kv-cache-tester replayer (see -agentx-harness: src/aiperf/dataset/loader/weka_trace_models.py). +aiperf: src/aiperf/dataset/loader/weka_trace_models.py). Subagent grouping mirrors the conversation-view algorithm from the SemiAnalysis claude-code-proxy: diff --git a/inferencex-e2e/infx/launch/event.py b/inferencex-e2e/infx/launch/event.py index b8ec37664a..52edb99008 100644 --- a/inferencex-e2e/infx/launch/event.py +++ b/inferencex-e2e/infx/launch/event.py @@ -28,7 +28,7 @@ AGENTIC = "agentic-coding" _SHA = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?") _PINNED = re.compile(r"@(sha256:[0-9a-f]{64})$") -_HARNESS = re.compile(r"^agentx-harness==(\S+)", re.MULTILINE) +_AIPERF = re.compile(r"^aiperf @ \S+/archive/([0-9a-f]{40})\.tar\.gz$", re.MULTILINE) _RESULT_GPUS = re.compile(r"_gpus_([0-9]+)") @@ -71,6 +71,7 @@ class Versions(_Record): srt_slurm_sha: str | None = None srtctl_version: str | None = None agentx_harness: str | None = None + aiperf_sha: str | None = None image_digest: str | None = None @@ -513,15 +514,15 @@ def _versions(env: Mapping[str, str], *, agentx: bool) -> Versions: head = _git_head(repository_root()) workflow = _text(env, "GITHUB_SHA") pinned = _PINNED.search(env.get("IMAGE", "")) - harness = None + aiperf = None if agentx: with contextlib.suppress(OSError): - match = _HARNESS.search(REQUIREMENTS.read_text()) - harness = match.group(1) if match else None + match = _AIPERF.search(REQUIREMENTS.read_text()) + aiperf = match.group(1) if match else None return Versions( inferencex_sha=head, workflow_sha=workflow if workflow != head else None, - agentx_harness=harness, + aiperf_sha=aiperf, image_digest=pinned.group(1) if pinned else None, ) diff --git a/inferencex-e2e/infx/tests/bench/test_agentic_command.py b/inferencex-e2e/infx/tests/bench/test_agentic_command.py index a60d67d2fe..854e3a5bbe 100644 --- a/inferencex-e2e/infx/tests/bench/test_agentic_command.py +++ b/inferencex-e2e/infx/tests/bench/test_agentic_command.py @@ -108,7 +108,7 @@ def _point(tmp_path: Path, **overrides: str | None) -> dict[str, str]: def _runtime(tmp_path: Path) -> Runtime: runtime = Runtime(tmp_path / "runtime") runtime.python.parent.mkdir(parents=True) - stubs = {runtime.python: FAKE_PYTHON, runtime.agentx: FAKE_AIPERF, runtime.hf: FAKE_HF} + stubs = {runtime.python: FAKE_PYTHON, runtime.aiperf: FAKE_AIPERF, runtime.hf: FAKE_HF} for path, text in stubs.items(): executable(path, text) return runtime @@ -350,6 +350,7 @@ def test_signal_during_the_replay_skips_scoring_and_exits_128_plus_n(tmp_path): out = pathlib.Path(argv[argv.index("--output-artifact-dir") + 1]) out.mkdir(parents=True, exist_ok=True) pathlib.Path(os.environ["REPLAY_ARGV"]).write_text(shlex.join([sys.argv[0], *argv])) +print("fake aiperf: configure timeout", os.environ.get("AIPERF_DATASET_CONFIGURATION_TIMEOUT")) with open(os.environ["EVENTS"], "a") as events: events.write("replay\n") record = { @@ -379,10 +380,10 @@ def test_signal_during_the_replay_skips_scoring_and_exits_128_plus_n(tmp_path): venv) for target; do :; done; exec "$REAL_PYTHON" -m venv --without-pip "$target" ;; pip) bin="$(dirname "$4")" - printf '#!%s\n' "$bin/python" > "$bin/agentx" - cat "$FAKE_AIPERF_SCRIPT" >> "$bin/agentx" + printf '#!%s\n' "$bin/python" > "$bin/aiperf" + cat "$FAKE_AIPERF_SCRIPT" >> "$bin/aiperf" printf '#!/bin/sh\necho "hf $*" >> "$EVENTS"\n' > "$bin/hf" - chmod +x "$bin/agentx" "$bin/hf" ;; + chmod +x "$bin/aiperf" "$bin/hf" ;; esac """ @@ -433,8 +434,9 @@ def frontend(method: str, path: str) -> tuple[int, object]: results = tmp_path / "results" command = (results / "benchmark_command.txt").read_text() assert command == (tmp_path / "replay-argv").read_text() + "\n" - assert shlex.split(command)[0] == str(tmp_path / "runtime/venv/bin/agentx") + assert shlex.split(command)[0] == str(tmp_path / "runtime/venv/bin/aiperf") assert "fake aiperf: profiled 1 request" in (results / "benchmark.log").read_text() + assert "fake aiperf: configure timeout 1800" in result.stdout assert "fake aiperf: profiled 1 request" in result.stdout aggregate = json.loads((tmp_path / "agentx.json").read_text()) assert (aggregate["conc"], aggregate["num_requests_total"]) == (8, 1) diff --git a/inferencex-e2e/infx/tests/bench/test_agentic_replay.py b/inferencex-e2e/infx/tests/bench/test_agentic_replay.py index 46eb7d2163..fb91c2ecfa 100644 --- a/inferencex-e2e/infx/tests/bench/test_agentic_replay.py +++ b/inferencex-e2e/infx/tests/bench/test_agentic_replay.py @@ -8,7 +8,7 @@ import pytest from infx.bench.agentic import traces -from infx.bench.agentic.replay import ReplayConfig, replay_argv +from infx.bench.agentic.replay import ReplayConfig, replay_argv, replay_environ from infx.bench.env import InputError BASE_ENV = { @@ -27,23 +27,46 @@ def _argv(**overrides: str | None) -> list[str]: env = {name: value for name, value in {**BASE_ENV, **overrides}.items() if value is not None} - return replay_argv(ReplayConfig.from_env(env, Path("/results")), "/venv/bin/agentx") + return replay_argv(ReplayConfig.from_env(env, Path("/results")), "/venv/bin/aiperf") def _value(argv: list[str], flag: str) -> str | None: return argv[argv.index(flag) + 1] if flag in argv else None -def test_default_point_leaves_the_replay_policy_to_the_scenario(): +PRESET = [ + "--endpoint-type", "chat", "--streaming", "--stats-interval", "30", "--random-seed", "42", + "--trajectory-start-min-ratio", "0.25", "--trajectory-start-max-ratio", "0.75", + "--trace-idle-gap-cap-seconds", "300", "--use-server-token-count", "--no-gpu-telemetry", + "--num-dataset-entries", "393", "--slice-duration", "1.0", +] # fmt: skip + + +def test_default_point_spells_out_the_agentx_preset_on_the_upstream_scenario(): assert _argv(MAX_MODEL_LEN="131072") == [ - "/venv/bin/agentx", "profile", "--url", "http://localhost:8000", + "/venv/bin/aiperf", "profile", "--scenario", "inferencex-agentx-mvp", + "--url", "http://localhost:8000", "--model", "deepseek-ai/DeepSeek-V4-Pro", "--tokenizer", "deepseek-ai/DeepSeek-V4-Pro", - "--concurrency", "8", "--benchmark-duration", "3600", "--tokenizer-trust-remote-code", + "--concurrency", "8", "--benchmark-duration", "3600", *PRESET, + "--failed-request-threshold", "0.10", "--warmup-requests-per-lane", "10", + "--warmup-grace-period", "1800", "--tokenizer-trust-remote-code", "--output-artifact-dir", "/results/aiperf_artifacts", "--public-dataset", "semianalysis_cc_traces_weka_062126", ] # fmt: skip +def test_preset_environment_fills_only_what_the_caller_left_unset(): + environ = replay_environ({"AIPERF_HTTP_TCP_USER_TIMEOUT": "60000", "PATH": "/bin"}) + + assert environ == { + "AIPERF_DATASET_CONFIGURATION_TIMEOUT": "1800", + "AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT": "1800", + "AIPERF_UI_REALTIME_METRICS_ENABLED": "true", + "AIPERF_HTTP_TCP_USER_TIMEOUT": "60000", + "PATH": "/bin", + } + + def test_a_recipe_tokenizer_replaces_the_model_id_as_tokenizer_only(): argv = _argv(TOKENIZER="/model", SERVED_MODEL_NAME="glm5_2") assert (_value(argv, "--model"), _value(argv, "--tokenizer")) == ("glm5_2", "/model") @@ -65,10 +88,12 @@ def test_opt_in_inputs_add_their_flags_in_place(): ) assert argv == [ - "/venv/bin/agentx", "profile", "--url", "http://localhost:8000", + "/venv/bin/aiperf", "profile", "--scenario", "inferencex-agentx-mvp", + "--url", "http://localhost:8000", "--model", "DeepSeek-V4-Pro", "--tokenizer", "deepseek-ai/DeepSeek-V4-Pro", - "--concurrency", "8", "--benchmark-duration", "3600", - "--failed-request-threshold", "0.25", "--warmup-grace-period", "3600", + "--concurrency", "8", "--benchmark-duration", "3600", *PRESET, + "--failed-request-threshold", "0.25", "--warmup-requests-per-lane", "10", + "--warmup-grace-period", "3600", "--extra-inputs", "temperature:0.7", "top_p:0.9", "--use-dynamo-conv-aware-routing", "--dynamo-session-timeout-seconds", "14400", "--tokenizer-trust-remote-code", @@ -83,8 +108,8 @@ def test_opt_in_inputs_add_their_flags_in_place(): @pytest.mark.parametrize( ("overrides", "duration", "warmup", "unsafe"), [ - ({"DURATION": "899"}, "899", None, True), - ({"DURATION": "900"}, "900", None, False), + ({"DURATION": "899"}, "899", "10", True), + ({"DURATION": "900"}, "900", "10", False), ({"AIPERF_EXPERIMENTAL_FAST": "1", "DURATION": "300"}, "1200", "1", False), ], ) @@ -201,4 +226,4 @@ def test_corpus_is_uncapped_unless_explicitly_capped_or_pinned(prefix, override, config = ReplayConfig.from_env(env, Path("/results")) assert (config.loader, config.dataset) == (loader, dataset) - assert _value(replay_argv(config, "/venv/bin/agentx"), "--public-dataset") == loader + assert _value(replay_argv(config, "/venv/bin/aiperf"), "--public-dataset") == loader diff --git a/inferencex-e2e/infx/tests/bench/test_agentic_runtime.py b/inferencex-e2e/infx/tests/bench/test_agentic_runtime.py index 8237b0ebab..16131a48b2 100644 --- a/inferencex-e2e/infx/tests/bench/test_agentic_runtime.py +++ b/inferencex-e2e/infx/tests/bench/test_agentic_runtime.py @@ -25,9 +25,9 @@ [ "$FAIL" = pip ] && exit 2 [ "$FAIL" = incomplete ] && exit 0 bin="$(dirname "$4")" - printf '#!/bin/sh\nexit 0\n' > "$bin/agentx" + printf '#!/bin/sh\nexit 0\n' > "$bin/aiperf" printf '#!/bin/sh\nexit 0\n' > "$bin/hf" - chmod +x "$bin/agentx" "$bin/hf" + chmod +x "$bin/aiperf" "$bin/hf" fi """ @@ -110,7 +110,7 @@ def test_failed_venv_creation_returns_uv_status_without_installing(tmp_path, too ("failure", "message"), [ ("pip", "benchmark client dependency bootstrap failed"), - ("incomplete", "isolated AgentX environment is incomplete"), + ("incomplete", "isolated AIPerf environment is incomplete"), ("installer", "uv installation did not create"), ], ) diff --git a/inferencex-e2e/infx/tests/launch/test_job_event.py b/inferencex-e2e/infx/tests/launch/test_job_event.py index e357955423..5ad41740a4 100644 --- a/inferencex-e2e/infx/tests/launch/test_job_event.py +++ b/inferencex-e2e/infx/tests/launch/test_job_event.py @@ -23,7 +23,10 @@ def test_annotation_escapes_workflow_command_data_and_title(): def test_an_agentic_point_records_no_sequence_lengths_and_its_pinned_versions(tmp_path, monkeypatch): requirements = tmp_path / "requirements.txt" - requirements.write_text("agentx-harness==9.8.7\ndatasets>=4.7.0\n") + sha = "0123456789abcdef" * 2 + "01234567" + requirements.write_text( + f"aiperf @ https://github.com/ai-dynamo/aiperf/archive/{sha}.tar.gz\ndatasets>=4.7.0\n" + ) monkeypatch.setattr(job_event, "REQUIREMENTS", requirements) digest = "sha256:" + "ab" * 32 env = { @@ -35,7 +38,7 @@ def test_an_agentic_point_records_no_sequence_lengths_and_its_pinned_versions(tm } # fmt: skip record = JobEventBuilder.begin(env).event assert (record.isl, record.osl) == (None, None) - assert (record.versions.agentx_harness, record.versions.image_digest) == ("9.8.7", digest) + assert (record.versions.aiperf_sha, record.versions.image_digest) == (sha, digest) assert record.topology.model_dump() == { "tp": None, "ep": None, "pp": None, "dp_attn": None, "gpu_count": None, "node_count": 3, "prefill": {"num_workers": 2, "tp": 4, "ep": 4, "dp_attn": True}, @@ -44,7 +47,7 @@ def test_an_agentic_point_records_no_sequence_lengths_and_its_pinned_versions(tm assert (record.github.pr_number, record.github.event_name) == (3807, "pull_request") fixed = JobEventBuilder.begin({**env, "SCENARIO_TYPE": "fixed-seq-len", "ISL": "8192"}).event - assert (fixed.isl, fixed.versions.agentx_harness) == (8192, None) + assert (fixed.isl, fixed.versions.aiperf_sha) == (8192, None) FIXED = { diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 73dc1a143c..9ee4f29c25 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10245,3 +10245,13 @@ - "The new image brings Kimi-K3 ROCm changes vllm-project/vllm#57392 (AITER Gluon KDA kernels for decode and spec decode), #61207 (stage fused KDA decode weights after loading), #61654 (MLA wrapper no longer reads W_UK_T before weight loading), #61895 (any-shape low-latency MoE router GEMM on MI355), #59406 (preshuffled FP4 GEMM config probed by logical K), #58381 (lighter MLA metadata build), #59070 (DCP prefill context kept in FP8 through AllGather), #59468 (DSpark draft slots no longer reserved in the scheduler token budget) and #61230 (AITER v0.1.25 base)." - "No change to the Inferact/Kimi-K3-DSpark draft's precision: the checkpoint is all BF16 with no quantization_config, and at both images the draft takes its own quantization config (get_draft_quant_config()), so the target quantization-config does not reach it. The draft runs BF16 weights with an fp8 KV cache." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4215 + +- config-keys: + - "*" + scenario-type: + - agentic-coding + description: + - "Replace the agentx-harness 1.0.12 client (SemiAnalysisAI/agentx-harness fork of AIPerf) with upstream ai-dynamo/aiperf at commit 8aa7039f872cd029f05a4c313fb8759e9a482cd3, run as aiperf profile --scenario inferencex-agentx-mvp." + - "The fork's agentx preset is passed explicitly: chat endpoint, streaming, stats interval 30 s, random seed 42, trajectory start ratios 0.25-0.75, trace idle-gap cap 300 s, server token counts, no GPU telemetry, 393 dataset entries, 1 s slices, failed-request threshold 0.10, 10 warmup requests per lane and a 1800 s warmup grace period, plus AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800, AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800, AIPERF_UI_REALTIME_METRICS_ENABLED=true and AIPERF_HTTP_TCP_USER_TIMEOUT=900000. Explicit recipe and workflow inputs still override them; the resolved benchmark configuration matches 1.0.12." + - "Upstream behavior that differs from 1.0.12: a SPAWN subagent's first turn keeps the parent's messages (ai-dynamo/aiperf#1555), every request also carries X-Session-Affinity with the correlation ID, a stall watchdog no longer cancels streaming requests (#1568) and a stalled benchmark fails instead of waiting indefinitely (#1501). The fork's SPAWN rollback at the sending cutoff and per-subagent sticky routing are upstream as #1540 and #1567." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX