diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md index 2cf40a553a..f2087cfb7f 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md @@ -2,7 +2,7 @@ InferenceX owns the recipes in this directory. For every NVIDIA srt-slurm launch, the srt driver ([`infx/launch/drivers/srt/checkout.py`](../../../infx/launch/drivers/srt/checkout.py)) makes a job-local Git clone of the pinned submodule and copies this entire tree into `recipes/`. It records the actual revision in `srt-slurm-sha.txt`; power lanes copy that revision into `power-producer-sha.txt` for result validation. -The shared version is the Git submodule pointer at [`utils/srt-slurm`](../../../utils/srt-slurm), currently [v2.59.1](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.59.1) (`ae3d6877c46f3147e3a5dafa46ee4cd89322e52f`). Update that submodule pointer when upgrading, then run the recipe and integration checks. Do not add model-specific checkout branches to launchers. +The shared version is the Git submodule pointer at [`utils/srt-slurm`](../../../utils/srt-slurm), currently [v2.65.2](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.65.2) (`59920ea16a13017a990e449d034a495c0b05bc57`). Update that submodule pointer when upgrading, then run the recipe and integration checks. Do not add model-specific checkout branches to launchers. InferenceX requires srt-slurm 2.0 or newer and `schema: 2` recipes. Legacy recipe layouts are unsupported; migrate them before adding them to this tree. diff --git a/inferencex-e2e/configs/CONFIGS.md b/inferencex-e2e/configs/CONFIGS.md index 9a1aaca4cd..c329222f93 100644 --- a/inferencex-e2e/configs/CONFIGS.md +++ b/inferencex-e2e/configs/CONFIGS.md @@ -287,7 +287,7 @@ schema; unknown keys fail. measurement fails the job under `REQUIRE_POWER=1` and otherwise records an invalid verdict without energy metrics. NVIDIA clusters run `dcgm-exporter`; AMD clusters use srt-slurm's - [`kind: custom` schema](https://github.com/NVIDIA/srt-slurm/blob/ae3d6877c46f3147e3a5dafa46ee4cd89322e52f/docs/power-telemetry.md#gpu-exporter-labels-and-metrics), + [`kind: custom` schema](https://github.com/NVIDIA/srt-slurm/blob/59920ea16a13017a990e449d034a495c0b05bc57/docs/power-telemetry.md#gpu-exporter-labels-and-metrics), which requires `gpu_labels` and `gpu_metrics` and rejects unknown keys such as the former `power_profile`. The AMD exporter reads [`runners/srt-slurm/exporters/amd-power.json`](../runners/srt-slurm/exporters/amd-power.json); diff --git a/inferencex-e2e/utils/srt-slurm b/inferencex-e2e/utils/srt-slurm index ae3d6877c4..59920ea16a 160000 --- a/inferencex-e2e/utils/srt-slurm +++ b/inferencex-e2e/utils/srt-slurm @@ -1 +1 @@ -Subproject commit ae3d6877c46f3147e3a5dafa46ee4cd89322e52f +Subproject commit 59920ea16a13017a990e449d034a495c0b05bc57