Repository navigation
fix(router): pin SPAWN subagents under their own id so they stay on one worker - #52
Merged
cquil11 merged 2 commits intoOct 15, 2026
Merged
Conversation
…ne worker In agentic replay, SPAWN subagent credits were routed only through the parent's sticky entry. When the parent had no entry (it already sent its final turn, it is gated on a join at replay start, or the lane is rootless), every subagent turn was routed least-loaded and bounced across workers. Each worker that served the subagent but not its final turn kept a UserSession copy until process exit, leaking memory over long runs. A SPAWN child now pins under its own x_correlation_id on its first turn, on the parent's worker when the parent's entry is live, and releases the entry on its own final turn exactly as roots do. A SPAWN child that stops before its final turn keeps the entry until run end, as roots already do. Multi-turn SPAWN children now count toward active_sessions, the first least-loaded tie-break, like roots. FORK children of a SPAWN child now route via that child's own entry and co-locate with it (previously they routed least-loaded on every turn and lost the inherited context). FORK routing under roots is unchanged. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Also renames the SampledSession routing_key test, whose name claimed SPAWN children route to the parent's worker on every turn. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@b5d7652b50aceb9e0c4ca77daf9ae8f472ac77fdRecommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@b5d7652b50aceb9e0c4ca77daf9ae8f472ac77fdLast updated for commit: |
cquil11
approved these changes
Oct 15, 2026
cquil11
left a comment
cquil11
left a comment
There was a problem hiding this comment.
Approved following the PR52-only c6656 AgentX run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38683542278/job/116253300866. The run completed with zero profiling errors and no child-worker migrations. Independent reconstruction from all 781,455 profiling records reproduced the core exported statistics and all 186,324,783 inter-chunk samples, including percentiles, within floating-point/export precision. Final metric and artifact coverage was retained.
cquil11
pushed a commit
that referenced
this pull request
Oct 16, 2026
Keep each SPAWN child on its initial worker until its final turn, avoiding stale session copies across workers. (cherry picked from commit 9adff5e)
FrankD412
added a commit
to ai-dynamo/aiperf
that referenced
this pull request
Oct 16, 2026
SPAWN children routed only through their parent's sticky entry (``routing_key = parent_correlation_id``) and were forbidden to mint one of their own. Whenever the parent entry was missing, every child turn went to the least-loaded worker: after the root's final turn (``has_forks`` is computed from FORK branches only, so a final turn with SPAWN branches pops the entry), when the root is gated on a join at replay start, and for rootless replay lanes, where ``trajectory_source._snapshot_for`` samples a trace after its root finished but while background ``::fa:`` chains are still alive, so the children's ``parent_correlation_id`` names a root that never sends a request and the entry never exists at all. Each worker a child visited decoded and cached the full conversation in ``UserSessionManager`` and only the worker serving the final turn evicted, so every other copy stayed until process exit. In a one-hour agentic replay at concurrency 256 against the mock server, 300 of 3,998 sessions (all SPAWN children) were served by more than one worker, 5,869 switches in total, roughly 1,300 orphaned session copies; worker private heap hit a 20 GiB cgroup cap in about 18 minutes and three workers were OOM-killed. A SPAWN child now pins under its own ``x_correlation_id`` from its first turn: on the parent's worker when the parent's entry is live, otherwise least-loaded, and releases the entry on its own final turn exactly as roots do. Multi-turn SPAWN children therefore count toward ``active_sessions``. FORK children of a SPAWN child alias onto the child's entry via ``register_child_routing`` and co-locate with it; FORK routing under roots is unchanged. With the same fix applied to the vendored 0.12.0 router, the same run had 0 of 4,777 sessions on more than one worker and 0 OOM kills. Ported from SemiAnalysisAI#52 by Anthony Casagrande onto main's alias-aware router. Not addressed here: a SPAWN child that stops before its final turn keeps its router entry until run end, as roots already do; and ``PhaseRunner`` still hands ``BranchOrchestrator`` ``sticky_router=None`` (it reads a ``sticky_router`` attribute off the router itself), so FORK refcount release never runs in production. Both are pre-existing. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
AgentX subagents (SPAWN children) bounced between workers whenever their parent had no sticky entry, and every worker they touched kept a copy of the session until the run ended. A memory leak that grows with run length and worker count. Subagents now pin to one worker by their own id.
Problem
SPAWN children routed only via the parent's sticky entry. That entry is often missing (parent already finished, gated on a join at replay start, or a rootless lane), so each subagent turn went to the least-loaded worker. In a live
--scenario agentxrun about half of all subagent turns had no parent entry, and 22 stale session copies leaked in 2 minutes. Each copy is 448 KB–4 MB (measured in a separate 15-minute run).Fix
A SPAWN child pins under its own
x_correlation_idon its first turn (on the parent's worker if that entry is live) and releases it on its own final turn, the same as roots.Performance
send_creditrouting cost, network send stubbed out (40k credits, 8 workers):masterSubagent credits that hit their pinned worker cost ~25 ns more (entry create/pop), outweighed by no longer scanning for the least-loaded worker.
Follow-up
BranchOrchestratornever receives the router in production (PhaseRunnerlooks up asticky_routerattribute the router doesn't have), so FORK refcounting and early-stop release are dead code. Fixing that surfaces separate FORK refcount bugs; it'll be its own PR.Testing
Unit, component-integration (DAG/agentic) and integration (DAG/AgentX) suites pass; the two unit failures also fail on
master. Live AgentX leak check before/after.