Skip to content

[rush-daemon] Don't restart the daemon for volatile per-shell environment variables - #6062

Merged
Sean Larkin (TheLarkInn) merged 4 commits into
mainfrom
thelarkinn-fix-rushd-env-fingerprint-restarts
Sep 24, 2026
Merged

Sean Larkin (TheLarkInn) merged 4 commits into
mainfrom
thelarkinn-fix-rushd-env-fingerprint-restarts

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

The Rush daemon restarted its whole process (tier 2) whenever a request's environment differed from the daemon's startup environment in any variable. Per-shell variables such as PWD (after cd), OLDPWD, SHLVL, _, TERM, COLUMNS and WSL_INTEROP (new for every wsl.exe) each caused a restart. The warm graph was lost and alternating shells thrashed the daemon. This PR excludes a centrally defined, documented set of volatile variables from environment fingerprints. Variables that can affect the engine, the graph or operation hashes still trigger a restart.

Root cause

  • WorkspaceInputFingerprint.ts computed environmentHash from every defined entry of the environment.
  • WorkspaceRequestLifecycle.#classify maps any environmentHash difference from the startup fingerprint (the daemon's process.env) to Restart. Only _RUSH_LIB_PATH was normalized.
  • ProductionDaemonRequestResolver ran a second, independent whole-environment comparison (environmentIdentity) against the daemon's startup environment.
  • The client sends {...process.env} verbatim.

Fix

  • rush-lib: new @alpha APIs workspaceFingerprintIgnoredEnvironmentVariables and getWorkspaceFingerprintEnvironmentEntries(). captureWorkspaceInputFingerprintAsync now hashes only the normalized entries. The daemon startup fingerprint and every request fingerprint go through this function, so both sides get the same normalization.
  • rush-daemon: ProductionDaemonRequestResolver's environment check uses the same normalization, so the two checks can't disagree.
  • Ignored variables (names match case-insensitively on Windows):
    • shell bookkeeping: _ PWD OLDPWD SHLVL PS1 HISTFILE HISTSIZE
    • terminal presentation: TERM TERM_PROGRAM TERM_PROGRAM_VERSION TERM_SESSION_ID COLORTERM COLUMNS LINES LS_COLORS WINDOWID
    • session handles: WSL_INTEROP WSLENV SSH_CLIENT SSH_CONNECTION SSH_TTY SSH_AUTH_SOCK TMUX TMUX_PANE STY XDG_SESSION_ID XDG_SESSION_TYPE DBUS_SESSION_BUS_ADDRESS
    • INIT_CWD: Rush strips it from every lifecycle env and sets it explicitly where needed
    • client routing: RUSH_DAEMON and RUSH_DAEMON_AUTO_START only select or start a daemon; RUSH_DAEMON_EXPERIMENTAL is read per request from the envelope
  • Still fingerprinted:
    • all other RUSH_* variables, e.g. RUSH_BUILD_CACHE_*, RUSH_PARALLELISM, and daemon resource settings such as RUSH_DAEMON_WATCH and RUSH_DAEMON_WARM_*
    • NODE_OPTIONS and other NODE_* variables
    • npm/pnpm config
    • PATH (exact, since order matters), HOME
    • any unknown variable
  • I grepped rush-lib, rush-daemon, terminal and node-core-library: none of them reads any ignored variable to configure the engine, build the graph or compute operation hashes.

Which environment do operations receive?

This part is unchanged; I'm documenting it here and in the rush-daemon README:

  • Phased operations get the daemon's own environment. OperationGraph builds the operation env from { ...process.env }, and dependsOnEnvVars hashing uses the same environment, so cache keys stay consistent with what operations see.
  • So for the ignored variables, an operation sees the daemon's startup value, not the submitting shell's value. For PWD/SHLVL/_, a child shell recomputes them anyway. INIT_CWD is stripped by Rush. The rest are presentation or session handles.
  • Caveat: WSL_INTEROP and SSH_AUTH_SOCK can go stale if the daemon's original shell exits. That was already true for any long-lived daemon.
  • Rushx and global commands still get the full request environment, as with native Rush.

Tests

  • rush-lib WorkspaceInputFingerprint.test.ts:
    • changes to PWD/OLDPWD/SHLVL/_/TERM/COLUMNS/WSL_INTEROP/SSH_*/INIT_CWD/RUSH_DAEMON* routing vars give the same hash
    • FOO, RUSH_BUILD_CACHE_ENABLED, RUSH_BUILD_CACHE_WRITE_ALLOWED, RUSH_DAEMON_WATCH, NODE_OPTIONS, NPM_CONFIG_REGISTRY, a reordered PATH and HOME each give a different hash
  • rush-daemon WorkspaceReloadTierStatus.test.ts: on a warm daemon with a successor launcher configured, requests that differ only in volatile variables stay at tier 0 with the same generation, graph and pid, and no operation re-runs. The existing test for a real environment change still covers the restart path.
  • Updated the API reports (rush-lib, rush-daemon) and the rush-sdk export snapshot for the two new exports.

Linux validation

Lab: WSL Ubuntu 24.04, Node 22. The patch was applied to a Linux-native clone of main.

  • rush build --to @rushstack/rush-cli-client --to @rushstack/rush-sdk: SUCCESS (lint clean)
  • rush test --only @rushstack/rush-daemon --only @rushstack/rush-sdk: SUCCESS
  • heft test --test-path-pattern WorkspaceInputFingerprint (rush-lib): 4/4 passed

Repro ($LAB/fixes/env-fingerprint-repro.sh): 12-project mkws-synth workspaces, all steps in one bash process with _ pinned and RUSH_DAEMON=1. BEFORE = the unfixed rush-client from main; AFTER = node <fix>/apps/rush-cli-client/bin/rush-client. "restored" = operations restored from the build cache; "0 ops" = warm skip with no output.

step BEFORE pid / wall / ops AFTER pid / wall / ops
base (cold) 2942347 / 3.61s / 12 restored 2965044 / 3.73s / 12 restored
base (warm) same / 1.10s / 0 ops same / 1.23s / 0 ops
OLDPWD=/x new / 4.13s / 12 restored same / 1.17s / 0 ops
SHLVL=7 new / 4.06s / 12 restored same / 1.04s / 0 ops
TERM=dumb COLUMNS=91 new / 4.32s / 12 restored same / 0.89s / 0 ops
cd packages/p03 (PWD/OLDPWD) new / 4.14s / 12 restored same / 0.99s / 0 ops
back to root new / 4.52s / 12 restored same / 0.92s / 0 ops
WSL_INTEROP=/run/WSL/999_interop new / 4.90s / 12 restored same / 0.88s / 0 ops
FOO=1 (unknown var, restarts by design) new / 4.28s / 12 restored new / 3.95s / 12 restored
back without FOO new / 4.27s new / 3.86s
RUSH_BUILD_CACHE_ENABLED=0 (engine setting) new / 8.13s new / 8.10s

After the fix, volatile-only differences keep the same daemon pid and re-run no operations (a warm skip at ~1s instead of a ~4s restart). Real engine inputs still restart the daemon.

Test note: the lifecycle test sends the same argv as the warm-up build. A different argv changes the command-parameter identity and is a tier-1 reload by design. A same control row in the test guards against confusing the two. PWD does not feed the command identity.

Follow-ups (out of scope)

  • lastRestartReason in daemon status (e.g. "environment: FOO added"). This needs a protocol/status field in the strict-lint protocol packages, so I left it for a separate PR.
  • A request-scoped overlay of the ignored variables onto the phased operation environment, so operations see the submitting shell's TERM/WSL_INTEROP/SSH_AUTH_SOCK.
  • Keeping variables fingerprinted when a project names them in dependsOnEnvVars.

Fixes #6047

This came out of the automated rushd Linux performance/behavior analysis (Rushd Hive, board BUG #20).

…rints

Fixes #6047

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@TheLarkInn
Sean Larkin (TheLarkInn) merged commit c086161 into main Sep 24, 2026
11 checks passed
@TheLarkInn
Sean Larkin (TheLarkInn) deleted the thelarkinn-fix-rushd-env-fingerprint-restarts branch September 24, 2026 18:00
@github-project-automation github-project-automation Bot moved this from Needs triage to Closed in Bug Triage Sep 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unconditional exclusions can invalidate dependsOnEnvVars semantics and preserve stale WSL or SSH session handles.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Centralizes daemon environment normalization to avoid restarts caused by volatile shell variables.

Changes:

  • Adds shared environment-filtering APIs.
  • Applies normalization to fingerprints and daemon checks.
  • Adds tests, documentation, API reports, and change records.
File Description
libraries/​rush-sdk/​src/​test/​__snapshots__/​script.test.ts.snap Records new exports.
libraries/​rush-lib/​src/​index.ts Exports normalization APIs.
libraries/​rush-lib/​src/​api/​WorkspaceInputFingerprint.ts Filters environment fingerprints.
libraries/​rush-lib/​src/​api/​test/​WorkspaceInputFingerprint.test.ts Tests filtering behavior.
libraries/​rush-daemon/​src/​test/​WorkspaceReloadTierStatus.test.ts Tests warm daemon reuse.
libraries/​rush-daemon/​src/​ProductionDaemonRequestResolver.ts Shares normalized comparison.
libraries/​rush-daemon/​README.md Documents environment semantics.
common/​reviews/​api/​rush-lib.api.md Updates rush-lib API report.
common/​reviews/​api/​rush-daemon.api.md Updates daemon API report.
common/​changes/​@rushstack/​rush-daemon/​env-fingerprint_2026-09-23.json Adds daemon change record.
common/​changes/​@microsoft/​rush/​env-fingerprint_2026-09-23.json Adds Rush change record.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

*
* @alpha
*/
export const workspaceFingerprintIgnoredEnvironmentVariables: ReadonlySet<string> = new Set([
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Closed

3 participants