Skip to content

Stop trusting the emSign roots Mozilla removed from NSS - #467

Open
adstastic wants to merge 2 commits into
omnesis-dev:mainfrom
adstastic:fix/distrust-emsign-roots
Open

adstastic wants to merge 2 commits into
omnesis-dev:mainfrom
adstastic:fix/distrust-emsign-roots

Conversation

@adstastic

Copy link
Copy Markdown

The issue

On 20 October 2026, eMudhra disclosed that the private keys of eight of its intermediate CAs (EM DV/OV/EV TLS CA - G3B/G3C and two code-signing CAs, plus their Sectigo cross-signs) can be recovered from the certificates themselves. Whoever holds one of those keys can issue a certificate for any hostname, and that certificate chains to roots that most trust stores accept (disclosure, impacted list).

Browsers are already protected: Firefox and Chrome blocked the intermediates on 17 October through OneCRL, CRLite and CRLSets, and Apple did on 19 October. Node.js is not. It checks no revocation, reads no browser blocklist, and its bundled root store still contains emSign ECC Root CA - G3. The test chain is leaf → EM DV TLS CA - G3B → emSign Root TLS CA - G3 → emSign ECC Root CA - G3 (bundled).

So every Omnesis Node process accepts a forged certificate for any host it connects to, including model APIs, OAuth and provider APIs, APNs/FCM, the npm registry during install and update, and a gateway in system-trust mode. On main today, fetch("https://dv-g3b-emudhra.we-have-a-problem.com/") returns 200 from the gateway's runtime, from a worker thread, and from npm.

Mozilla removed emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3 in NSS 3.131.1. Node took that set into main on 20 October (nodejs/node#66877). The October 2026 security releases ship it as "ca-cert (eMudhra Intermediate CA Compromise)" in v24.22.1, v26.12.1 and v22.23.4. The node_root_certs.h at those tags carries none of the three roots; v24.22.0 carried all three. The same releases fix several high-severity TLS hostname-verification bypasses (CVE-2026-105483, CVE-2026-106069).

The fix

Upgrading Node is the real fix, but the engines range still admits Node 24.16 to 24.22.0 and 26.1 to 26.12.0, and operators run whatever Node they installed. This PR keeps those installs safe and moves the repo's own Node forward.

  • .nvmrc → 24.22.1. CI (node-version: "24") and the Docker images (node:24-slim, node:24-bookworm-slim) use floating tags, so they pick the release up on their next run or build.

  • packages/core/src/tls-distrust.ts holds the fingerprints of the three roots NSS 3.131.1 removed. On import it drops them from the thread's default CA store (tls.setDefaultCACertificates). It matches on SHA-256 fingerprints, not on subject names.

    • AAA Certificate Services stays trusted. Node does not bundle it, and removing it elsewhere would break legitimate Sectigo-served hosts.
    • emSign Root CA - G1 stays trusted. NSS kept it.
  • Every thread gets the filter. The CA store is per thread: a worker starts with the full bundled set even after the main thread changed its own. The core barrel therefore imports tls-distrust as its first statement, so the main process and every worker or subprocess that loads @omnesis/core (writer, io, cpu, search, indexer, auth subprocess, whisper, relay) drops the roots before it can open a socket. A test pins that ordering.

  • Explicit ca: lists use the filtered roots. Code that passes its own ca: list bypasses the default store, so these now use publicRoots() (the bundled roots minus the distrusted ones):

    • verifiesUnderSystemTrust (tofu.ts), which decides whether a pinned client may move to verifying by name;
    • probeServedCertificate (served-by-gateway.ts);
    • GatewayTrust system roots in agent-integration. That package imports nothing of Omnesis at runtime, so it restates the set, and a parity test holds the two copies equal.
  • npm children. npm builds its own store from the same bundled set. omnesis update (spawnRunner), the Codex runtime installer and install.sh (and its website/ mirror) now run npm with NODE_OPTIONS=--import=<preload>. The preload applies the same filter inside npm's process.

    • The preload sits in a private mkdtemp directory, so no other user can replace it.
    • It filters only the default store, so a ca or cafile the user configured for npm still wins. This is the reason for a preload rather than npm_config_cafile, which would override such a setting.
    • It does nothing on a Node without setDefaultCACertificates.
  • Docs and changeset. A short "Public root certificates" section in security.html, plus a patch changeset.

On Node 24.22.1 or 26.12.1 the module finds nothing to remove and does nothing. Delete the entries, and with them the module, once the engines floor excludes every release that bundles them.

Verified against the live test host

Measured on a Node that still bundles the roots (v26.5.0):

Client main This branch
Gateway main thread (fetch) 200 (accepted) UNABLE_TO_GET_ISSUER_CERT_LOCALLY
Worker started with resolveWorkerEntry 200 UNABLE_TO_GET_ISSUER_CERT_LOCALLY
npm view --registry <test host> TLS accepted (FETCH_ERROR) UNABLE_TO_GET_ISSUER_CERT_LOCALLY
registry.npmjs.org, api.push.apple.com ok ok

What cannot be done today

  • Raising the engines floor to 24.22.1 / 26.12.1. This PR does not do it. A higher floor would make the installer and omnesis update upgrade Node on every host, which would also pick up the TLS hostname fixes. It is still a fleet-wide runtime change, so it belongs in its own PR. Until then, operators should upgrade Node themselves.
  • Blocking the intermediates themselves. Node has no intermediate blocklist; setDefaultCACertificates sets trust anchors only. Removing the roots breaks every chain Node can build from its bundled store. An operator who starts Omnesis with --use-system-ca / NODE_USE_SYSTEM_CA adds the OS store, and an OS store that still has AAA Certificate Services also accepts the Sectigo cross-signs. Blocking those would need a checkServerIdentity chain check on every connection, so this PR leaves them out, and the OS store has to be updated.
  • Pinning the gateway instead of adding it to the public roots. applyCaTrustInProcess (tofu.ts) adds the saved gateway certificate to the default store instead of trusting only that certificate. A collector or CLI pinned to a self-signed gateway therefore still accepts any publicly trusted certificate for the gateway's name or IP. This PR removes the roots behind the current incident; making gateway-client connections trust only the saved certificate is the root-cause fix. That is a separate, larger change and should be its own PR.
  • Android phones paired with system trust use the phone's CA store (Conscrypt), and Chrome's CRLSet does not protect apps. They depend on an Android or Mainline update; Android 8 and 9 may never get one. A trust-manager wrapper that rejects the intermediates' SPKIs is a separate native change. Phones paired by fingerprint are not affected, and neither is iOS: Apple blocked the intermediates on 19 October.
  • Hermes in system-trust mode uses Python's ssl.create_default_context(), so it trusts the OS store or certifi. Python's ssl cannot remove one root from a default context, so the operator has to update the host (certifi>=2026.10.19 or the OS ca-certificates). Hermes pairings that pin the gateway are not affected.
  • Tools Omnesis does not own. curl and git read the OS store. On macOS, /usr/bin/curl reads /etc/ssl/cert.pem, which Apple has not updated. The Claude Code (Bun) and Codex (Rust) agent backends use their own stores, and the curl | sh installer bootstrap uses the store of the curl that fetches it. None of these can be fixed in this repo.
  • Docker images. Rebuilding an older release's image to pick up a newer ca-certificates changes nothing, because Node ignores the OS store. The fix arrives with the next image release.

Operator notes

  • Upgrade Node to 24.22.1 or 26.12.1 on every host. That fixes the root store and the TLS hostname CVEs in the same step.
  • Run omnesis update on every host, including remote collectors and CLIs.
  • Rebuild the push relay image by hand; no workflow builds it.
  • Until then, keep collectors and agents on the tailnet rather than an untrusted LAN.
  • Do not remove AAA Certificate Services from system stores; it anchors legitimate Sectigo chains.

Test plan

  • packages/core/src/tls-distrust.test.ts:
    • the default store and publicRoots() are free of the roots, and nothing else is removed;
    • the barrel imports the module first;
    • agent-integration and install.sh restate the same fingerprints;
    • a child Node process trusts the roots without the preload and drops them with it, and existing NODE_OPTIONS are kept.
  • update.test.ts: a stand-in npm run by spawnRunner sees no distrusted root and keeps its own NODE_OPTIONS.
  • codex-runtime-installer.test.ts: the npm environment carries the preload.
  • trust.integration.test.ts: agent-integration's publicRoots() is the bundled set minus the emSign roots.
  • Live probe against the test host from the main thread, a worker thread and npm (table above).
  • packages/core, packages/agent-integration and the installer suites, typecheck:fast, lint on the changed files.
  • npm run checks:affected -- --base origin/main: 143 of 144 tasks passed. @omnesis/gateway:nx-unit failed 3 of 16,616 tests, none caused by this change:
    • model-credentials.test.ts (2) reads a TypeSafe key that the dev shell exports, and passes with it unset. This is a separate isolation gap in that test.
    • query-runner.test.ts (1) timed out at 10 s under the full gate's load, and passes in isolation.
  • install-sh-first-run.test.mjs: npm ci runs with the trust preload, and the installer removes the preload afterwards. npm run build keeps exactly the NODE_OPTIONS it was given.

🤖 Generated with Claude Code

eMudhra's G3B/G3C intermediate keys are recoverable from the certificates,
so anyone can issue a certificate for any host that chains to emSign ECC
Root CA - G3. Browsers block the intermediates through their own
revocation lists; Node checks no revocation and still bundles the root, so
every Omnesis process and the npm it runs accepted such a certificate.

Drop the three roots NSS 3.131.1 removed from each thread's default store
(the store is per thread, hence the barrel's first import), from the
explicit public-root lists, and from npm via a NODE_OPTIONS preload that
leaves a user-configured npm ca/cafile in charge. Remove the entries once
the pinned Node release stops bundling them.
The October 2026 Node security releases (24.22.1, 26.12.1, 22.23.4) ship the
NSS 3.131.1 root store. The runtime filter stays for operators on an older
Node the engines range still admits; it is a no-op on the new releases.
@adrienconrath

Copy link
Copy Markdown
Contributor

Thanks for this. Dropping the emSign roots Node still bundles is the right call, and putting it at the top of the core barrel is a clean way to make it hold for every thread. I reviewed the branch and reproduced each point below. Two of them break installs today, so I'd like those fixed before we merge.

Blocking

1. A first-run Linux install without Node fails at the first npm call

ensure_node creates NPM_TRUST_DIR (scripts/install.sh:768). On Linux with no Node, run_nodesource_setup ends with cleanup_temp_files (line 848), which now rm -rfs that directory. The variable stays set, so the npm() wrapper's := keeps the dead path and the heredoc write fails.

Reproduced in a fresh debian:bookworm-slim container (no Node) with sh install.sh --method package --client-only:

[omnesis] Node v24.22.1 ready

Installing the CLI
/t/install-pr.sh: 989: cannot create /tmp/tmp.PnEjlSex0C/distrust-roots.mjs: Directory nonexistent
EXIT=2

With main's install.sh, the same container gets past this point and reaches the registry. CI stays green because every install-e2e lane runs actions/setup-node first, so none of them takes the NodeSource path. Clearing NPM_TRUST_DIR in cleanup_temp_files, or creating it after Node is installed, would fix it. A first-run test on a host without Node would keep it fixed.

2. Every process fails to start on Node 24.0–24.4

tls-distrust.ts statically imports setDefaultCACertificates from node:tls, which only exists from Node 24.5.0, and the barrel imports the module first. Loading the module unchanged in the official images:

Node Result
24.4.0 SyntaxError: The requested module 'node:tls' does not provide an export named 'setDefaultCACertificates'
24.5.0 / 24.15.0 / 24.21.0 loads

The published manifests declare engines: >=24.0.0 (scripts/release/transform-manifest.mjs:120) and the installer's runtime check accepts any Node 24, so a user on 24.0–24.4 gets a gateway, collector and CLI, omnesis update included, that can't start. Either feature-detect it the way the preload already does (import tls from "node:tls" plus typeof tls.setDefaultCACertificates === "function"), or raise the floor everywhere it's declared.

Should fix

3. The preload file can disappear under a long-running process

distrustingNodeOptions writes the preload once into an mkdtemp directory and caches the URL for the life of the process. If that directory is deleted (systemd-tmpfiles or macOS periodic cleanup on a gateway that runs for weeks), every later npm the process starts fails. On Node 24.21: delete the directory after the first call, call again, and you get the same path, and a child started with that NODE_OPTIONS fails with ERR_MODULE_NOT_FOUND. Each process that calls it also leaves an omnesis-tls-* directory behind (two runs, two leftover directories). Checking that the file exists before reusing the cached URL, and removing the directory on exit, would cover both.

4. On the Node this PR pins, the tests pass without the filter

Node 24.22.1, the new .nvmrc and what CI's node-version: "24" currently resolves to, bundles none of the three roots (24.21.0: 3 of 121; 24.22.1: 0 of 118). As a mutation test, I made isDistrustedRoot return false:

  • On Node 24.22.1, all 8 tests in tls-distrust.test.ts still pass.
  • On Node 24.21.0, 3 of them fail, as they should.

So in CI these tests can't catch a wrong fingerprint or a broken PEM decode. Embedding the three emSign PEMs as fixtures, and asserting that isDistrustedRoot matches them and that the preload removes them from a store that contains them, would make them independent of the Node version. The publicRoots check in trust.integration.test.ts has the same gap.

5. Two download paths still use Node's full bundled store

  • The source launcher that install.sh writes (the heredoc ending at line 1938) runs npm ci from its recovery path (install_source_dependencies). The npm() wrapper only exists in the installer's own shell, so the launcher's npm ci runs without the preload.
  • The node-gyp header prefetch (packages/cli/src/update/node-gyp-headers.ts, and the same snippet in install.sh around line 958) starts node-gyp install --ensure through process.execPath, not npm. It downloads Node headers and their SHASUMS without the preload.

The new paragraph in website/docs/security.html says the npm runs from the installer and omnesis update don't trust these roots, so it would be accurate once both paths are covered.

6. omnesis update gives the preload to npm run build; the installer deliberately doesn't

install.sh skips npm run ("npm run reaches no registry, so a build keeps exactly the NODE_OPTIONS it was given"), but spawnRunner matches any npm/npx command. Calling spawnRunner with a stand-in npm and args: ["run", "build"] shows:

args=run build NODE_OPTIONS=--max-old-space-size=3072 --import=file:///tmp/omnesis-tls-…/distrust-roots.mjs

so every Node process in the build (tsc, workers) loads it. The two update paths should match; skipping run in spawnRunner would align them.

Minor

  • 7. The comment in packages/agent-integration/src/trust.ts:53 names distrusted-roots-parity.test.ts, which doesn't exist. The parity check is in packages/core/src/tls-distrust.test.ts.
  • 8. The preload's logic exists twice, in PRELOAD_SOURCE and in the install.sh heredoc (website/install.sh is the enforced mirror, so that one's fine). The parity test compares only the fingerprints, so a change to the regex or the feature check can land in one copy and not the other. Comparing the two preload bodies in the test would close that.
  • 9. NPM_TRUST_DIR is taken from the environment if it's set. I ran the installer with NPM_TRUST_DIR pointing at a directory holding a planted distrust-roots.mjs. The planted file was imported into the installer's npm prefix and npm view runs, and the whole directory was deleted on exit. Initialising it to empty at the top of the script, as SOURCE_CHECKOUT_TMP="" is, avoids both.

CI

Install e2e passed, but it doesn't take the NodeSource path (see 1). full-validation failed in node / unit on two tests that don't look related to this change: watch-firing-thread.test.ts (an ENOTEMPTY in its temp-directory cleanup) and agent-reading-anchor.test.ts (a portal scroll position). Both pass on main's latest run, and both files passed three times out of three when I ran them against this branch, so they look like flakes. A re-run after the fixes should confirm.

Happy to look again once these are addressed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants