Repository navigation
fix: mitigate compromised eMudhra CA trust across AWF - #10343
Conversation
Apply a shared SPKI distrust policy to controlled Node and OpenSSL trust paths, stage sanitized host CA stores without modifying host inputs, build gh with patched Go, and authenticate release bootstrap with constrained provenance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89097bc0-0800-4855-965e-ceaab0cef6d3
Documentation PreviewDocumentation has been built for this PR. To view locally:
Built from commit a61346c |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
@copilot address review feedback |
…zation' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Use isolated signed action fixtures while preserving mandatory production provenance. Retain an optional real-release verification job. Support Node's null CA default semantics using filtered roots, and exercise the Node 20 fallback without newer TLS APIs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89097bc0-0800-4855-965e-ceaab0cef6d3
| if grep -Fqx "$key" "$policy_dir/distrusted-spki.txt"; then | ||
| echo "AWF trust policy: removed distrusted CA key $key" >&2 | ||
| continue | ||
| fi |
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 93.40% | 93.41% | ➡️ +0.01% |
| Statements | 91.96% | 91.96% | ➡️ +0.00% |
| Functions | 91.16% | 91.15% | ➡️ -0.01% |
| Branches | 85.99% | 85.95% | 📉 -0.04% |
📁 Per-file Coverage Changes (3 files)
| File | Lines (Before → After) | Statements (Before → After) |
|---|---|---|
src/enclave/host-executor-server.ts |
91.0% → 91.0% (+0.00%) | 89.7% → 90.0% (+0.29%) |
src/nvx/one-shot-adapter.ts |
82.9% → 83.4% (+0.56%) | 80.2% → 80.7% (+0.52%) |
src/cli.ts |
85.7% → 87.5% (+1.79%) | 85.7% → 87.5% (+1.79%) |
✨ New Files (1 files)
src/tls-trust-policy.ts: 94.1% lines
Coverage comparison generated by scripts/ci/compare-coverage.ts
There was a problem hiding this comment.
🟡 Changes recommended
Confirmed trust-handling and runtime regressions need correction before this security mitigation can be deployed safely.
10 open issues
What changed in this PR
Mitigates disclosed eMudhra keys across AWF-controlled TLS trust and release installation paths.
Changes:
- Centralizes CA distrust and distributes filtered OpenSSL/Node trust across images and chroots.
- Updates image-owned
ghtoolchains and requires authenticated release metadata and image pins. - Adds regression checks and rollout documentation.
| File | Description |
|---|---|
| tests/entrypoint-phase-functions.test.sh | Tests filtered chroot trust staging. |
| src/tls-trust-policy.ts | Defines canonical Node distrust policy. |
| src/squid/ssl-bump.ts | Configures filtered upstream trust. |
| src/squid-config-ssl-bump.test.ts | Checks Squid trust configuration. |
| src/cli.ts | Loads host CLI trust policy. |
| scripts/security/tls-runtime-smoke.cjs | Exercises Node TLS enforcement. |
| scripts/security/install-system-trust.sh | Installs filtered system trust. |
| scripts/security/install-node-trust.sh | Installs Node preload launcher. |
| scripts/security/generate-trust-policy.cjs | Generates synchronized component policies. |
| scripts/security/fixtures/emsign-ecc-g3.pem | Supplies distrusted-root fixture. |
| scripts/security/filter-ca-store.sh | Filters certificates by key. |
| scripts/security/chroot-trust-smoke.sh | Checks Linux trust overlays. |
| scripts/security/ca-trust.test.ts | Covers filtering and image wiring. |
| scripts/ci/release-bootstrap-provenance.test.ts | Covers authenticated release bootstrap. |
| README.md | Links distrust documentation. |
| install.sh | Requires authenticated release provenance. |
| docs/security-ca-trust.md | Documents coverage and rollout boundaries. |
| docs/releasing.md | Documents release authentication requirements. |
| containers/trust/node-trust.cjs | Shared generated Node policy. |
| containers/trust/install-system-trust.sh | Shared system-trust installer. |
| containers/trust/install-node-trust.sh | Shared Node launcher installer. |
| containers/trust/filter-ca-store.sh | Shared certificate filter. |
| containers/trust/distrusted-spki.txt | Shared key denylist. |
| containers/squid/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/squid/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/squid/trust/distrusted-spki.txt | Generated key denylist. |
| containers/squid/Dockerfile | Filters Squid image trust. |
| containers/router/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/router/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/router/trust/distrusted-spki.txt | Generated key denylist. |
| containers/router/Dockerfile | Configures filtered router trust. |
| containers/router/.dockerignore | Includes trust build files. |
| containers/gh-aw-node/trust/node-trust.cjs | Generated Node policy. |
| containers/gh-aw-node/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/gh-aw-node/trust/install-node-trust.sh | Generated Node launcher installer. |
| containers/gh-aw-node/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/gh-aw-node/trust/distrusted-spki.txt | Generated key denylist. |
| containers/gh-aw-node/Dockerfile | Installs system and Node filtering. |
| containers/enclave/Dockerfile | Adds trust filtering across enclave stages. |
| containers/dind-ubuntu/trust/node-trust.cjs | Generated Node policy. |
| containers/dind-ubuntu/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/dind-ubuntu/trust/install-node-trust.sh | Generated Node launcher installer. |
| containers/dind-ubuntu/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/dind-ubuntu/trust/distrusted-spki.txt | Generated key denylist. |
| containers/dind-ubuntu/Dockerfile | Installs DinD trust filtering. |
| containers/cloud-hypervisor/Dockerfile | Filters guest-image trust stores. |
| containers/cli-proxy/trust/node-trust.cjs | Generated Node policy. |
| containers/cli-proxy/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/cli-proxy/trust/install-node-trust.sh | Generated Node launcher installer. |
| containers/cli-proxy/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/cli-proxy/trust/distrusted-spki.txt | Generated key denylist. |
| containers/cli-proxy/entrypoint.sh | Filters combined gateway trust. |
| containers/cli-proxy/Dockerfile | Updates Go and installs filtering. |
| containers/build-tools/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/build-tools/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/build-tools/trust/distrusted-spki.txt | Generated key denylist. |
| containers/build-tools/Dockerfile | Updates Go and filters trust. |
| containers/api-proxy/trust/node-trust.cjs | Generated Node policy. |
| containers/api-proxy/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/api-proxy/trust/install-node-trust.sh | Generated Node launcher installer. |
| containers/api-proxy/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/api-proxy/trust/distrusted-spki.txt | Generated key denylist. |
| containers/api-proxy/Dockerfile | Installs proxy trust enforcement. |
| containers/agent/trust/node-trust.cjs | Generated Node policy. |
| containers/agent/trust/install-system-trust.sh | Generated system-trust installer. |
| containers/agent/trust/install-node-trust.sh | Generated Node launcher installer. |
| containers/agent/trust/filter-ca-store.sh | Generated certificate filter. |
| containers/agent/trust/distrusted-spki.txt | Generated key denylist. |
| containers/agent/entrypoint.sh | Stages filtered host-trust overlays. |
| containers/agent/Dockerfile.minimal | Adds minimal-image trust filtering. |
| containers/agent/Dockerfile | Updates Go and installs filtering. |
| action.yml | Verifies releases and requires image digests. |
| .github/workflows/release.yml | Attests payloads and release identity. |
Review effort: Max
| echo "[entrypoint][WARN] SSL Bump enabled but CA certificate not found" | ||
| fi | ||
| fi | ||
| sh /opt/awf/trust/install-system-trust.sh |
| for candidate in "${directories[@]}" \ | ||
| /host/etc/ssl/cert.pem /host/etc/pki/tls/cert.pem; do |
| cp /opt/awf/trust/node-trust.cjs "$trust/node-trust.cjs" | ||
| # The /run staging root is container-local; none of these operations writes | ||
| # through the read-only host /etc bind mounts. | ||
| for candidate in "${directories[@]}"; do |
| RUN apk add --no-cache openssl ca-certificates \ | ||
| && sh /opt/awf/trust/install-system-trust.sh \ | ||
| && sh /opt/awf/trust/install-node-trust.sh |
| # | ||
| # Requirements: | ||
| # - curl | ||
| # - A trusted gh CLI with attestation verify support (and GitHub authentication) |
| const script = ` | ||
| const tls = require('tls'), {X509Certificate,createHash} = require('crypto'); | ||
| const {distrustedSpkis} = require(${JSON.stringify(path.join(policyDir, 'node-trust.cjs'))}); | ||
| for (const pem of tls.getCACertificates('default')) { |
| awk -v dir="$work" ' | ||
| /-----BEGIN CERTIFICATE-----/ { if (active) exit 1; active=1; count++; file=dir "/cert-" count ".pem" } | ||
| active { print > file } | ||
| /-----END CERTIFICATE-----/ { if (!active) exit 1; close(file); active=0 } | ||
| /-----BEGIN TRUSTED CERTIFICATE-----/ { exit 1 } | ||
| END { if (active || !count) exit 1 } | ||
| ' "$work/input.pem" || { echo "AWF trust policy: invalid or empty PEM CA store" >&2; exit 1; } |
| const defaults = tls.getCACertificates | ||
| ? [...tls.getCACertificates('default')] | ||
| : [...tls.rootCertificates]; |
| tls.createSecureContext = (options = {}) => createSecureContext({ | ||
| ...options, | ||
| ca: options.ca === undefined ? filteredDefaults : filterCaCertificates(options.ca), | ||
| }); |
| cmp "${host_root}/etc/pki/tls/certs/ca-bundle.crt" "${tmp_dir}/original.crt" | ||
| grep -q 'remount,ro,bind' "${tmp_dir}/mounts" | ||
| grep -q '/etc/pki/tls/certs' "${tmp_dir}/mounts" | ||
| [ "${SSL_CERT_DIR}" = "/run/awf-lib/trust/certs" ] |
Exercise modern and legacy Node trust APIs, null/default/explicit CA options, malformed or unreadable stores, empty results, and idempotent installation. Cover every canonical policy branch without weakening coverage gates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89097bc0-0800-4855-965e-ceaab0cef6d3
|
✅ Copilot review passed with no inline comments. @lpcox Add the |
|
✅ Build Test Suite completed successfully!
|
|
✅ Smoke Claude passed
|
|
✅ Security Guard completed successfully! Security GuardOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅
|
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.
|
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤
|
|
🪪 Smoke Copilot BYOK AOAI (Entra) is testing Azure OpenAI BYOK (Entra / GitHub OIDC) mode on this pull request... |
|
🔑 Smoke Copilot BYOK AOAI (api-key) is testing Azure OpenAI BYOK (api-key) mode on this pull request... |
|
🪪 Smoke Copilot BYOK AOAI (Entra) is testing Azure OpenAI BYOK (Entra / GitHub OIDC) mode on this pull request... |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅
|
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.
|
|
🔑 Smoke Copilot BYOK AOAI (api-key) is testing Azure OpenAI BYOK (api-key) mode on this pull request... |
|
✅ Build Test Suite completed successfully!
|
|
✅ Smoke Gemini completed. All facets verified. 💎
|
|
🚀 Security Guard has started processing this pull request |
|
Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.
|
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 Smoke CodexOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓
|
| local directories=( | ||
| /host/etc/ssl/certs | ||
| /host/etc/pki/ca-trust/extracted/pem | ||
| /host/etc/pki/tls/certs | ||
| ) |
|
Smoke test (Copilot):
|
|
Cloud Hypervisor + Copilot smoke test
|
|
EGRESS_RESULT allow=pass deny=pass
cc @lpcox
|
Smoke Claude Results
Overall result: PASS
|
|
Smoke services: ✅ Redis PONG · ✅ pg_isready · ✅ SELECT 1 → PASS
|
|
OTEL smoke test
|
Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS
Running in direct BYOK mode via api-proxy → api.githubcopilot.com
|
🏗️ Build Test Suite Results
Overall: 4/8 ecosystems passed — FAIL Failures
All 8 repositories cloned successfully. The
|
Chroot version comparison
Node.js differs, so the tests did not all pass and the
|
|
PR titles:
✅ GitHub merged PRs Overall status: FAIL
|
Smoke Test: Gemini Engine Validation
|
Summary
Mitigate the October 2026 eMudhra intermediate-key disclosure across AWF-controlled TLS and release-bootstrap paths. There is no evidence of AWF exploitation. The disclosed TLS issuer keys can support interception of traffic to already allowed domains; the two code-signing issuers are not themselves TLS issuers.
NODE_OPTIONSand propagates it to child Node processes; the host AWF CLI loads the same policy. This covers HTTPS, direct TLS/websocket connections, OIDC/model discovery, fetch, and telemetry using those clients./run/awf-lib/trust, overlay mounted Debian/RHEL trust directories read-only, and preserve legitimate private/AWF interception CAs. Never rewrite the host trust store. Filter dynamically combined CLI-proxy gateway trust as well.ghvariants with digest-pinned Go 1.27.3 (CVE-2026-111943), asserting the compiled binary's toolchain metadata while retaining existing source checksums and module assertions.This branch was created from
mainin a separate worktree. It does not include edits or unrelated commits from the original working directory. The subsequently user-applied CodeQL autofix is preserved.Validation
mainand this branch; Linux is clean.NODE_OPTIONSwas replaced.CAP_SYS_ADMINchroot-overlay smoke passed: disclosed emSign root rejected, private CA accepted, overlay read-only, original source store unchanged.gh-buildtargets built successfully; resulting binary reportsgo1.27.3.--versionpassed.CI follow-up
Historical positive action tests installed releases lacking the newly required signed identity. They now exercise the real action shell using isolated, locally signed fixture assets and test-only transport/verifier/Docker stand-ins. These fixtures do not validate real GitHub/Sigstore provenance or pull real images. An optional manual
attested-versionjob retains live verification for a genuinely published attested release; production action/installer code has no fixture mode or legacy-release exception.Fixed npm's native-supported
ca: nulloption to select filtered defaults, rather than throwing or restoring unfiltered trust. Fixed the regression assertion to support Node 20 withouttls.getCACertificates. Both Linux Node 20 and Node 26 pass HTTPS/direct TLS/fetch tests including null/default/explicit CA options, and Linux action scenarios accept valid fixtures while rejecting missing metadata and tampered signatures. 42 targeted follow-up regressions passed; a further 10 canonical policy unit cases achieve 100% lines/statements/functions/branches, without relaxing coverage gates.Compatibility and remaining boundaries
Emergency root distrust is intentionally broader than intermediate revocation: unrelated chains under the four listed roots will also stop validating. Revisit that fallback when upstream runtimes provide adequate intermediate-key revocation. Existing custom/private roots continue to work unless their keys are denylisted.
Older unattested releases now fail bootstrap closed. A trusted, independently installed, attestation-capable
ghverifier and trusted initial installer/action are prerequisites. An installer cannot authenticate the script already executing; host clients and Docker-daemon trust must be updated separately.Full API-proxy/enclave Node builds are blocked locally by npm-registry TLS EOF/ECONNRESET errors; the same transport problem reproduces on the unmodified pinned Node base. The enclave build's distinct null-CA policy regression was reproduced and fixed. Actual Linux trust/runtime paths and the complete API-proxy application suite passed. Not every image or KVM/guest-boot conformance path was built/run locally; fixture tests do not substitute for first-release live provenance validation.
Deployment requires rebuilding/publishing the affected images/rootfs artifacts and updating AWF plus image pins together; existing immutable images and running containers are unchanged. AWF cannot universally repair arbitrary Java keystores, certifi/vendored Python stores, Rust/browser embedded roots, independently selected CA paths, custom trust contexts, or applications disabling verification. This policy is not a new boundary against user commands deliberately overriding their own TLS trust. See
docs/security-ca-trust.mdanddocs/releasing.mdfor the precise coverage and rollout requirements.References: Mozilla incident discussion, Go advisory, Node upstream tracking.