Skip to content

Stop trusting compromised eMudhra roots in the musl build images - #61

Merged
andrewlock merged 1 commit into
DataDog:mainfrom
eliottness:eliottness/emudhra-ca-roots
Oct 30, 2026
Merged

andrewlock merged 1 commit into
DataDog:mainfrom
eliottness:eliottness/emudhra-ca-roots

Conversation

@eliottness

@eliottness eliottness commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Stops the musl build images from trusting two compromised roots, emSign ECC Root CA - G3 and AAA Certificate Services (the roots behind the compromised eMudhra intermediates, incident IR-62343).

  • tools/docker/Dockerfile.musl-x64 (alpine:3.17): removes both roots from the CA store and regenerates the bundle. The base is EOL, so no fixed ca-certificates package ships.
  • tools/docker/Dockerfile.musl-aarch64 (alpine:3.18): same patch for the aarch64 musl image. The base is EOL too.

Of the 4 impacted image references in the APPSEC-71273 ledger, these 2 are fixed here. The other 2 are follow-ups (see below).

Motivation

Incident IR-62343 / Jira APPSEC-71292 (parent APPSEC-71273). Images used by this repository's CI still ship CA stores that trust the compromised roots. Images that CI never builds or pulls are left alone.

Base-image choice: alpine:3.22 and alpine:latest (v3.24) scan clean for these roots, so a newer base is possible. The bases stay at 3.17 / 3.18 on purpose: dd-trace-dotnet's musl build images pin alpine:3.17 (x64) and alpine:3.18 (aarch64), and these Dockerfiles mirror them. dd-trace-dotnet's NativeValidation enforces an alpine 3.14 musl symbol floor; a newer base could change the symbols the artifact requires, and that check runs in dd-trace-dotnet, not in this repo's CI. Maintainer decision needed: confirm the patched 3.17 / 3.18 bases stay, or move to a clean newer base and run the dd-trace-dotnet NativeValidation check.

Additional Notes

  • Coverage: both changed Dockerfiles are built by the Build all platforms jobs x86_64-alpine-linux-musl and aarch64-alpine-linux-musl on the Build workflow (pull_request, head 1ee744b). Run: https://github.com/DataDog/libdatadog-dotnet/actions/runs/39995729540
  • Built images were not re-scanned in this follow-up, and no local Docker builds were run; the CI build jobs are the build evidence.
  • Follow-ups, not in this PR (2 of 4 ledger refs, no fix published yet):
    • ghcr.io/cross-rs/x86_64-unknown-linux-gnu:main-centos: no fix is published yet because NSS 3.131.1 is unavailable for CentOS 7, and a rebuilt blocklist image would still keep both roots in NSS and the extracted CA bundles.
    • quay.io/pypa/manylinux2014_aarch64: no fix is published yet because NSS 3.131.1 is unavailable for CentOS 7, and a rebuilt blocklist image would still keep the roots in NSS, the extracted CA bundles, and the Python 3.9 pip-vendored certifi.
    • Both were re-scanned on 2026-10-26 and still report the roots.

The musl build images stay on alpine 3.17/3.18 (pinned to match dd-trace-dotnet's build env) and drop
emSign ECC Root CA - G3 / AAA Certificate Services from the CA bundle (IR-62343).

Jira: APPSEC-71292
@eliottness
eliottness marked this pull request as ready for review October 26, 2026 13:33
@eliottness
eliottness requested a review from a team as a code owner October 26, 2026 13:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T13:38:10.413198Z 1ee744b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static inspection supports the CA-removal logic for both musl images. Removal of both compromised roots from the final images remains unverified without rebuilt-image trust-store scans.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 1ee744b

@eliottness

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-29 14:40:15 UTC ℹ️ Start processing command /merge


2026-10-29 14:40:20 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 0s (p90).


2026-10-29 16:40:38 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 0 took longer than expected. The current limit for the base branch 'main' is 120 minutes.

@eliottness

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-29 16:51:16 UTC ℹ️ Start processing command /merge


2026-10-29 16:51:21 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 0s (p90).


2026-10-29 18:51:39 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 0 took longer than expected. The current limit for the base branch 'main' is 120 minutes.

@eliottness

Copy link
Copy Markdown
Contributor Author

Thanks for the approval @andrewlock! The merge queue can't land this one: both /merge attempts timed out after 120 minutes waiting for GitLab pipeline 0, because this repo only runs GitHub Actions, so devflow never gets a GitLab pipeline. Every earlier PR here (#48 to #54) was merged directly by a maintainer. All checks are green; could you merge it directly? Nothing needs to happen after merge: the musl build images are built in CI on every run.

@andrewlock
andrewlock merged commit 9bf90ca into DataDog:main Oct 30, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants