Skip to content

trigger deploy fails on custom DEPLOY_BASE_IMAGES since 4.7.6: emSign distrust step assumes a root Debian base #5054

Open

Description

@vlakshmanan-silkline

Provide environment information

  • Self-hosted Trigger.dev v4.7.7 (webapp, supervisor, Helm chart 4.7.7)
  • CLI trigger.dev 4.7.7, runtime: "node-26"
  • Webapp configured with custom base images (feat(webapp,cli): let self-hosted instances require deploy base images #5005):
    • DEPLOY_BASE_IMAGES=node-26=cgr.dev/chainguard/node@sha256:…
    • DEPLOY_BUILD_BASE_IMAGES=node-26=cgr.dev/chainguard/node:latest-dev@sha256:…
  • No image.instructions / aptGet extensions in trigger.config.ts

Describe the bug

Since 4.7.6, every trigger deploy fails building the task image when the webapp supplies custom base images. 4.7.4 and 4.7.5 deploy the same project fine.

#14 [base 2/3] RUN mkdir -p /usr/local/share/ca-certificates/emsign-block &&   printf '%s\n' …
#14 0.229 mkdir: can't create directory '/usr/local/share/': Permission denied
ERROR: process "/bin/sh -c mkdir -p /usr/local/share/ca-certificates/emsign-block && …" did not complete successfully: exit code: 1

generateContainerfile inserts distrustCompromisedRoots into the base stage and into the build stage when a build base is set (buildImage.ts#L748-L762, used at L844, L868 and L971). The step runs before the template's USER node and assumes the triggerdotdev/node Debian base:

The step needs A typical custom base (e.g. cgr.dev/chainguard/node)
to run as root defaults to uid 65532 (node)
/usr/local/share/ca-certificates/ no /usr/local/share
/etc/ca-certificates.conf to sed absent
update-ca-certificates absent (the image ships only the finished bundle)

The step has no opt-out, so a self-hosted install that requires custom bases cannot deploy at all on 4.7.6 or 4.7.7. main has the same code.

Reproduction repo

None needed. The failing instruction reproduces on its own:

FROM cgr.dev/chainguard/node:latest
RUN mkdir -p /usr/local/share/ca-certificates/emsign-block
mkdir: can't create directory '/usr/local/share/': Permission denied

To reproduce

  1. Self-host v4.7.6+ with DEPLOY_BASE_IMAGES / DEPLOY_BUILD_BASE_IMAGES pointing at a non-root image without Debian's ca-certificates tooling, e.g. cgr.dev/chainguard/node and :latest-dev.
  2. Run trigger deploy with CLI 4.7.6 or 4.7.7.
  3. The image build fails at [base 2/3] (and [build 2/…]) as above.

Suggested fix

Skip the step for stages built on a server-supplied base, and leave the base image's trust store to whoever supplies it. We run this as a local yarn patch:

+const distrustUnlessServerBase = (serverBase) => (serverBase ? "" : distrustCompromisedRoots);
 …
-${distrustCompromisedRoots}`;
+${distrustUnlessServerBase(options.image?.buildBase)}`;
 …
-${distrustCompromisedRoots}
+${distrustUnlessServerBase(options.image?.base)}

Other options, in case you'd rather keep the step on custom bases:

  • An explicit opt-out, e.g. a webapp env var or an image option.
  • Running the step as USER root and skipping it when /etc/ca-certificates.conf or update-ca-certificates is missing. This still leaves distroless or apk-based bases unprotected, so it seems weaker than documenting that a custom base owns its trust store.

Happy to open a PR for whichever you prefer.

Additional information

  • Custom bases aren't necessarily exposed by skipping the step. On cgr.dev/chainguard/node (Node 26.12.1, ca-certificates-bundle 20260909-r3), neither tls.rootCertificates nor the system store contains emSign ECC Root CA - G3, ECC Root CA - C3 or Root CA - C1.
  • A runtime safeguard that works on any base is tls.setDefaultCACertificates(getCACertificates('default').filter(…)), dropping those three roots at worker start. We run that in our worker init alongside the patch.
  • Introduced by dac3b033eb and 1fb4c7608e (shipped in 4.7.6), which postdate custom base images in feat(webapp,cli): let self-hosted instances require deploy base images #5005.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions