Repository navigation
trigger deploy fails on custom DEPLOY_BASE_IMAGES since 4.7.6: emSign distrust step assumes a root Debian base #5054
Description
Provide environment information
- Self-hosted Trigger.dev v4.7.7 (webapp, supervisor, Helm chart 4.7.7)
- CLI
trigger.dev4.7.7,runtime: "node-26" - Webapp configured with custom base images (feat(webapp,cli): let self-hosted instances require deploy base images #5005):
DEPLOY_BASE_IMAGES=node-26=cgr.dev/chainguard/node@sha256:…DEPLOY_BUILD_BASE_IMAGES=node-26=cgr.dev/chainguard/node:latest-dev@sha256:…
- No
image.instructions/aptGetextensions intrigger.config.ts
Describe the bug
Since 4.7.6, every trigger deploy fails building the task image when the webapp supplies custom base images. 4.7.4 and 4.7.5 deploy the same project fine.
#14 [base 2/3] RUN mkdir -p /usr/local/share/ca-certificates/emsign-block && printf '%s\n' …
#14 0.229 mkdir: can't create directory '/usr/local/share/': Permission denied
ERROR: process "/bin/sh -c mkdir -p /usr/local/share/ca-certificates/emsign-block && …" did not complete successfully: exit code: 1
generateContainerfile inserts distrustCompromisedRoots into the base stage and into the build stage when a build base is set (buildImage.ts#L748-L762, used at L844, L868 and L971). The step runs before the template's USER node and assumes the triggerdotdev/node Debian base:
| The step needs | A typical custom base (e.g. cgr.dev/chainguard/node) |
|---|---|
| to run as root | defaults to uid 65532 (node) |
/usr/local/share/ca-certificates/ |
no /usr/local/share |
/etc/ca-certificates.conf to sed |
absent |
update-ca-certificates |
absent (the image ships only the finished bundle) |
The step has no opt-out, so a self-hosted install that requires custom bases cannot deploy at all on 4.7.6 or 4.7.7. main has the same code.
Reproduction repo
None needed. The failing instruction reproduces on its own:
FROM cgr.dev/chainguard/node:latest
RUN mkdir -p /usr/local/share/ca-certificates/emsign-blockmkdir: can't create directory '/usr/local/share/': Permission denied
To reproduce
- Self-host v4.7.6+ with
DEPLOY_BASE_IMAGES/DEPLOY_BUILD_BASE_IMAGESpointing at a non-root image without Debian'sca-certificatestooling, e.g.cgr.dev/chainguard/nodeand:latest-dev. - Run
trigger deploywith CLI 4.7.6 or 4.7.7. - The image build fails at
[base 2/3](and[build 2/…]) as above.
Suggested fix
Skip the step for stages built on a server-supplied base, and leave the base image's trust store to whoever supplies it. We run this as a local yarn patch:
+const distrustUnlessServerBase = (serverBase) => (serverBase ? "" : distrustCompromisedRoots);
…
-${distrustCompromisedRoots}`;
+${distrustUnlessServerBase(options.image?.buildBase)}`;
…
-${distrustCompromisedRoots}
+${distrustUnlessServerBase(options.image?.base)}Other options, in case you'd rather keep the step on custom bases:
- An explicit opt-out, e.g. a webapp env var or an
imageoption. - Running the step as
USER rootand skipping it when/etc/ca-certificates.conforupdate-ca-certificatesis missing. This still leaves distroless or apk-based bases unprotected, so it seems weaker than documenting that a custom base owns its trust store.
Happy to open a PR for whichever you prefer.
Additional information
- Custom bases aren't necessarily exposed by skipping the step. On
cgr.dev/chainguard/node(Node 26.12.1,ca-certificates-bundle20260909-r3), neithertls.rootCertificatesnor the system store contains emSign ECC Root CA - G3, ECC Root CA - C3 or Root CA - C1. - A runtime safeguard that works on any base is
tls.setDefaultCACertificates(getCACertificates('default').filter(…)), dropping those three roots at worker start. We run that in our worker init alongside the patch. - Introduced by
dac3b033eband1fb4c7608e(shipped in 4.7.6), which postdate custom base images in feat(webapp,cli): let self-hosted instances require deploy base images #5005.
Activity