Skip to content

[2026.3] Pin Mercure to v0.24 in the API test stack and keep container logs on failure - #555

Merged
kingjia90 merged 4 commits into
2026.3from
fix/api-tests-supervisord
Sep 30, 2026
Merged

kingjia90 merged 4 commits into
2026.3from
fix/api-tests-supervisord

Conversation

@kingjia90

@kingjia90 kingjia90 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Since 2026-09-17 the API-test nightly fails four specs on every leg: the three ZIP downloads
(HTTP 500) and ExecutionEngineTest › HideCompletedJobSucceeds (timeout). All four come from one fact:
execution-engine jobs are created and then stay running forever. The ZIP 500 is just the archive
never being written. The installed composer package versions and the test code were identical between
the last green and the first red run. Only floating Docker images changed.

Cause

docker-compose.yaml runs the hub as untagged dunglas/mercure, and that tag moved to
Mercure v1.0.0 on 2026-09-16. Mercure 1.0 is a breaking protocol change. Publishers must now send
an OAuth 2.0 access token (typ: at+jwt, an iss bound to a configured issuer block, aud, exp,
authorization_details). Pimcore still publishes with the legacy mercure-claim JWT, so every publish
is rejected. The logs this PR now collects show it directly:

mercure-1      | "msg":"Authorization error","error":"invalid JWT: untrusted issuer \"\""
supervisord-1  | ERROR [generic_execution_engine] [JobRun 1]: Symfony\Component\HttpClient\Exception\ClientException:
               |   HTTP/1.1 401 Unauthorized returned for "http://mercure/.well-known/mercure".
               |   … ElementDeleteHandler->updateProgress(…)
supervisord-1  | CRITICAL [messenger] Error thrown while handling message …ElementDeleteMessage.
               |   Removing from transport after 0 retries. Error: "… failed: Failed to send an update."

The execution-engine handlers publish progress through Mercure. pimcore_generic_execution_engine
runs with max_retries: 0, so the failed message is dropped and the job row stays running.

Change

  • Pin the hub to dunglas/mercure:v0.24 (0.x protocol, still receives 0.24 patch releases). This
    is the fix.
  • Restart supervisord at the end of 02-install-pimcore.sh. The worker is started in step 1,
    before Pimcore is installed, and crash-loops until the install finishes (seen in the logs: exit 255
    and 1, and cache_items doesn't exist). The restart makes it boot cleanly against the installed app.
    It did not fix the failures on its own; see validation below.
  • 07-collect-logs.sh plus two if: failure() steps upload docker compose ps, every service's log
    and var/log as container-logs-<platform>. Nothing captured the worker's output before, which is
    why this took so long to find. The product key, encryption secret, instance identifier and composer
    token are replaced with [REDACTED] before upload, because the artifact is public.

Validation

Both runs installed 2026.3.x, dispatched from a temporary branch that is this PR plus a one-line
dispatch-choice commit:

Run Change Result
36680113994 restart + log capture 4 failed, 754 passed, 4 skipped (30 min). The logs show the Mercure 401.
36683773565 + Mercure v0.24 pin 0 failed, 758 passed, 4 skipped (11.3 min)

The 4 skipped specs are the same in both runs.

Follow-ups (not in this PR)

  • The same untagged dunglas/mercure image is used by skeleton, demo-enterprise and paas-skeleton
    (2026.3 and 2026.x), the devcontainer, demo-server and studio-e2e-tests. Fresh Docker installs since
    2026-09-16 are very likely affected the same way (Studio background jobs never finish). The long-term
    fix is Mercure 1.0 token support in Pimcore's publisher.
  • Test side: ZipDownloadByFolderIdTest polls in an unbounded loop under a 5-minute timeout, and
    ExportTests accept 500 on download, which hid this stall.

Relates to pimcore/service-operations#1397.

🤖 Generated with Claude Code

… failure

The supervisord container is started in step 1, before Pimcore is installed, and
nothing restarts it afterwards. Its messenger:consume worker (time-limit 3600)
therefore keeps whatever state it booted with. Since 2026-09-17 every
execution-engine job in the API tests stays "running" forever, which is what
fails the three ZIP downloads and HideCompletedJobSucceeds.

- restart supervisord at the end of 02-install-pimcore.sh
- add 07-collect-logs.sh and upload container logs + var/log on failure, so the
  worker's output is no longer discarded by 04-shutdown.sh; secret values are
  redacted because the artifact is public

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The untagged dunglas/mercure image moved to v1.0.0 on 2026-09-16. Mercure 1.0
only accepts OAuth 2.0 access tokens (typ at+jwt, a trusted iss, aud, exp,
authorization_details) and rejects the legacy mercure-claim JWT that Pimcore
publishes with: the hub logs "invalid JWT: untrusted issuer" and answers 401.
The execution-engine handlers publish progress to Mercure, so every job's
message fails and, with max_retries 0, is dropped, leaving the job running
forever. That is what fails the three ZIP downloads and HideCompletedJobSucceeds
since 2026-09-17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Redaction failures can currently be masked while potentially unredacted public artifacts are still uploaded.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Restarts the messenger worker after installation and preserves failure diagnostics before container teardown.

Changes:

  • Restarts supervisord after Pimcore setup.
  • Collects, redacts, and uploads container/application logs on failure.
  • Documents and ignores generated log dumps.
File Description
.github/​scripts/​02-install-pimcore.sh Restarts the worker after installation.
.github/​scripts/​07-collect-logs.sh Collects and redacts diagnostic logs.
.github/​workflows/​api-tests.yml Uploads failure logs as artifacts.
doc/​04_API_Testing/​01_API_TESTS.md Documents log collection.
.gitignore Excludes generated log dumps.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/scripts/07-collect-logs.sh Outdated
Comment thread .github/workflows/api-tests.yml Outdated
…load then

perl -pi only warns and exits 0 when it cannot open a file, so a skipped
file kept the secret while the collector reported success. Verify with
grep afterwards and drop the logs if any secret is left. Upload the
public artifact only when the collector step succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kingjia90 kingjia90 changed the title [2026.3] Restart the messenger worker after install and keep container logs on failure [2026.3] Pin Mercure to v0.24 in the API test stack and keep container logs on failure Sep 30, 2026
JSON log lines (the Mercure hub, Monolog context) write "/" as "\/", so a
secret containing a slash survived both the literal redaction and the
post-redaction grep check. Scrub and verify that form too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kingjia90 kingjia90 removed this from the 2026.3.1 milestone Sep 30, 2026
@kingjia90 kingjia90 closed this Sep 30, 2026
@kingjia90
kingjia90 deleted the fix/api-tests-supervisord branch September 30, 2026 08:21
@kingjia90
kingjia90 restored the fix/api-tests-supervisord branch September 30, 2026 09:04
@kingjia90 kingjia90 reopened this Sep 30, 2026
@kingjia90 kingjia90 added this to the 2026.3 milestone Sep 30, 2026
@kingjia90
kingjia90 merged commit 2ca8cbe into 2026.3 Sep 30, 2026
26 of 28 checks passed
@kingjia90
kingjia90 deleted the fix/api-tests-supervisord branch September 30, 2026 09:09
kingjia90 added a commit that referenced this pull request Sep 30, 2026
Only the Mercure v0.24 pin is needed to restore the API tests; the
failure-time log collection was a diagnostic aid and is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kingjia90 added a commit that referenced this pull request Sep 30, 2026
Only the Mercure v0.24 pin is needed to restore the API tests; the
failure-time log collection was a diagnostic aid and is dropped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants