Add E2E test verifying OTEL tracing across Zenko services by delthas · Pull Request #2378 · scality/Zenko

delthas · 2026-04-17T13:00:54Z

Summary

Deploy Jaeger all-in-one (memory-only, OTLP-enabled, pinned to 1.76.0)
in the kind CI cluster alongside existing dependencies (Keycloak,
Prometheus, etc.)
Enable tracing on the Zenko CR (spec.tracing, ratio "0") so the
cluster runs in production-mode (no ambient traces). The test sends
a single PUT through the existing S3 ingress with a W3C traceparent
header injected via SDK middleware. Cloudserver's parent-based sampler
honors the header → only that one request is traced.
Add a @PreMerge CTST scenario that asserts the trace appears in
Jaeger and contains spans from both connector-cloudserver and
connector-vault

What changed

File	What
`solution/deps.yaml`	Bump `vault` `8.11.6 → 8.11.7` (OTEL support) and `cloudserver` (OTEL support, parent-based sampler)
`configs/zenko.yaml`	Add `spec.tracing` (enabled, ratio `"0"`, jaeger endpoint)
`configs/jaeger.yaml`	Jaeger Deployment + Service (pinned `1.76.0`, memory-only, requests/limits)
`install-kind-dependencies.sh`	`kubectl apply -f configs/jaeger.yaml`
`setup-e2e-env.sh`	Port-forward Jaeger query API + `JaegerQueryEndpoint` in world params
`world/Zenko.ts`	`JaegerQueryEndpoint` world parameter
`features/otel-tracing.feature`	New `@PreMerge` scenario
`steps/otel-tracing.ts`	Generates W3C `traceparent`, adds an SDK middleware that injects the header on the CTST S3 client, polls `GET /api/traces/<id>` until the trace appears, asserts cloudserver + vault spans

Why

Parent ticket OS-1072
tracks adding OpenTelemetry tracing across the Zenko stack. This PR
adds CI coverage to verify that traces actually propagate end-to-end
(cloudserver → vault) once tracing is enabled.

Note on trust boundary: production runs metalk8s nginx which strips
incoming traceparent headers; Zenko CI uses upstream
kubernetes/ingress-nginx which forwards them. So in CI the test
injects via the regular ingress; the production trust-boundary
(external traceparent is dropped at the edge) is metalk8s-specific
and out of scope for this CI test.

Issue: ZENKO-5258

bert-e · 2026-04-17T13:00:57Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-04-17T13:01:05Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

bert-e · 2026-05-29T14:38:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

francoisferrand

overall not sure where to stand here: there is a fundamental compromise: do we want to test the "production" case first and foremost (i.e. without tracing), or is it safe-enough to enable Otel all the time (i.e. not testing production anymore, which could hide crashes or introduce subtle delays...)

SylvainSenechal

side question, here the jaeger server will be killed at the end the ci run, but would it be possible to have the traces data retained, so that maybe we could export these traces data in the artifact, and spin up a server to explore them with a ui ?

delthas · 2026-06-03T10:55:57Z

Moved to a ratio of 0% (no traces, but enabled); and injecting a forcefully enabled trace using a traceparent header directly made against cloudserver (bypassing the Ingress that strips it)

This should bring no performance overhead, no overlay reconfiguration, and nicely isolate our test trace.

Includes OTEL tracing support needed by the E2E test in this PR. Issue: ZENKO-5258

bert-e · 2026-06-23T10:36:59Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

claude · 2026-06-23T10:40:09Z

  dashboard: cloudserver/cloudserver-dashboards
  image: cloudserver
-  tag: 9.3.9
+  tag: 9.4.0-preview.1


9.4.0-preview.1 is a pre-release tag. deps.yaml defines the component versions for the solution — the review criteria require pinning to a concrete released tag or content hash. If this PR is meant to land only after cloudserver 9.4.0 ships, consider gating the merge on that release and updating the tag then. If the preview tag is intentional for CI validation, document that constraint so it isn't accidentally shipped.

claude · 2026-06-24T14:50:34Z

  dashboard: cloudserver/cloudserver-dashboards
-  image: cloudserver
-  tag: 9.3.9
+  image: playground/delthas/cloudserver


image: playground/delthas/cloudserver points to a personal playground namespace, not the official cloudserver image. deps.yaml is the source of truth for solution images — merging this as-is ships a non-release, non-official image. The previous review flagged 9.4.0-preview.1; the current state is more severe since the image path itself changed. Restore image: cloudserver and pin to a released tag before merge (or gate merge on the cloudserver release).

Suggested change

image: playground/delthas/cloudserver

image: cloudserver

Temporary bump to ghcr.io/scality/playground/delthas/cloudserver:9.4.1-h20bb7d391a725278da84ec1e7bad1d99cc9e94a1 to verify CI works against this build. To be reverted / replaced before merge. Issue: ZENKO-5258

claude · 2026-06-24T16:01:17Z

-  dashboard: cloudserver/cloudserver-dashboards
-  image: cloudserver
-  tag: 9.3.9
+  dashboard: playground/delthas/cloudserver/cloudserver-dashboards


Same playground namespace issue as the image field below — this also needs to be restored to the official path before merge.

Suggested change

dashboard: playground/delthas/cloudserver/cloudserver-dashboards

dashboard: cloudserver/cloudserver-dashboards

Temporary bump to ghcr.io/scality/playground/delthas/backbeat:9.4.1-h736f50680341e71275532cc3b7ef2777a7768bec (with matching dashboards + policies) to verify CI works against this build. To be reverted / replaced before merge. Issue: ZENKO-5258

@premerge

- Deploy Jaeger all-in-one (memory-only, OTLP-enabled) in the kind CI cluster alongside existing dependencies - Patch the Zenko CR with `spec.otel` (enabled, sampling ratio 1.0) so every request is traced during CI — also acts as a smoke test that OTEL doesn't break existing @premerge tests - Add a new @premerge CTST scenario that puts an S3 object and then polls the Jaeger query API to assert a trace exists with spans from both cloudserver and vault Issue: ZENKO-5258

claude · 2026-06-25T13:17:22Z

-  policy: backbeat/backbeat-policies
-  tag: 9.4.0
+  dashboard: playground/delthas/backbeat/backbeat-dashboards
+  image: playground/delthas/backbeat


Same playground namespace issue as cloudserver (already flagged): image, dashboard, and policy all point to playground/delthas/backbeat/.... Restore to official paths before merge.

Suggested change

image: playground/delthas/backbeat

image: backbeat

delthas · 2026-06-26T10:39:55Z

CI failing for unrelated reasons; OTEL test succeeding.

francoisferrand · 2026-06-26T14:32:43Z

+
+        const { traceparent, traceId } = generateTraceContext();
+        const client = this.createS3Client();
+        injectTraceparent(client, traceparent);


how come CTST can inject a trace (and esp. cloudserver/... accepts it) ?
CTST is not running k8s in-cluster AFAIK : it runs on the "host" now, while zenko runs in a "kind" k8s cluster

→ is the "trusted hosts" protection not working? I would expect we need to explicitely fill some field in zenko CR to end up trusting ctst?

SylvainSenechal · 2026-06-26T16:28:01Z

Wait so, the CR is configured with samplingRatio = 0, so no traces are generated by the services.
Then you inject the trace yourself by manipulating the header of the request 🤔

Wouldn't we want to test it with a non zero samplingRatio ?
Or I guess you did it because you don't wanna enable tracing for every tests, and you don't want to patch the cr during the tests ?

SylvainSenechal

Just 1 question + update the /playground deps

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from c362986 to 9a3a251 Compare April 21, 2026 13:20

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from 9a3a251 to f32a16b Compare May 29, 2026 14:22

delthas changed the base branch from development/2.14 to development/2.15 May 29, 2026 14:22

claude Bot reviewed May 29, 2026

View reviewed changes

Comment thread .github/scripts/end2end/install-kind-dependencies.sh Outdated

claude Bot reviewed May 29, 2026

View reviewed changes

Comment thread tests/functional/ctst/features/otel-tracing.feature Outdated

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from f32a16b to a7b196c Compare May 29, 2026 14:34

scality deleted a comment from bert-e May 29, 2026

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from a7b196c to c9ae52e Compare May 29, 2026 14:38

scality deleted a comment from bert-e May 29, 2026

delthas marked this pull request as ready for review May 29, 2026 14:41

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from c9ae52e to 79d656d Compare May 29, 2026 14:41

delthas requested review from a team, SylvainSenechal, benzekrimaha and maeldonn and removed request for benzekrimaha May 29, 2026 14:43

francoisferrand reviewed Jun 2, 2026

View reviewed changes

SylvainSenechal reviewed Jun 3, 2026

View reviewed changes

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from 79d656d to 648091c Compare June 3, 2026 10:41

claude Bot reviewed Jun 3, 2026

View reviewed changes

Comment thread .github/scripts/end2end/configs/jaeger.yaml

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch 2 times, most recently from 8f05574 to 5160774 Compare June 3, 2026 10:54

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from 5160774 to 4a072e5 Compare June 5, 2026 08:13

maeldonn reviewed Jun 15, 2026

View reviewed changes

Comment thread .github/scripts/end2end/setup-e2e-env.sh Outdated

scality deleted a comment from bert-e Jun 15, 2026

delthas mentioned this pull request Jun 17, 2026

remove unused port-forward PID captures #2446

Merged

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch 2 times, most recently from 3b4b0fa to dcf5457 Compare June 23, 2026 09:28

claude Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread solution/deps.yaml

delthas mentioned this pull request Jun 23, 2026

CLDSRV-884: Bump version to 9.4.0-preview.1 scality/cloudserver#6202

Merged

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from dcf5457 to 69afb02 Compare June 23, 2026 10:31

Bump vault 8.11.7

c1778af

Includes OTEL tracing support needed by the E2E test in this PR. Issue: ZENKO-5258

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from 69afb02 to 43b1e51 Compare June 23, 2026 10:36

scality deleted a comment from bert-e Jun 23, 2026

claude Bot reviewed Jun 23, 2026

View reviewed changes

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from 43b1e51 to dc9fb9c Compare June 24, 2026 14:48

claude Bot reviewed Jun 24, 2026

View reviewed changes

[Do not merge] WIP: bump cloudserver to playground build

59abbeb

Temporary bump to ghcr.io/scality/playground/delthas/cloudserver:9.4.1-h20bb7d391a725278da84ec1e7bad1d99cc9e94a1 to verify CI works against this build. To be reverted / replaced before merge. Issue: ZENKO-5258

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from dc9fb9c to fa02bea Compare June 24, 2026 15:57

claude Bot reviewed Jun 24, 2026

View reviewed changes

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from fa02bea to b936735 Compare June 24, 2026 17:32

delthas mentioned this pull request Jun 25, 2026

Bump arsenal to 8.5.1 scality/cloudserver#6206

Open

delthas added 2 commits June 25, 2026 15:15

delthas force-pushed the improvement/ZENKO-5258/otel-tracing-e2e-test branch from b936735 to 1245e78 Compare June 25, 2026 13:15

claude Bot reviewed Jun 25, 2026

View reviewed changes

delthas requested review from a team, SylvainSenechal, francoisferrand and maeldonn June 26, 2026 10:38

francoisferrand approved these changes Jun 26, 2026

View reviewed changes

SylvainSenechal reviewed Jun 26, 2026

View reviewed changes

SylvainSenechal approved these changes Jun 26, 2026

View reviewed changes

	dashboard: playground/delthas/cloudserver/cloudserver-dashboards
	dashboard: cloudserver/cloudserver-dashboards

Uh oh!

Conversation

delthas commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Why

Uh oh!

bert-e commented Apr 17, 2026

Hello delthas,

Uh oh!

bert-e commented Apr 17, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

bert-e commented May 29, 2026

Waiting for approval

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SylvainSenechal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

delthas commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Jun 23, 2026

Waiting for approval

Uh oh!

claude Bot Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

delthas commented Jun 26, 2026

Uh oh!

francoisferrand Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

delthas commented Apr 17, 2026 •

edited

Loading

delthas commented Jun 3, 2026 •

edited

Loading