Add CRR cascaded tests by SylvainSenechal · Pull Request #2447 · scality/Zenko

SylvainSenechal · 2026-06-22T14:45:39Z

Crr cascaded tests, didn't add very complex scenarios, let me know if you can come up with other test ideas but really not sure. I think there is only one happy path to test, and for the potential problematic situation, there is loop which I tested, and the rest not really sure we can test easily (like concurrency on write and stale micro version id)

Wont work in ci, as deps.yaml isn't updated with cloudserver/backbeat.
BUT : "It works on my ~~computer~~ codespace"

bert-e · 2026-06-22T14:45:44Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-06-22T14:45:52Z

Incorrect fix version

The Fix Version/s in issue ZENKO-5263 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

2.15.2

Please check the Fix Version/s of ZENKO-5263, or the target
branch of this pull request.

claude · 2026-06-22T14:47:40Z

 # yq eval 'sortKeys(.)' -i deps.yaml
 backbeat:
-  sourceRegistry: ghcr.io/scality
+  sourceRegistry: ghcr.io/scality/playground/sylvainsenechal


backbeat and cloudserver point to a personal playground registry (ghcr.io/scality/playground/sylvainsenechal) with dev tags (9.4.101, 9.3.101). Per review criteria, Scality-internal images must resolve to tags published on ghcr.io/scality. These need to be reverted before merge.

SylvainSenechal · 2026-06-23T22:11:20Z

 CRR_DESTINATION_LOCATION_NAME=${CRR_DESTINATION_LOCATION_NAME} \
 CRR_SOURCE_ACCOUNT_NAME=${CRR_SOURCE_ACCOUNT_NAME} \
 CRR_DESTINATION_ACCOUNT_NAME=${CRR_DESTINATION_ACCOUNT_NAME} \
+CRR_CASCADE_LOCATION_A_NAME=${CRR_CASCADE_LOCATION_A_NAME} \


I know, in the past, we talked about avoiding creating too many locations but 🤔
I don't know, I think it's probably fine ?

we still don't have that many locations

It's probably not a bad idea to create many locations in ci, to see if it actually creates issues or not

SylvainSenechal · 2026-06-23T22:17:08Z

+        while (Date.now() < deadline) {
+            Identity.useIdentity(IdentityEnum.ACCOUNT, location);
+            try {
+                await this.createS3Client().send(new HeadObjectCommand({ Bucket: bucket, Key: objectName }));


Kept it light and very functional for this test : only check that the object is here, not checking further thing like specific object metadata/status. Could add, not sure it's super useful for functional tests, although I know sometimes we did it in the past

I think we should test replication-status ; but not use if this can be check here (may be race conditions -esp. when using cascade- so the status may have intermediate value...)

yeah there are 2 things to check :

Metadata: { 'crr-location-c-replication-status': 'PENDING' },
ReplicationStatus: 'REPLICA',

ReplicationStatus will always be REPLICA, because its based on the new isReplica boolean
But in metadata, depending how fast we check, it will be either PENDING or REPLICA, as a cascade transition will turn a replica into an object thats pending a replication.

I'm adding some check here, and i will add a step at the end to check all the status

Issue: ZENKO-5263

francoisferrand · 2026-06-24T14:39:40Z

+        When an object "cascade-loop-obj" is put in location "crr-location-a"
+        Then the object should replicate to location "crr-location-b" within 300 seconds
+        And the object should replicate to location "crr-location-c" within 300 seconds
+        And the object at location "crr-location-a" should never have replication status PENDING within 30 seconds


should also check the "final" replication status on each location : SUCCESS / REPLICA / REPLICA ?

francoisferrand · 2026-06-24T14:41:57Z

+        When an object "cascade-loop-obj" is put in location "crr-location-a"
+        Then the object should replicate to location "crr-location-b" within 300 seconds
+        And the object should replicate to location "crr-location-c" within 300 seconds
+        And the object at location "crr-location-a" should never have replication status PENDING within 30 seconds


should we have more complex scenario, where we write on multiple points in the loop?
may not be able to control how he operations race to ensure a specific scenario, but worth adding to ensure the system is "eventually" consistent and stable?

I came up with a more complex scenario which I think is decent :

Loop config

Concurrent writes on all location, writing a specific random metadata marker
=> Assert that they all converge on the same metadataMarker

Tbh, still not the perfect test but I think worth doing

francoisferrand · 2026-06-24T14:42:08Z

+        And replication is configured from location "crr-location-a" to "crr-location-b"
+        When an object "cascade-obj" is put in location "crr-location-a"
+        Then the object should replicate to location "crr-location-b" within 300 seconds
+        And the object should replicate to location "crr-location-c" within 300 seconds


should also check the "final" replication status on each location : SUCCESS / REPLICA / REPLICA ?

Yes I'm gonna add a step at the end to check all the status

francoisferrand · 2026-06-24T14:43:35Z

      bucketMatch: false
      repoId: []
+  - name: "${CRR_LOCATION_A_NAME}"
+    locationType: "location-scality-crr-v1"


nit: do we really need multiple locations to test the feature? could we not test with a single location and multiple buckets, yet get a similar result & good confidence?

I think I discussed this with Maël too and I was thinking the same thing, technically it should be possible since the destination bucket is not bounded to the location, but in the end I think I prefer working with multiple locations as it forces us to build more realistic test scenarios

francoisferrand · 2026-06-24T14:50:17Z

+        while (Date.now() < deadline) {
+            Identity.useIdentity(IdentityEnum.ACCOUNT, location);
+            try {
+                await this.createS3Client().send(new HeadObjectCommand({ Bucket: bucket, Key: objectName }));


I think we should test replication-status ; but not use if this can be check here (may be race conditions -esp. when using cascade- so the status may have intermediate value...)

francoisferrand · 2026-06-24T14:51:34Z

+
+Then(
+    'the object at location {string} should never have replication status PENDING within {int} seconds',
+    { timeout: 120_000 },


kind of redundant, deadline/timeout is implemented in the loop itself ?

yeah but prefer to keep it, it could avoid ending up with 5h ci in case of a bug

SylvainSenechal · 2026-06-26T14:08:12Z

Realising here I have 3 scenarios, and each new scenario pretty much covers the case of the previous scenario so in theory we could just run the last one to test the feature, I think still good to have a few scenarios

Issue: ZENKO-5263

claude · 2026-06-26T14:19:37Z

+    @PreMerge
+    @ReplicationTest
+    @CRRCascade
+    @haha


@haha is a leftover debug tag (renamed from @hihi but still present). Remove before merge.

Suggested change

@haha

claude Bot reviewed Jun 22, 2026

View reviewed changes

Comment thread .github/scripts/end2end/setup-e2e-env.sh Outdated

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch 3 times, most recently from ede39d5 to 56052d7 Compare June 23, 2026 16:07

claude Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/functional/ctst/steps/crrCascade.ts Outdated

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch from 56052d7 to ee17d07 Compare June 23, 2026 19:08

scality deleted a comment from claude Bot Jun 23, 2026

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch 2 times, most recently from 32f34cc to 664e381 Compare June 23, 2026 22:09

SylvainSenechal commented Jun 23, 2026

View reviewed changes

claude Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread solution/deps.yaml Outdated

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch from 664e381 to c3043bf Compare June 23, 2026 22:15

SylvainSenechal commented Jun 23, 2026

View reviewed changes

claude Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/functional/ctst/steps/crrCascade.ts Outdated

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch 2 times, most recently from d7560c3 to c8df3be Compare June 23, 2026 22:26

SylvainSenechal marked this pull request as ready for review June 23, 2026 22:28

SylvainSenechal requested review from a team, delthas and maeldonn June 23, 2026 22:28

move replication tests into their own folder

86ad4cb

Issue: ZENKO-5263

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch from c8df3be to c70930c Compare June 24, 2026 13:39

francoisferrand reviewed Jun 24, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch 2 times, most recently from 0f6db0b to b3de8ff Compare June 26, 2026 14:03

claude Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread tests/functional/ctst/features/replication/crrCascade.feature Outdated

claude Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread tests/functional/ctst/steps/crrCascade.ts Outdated

SylvainSenechal commented Jun 26, 2026

View reviewed changes

SylvainSenechal requested a review from francoisferrand June 26, 2026 14:08

Add tests for Crr Cascade feature

900f7de

Issue: ZENKO-5263

SylvainSenechal force-pushed the improvement/ZENKO-5263 branch from b3de8ff to 900f7de Compare June 26, 2026 14:15

claude Bot reviewed Jun 26, 2026

View reviewed changes

francoisferrand removed their request for review June 26, 2026 17:00

Uh oh!

Conversation

SylvainSenechal commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Jun 22, 2026

Hello sylvainsenechal,

Uh oh!

bert-e commented Jun 22, 2026

Incorrect fix version

Uh oh!

claude Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SylvainSenechal commented Jun 22, 2026 •

edited

Loading