Skip to content

ARSN-617: clean-read filter hiding non-localized versions - #2688

Draft
DarkIsDude wants to merge 3 commits into
development/8.5from
improvement/ARSN-617/clean-read-filter
Draft

ARSN-617: clean-read filter hiding non-localized versions#2688
DarkIsDude wants to merge 3 commits into
development/8.5from
improvement/ARSN-617/clean-read-filter

Conversation

@DarkIsDude

Copy link
Copy Markdown
Contributor

Context

Part of the clean-room D/R feature (epic ZENKO-5238, design Artesca Full D/R).

In a clean room, object metadata is replicated before the object data is copied locally: until then the version's location/dataStoreName still refers to the remote production site. Such a version is non-localized, the condition being locations[objMD.dataStoreName].isCRR. Clients of the clean room must never observe those versions.

Changes

  • new per-call cleanRead flag on the metadata read and listing APIs (getObject, getObjects, getBucketAndObject, listObject, listMultipartUploads), a backend-agnostic contract implemented by MongoClientInterface for now;
  • the location configuration reaches the metadata layer through a new getLocationConstraints parameter, evaluated per call so configuration updates are picked up (same channel as the existing isLocationTransient predicate);
  • filtering happens in the MongoDB query (value.dataStoreName: { $nin: [...] }) rather than after the fetch: listing cursors are limited by maxKeys, so filtering documents after the limit would under-fill pages and report a complete listing while entries remain. In a metadata search the condition is added as an $and element after the search query is merged, so a search on dataStoreName neither overwrites it nor is overwritten by it;
  • getLatestVersion applies the same filter, so a hidden master (and a PHD key) resolves to the newest localized version;
  • DelimiterMaster in v0 skips the version keys of an object whose master key is hidden, instead of exposing them as master entries (v0 streams master and version keys in the same range); gated on the flag, no change for other deployments;
  • MetadataWrapper sets the flag on every read and listing call when clean read is enabled, and refuses to start when the backend does not implement it — a silently dropped filter would expose the very versions it is meant to hide. Backbeat-internal listings (listLifecycleObject) and all write paths are untouched;
  • delete markers and PHD keys carry no dataStoreName, are matched by $nin, and are therefore never hidden.

Activation is out of scope here: CLDSRV-957 surfaces the environment variable through Cloudserver's configuration, ZKOP-564 sets it on the user-facing Cloudserver only. Write-time master-key handling is ARSN-618.

Tests

  • tests/unit/storage/metadata/mongoclient/cleanRead.spec.js — generated queries: flag off, flag on, no CRR location, no location configuration, filter propagation to getLatestVersion, and MongoReadStream $and merging against a colliding search query;
  • tests/unit/storage/metadata/MetadataWrapperCleanRead.spec.js — flag injection on read/listing calls, absence on internal and write calls, refusal on an unsupported backend;
  • tests/functional/metadata/mongodb/cleanRead.spec.js — end-to-end on mongodb-memory-server, both v0 and v1, over a mix of localized versions, non-localized versions, a delete marker and a mixed object: master and version listings, paged version listing with maxKeys: 1, getObject by version and by master, getObjects batch, plus every case with the flag off.

yarn build, yarn lint, yarn lint_md, yarn prettier:diff, yarn test (3941 passing) and the tests/functional/metadata/mongodb suite (282 passing) are green.

The first commit is a prettier reformat of the files touched, kept separate from the functional change.

🤖 Generated with Claude Code

DarkIsDude and others added 2 commits August 19, 2026 15:54
Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a per-call `cleanRead` flag on the metadata read and listing APIs,
hiding the object versions whose data still lives on a remote source
site (`locations[objMD.dataStoreName].isCRR`), as needed by the
clean-room D/R deployments. MongoClientInterface implements it in the
MongoDB query itself, so that listing limits and truncation stay
correct, and MetadataWrapper sets the flag on every read and listing
call when the deployment enables it.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bert-e

bert-e commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Hello darkisdude,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue ARSN-617 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 8.5.15

Please check the Fix Version/s of ARSN-617, or the target
branch of this pull request.

@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.73239% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.73%. Comparing base (d6ef4ed) to head (6e7339a).
⚠️ Report is 3 commits behind head on development/8.5.

Files with missing lines Patch % Lines
lib/storage/metadata/mongoclient/readStream.js 73.07% 7 Missing ⚠️
lib/algos/list/delimiterMaster.ts 90.74% 5 Missing ⚠️
...orage/metadata/mongoclient/MongoClientInterface.ts 90.24% 4 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                 @@
##           development/8.5    #2688      +/-   ##
===================================================
+ Coverage            74.66%   74.73%   +0.06%     
===================================================
  Files                  227      227              
  Lines                18606    18640      +34     
  Branches              3880     3857      -23     
===================================================
+ Hits                 13892    13930      +38     
+ Misses                4709     4705       -4     
  Partials                 5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants