Skip to content

HDDS-16673. Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end - #11397

Open
rich7420 wants to merge 2 commits into
apache:masterfrom
rich7420:HDDS-16673
Open

rich7420 wants to merge 2 commits into
apache:masterfrom
rich7420:HDDS-16673

Conversation

@rich7420

@rich7420 rich7420 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Paginated ListBuckets and ListDirectoryBuckets can call hasNext() again after the bucket iterator is exhausted, causing another empty ListBuckets RPC.

Check the page limit before advancing the iterator, and perform the lookahead only when the page is full. This avoids the repeated fetch on non-empty final pages while preserving continuation tokens and directory-bucket filtering.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16673

How was this patch tested?

  • TestRootList 10×10 passed.
  • Real HTTP/OM RPC tracing on master ae281be314, before/after applying the patch: one fewer ListBuckets RPC on exact and short final pages for both APIs. Continuing pages and unpaginated listing kept the same counts. Response sizes, directory filtering, and token presence were checked; each scenario was repeated.
  • https://github.com/rich7420/ozone/actions/runs/36992173530

Copilot AI balanced review requested due to automatic review settings October 3, 2026 08:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

s3 S3 Gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants