Repository navigation
HDDS-16723. Drop queued ICRs for re-registering endpoint to avoid oversized heartbeat. - #11423
Open
hani-fouladgar wants to merge 3 commits into
Open
hani-fouladgar wants to merge 3 commits into
hani-fouladgar wants to merge 3 commits into
Conversation
…rsized heartbeat.
hani-fouladgar
marked this pull request as draft
October 6, 2026 21:43
…void oversized heartbeat." This reverts commit dafb4b6.
…rtbeats Co-Authored-By: Claude Opus 4.8
hani-fouladgar
force-pushed
the
HDDS-16723
branch
from
October 7, 2026 18:07
5d72569 to
9d55100
Compare
hani-fouladgar
marked this pull request as ready for review
October 7, 2026 23:16
Contributor
Author
|
@jojochuang @ptlrs @yandrey321 @errose28 - Please review this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A datanode heartbeats each SCM endpoint with the reports queued for that endpoint. For incremental container reports (ICRs) and command status reports,
StateContext.getAllAvailableReports()was called with no size limit (Integer.MAX_VALUE), so every queued report was packed into a single heartbeat.This PR:
hdds.container.report.max.limit(default10000) and documents it inozone-default.xml.StateContext.getAllAvailableReports(HostAndPort endpoint, int maxLimit)andStateContext.hasPendingReports(HostAndPort endpoint). The existing no-arg method delegates withInteger.MAX_VALUE, so other callers are unaffected.HeartbeatEndpointTaskreads the limit and passes it to report collection. After a successful heartbeat, if reports remain queued for the endpoint it callssetNextHB(Time.monotonicNow())so the next heartbeat fires immediately, draining a backlog in a few bounded messages rather than one cap-sized chunk per heartbeat interval.Design notes:
hdds.container.action.max.limit,hdds.pipeline.action.max.limit) and the reportlimitparameter thatgetAllAvailableReportsUpToLimitalready accepted but was never given a real value.ContainerReplicaProto, a capped heartbeat is on the order of ~1 MB — well under the 128 MBipc.maximum.data.lengthceiling, yet far above steady-state ICR volume, so the cap only engages during backlog recovery and leaves normal heartbeats unchanged.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16723
If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.
(Please replace this section with the link to the Apache JIRA)
How was this patch tested?
TestStateContext#testGetAllAvailableReportsRespectsLimit: queues more ICRs than the limit, asserts a limited collection returns at most the limit and leaves the remainder queued (hasPendingReports), and that successive calls drain the queue.TestHeartbeatEndpointTask#testHeartbeatCapsReportsAndSchedulesFollowup: queues more ICRs than the configured limit, runs a heartbeat, and asserts the heartbeat request carries only the capped number of ICRs, the remainder stays queued, and an immediate follow-up heartbeat is scheduled.TestStateContextandTestHeartbeatEndpointTasksuites