Skip to content

HDDS-16723. Drop queued ICRs for re-registering endpoint to avoid oversized heartbeat. - #11423

Open
hani-fouladgar wants to merge 3 commits into
apache:masterfrom
hani-fouladgar:HDDS-16723
Open

hani-fouladgar wants to merge 3 commits into
apache:masterfrom
hani-fouladgar:HDDS-16723

Conversation

@hani-fouladgar

@hani-fouladgar hani-fouladgar commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

A datanode heartbeats each SCM endpoint with the reports queued for that endpoint. For incremental container reports (ICRs) and command status reports, StateContext.getAllAvailableReports() was called with no size limit (Integer.MAX_VALUE), so every queued report was packed into a single heartbeat.

This PR:

  • Adds config hdds.container.report.max.limit (default 10000) and documents it in ozone-default.xml.
  • Adds StateContext.getAllAvailableReports(HostAndPort endpoint, int maxLimit) and StateContext.hasPendingReports(HostAndPort endpoint). The existing no-arg method delegates with Integer.MAX_VALUE, so other callers are unaffected.
  • HeartbeatEndpointTask reads the limit and passes it to report collection. After a successful heartbeat, if reports remain queued for the endpoint it calls setNextHB(Time.monotonicNow()) so the next heartbeat fires immediately, draining a backlog in a few bounded messages rather than one cap-sized chunk per heartbeat interval.

Design notes:

  • The cap and follow-up-heartbeat behavior reuse the existing per-heartbeat-limit pattern already used for container/pipeline actions (hdds.container.action.max.limit, hdds.pipeline.action.max.limit) and the report limit parameter that getAllAvailableReportsUpToLimit already accepted but was never given a real value.
  • The cap is a report count. In practice the heavyweight full container report is a single prioritized message bounded by container count, while ICRs are small per-container deltas, so a count cap is a sound proxy for message size. At the default of 10000 and ~120 bytes per ContainerReplicaProto, a capped heartbeat is on the order of ~1 MB — well under the 128 MB ipc.maximum.data.length ceiling, yet far above steady-state ICR volume, so the cap only engages during backlog recovery and leaves normal heartbeats unchanged.
  • No report is dropped: reports beyond the limit stay queued and are sent in immediately-scheduled follow-up heartbeats. There is no wire/protobuf change and no cross-version compatibility impact.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16723

If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.

(Please replace this section with the link to the Apache JIRA)

How was this patch tested?

  • TestStateContext#testGetAllAvailableReportsRespectsLimit: queues more ICRs than the limit, asserts a limited collection returns at most the limit and leaves the remainder queued (hasPendingReports), and that successive calls drain the queue.
  • TestHeartbeatEndpointTask#testHeartbeatCapsReportsAndSchedulesFollowup: queues more ICRs than the configured limit, runs a heartbeat, and asserts the heartbeat request carries only the capped number of ICRs, the remainder stays queued, and an immediate follow-up heartbeat is scheduled.
  • Ran the full TestStateContext and TestHeartbeatEndpointTask suites

@hani-fouladgar
hani-fouladgar marked this pull request as draft October 6, 2026 21:43
@hani-fouladgar
hani-fouladgar marked this pull request as ready for review October 7, 2026 23:16
@hani-fouladgar

Copy link
Copy Markdown
Contributor Author

@jojochuang @ptlrs @yandrey321 @errose28 - Please review this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant