HDDS-16040. Make DN DiskBalancerService StorageType Aware for Container Migration. by devmadhuu · Pull Request #11322 · apache/ozone

devmadhuu · 2026-09-25T04:00:15Z

What changes were proposed in this pull request?

The datanode disk balancer treats all volumes as one pool, so it can move a container from an SSD volume to a DISK volume. Now that buckets carry a storage policy, that silently breaks the policy the data was placed for.

This change groups volumes by StorageType in DefaultContainerChoosingPolicy and balances each group on its own, so a container never moves between volumes of different storage types. The existing selection logic — ideal usage, lower/upper thresholds, source-is-highest-utilization, destination-walk, commit-on-selection — is unchanged; it was extracted into a private method and now runs per group.

A storage type with fewer than two usable volumes is skipped rather than paired across types. The stopAfterDiskEven log message was updated, since "disks are even" is now also reported when every storage type has fewer than two volumes.

Most of the diff in DefaultContainerChoosingPolicy is re-indentation from the method extraction; the new logic is few lines.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16040

How was this patch tested?

Three scenarios added to TestDefaultContainerChoosingPolicy:

Mixed SSD and DISK volumes → the chosen pair stays within one storage type.
One SSD volume plus two DISK volumes → the lone SSD volume is skipped, DISK is balanced.
One volume per storage type → returns null, nothing moves across types.

…er Migration.

devmadhuu · 2026-09-25T06:35:41Z

@xichen01 @Gargi-jais11 @ivandika3 Kindly review.
Thanks to @xichen01 for contribution. This PR is adaptation of previous work carried out on old ozone version.

Gargi-jais11

Thanks @devmadhuu for the patch.
Left few review comments.

Gargi-jais11 · 2026-10-06T11:58:06Z

+      final Map<StorageType, List<VolumeFixedUsage>> usagesByStorageType = volumeUsages.stream()
+          .collect(Collectors.groupingBy(v -> v.getVolume().getStorageType()));
+      for (StorageType storageType : StorageType.values()) {


DiskBalancerService#getTasks calls the policy availableTaskCount times per loop. Each call returns the first type that has a candidate. While SSD is out of balance, every thread slot goes to SSD moves. DISK gets nothing until SSD is within threshold, or until SSD has no movable container left.

For Example: a datanode has two small SSD volumes at 40% and 25%, and ten large DISK volumes ranging from 95% down to 20%, with a 10% threshold. DISK is the urgent group: its volumes are much larger and close to full. It still waits for SSD to finish, and SSD moves can take hours when bandwidthInMB is throttled. With parallelThread=5, all five slots go to SSD.

Might be we can discuss on this how we will handle. What I think right now is:

It should pick the group with the largest imbalance (highest.utilization - idealUsage).

Rotate the starting storage type on each call.
There might be some better options as well, I am exploring more.

Thanks, this is a valid concern. getTasks() invokes the policy once per available worker, and the policy currently restarts StorageType.values() on every invocation. Therefore, the first eligible type can consume every worker slot. I propose ranking eligible storage-type groups by their current threshold violation. We would then try groups in descending severity order, falling through when a group has no movable container. The source delta and destination committed-byte accounting are already updated between policy calls, so the ranking naturally changes as moves are scheduled. I prefer deterministic severity ranking over unconditional rotation because rotation can prioritize a mildly imbalanced tier over a volume close to full. StorageType order can remain only as a deterministic tie-breaker. I have also updated the test so DISK has the larger imbalance while SSD still appears first in the list; that will verify that selection is based on imbalance rather than list order.

I propose ranking eligible storage-type groups by their current threshold violation.

This makes sense, agreed.

…ut and computation of threshold violation within storage type and avoiding worker threads starvation for different storage types.

Gargi-jais11

Thanks @devmadhuu for updating the PR.
Few more comments.

Gargi-jais11 · 2026-10-07T04:06:58Z

    return info;
  }

+  @VisibleForTesting


@VisibleFprTesting is no more supported. Refernce: https://issues.apache.org/jira/browse/HDDS-12725

Gargi-jais11 · 2026-10-07T09:36:43Z

   * from each source volume during container moves
   * @return Ideal usage as a ratio (used space / total capacity)
   */
  public static double getIdealUsage(List<VolumeFixedUsage> volumes) {


getIdealUsage is not considering Storage Type. It should be per Storage type evaluated.

Only 1 caller method was not passing per storage type list which is fixed now including CLI caller.

Gargi-jais11 · 2026-10-07T09:38:48Z

-          lowestUsage.getUtilization() > lowerThreshold) {
-        return null;
-      }
+  private static double getThresholdViolation(List<VolumeFixedUsage> volumeUsages,


Many new helper methods are added and calling VolumeUsages. These helper methods should be moved to DiskBalancerVolumeCalaculation class as the logic can be reused and Each calculation will get one time snapshot of the VolumeUsages.

Gargi-jais11 · 2026-10-07T09:39:57Z

                "name": "ozoneAvailable",
                "type": "uint64",
                "optional": true
+              },


Do we really need in this PR?

I think this is needed as per proto-backwards-compatibility plugin else build will fail. This is auto generated file.

Gargi-jais11 · 2026-10-07T09:40:54Z

+    optional StorageTypeProto storageType = 9;
+}
+
+message StorageTypeDiskBalancerInfoProto {


Please check the existing proto messages it those can be reused else we can keep the new ones.

Checked the existing messages. VolumeReportProto describes a single physical volume (storageId, storagePath), and DatanodeUsageInfoProto carries per-datanode absolute byte counts plus DatanodeDetailsProto. The new message is an aggregate across all volumes of one storage type, and four of its six fields — currentVolumeDensitySum, idealUsage, usableVolumeCount, balanceable - have no counterpart in either. Reusing one would mean adding those fields to a message where its own existing fields don't apply, so I've kept the new message.

Gargi-jais11 · 2026-10-07T09:41:55Z

+        appendStorageTypeDetails(header, p);
+      } else if (p.hasIdealUsage() && p.hasDiskBalancerConf()
          && p.getDiskBalancerConf().hasThreshold()) {
        double idealUsage = p.getIdealUsage();


here in report we should display idealUsage for each storageType so that it's easy for user to compare.

HDDS-16040. Make DN DiskBalancerService StorageType Aware for Contain…

78904f7

…er Migration.

devmadhuu requested review from Gargi-jais11 and xichen01 September 25, 2026 04:00

github-actions Bot added the datanode label Sep 25, 2026

devmadhuu marked this pull request as ready for review September 25, 2026 06:35

devmadhuu requested a review from ChenSammi October 1, 2026 13:37

Gargi-jais11 reviewed Oct 6, 2026

View reviewed changes

Comment thread ...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java

Gargi-jais11 reviewed Oct 6, 2026

View reviewed changes

Comment thread .../java/org/apache/hadoop/ozone/container/diskbalancer/TestDefaultContainerChoosingPolicy.java Outdated

HDDS-16040. Fixed review comments for computing diskbalancer CLI outp…

ca84d0e

…ut and computation of threshold violation within storage type and avoiding worker threads starvation for different storage types.

devmadhuu requested a review from Gargi-jais11 October 6, 2026 16:19

github-actions Bot added the command-line label Oct 6, 2026

devmadhuu marked this pull request as draft October 7, 2026 04:20

devmadhuu marked this pull request as ready for review October 7, 2026 06:32

HDDS-16713. Bump develocity-maven-extension to 2.6.0 (apache#11405)

9274ed3

Gargi-jais11 reviewed Oct 7, 2026

View reviewed changes

HDDS-16040. Fixed review comments for computing diskbalancer CLI output.

b0accc0

devmadhuu requested a review from Gargi-jais11 October 7, 2026 17:36

Conversation

devmadhuu commented Sep 25, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

devmadhuu commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devmadhuu commented Sep 25, 2026 •

edited

Loading