From d0377eb4b9a915b067c60ab3c8a43fa4b7976150 Mon Sep 17 00:00:00 2001 From: Ethan Rose Date: Tue, 22 Sep 2026 16:54:14 -0400 Subject: [PATCH 1/3] Add initial zdu branch merge checklist --- .../03-merged-branches/19-hdds-14496-zdu.md | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md diff --git a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md new file mode 100644 index 0000000000..a5e35feca7 --- /dev/null +++ b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md @@ -0,0 +1,46 @@ +# HDDS-14496: Zero Downtime Upgrade + +Epic: [HDDS-14496](https://issues.apache.org/jira/browse/HDDS-14496) +Feature branch: https://github.com/apache/ozone/tree/HDDS-14496-zdu + +## 1. Builds/intermittent test failures + +## 2. Documentation + + +## 3. Design, attached the docs + + +## 4. S3 compatibility + + + +## 5. Docker-compose / Acceptance tests + + +## 6. Support of containers / Kubernetes + +No addition. No change in existing support. + +## 7. Coverage / Code quality + +TODO add coverage of new code and existing master coverage + +## 8. Build time + +TODO add build time and full CI time for master and branch + +## 9. Possible incompatible changes/used feature flag + +TODO point to dev docs, but also flag that a new framework is in progress + +## 10. Third-party dependencies/License changes + +There are no third party dependencies introduced by this feature. + +## 11. Performance + + +## 12. Security considerations + +TODO admin checks on all APIs From a127a92bc3d330c8a4a274b0031efb182043b769 Mon Sep 17 00:00:00 2001 From: Ethan Rose Date: Fri, 25 Sep 2026 19:51:38 -0400 Subject: [PATCH 2/3] Completely fill in merge checklist --- .../03-merged-branches/19-hdds-14496-zdu.md | 75 +++++++++++++++++-- 1 file changed, 69 insertions(+), 6 deletions(-) diff --git a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md index a5e35feca7..edc47b303a 100644 --- a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md +++ b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md @@ -5,34 +5,80 @@ Feature branch: https://github.com/apache/ozone/tree/HDDS-14496-zdu ## 1. Builds/intermittent test failures +Currently no intermittent test failures are observed on the branch. The following integration tests were run 400 times and passed on the branch: + +- `TestDatanodeCurrentVersionEndToEnd` +- `TestOMUpgradeFinalization` +- `TestBlockDeletionService` +- `TestUpgradeFinalizationWithOldClients` +- `TestScmHAFinalization` + - Previously flaky on the branch, but fixed by https://github.com/apache/ozone/pull/11200 + +Additionally, `TestScmDataDistributionFinalization` was marked flaky on master but fixed on the ZDU branch by https://github.com/apache/ozone/pull/11228 + +AI identified test stabilization changed that were proactively applied in https://github.com/apache/ozone/pull/11258 as well. + ## 2. Documentation +- Documentation for admins to execute ZDU is open at https://github.com/apache/ozone-site/pull/545 +- Documentation for developers to maintain upgrade compatibility while working on features is open at https://github.com/apache/ozone-site/pull/560 + +Both of these PRs are currently works in progress and will continue to be updated during the branch review. ## 3. Design, attached the docs +Design document is open at https://github.com/apache/ozone/pull/9664. It has been updated as the feature was developed, but we will do a final check for consistency with the implementation before it is merged with the branch. ## 4. S3 compatibility - +ZDU happens within an Ozone cluster and does not alter any user facing S3 APIs. When doing a rolling upgrade of S3 gateways, there will be a period where the gateways are in mixed versions and may support different sets of APIs. If this is not desirable, a load balancer and upgrade order must be configured outside of Ozone to direct all traffic to gateways in one version while another set is upgraded. ## 5. Docker-compose / Acceptance tests +A new `rolling` upgrade acceptance test execution has been added in addition to the existing `non-rolling` flow. Due to the excessive amount of time taken to restart containers one-by-one in the test environment, `rolling` executes as follows: + +1. Rolling upgrade of all services from old to new version +2. Non-rolling downgrade of all services from new to old version +3. Non-rolling upgrade of all services from old to new version + +The initial rolling upgrade covers the same set of mixed component versions as the downgrade, since restarts for downgrade must be done in reverse order of the upgrade. Therefore we still get coverage of all component version interactions through just the initial rolling upgrade. + +This simplification allows the execution of the rolling upgrade suite to finish in [20 minutes](https://github.com/apache/ozone/actions/runs/33115384634/job/98781266395), compared to doing a rolling restart for all three steps which took [1.5 hours](https://github.com/errose28/ozone/actions/runs/32282383715/job/96194185059). Note that the original non-rolling upgrade suite finished in about [15 minutes](https://github.com/apache/ozone/actions/runs/32365620238/job/96418094876). Since acceptance test suites are run in parallel and the upgrade suite was not the longest, the additional 5 minutes of execution time does not affect overall time for CI completion. + +Since rolling upgrade requires the initial version to support the feature, it can't actually be run yet since there is currently no supported release. On the branch we tested it by using the same version for old and new, basically doing a rolling restart to ensure the test was runnable. When the branch is merged, we will continue to run only the existing non-rolling upgrade tests until the first ZDU enabled release goes out. ## 6. Support of containers / Kubernetes -No addition. No change in existing support. +No additions or changes in existing support. ## 7. Coverage / Code quality -TODO add coverage of new code and existing master coverage +- [Master has 79% code coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&id=hadoop-ozone) +- [ZDU branch has 78.7% code coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&branch=HDDS-14496-zdu&id=hadoop-ozone) ## 8. Build time -TODO add build time and full CI time for master and branch +- Latest successful ZDU CI runs finished in [1 hour 26 minutes](https://github.com/apache/ozone/actions/runs/34383195672) and [1 hour 30 minutes](https://github.com/apache/ozone/actions/runs/35762034464) +- This is in line with recent master CI runs which show executions ranging from [1 hour 26 minutes](https://github.com/apache/ozone/actions/runs/36069833598) to + [1 hour 31 minutes](https://github.com/apache/ozone/actions/runs/35985080804) ## 9. Possible incompatible changes/used feature flag -TODO point to dev docs, but also flag that a new framework is in progress +### Compatibility with existing releases + +This branch introduces a new unified versioning framework as outlined in the design doc, where a single version per component is used to track all compatibility concerns. The change from the old to new framework requires compatibility handling from the framework itself, so it takes effect after the new `ZDU` component version added on this branch is finalized. + +Additionally, new simplified CLIs to start and monitor upgrade finalization have been added, and prepare for upgrade has been removed. The old CLIs have been retained in a way that still allows old upgrade scripts to finalize a ZDU enabled cluster. + +### Compatibility with future releases + +From the point this branch is merged onward (component version `ZDU`), all commits must maintain compatibility with zero-downtime upgrade. Instructions for developers to maintain this are added to the website in https://github.com/apache/ozone-site/pull/560. There are multiple initiatives that will be investigated after the branch merge to make this process easier: + +- Static analysis to comment on PRs and notify reviewers when potentially incompatible parts of the code are changed +- An AI skill that can inspect code diff between two releases and flag potentially unaddressed incompatible changes before they are shipped +- A new OM request versioning framework that better aligns with the more strict requirements of ZDU on OM requests. + +There are no changes to our external client/server cross-compatibility guarantees as part of this feature. ## 10. Third-party dependencies/License changes @@ -40,7 +86,24 @@ There are no third party dependencies introduced by this feature. ## 11. Performance +With ZDU, checking versions for feature compatibility is still a constant time operation and should not change request processing performance. Finalization still executes as a single Ratis request and upgrade actions are still required to be constant time, which is the same as master currently. + +Upgrade finalization no longer requires closing all write pipelines, which should improve performance during finalization compared to non-rolling upgrades. ## 12. Security considerations -TODO admin checks on all APIs +Two new user facing APIs were added: + +- [finalize upgrade](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMFinalizeUpgradeRequestBase.java#L69) + - The pre-execute method of finalize upgrade checks for admin privileges. +- [query upgrade status](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L3780) + - This is a read-only operation that does not expose sensitive information and currently does not have an admin check, although one could be added for consistency. + +These commands forward to internal APIs that OM invokes on SCM through the [`ScmBlockLocationProtocol`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java#L45), which is restricted to `hdds.scm.kerberos.principal` the same as existing operations like block deletion. + +Internal APIs were added in OM and SCM to check that all peers have been upgraded before allowing finalization to proceed: + +- `StorageContainerLocationProtocol#getPeerUpgradeStatus` + - Restricted to [`hdds.scm.kerberos.principal`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java#L60) +- `OMAdminProtocol#getPeerUpgradeStatus` + - Restricted to [`ozone.om.kerberos.principal`](https://github.com/apache/ozone/blob/53f472a7c04f58a021625cdc6f060bf629e9d4a7/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OMAdminProtocol.java#L31) From b47ada398d01dcfc7a8b0b1b2f567cb8ced94eff Mon Sep 17 00:00:00 2001 From: Ethan Rose Date: Tue, 29 Sep 2026 13:04:44 -0400 Subject: [PATCH 3/3] Update admin check for upgrade status --- .../03-merged-branches/19-hdds-14496-zdu.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md index edc47b303a..4a2afc9438 100644 --- a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md +++ b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md @@ -92,12 +92,10 @@ Upgrade finalization no longer requires closing all write pipelines, which shoul ## 12. Security considerations -Two new user facing APIs were added: +Two new user facing APIs were added which both have admin checks: - [finalize upgrade](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMFinalizeUpgradeRequestBase.java#L69) - - The pre-execute method of finalize upgrade checks for admin privileges. -- [query upgrade status](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L3780) - - This is a read-only operation that does not expose sensitive information and currently does not have an admin check, although one could be added for consistency. +- [query upgrade status](https://github.com/apache/ozone/blob/709ac02990fe3a7a6fd99a781df53e95e33d2bfd/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L3781) These commands forward to internal APIs that OM invokes on SCM through the [`ScmBlockLocationProtocol`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java#L45), which is restricted to `hdds.scm.kerberos.principal` the same as existing operations like block deletion.