HDDS-16022. Branch merge checklist for ZDU

docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md

@@ -0,0 +1,107 @@
+# HDDS-14496: Zero Downtime Upgrade
+
+Epic: [HDDS-14496](https://issues.apache.org/jira/browse/HDDS-14496)
+Feature branch: https://github.com/apache/ozone/tree/HDDS-14496-zdu
+
+## 1. Builds/intermittent test failures
+
+Currently no intermittent test failures are observed on the branch. The following integration tests were run 400 times and passed on the branch:
+
+- `TestDatanodeCurrentVersionEndToEnd`
+- `TestOMUpgradeFinalization`
+- `TestBlockDeletionService`
+- `TestUpgradeFinalizationWithOldClients`
+- `TestScmHAFinalization`
+  - Previously flaky on the branch, but fixed by https://github.com/apache/ozone/pull/11200
+
+Additionally, `TestScmDataDistributionFinalization` was marked flaky on master but fixed on the ZDU branch by https://github.com/apache/ozone/pull/11228
+
+AI identified test stabilization changed that were proactively applied in https://github.com/apache/ozone/pull/11258 as well.
+
+## 2. Documentation
+
+- Documentation for admins to execute ZDU is open at https://github.com/apache/ozone-site/pull/545
+- Documentation for developers to maintain upgrade compatibility while working on features is open at https://github.com/apache/ozone-site/pull/560
+
+Both of these PRs are currently works in progress and will continue to be updated during the branch review.
+
+## 3. Design, attached the docs
+
+Design document is open at https://github.com/apache/ozone/pull/9664. It has been updated as the feature was developed, but we will do a final check for consistency with the implementation before it is merged with the branch.
+
+## 4. S3 compatibility
+
+ZDU happens within an Ozone cluster and does not alter any user facing S3 APIs. When doing a rolling upgrade of S3 gateways, there will be a period where the gateways are in mixed versions and may support different sets of APIs. If this is not desirable, a load balancer and upgrade order must be configured outside of Ozone to direct all traffic to gateways in one version while another set is upgraded.
+
+## 5. Docker-compose / Acceptance tests
+
+A new `rolling` upgrade acceptance test execution has been added in addition to the existing `non-rolling` flow. Due to the excessive amount of time taken to restart containers one-by-one in the test environment, `rolling` executes as follows:
+
+1. Rolling upgrade of all services from old to new version
+2. Non-rolling downgrade of all services from new to old version
+3. Non-rolling upgrade of all services from old to new version
+
+The initial rolling upgrade covers the same set of mixed component versions as the downgrade, since restarts for downgrade must be done in reverse order of the upgrade. Therefore we still get coverage of all component version interactions through just the initial rolling upgrade.
+
+This simplification allows the execution of the rolling upgrade suite to finish in [20 minutes](https://github.com/apache/ozone/actions/runs/33115384634/job/98781266395), compared to doing a rolling restart for all three steps which took [1.5 hours](https://github.com/errose28/ozone/actions/runs/32282383715/job/96194185059). Note that the original non-rolling upgrade suite finished in about [15 minutes](https://github.com/apache/ozone/actions/runs/32365620238/job/96418094876). Since acceptance test suites are run in parallel and the upgrade suite was not the longest, the additional 5 minutes of execution time does not affect overall time for CI completion.
+
+Since rolling upgrade requires the initial version to support the feature, it can't actually be run yet since there is currently no supported release. On the branch we tested it by using the same version for old and new, basically doing a rolling restart to ensure the test was runnable. When the branch is merged, we will continue to run only the existing non-rolling upgrade tests until the first ZDU enabled release goes out.
+
+## 6. Support of containers / Kubernetes
+
+No additions or changes in existing support.
+
+## 7. Coverage / Code quality
+
+- [Master has 79% code coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&id=hadoop-ozone)
+- [ZDU branch has 78.7% code coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&branch=HDDS-14496-zdu&id=hadoop-ozone)
+
+## 8. Build time
+
+- Latest successful ZDU CI runs finished in [1 hour 26 minutes](https://github.com/apache/ozone/actions/runs/34383195672) and [1 hour 30 minutes](https://github.com/apache/ozone/actions/runs/35762034464)
+- This is in line with recent master CI runs which show executions ranging from [1 hour 26 minutes](https://github.com/apache/ozone/actions/runs/36069833598) to
+ [1 hour 31 minutes](https://github.com/apache/ozone/actions/runs/35985080804)
+
+## 9. Possible incompatible changes/used feature flag
+
+### Compatibility with existing releases
+
+This branch introduces a new unified versioning framework as outlined in the design doc, where a single version per component is used to track all compatibility concerns. The change from the old to new framework requires compatibility handling from the framework itself, so it takes effect after the new `ZDU` component version added on this branch is finalized.
+
+Additionally, new simplified CLIs to start and monitor upgrade finalization have been added, and prepare for upgrade has been removed. The old CLIs have been retained in a way that still allows old upgrade scripts to finalize a ZDU enabled cluster.
+
+### Compatibility with future releases
+
+From the point this branch is merged onward (component version `ZDU`), all commits must maintain compatibility with zero-downtime upgrade. Instructions for developers to maintain this are added to the website in https://github.com/apache/ozone-site/pull/560. There are multiple initiatives that will be investigated after the branch merge to make this process easier:
+
+- Static analysis to comment on PRs and notify reviewers when potentially incompatible parts of the code are changed
+- An AI skill that can inspect code diff between two releases and flag potentially unaddressed incompatible changes before they are shipped
+- A new OM request versioning framework that better aligns with the more strict requirements of ZDU on OM requests.
+
+There are no changes to our external client/server cross-compatibility guarantees as part of this feature.
+
+## 10. Third-party dependencies/License changes
+
+There are no third party dependencies introduced by this feature.
+
+## 11. Performance
+
+With ZDU, checking versions for feature compatibility is still a constant time operation and should not change request processing performance. Finalization still executes as a single Ratis request and upgrade actions are still required to be constant time, which is the same as master currently.
+
+Upgrade finalization no longer requires closing all write pipelines, which should improve performance during finalization compared to non-rolling upgrades.
+
+## 12. Security considerations
+
+Two new user facing APIs were added which both have admin checks:
+
+- [finalize upgrade](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMFinalizeUpgradeRequestBase.java#L69)
+- [query upgrade status](https://github.com/apache/ozone/blob/709ac02990fe3a7a6fd99a781df53e95e33d2bfd/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L3781)
+
+These commands forward to internal APIs that OM invokes on SCM through the [`ScmBlockLocationProtocol`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java#L45), which is restricted to `hdds.scm.kerberos.principal` the same as existing operations like block deletion.
+
+Internal APIs were added in OM and SCM to check that all peers have been upgraded before allowing finalization to proceed:
+
+- `StorageContainerLocationProtocol#getPeerUpgradeStatus`
+  - Restricted to [`hdds.scm.kerberos.principal`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java#L60)
+- `OMAdminProtocol#getPeerUpgradeStatus`
+  - Restricted to [`ozone.om.kerberos.principal`](https://github.com/apache/ozone/blob/53f472a7c04f58a021625cdc6f060bf629e9d4a7/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OMAdminProtocol.java#L31)