Skip to content

HDDS-15031. Add troubleshooting page for slow datanode decommission - #552

Open
Eason09053360 wants to merge 2 commits into
apache:masterfrom
Eason09053360:HDDS-15031
Open

Eason09053360 wants to merge 2 commits into
apache:masterfrom
Eason09053360:HDDS-15031

Conversation

@Eason09053360

Copy link
Copy Markdown

What changes were proposed in this pull request?

This fills in the Troubleshooting → Decommissioning and Maintenance page, which was a draft: true placeholder under HDDS-9860.

The page covers a Datanode decommission that is slow or stuck:

  • How the SCM admin monitor decides a node is done (pipelines closed, state confirmed by heartbeat, every container sufficiently replicated) and how the Replication Manager throttles the copies.
  • How to check progress with ozone admin datanode status decommission, the per-node SCM log summary line, and the NodeDecommissionMetrics / ReplicationManagerMetrics JMX metrics.
  • Common causes for a stuck workflow: pipelines that do not close, containers that do not close, no suitable target, no source with capacity, replication command timeouts, and the node dying mid-workflow, each with the SCM log message to look for.
  • The four replication throttles (hdds.scm.replication.datanode.replication.limit, hdds.datanode.replication.outofservice.limit.factor, hdds.scm.replication.inflight.limit.factor, hdds.datanode.replication.streams.limit), their defaults, where they apply, and which are reconfigurable at runtime.

Property defaults, log messages, metric names and CLI output were checked against the current Ozone source.

The Datanode Decommission page in the administrator guide gets a "See also" link to the new page, since nothing linked to it before.

What is the link to the Apache Jira?

https://issues.apache.org/jira/browse/HDDS-15031

How was this patch tested?

  • markdownlint on the two changed pages.
  • npm run build, which fails on broken links and anchors in this site, completes successfully.
  • Checked the rendered page locally.

Generated-by: Claude Code (Claude Fable 5.1), used to cross-check the page against the Ozone source and to apply the resulting fixes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The page contains an invalid CLI example and misleading maintenance and replication-throttling guidance.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 Medium severity

Open (5)
What changed in this PR

Completes the troubleshooting documentation for slow or stuck Datanode decommissioning and links it from the administrator guide.

Changes:

  • Documents workflow progress checks using CLI, logs, and metrics.
  • Describes common replication blockers and tuning properties.
  • Adds a troubleshooting link to the Datanode decommission guide.
File Description
docs/​06-troubleshooting/​10-decommission-maintenance.md Adds the troubleshooting guide.
docs/​05-administrator-guide/​.../​01-datanode-decommission.md Adds a “See also” link.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/06-troubleshooting/10-decommission-maintenance.md Outdated
Comment thread docs/06-troubleshooting/10-decommission-maintenance.md Outdated
Comment thread docs/06-troubleshooting/10-decommission-maintenance.md
Comment thread docs/06-troubleshooting/10-decommission-maintenance.md Outdated
Comment thread docs/06-troubleshooting/10-decommission-maintenance.md Outdated

@jojochuang jojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a fine start. A good troubleshooting guide needs to be more concise and actionable steps.

Comment thread docs/06-troubleshooting/10-decommission-maintenance.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants