Skip to content

Optimize goal state updates: batch writes and fix transit limit - #63

Merged
TejasNaikk merged 5 commits into
mainfrom
fix-goalstate-unnecessary-updates
Feb 11, 2026
Merged

TejasNaikk merged 5 commits into
mainfrom
fix-goalstate-unnecessary-updates

Conversation

@TejasNaikk

Copy link
Copy Markdown
Owner

Problem

  1. Each shard assignment was triggering a separate etcd write, causing excessive watcher triggers on data nodes during node replacement (130+ writes for a single node)
  2. Rolling update was updating 4 nodes at once instead of respecting the 20% limit because INITIALIZING shards were incorrectly counted as "converged"

Solution

Batch goal state writes

  • Accumulate all shard assignments for a node in memory during orchestration
  • Flush to etcd in a single write per node at the end of the cycle
  • Reduces etcd writes from O(shards) to O(nodes)

Fix convergence check

  • hasActualStateConverged now checks for ShardState.STARTED instead of just shard existence
  • Shards in INITIALIZING state are correctly treated as "in transit"
  • Uses existing ShardState enum for type safety

Test Plan

Added Unit Tests

Tested in staging

GeorgeZhai
GeorgeZhai previously approved these changes Feb 11, 2026

@GeorgeZhai GeorgeZhai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TejasNaikk
TejasNaikk merged commit 08000f4 into main Feb 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants