[Patrick W. Healy (phealy)]

Summary

Have the node agent adopt its Site and GatewayPool assignment once per process and acknowledge it.

• Adoption. At startup, the agent reads its own Node and fixes the Site and GatewayPool for the life of the process. Managed CNI setup is tied to that assignment. Gateways still get their CNI configuration from the Site.
• Acknowledgment. After its first successful reconciliation, the agent records the assignment it is running on its own Pod. The controller from net: publish node assignments and replace stale agents #916 uses this to replace outdated agents.
• Startup-only Node watch:
  • The agent watches only its own Node, from adoption until the acknowledgment is saved. The watch uses a field selector and starts from a validating GET.
  • Any of these restarts the container: a UID, Site or pool change, or the Node being deleted.
  • This prevents an agent whose first reconciliation keeps failing from being stuck on an old assignment, a case the controller skips because the Pod hasn't acknowledged.
  • The watch stops once the acknowledgment is written, so no long-lived Node watch remains.
• Taint removal. When a gateway is demoted, removing its taint retries only on patch races, with a capped exponential backoff. An admission denial is returned immediately.

Testing

• Unit tests for:
  • adoption and acknowledgment ownership
  • startup watch cancellation and revalidation after a disconnect
  • which taint-removal errors are retried, and the retry limits
• Tested on an isolated AKS cluster with remote workers:
  • Set a GatewayPool label that didn't exist, so the agent couldn't acknowledge.
  • Removed the label. The startup watch restarted the container without the controller replacing the Pod, and the agent then acknowledged the correct assignment.
  • Audit logs showed every watch on the agent's own Node closed. The last one closed 1.8 ms after the acknowledgment.

Stack

This is one of five stacked PRs that reduce API server load from unbounded-net-controller and unbounded-net-node. Kubernetes remains the source of truth. Review them bottom to top:

1. net: skip no-op managed kube-proxy reconciliation #914 Controller no-op reconciliation
2. net: reconcile SiteNodeSlices per Site #915 Targeted SiteNodeSlice reconciliation
3. net: publish node assignments and replace stale agents #916 Controller-published node assignments
4. net: adopt and acknowledge node assignments in the agent #917 Node agent assignment adoption
5. net: scope worker gateway topology watches #918 Scoped worker gateway watches

Motivation. A large cluster running v0.8.0 had the controller continuously writing objects that hadn't changed. Every node agent also watched the whole cluster's gateway topology. On a canary cluster, the full stack cut controller API calls by 54.4% and node agent API calls by 73.8%.

[Copilot]

🟡 Changes recommended

Transition cleanup has unresolved concurrency and recovery issues that can leave stale networking configuration, advertisements, or taints.

4 open findings

• Wait for reconciliation workers before transition cleanup · New
• Clear all advertised routes when syncing nil routes · New
• Retry gateway taint removal before acknowledging worker adoption · New
• Disable stale managed CNI configuration during handoff · New

What changed in this PR

Pins each node agent’s assignment for its process lifetime and acknowledges it for controller-driven replacement, supporting the API-load reduction stack.

Changes:

• Adopt assignments and monitor changes until acknowledgment succeeds.
• Tie CNI setup and transition cleanup to the adopted assignment.
• Add bounded taint-removal retries and lifecycle tests.

File	Description
cmd/​unbounded-net-node/​startup_assignment_watch_test.go	Tests watch cancellation and revalidation.
cmd/​unbounded-net-node/​site_watch_reconcile.go	Integrates acknowledgment and transition cleanup.
cmd/​unbounded-net-node/​node_types.go	Tracks adopted assignment and taint state.
cmd/​unbounded-net-node/​main.go	Wires adoption into startup.
cmd/​unbounded-net-node/​gateway_taint_test.go	Tests retry classification and limits.
cmd/​unbounded-net-node/​gateway_pool_peering_protocol_test.go	Tests gateway readiness requirements.
cmd/​unbounded-net-node/​bootstrap_helpers.go	Validates bootstrap assignments and removes taints.
cmd/​unbounded-net-node/​bootstrap_helpers_test.go	Checks taint removal.
cmd/​unbounded-net-node/​bootstrap_assignment_test.go	Tests assignment-aware CNI bootstrap.
cmd/​unbounded-net-node/​assignment.go	Implements adoption, monitoring, and acknowledgment.
cmd/​unbounded-net-node/​assignment_test.go	Tests assignment lifecycle helpers.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Assignment handoffs and networking teardown warrant human validation, and the startup health-server ordering regression remains unresolved.

1 open finding

• Start health server before waiting for Node Site label · New

4 resolved since last review

• Disable stale managed CNI configuration during handoff
• Retry gateway taint removal before acknowledging worker adoption
• Clear all advertised routes when syncing nil routes
• Wait for reconciliation workers before transition cleanup

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

The unresolved CNI startup issue and assignment-driven networking teardown require correction and final human review.

1 open finding

• Missing Site incorrectly defaults startup to managed CNI mode · New

1 resolved since last review

• Start health server before waiting for Node Site label

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Live-network teardown needs human review, and Node deletion still bypasses forced transition cleanup.

1 open finding

• NotFound leaves routes and interfaces configured after node deletion · New

1 resolved since last review

• Missing Site incorrectly defaults startup to managed CNI mode

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Assignment handoffs and forced host-network cleanup warrant human validation, and assignment-watch reconnect throttling remains unresolved.

0 open findings

1 resolved since last review

• NotFound leaves routes and interfaces configured after node deletion

Previously missed (1)

In code that hasn't changed since last review

Throttle watch reconnects when no Site assignment exists

cmd/​unbounded-net-node/​assignment.go:299

While a Node has no Site assignment, a closed watch or watch.Error event immediately starts another GET/WATCH cycle (lines 206 and 257-299). Repeated error events or immediate stream closures therefore cause an unthrottled request and warning-log loop on each affected agent. Call waitForAssignmentRetry after stopping the watch, as watchAdoptedAssignment already does, so reconnects are delayed and cancellation remains responsive.

🧠 Review effort: Balanced

[Patrick W. Healy (phealy)]

Fixed the latest review body's previously missed reconnect-throttling issue in bdc9069. Unassigned Node watches now wait for the cancellation-aware retry interval after closed streams, watch errors or deletion before issuing another GET/WATCH. Regression tests cover immediate closures and error events without a reconnect storm.

[Copilot]

🔵 Needs a closer look

Assignment handoff and forced host-network cleanup affect node connectivity and warrant final human validation.

0 open findings

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Concurrent assignment handoff and forced host-network cleanup warrant final human review despite no confirmed blocking findings.

0 open findings

🧠 Review effort: Balanced

The merge-base changed after approval.