Skip to content

net: scope worker gateway topology watches - #918

Open
Patrick W. Healy (phealy) wants to merge 19 commits into
phealy/net-api-assignmentsfrom
phealy/net-api-worker-watches
Open

net: scope worker gateway topology watches#918
Patrick W. Healy (phealy) wants to merge 19 commits into
phealy/net-api-assignmentsfrom
phealy/net-api-worker-watches

Conversation

@phealy

@phealy Patrick W. Healy (phealy) commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Stop worker agents from watching cluster-wide gateway topology they don't use.

  • Gateway informers. The agent starts the GatewayPool, GatewayPoolNode and SiteGatewayPoolAssignment informers only while its role or its Site's assignments need them. It stops them when they no longer do.
  • MTU.
    • MTU is resolved from actual forwarding paths rather than from undirected topology components.
    • SiteGatewayPoolAssignment MTUs no longer apply to ordinary worker mesh links. They still apply on the gateway path to an assigned Site.
  • Watch timeout. The randomized Node informer watch timeout is doubled, so watches are re-established half as often. LIST requests are unchanged.
  • Startup race. The gateway event callback is installed before the reconcile worker starts, and events go through a synchronized lookup. Events that arrive during startup are no longer lost.

Testing

  • Unit tests for:
    • informer start and stop across role and assignment changes
    • registering the callback after events start arriving (add, update and delete for all three resources)
    • MTU along forwarding paths
    • the watch timeout wrapper
  • Tested on an isolated AKS cluster with two gateways and two remote workers:
    • ICMP and HTTP succeeded in both directions, 10 of 10.
    • Removing the remote Site from the assignment and adding it back stopped and restarted exactly the three gateway informers per remote. No agents restarted.
    • Demoting a gateway and promoting it again replaced only that gateway's agent. The other gateway carried traffic with no loss.

Stack

This is one of five stacked PRs that reduce API server load from unbounded-net-controller and unbounded-net-node. Kubernetes remains the source of truth. Review them bottom to top:

  1. net: skip no-op managed kube-proxy reconciliation #914 Controller no-op reconciliation
  2. net: reconcile SiteNodeSlices per Site #915 Targeted SiteNodeSlice reconciliation
  3. net: publish node assignments and replace stale agents #916 Controller-published node assignments
  4. net: adopt and acknowledge node assignments in the agent #917 Node agent assignment adoption
  5. net: scope worker gateway topology watches #918 Scoped worker gateway watches

Motivation. A large cluster running v0.8.0 had the controller continuously writing objects that hadn't changed. Every node agent also watched the whole cluster's gateway topology. On a canary cluster, the full stack cut controller API calls by 54.4% and node agent API calls by 73.8%.

@phealy
Patrick W. Healy (phealy) added this pull request to stack #919 October 14, 2026 23:10
This was referenced Oct 14, 2026
Merged
Open
Open
Open
@phealy Patrick W. Healy (phealy) changed the title phealy/net api worker watches net: scope worker gateway topology watches Oct 14, 2026
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 14, 2026 23:12
Copilot AI reviewed Oct 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved startup concurrency, blocked reconciliation, MTU calculation, and status consistency issues prevent approval.

6 open findings
What changed in this PR

Reduces node-agent API server load by running gateway topology watches only when needed, completing the five-PR optimization stack.

Changes:

  • Adds gateway informer lifecycle management, synchronized callbacks, and readiness reporting.
  • Calculates MTU limits from forwarding paths rather than connected topology.
  • Doubles informer watch timeouts and adds tests and documentation.
File Description
docs/​net/​configuration.md Documents forwarding-path MTU limits.
docs/​net/​architecture.md Explains conditional gateway watches.
docs/​content/​reference/​networking/​configuration.md Updates public MTU guidance.
cmd/​unbounded-net-node/​status_server.go Adds lifecycle-aware readiness and status.
cmd/​unbounded-net-node/​status_server_http_test.go Tests gateway readiness responses.
cmd/​unbounded-net-node/​site_watch_reconcile.go Integrates informer lifecycle and transit MTUs.
cmd/​unbounded-net-node/​reconciliation_helpers.go Adds forwarding-path MTU calculations.
cmd/​unbounded-net-node/​reconciliation_helpers_test.go Tests revised MTU behavior.
cmd/​unbounded-net-node/​node_types.go Carries transit MTU limits.
cmd/​unbounded-net-node/​mtu.go Applies role-aware assignment limits.
cmd/​unbounded-net-node/​main.go Separates discovery and gateway informers.
cmd/​unbounded-net-node/​informer_factory.go Doubles watch timeouts.
cmd/​unbounded-net-node/​informer_factory_test.go Tests watch and LIST options.
cmd/​unbounded-net-node/​gateway_informer_lifecycle.go Manages conditional gateway caches.
cmd/​unbounded-net-node/​gateway_informer_lifecycle_test.go Tests lifecycle transitions and callbacks.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cmd/unbounded-net-node/reconciliation_helpers.go Outdated
Comment thread cmd/unbounded-net-node/reconciliation_helpers.go Outdated
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go Outdated
Comment thread cmd/unbounded-net-node/gateway_informer_lifecycle.go Outdated
Comment thread cmd/unbounded-net-node/status_server.go
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 16, 2026 13:38
Copilot AI reviewed Oct 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Gateway-side MTU resolution still omits local pool and same-Site assignment limits.

2 open findings
6 resolved since last review

🧠 Review effort: Balanced

Comment thread cmd/unbounded-net-node/reconciliation_helpers.go Outdated
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go Outdated
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 16, 2026 14:04
Copilot AI reviewed Oct 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

MTU resolution still omits applicable forwarding-path limits and includes unrelated assignment caps.

3 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread cmd/unbounded-net-node/reconciliation_helpers.go
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go Outdated
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go Outdated
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 16, 2026 14:19
Copilot AI reviewed Oct 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

MTU calculations still omit intermediate gateway Site caps and worker-to-gateway SitePeering caps.

2 open findings
3 resolved since last review

🧠 Review effort: Balanced

Comment thread cmd/unbounded-net-node/mtu.go Outdated
Comment thread cmd/unbounded-net-node/reconciliation_helpers.go
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 16, 2026 14:32
Copilot AI reviewed Oct 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cache retirement can precede successful cleanup, and transit MTU processing retains excluded-path caps and introduces quadratic assignment scanning.

3 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread cmd/unbounded-net-node/site_watch_reconcile.go
Comment thread cmd/unbounded-net-node/reconciliation_helpers.go Outdated
Comment thread cmd/unbounded-net-node/site_watch_reconcile.go Outdated
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 16, 2026 14:55
@phealy
Patrick W. Healy (phealy) force-pushed the phealy/net-api-worker-watches branch 2 times, most recently from c6fd7c2 to f4c85db Compare October 23, 2026 21:02
@phealy
Patrick W. Healy (phealy) added this pull request to the merge queue Oct 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue because a pull request earlier in the stack was removed Oct 23, 2026
@phealy
Optimize worker gateway topology watches
f05046b 
Use forwarding paths rather than undirected topology components for MTU resolution, and dynamically synchronize gateway-specific informers only while the pinned role or direct Site assignments require them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Double node informer watch lifetimes
48dcc1c 
Wrap dynamic watch requests so reflector-selected randomized timeouts are doubled for both watch-list initialization and normal watches without changing LIST requests or other watch options.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Scope assignment MTU to gateway paths
86c9564 
Keep SiteGatewayPoolAssignment MTUs off ordinary worker mesh links while retaining them for gateway-to-assigned-Site forwarding paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Install gateway event callbacks before reconciliation starts
d21d01a 
Install the update callback once before launching the reconcile worker and dispatch lifecycle events through a synchronized callback lookup. Cover late callback registration with immediate add, update, and delete events for all three gateway resources.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Bound gateway informer startup and honor forwarding path MTUs
babc06f 
Cancel obsolete unsynced generations, bound required-cache synchronization, preserve adjacent-hop MTU caps, and use the active generation in route summaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Honor gateway-local pool and same-Site assignment MTU caps
018254e 
Apply local pool caps to gateway-facing peers and fabric MTU without including unrelated pools, and preserve assignment caps for same-Site gateway-to-worker links.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Scope gateway transit MTUs to traversed Site and assignment edges
31db6e6 
Include the next-hop gateway Site and downstream gateway mesh-edge caps while separating worker assignments from gateway-local pool assignments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Honor first-hop and intermediate gateway Site MTU caps
0861bfe 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Fence gateway retirement on cleanup and scope transit MTU work
f1e495b 
Propagate link, route and policy teardown errors without forgetting failed interface cleanup. Index assignment MTUs once and exclude local routes before calculating transit caps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Require gateway objects and preserve pending cleanup state
24749cb 
Skip route destinations without GatewayPoolNode objects. Index SitePeering MTUs and retain discovered policy tables until cleanup succeeds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Publish gateway forwarding MTUs and retry partial policy setup
21aa8ef 
Advertise the verified local forwarding limit after successful dataplane work. Apply published gateway and intermediate-pool limits on workers only to avoid gateway feedback. Preserve pending policy setup across failed teardown, reinstallation, and expected-interface reconciliation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Fence dataplane retries and index remaining gateway MTU edges
bac8e56 
Bypass unchanged-state reconciliation after a partial programming failure, and propagate gateway policy installation errors. Index enabled pool-peering MTU caps once per reconcile. Avoid synchronous assignment-cache scans for unchanged resyncs and events unrelated to the adopted Site, preserving immediate local dependency cancellation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Share configured MTU limits within direct mesh components
9eb31c4 
Compute a common configured Site and mesh-peering MTU cap from existing caches. Shared pod-facing and tunnel interfaces require this constraint even without transitive routing. Exclude disabled, network-only, missing-Site and disconnected relationships; do not join components through gateway paths. Apply the cap to peers and the local fabric without adding API writes or watches, and cover the three-Site asymmetric case and upward recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Resolve gateway peering MTUs through lazy membership indexes
422e9db 
Replace eager all-pairs expansion with linear Site and GatewayPool memberships and memoized lookups for pairs actually used by accepted gateway paths. Build gateway-only indexes lazily so direct-mesh workers do not pay for them. Cover 1,000-member peerings and repeated reverse-pair cache hits without quadratic storage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Document the direct-mesh MTU cap in the public reference
8f200b9 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Recover fabric MTUs after failed gateway publication
5b67ce3 
Retain the attempted fabric MTU independently from committed routing state and retry CNI/pod MTU writes after any incomplete reconciliation. Mark all error exits dirty, including unchanged-state verification failures. Preserve tighten-before-tunnel and tunnel-before-relax ordering through failed writes and topology reversions. Extend production-path tests for failed status publication, partial pod MTU updates, verification drift, and both reversion directions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Isolate MTU policy tests from host network state
3ff3be3 
Use an empty netlink cache so policy-cap assertions do not inherit the host default-route MTU. Preserve the production underlay cap behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 46ff6261-c4cc-4fb0-9fb7-4e66d2b9c203
@phealy
Preserve node regression fixtures after linearizing the stack
b2f837d 
Retain advertised gateway objects for unmanaged Site coverage and dataplane retry assertions for neighbor sizing failures that were previously recorded in merge resolutions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 46ff6261-c4cc-4fb0-9fb7-4e66d2b9c203
@phealy
Buffer IPAM frames before writing to the socket
21c4dfb 
Avoid a second zero-length socket write after an empty protobuf frame has already been consumed and the peer has closed. Cover immediate peer closure for empty and nonempty requests and preserve write-error propagation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 46ff6261-c4cc-4fb0-9fb7-4e66d2b9c203

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@bcho hbc (bcho) bcho approved these changes
@cchildress Cameron Childress (cchildress) cchildress left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@phealy @bcho @cchildress