Repository navigation
feat(kg): add a configurable --check-in-period flag - #550
Open
IvanHunters wants to merge 1 commit into
Open
IvanHunters wants to merge 1 commit into
IvanHunters wants to merge 1 commit into
Conversation
A node's WireGuard IP is assigned by its rank in the currently Ready set, and a node counts as Ready only while its last-seen annotation is within twice the check-in period (60s by default). On large clusters the per-node last-seen writes generate enough watch churn that liveness cannot propagate within that window, so every agent's Ready set collapses and most nodes collide on the first address of the subnet. Add a --check-in-period flag so operators can widen the freshness window on large clusters. The default is unchanged at 30s, so existing deployments behave exactly as before. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add a
--check-in-periodflag tokgso the node check-in interval, and the Ready staleness window derived from it, can be tuned. The default is unchanged at 30s, so existing deployments behave exactly as before.Why
A node's WireGuard IP is assigned by its rank in the currently Ready set:
NewTopologyallocates addresses over the sorted segments, and the segment list is built only from nodes that passNode.Ready(). A node is Ready only while itslast-seenannotation is within2 * checkInPeriod(60s by default), and each node refreshes its ownlast-seeneverycheckInPeriod(30s).On large clusters (around 200 nodes) the per-node
last-seenwrites generate enough watch churn that liveness cannot propagate within the 60s window. Every agent's Ready set collapses to a small subset, allocation runs over that tiny set, and most nodes end up assigning their own node the first address of the subnet, colliding on a single WireGuard IP. Because the IP is a function of rank in the Ready set, a single Ready flap shifts many nodes' IPs, which triggers more rewrites and more churn, so the system oscillates around the Ready threshold.Making the period configurable lets operators widen the freshness window on large clusters so the Ready set stays close to the full node set. This is a mitigation, not a root-cause fix: the underlying problem is that IP allocation is coupled to the volatile Ready set. Full analysis and other fix directions are in #549.
Changes
pkg/mesh/backend.go:checkInPeriodbecomes a package variable defaulting to the new exportedDefaultCheckInPeriod(30s); semantics are unchanged.pkg/mesh/mesh.go:Newtakes acheckIn time.Duration; when it is greater than zero it sets the period, otherwise the default is kept, so existingNewcallers and tests are unaffected.cmd/kg/main.go: register the--check-in-periodflag with defaultmesh.DefaultCheckInPeriodand pass it tomesh.New.Testing
go build ./...(linux/amd64)go test ./pkg/mesh/go vet ./pkg/mesh/ ./cmd/kg/