Skip to content

fix(webhook): tolerate ClusterCreationPolicy CRD not installed - #110

Merged
atbagan merged 1 commit into
mainfrom
fix/controller-tolerate-missing-clustercreationpolicy-crd
May 30, 2026
Merged

atbagan merged 1 commit into
mainfrom
fix/controller-tolerate-missing-clustercreationpolicy-crd

Conversation

@atbagan

@atbagan atbagan commented May 30, 2026

Copy link
Copy Markdown
Contributor

What this changes

Adds an apimeta.IsNoMatchError guard around the ClusterCreationPolicyList listing in TenantClusterValidator.validatePolicy (internal/webhook/tenantcluster_webhook.go:677). When the CRD is not installed in the API server, the admission webhook now treats the situation identically to "no policies installed" (empty-list path) instead of returning an internal error that denies the request.

Four lines of production code plus one focused test case that uses a controller-runtime interceptor to inject the exact apimeta.NoKindMatchError the apiserver returns when the CRD is missing.

Why

The ClusterCreationPolicy CRD shipped in butler-api v0.21.0 and is distributed to clusters via the butler-crds Helm chart. butler-controller PR #105 (the same PR that introduced this validation path) bumped to v0.21.0 and started referencing ClusterCreationPolicyList on every TenantCluster CREATE and UPDATE admission. The butler-crds chart was never updated to ship the new CRD; the chart's templates directory has 20 CRD YAMLs and clustercreationpolicy-crd.yaml is not among them.

The result on butler-beta on 2026-05-29: every Flux Kustomization reconcile dry-runs every TenantCluster in clusters/butler-beta. Each dry-run hits this webhook. The webhook calls Client.List(ctx, &ClusterCreationPolicyList{}). The apiserver returns no matches for kind "ClusterCreationPolicy". The webhook returns the wrapped error. The dry-run is denied. The entire Kustomization apply fails. This blocked butler-server v0.17.0 from rolling out via the standing IUA-driven image patch. Full investigation in .secrets/butler-beta-vtenantcluster-2026-05-29.md.

The validation logic at line 683 already short-circuits when len(policies.Items) == 0 (no policies installed means no enforcement). This change makes "CRD not installed" share that same code path. The semantic is the same. The implementation now matches.

Verification

CGO_ENABLED=0 go vet ./... clean. go test ./internal/webhook/... pass including the new TestTCWebhook_ValidatePolicy_CRDMissing_TreatedAsNoPolicies. Diff is 63 insertions across two files.

The fix was reasoned about against butler-beta on 2026-05-29 but verified live only via the equivalent kubectl-apply-of-the-CRD short-term unblock; the defensive code path here was not exercised against a running cluster because rolling a new controller image through GitOps requires Flux to be unwedged first, and Flux is currently being unwedged by the direct CRD apply.

Dependencies and merge order

Do not merge until the companion butler-charts PR lands. The chart-side fix adds the missing CRD to the butler-crds chart. Once the chart is in good shape, this controller fix lands as durability hardening for the next cluster that takes a new controller before the chart upgrade.

The two PRs together close the missed-migration gap from both sides: the chart so the CRD ships, the controller so a future similar gap degrades gracefully instead of wedging admission.

Not in scope

  • Restoring the butler-crds chart to include this CRD. That is the companion butler-charts PR.
  • Auditing the butler-crds chart for other out-of-sync CRDs. Worth doing as part of the chart fix; flagged there.
  • Resolving butler-beta's Steward webhook outage and etcd-1 crash loop. Separate platform debt, separate session.

@atbagan
atbagan merged commit 301d36d into main May 30, 2026
8 checks passed
@atbagan
atbagan deleted the fix/controller-tolerate-missing-clustercreationpolicy-crd branch May 30, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant