design.md

Design: the core address-binding controller

• Component: address-controller — the class-agnostic core of "IP addresses as a first-class resource" (cozystack/community#35)
• API group: local.sdn.cozystack.io/v1alpha1
• Status: implemented (alpha)

This document records the contract: the resource model, the two state machines the core drives, the exact reconciliation algorithms, and the obligations a per-class driver must meet. It is the reference against which drivers (such as metallb-iad) are written.

1. Scope

The core is the analog of the generic PVC/PV binding controller in the storage subsystem. It owns:

• resolving a claim to a class,
• accepting and completing claim–address bindings,
• matching pre-provisioned addresses to claims,
• phase and condition bookkeeping on both sides,
• finalizer-driven reclaim when a claim is deleted,
• auto-claiming addresses for LoadBalancer Services that name no claim (§10).

It never allocates an address, never interprets class parameters, never associates an address with a Service, and never talks to a backend. Everything backend-specific is a per-class driver's job, in a separate deployment, discovered through the class's spec.provisioner — exactly as a StorageClass names a CSI driver.

2. Resource model

storage analog	kind	scope	one-line role
StorageClass	IPAddressClass	cluster	which pool, which driver, which reclaim policy
PersistentVolume	IPAddress	cluster	one concrete IP: the reservation and the inventory record
PersistentVolumeClaim	IPAddressClaim	namespaced	"give me one" — the whole tenant-facing API

IPAddressClass

• spec.provisioner (immutable) — the driver that fulfils claims of this class.
• spec.reclaimPolicy — Retain (default) or Delete; copied onto each provisioned IPAddress, where it takes effect. What each policy actually reclaims — the API object, the ledger entry, backend state — is spelled out in §6.
• spec.parameters — an opaque object (unknown fields preserved), never read by the core. Shape is defined by the driver.
• The annotation ipaddressclass.local.sdn.cozystack.io/is-default-class: "true" marks the class used by claims that name none.
• The annotation ipaddressclass.local.sdn.cozystack.io/is-auto-claim-class: "true" marks the class auto-claims draw from (§10). Independent of the default class; at most one class may carry it.

IPAddress

• spec.address (immutable) — the IP, one per object. The object is the reservation.

  Why spec and not status: the split here is request vs. record, not desired vs. observed. The claim is the request; an IPAddress is created only after an allocation is a fact — the driver performs or observes the backend operation first (carve from a range, allocate at a provider, adopt an existing reservation) and then records the result. The address is therefore always known at creation, and there is no observe-later phase for a status field to serve: an IPAddress is never a pending request for an address (that is what a Pending claim is), it is the ledger entry for one that exists.

  Given that, the field must provide identity, and identity cannot live in status. The object's name derives from the IP, one-object- per-IP is enforced by create-time collision on exactly this value, and conflict detection indexes it — all of which needs the value present from the first moment and immutable for the object's life. Status is the opposite by design: mutable, and legitimately absent at any moment (wiped status must be reconstructible from the world). For the providerRef arm the IP genuinely is re-observable — describe eipalloc-… and read it back — but for the fromClass arm the ledger entry is the only record in existence, and a union shares one schema: the arm with nothing to re-observe from decides the placement. The closest precedent is Service.spec.clusterIP: allocated by the system, not the user, yet spec — because it is an allocation result the system commits to and everything downstream references, not an observation that may drift. (Cluster API's IPAM contract records the address in spec likewise; KEP-1880 goes further and puts it in the object name.)

  Note what this does not imply: a provider where the IP cannot be chosen at allocation time — AWS hands you whatever AllocateAddress returns — is fully supported. Choosing at allocation is not the Pin capability; Pin is attaching a specific already-reserved address to a workload at association time, which such providers do natively (associate by the eipalloc-… handle). The driver allocates, observes the handle and the IP, and records both (spec.address, spec.source.providerRef). Only a backend with no pin mechanism at all — nothing that says "use this reserved address" at association time — is disqualified, because a reservation that can never be re-attached is not a reservation.

• spec.className — the class it belongs to.

• spec.claimRef {namespace, name, uid} — the binding. uid may be empty at creation (a driver pre-binding); the core completes it. This field is the single authoritative record of a binding; everything else (claim status, phases) is derived from it.

• spec.reclaimPolicy — what happens to this object when its claim goes.

• spec.source — a union, exactly one member set:

  • fromClass: {} — the driver carved it from the class's range; the driver is the IPAM of record;
  • providerRef: {id} — a reservation held by an external provider under the provider's own IAM, recorded here by its stable handle — whether the driver allocated it at the provider on demand or adopted one that already existed.

• status.phase — see the state machine below.

• status.associatedTo — the workload the address is currently announced for. nil means reserved but inert: held, attached to nothing. Written by drivers only.

IPAddressClaim

• spec.className — empty means the default class.
• spec.family — IPv4 | IPv6 | Dual. Dual binds two addresses.
• spec.addressName — optional pre-binding to one specific Available address (the PVC.spec.volumeName analog); meaningful for single-family claims.
• status.phase — Pending | Bound | Lost.
• status.className — the sticky record of which class the claim resolved to (see §4 step 6).
• status.addresses — a list of {name, address}, one entry per bound IPAddress, so a Dual claim reports both families. This is what a tenant reads and puts in DNS.

3. Ownership: who writes what

The contract is largely a discipline about who may write which field.

field	tenant	admin	core	driver
Claim.spec	✍ creates/edits		✍ creates (auto-claim only)
Claim provisioner annotation			✍ stamps	reads
Claim.status.*			✍	never
Claim protection finalizer			✍
Address object creation		✍ (static pre-provisioning)	never	✍ (provisioning)
Address.spec.claimRef		✍ may clear on Released	✍ sets on match / completes uid	✍ pre-sets at creation only
Address.status.phase: Available/Bound/Released			✍	never
Address.status.phase: Conflict/Lost			never (sticky)	✍ sets and clears
Address.status.associatedTo			never	✍
Address protection finalizer			✍
Address driver finalizer(s)				✍
IPAddressClass		✍	reads; ✍ protection finalizer	reads (incl. parameters)

Two invariants fall out of this table:

1. One binding record. Address.spec.claimRef is the only place a binding lives. Claim.status.addresses is a projection of it, always recomputed, never authoritative.
2. Phase partition. Available, Bound, Released belong to the core; Conflict and Lost belong to drivers, and the core treats them as sticky — it neither enters nor leaves them. This is what lets a driver flag backend-level facts (a collision, a lost provider reservation) without racing the core's bookkeeping.

4. The claim reconciliation algorithm

State machine:

            class resolved &        all requested families
            no address yet          bound & not Lost
  (new) ──────► Pending ─────────────────► Bound
                   ▲                        │
                   │   (never: Lost is      │  a bound address
                   │    not re-entered      ▼  disappears / goes Lost
                   │    from Pending)      Lost
                   └────────────────────────┘
                        all families satisfied again

Each reconcile pass runs the following sequence. Every step is idempotent; the pass re-derives everything from the cluster state.

1. Deletion? If the claim is being deleted, run the reclaim flow (§6) and stop.

2. Protection. Ensure the claim carries the local.sdn.cozystack.io/claim-protection finalizer, so deletion always passes through the reclaim flow.

3. Collect bound addresses. All IPAddress objects whose claimRef names this claim's namespace/name, and whose claimRef.uid is either empty or equal to the claim's UID. A UID mismatch means the address is bound to an earlier, deleted claim that happened to have the same name — a stale binding. It is never adopted; the address reconciler reclaims it (§5 step 6).

4. Accept pre-bindings. For collected addresses with an empty claimRef.uid (a driver or an admin pre-bound them), write the claim's UID if the address is of the claim's class, of a requested family, and of a family the claim does not already hold. This is the core's acceptance of the driver's provisioning. A claim that has not recorded a class resolves it first (step 6). A mismatched pre-binding is refused with a PreBindingRefused event and is not collected, as the PV controller leaves a pre-bound volume that does not satisfy its claim.

5. Anything unbound? Expand spec.family into concrete families (Dual → v4 + v6); a family is unbound if no collected address of that family exists, in any phase. A Lost address keeps its family: the claim waits for it to be restored or deleted and is never given a second address of that family. If no family is unbound, skip straight to status (step 8) — the class is not consulted at all. Binding state comes strictly before class state: a fully bound claim needs nothing from its class, so a deleted class never disturbs existing bindings (see Class deletion below).

6. Class resolution (only reached with families unbound, or with a pre-binding to judge). Resolve a class name, trying in order:

   1. spec.className, if set (an explicit spec always wins);
   2. status.className — the sticky record of a previous resolution, so a claim that resolved against a default class does not flap when the default annotation moves;
   3. the default class: the exactly-one IPAddressClass annotated as default. Zero or more than one is a terminal condition for this pass (NoDefaultClass / MultipleDefaultClasses on the ClassResolved condition); the claim is retried when any class changes. The core deliberately refuses to guess among multiple defaults.

   The named class must exist (ClassNotFound condition otherwise; the pass still falls through to step 8 so phase and addresses stay maintained). On success the core stamps the claim with the annotation local.sdn.cozystack.io/provisioner: <class.spec.provisioner>. The stamp always mirrors the resolved class — if a claim is re-targeted at a different class before binding, the stamp follows. This annotation is the driver's watch key; it is how a driver knows a claim is its to serve without ever resolving classes itself.

7. Match, per unbound family. For each unbound family:

   • candidate set: if spec.addressName is set, only that object; otherwise every address in phase Available (so: no claimRef, and already accepted by the address reconciler) with the resolved class and the wanted family;
   • bind the candidate with the lexicographically smallest name by writing claimRef {namespace, name, uid}. Determinism makes concurrent controllers converge; optimistic concurrency resolves races (the loser's write fails, it re-lists and takes the next candidate);
   • no candidate → the family stays unsatisfied and the claim waits for its driver to provision.

8. Status. Recompute from scratch:

   • every family has a collected address not in phase Lost → Bound;
   • previously Bound (or Lost) and now not → Lost — a binding degraded, which is surfaced, never silently re-provisioned around; the claim returns to Bound by itself if the address comes back, and binds a new one once an admin deletes the Lost address;
   • otherwise Pending, with the Bound condition explaining what it waits for (WaitingForProvisioning names the stamped provisioner). status.addresses lists every collected address (name + IP), sorted by object name. status.className records the resolution when one happened this pass and is left untouched otherwise.

Class deletion. The core holds a deleted class with the local.sdn.cozystack.io/class-protection finalizer until no IPAddress of the class is left, in any phase, and records a ClassInUse event naming the addresses it waits for. Drivers resolve their backend configuration through the class to hold and pin its addresses, so a class must outlive them. The claim side does not depend on this: satisfied claims skip class resolution entirely (step 5), and reclaim reads the reclaimPolicy copied onto each address at provisioning time (§6). A class removed by force leaves existing bindings, reclaim and status upkeep intact; still-unbound claims of that class stay Pending with ClassNotFound.

5. The address reconciliation algorithm

State machine (core-owned transitions solid, driver-owned dashed):

   (new) ──► Pending ──► Available ◄────────────────┐
   (invalid     │            │  ▲                   │ admin clears claimRef,
    address     │       core matches / driver       │ claimRef
    stays       │       pre-binds + core accepts    │
    Pending)    │            ▼  │                   │ or releaseDelay elapses
                │          Bound ───────────────► Released
                │            ┆      claim deleted     │
                │            ┆      (Retain)          │ (Delete: object
                │            ┆                        │  is deleted instead)
                │            ▼┄┄┄┄ driver-owned ┄┄┄┄┄▼
                └──────►  Conflict / Lost   (sticky for the core;
                          set and cleared by the driver only)

Each pass:

1. Deletion? The protection finalizer (local.sdn.cozystack.io/address-protection) is released only when no live claim holds the address — i.e. the address is not Bound, or the referenced claim is missing, being deleted, or has a different UID. Otherwise deletion blocks and waits: a Bound address cannot vanish from under a live claim. (Driver finalizers are independent of this and are the driver's own business.)
2. Protection. Ensure the finalizer.
3. Validation. An unparsable spec.address pins the object at Pending; nothing downstream trusts an invalid IP.
4. Sticky phases. Conflict and Lost short-circuit the pass — driver territory (§3).
5. Unbound. No claimRef → Available. This single rule is also the recycling path: an admin clears the claimRef of a Released address and it becomes matchable again — the deliberate, manual step PV semantics prescribe for Retain. A class may automate the step with spec.releaseDelay (step 7).
6. Bound. With a claimRef:
   • claim exists, UID matches → Bound;
   • claim exists, claimRef.uid empty → wait (the claim reconciler is about to complete the binding);
   • claim is being deleted → wait (the claim-side reclaim flow owns the transition);
   • claim gone, or UID mismatch → safety-net reclaim: apply this address's own reclaimPolicy (Delete → delete self, Retain → Released). This duplicates the claim-side flow on purpose: reclaim must happen even if the claim vanished without its finalizer running (e.g. the finalizer was force-removed).
7. Release delay. A Released address of a class with spec.releaseDelay has its claimRef cleared once the delay has passed since it was released, and step 5 makes it Available. The release time is the lastTransitionTime of the address's Bound condition, which the core sets with every phase it owns (True for Bound; False with reason Available, Released or Pending otherwise). An address annotated ipaddress.local.sdn.cozystack.io/held: "true" is exempt and stays Released until an admin acts. An address Released before this condition existed is dated from the first pass that sets it. Changing a class's delay re-evaluates its Released addresses.

6. The reclaim flow (claim deletion)

Runs behind the claim's protection finalizer, so it always runs before the claim disappears:

• For every address bound to the claim (same collection rule as §4 step 3, less pre-bindings step 4 would refuse), apply the address's reclaimPolicy — copied from the class at provisioning time, so a later class edit does not retroactively change the fate of existing addresses:
  • Retain → phase Released. The claimRef survives — the released address is not reusable until an admin clears it (§5 step 5). An address in Conflict or Lost keeps its phase: both belong to the driver (§3).
  • Delete → delete the IPAddress object. The core does not tear down any backend state; the driver's own finalizer on the address intercepts the deletion and deallocates first (§7 obligation 4).
• Exception: a claim with a controller owner — a Service for an auto-claim (§10), or any controller that mints claims for a tenant — lives and dies with that owner, so its addresses are deleted whatever their reclaimPolicy. Removing the controller reference turns it into an ordinary claim, and the policy applies again.
• Remove the finalizer; the claim goes away.

What each policy actually reclaims. "The address" is up to three things: the API object, the reservation it records, and whatever backend state stands behind it. The policies act on them differently:

• Retain acts on nothing but the phase. The object stays, so the reservation stays: a range-carved (fromClass) IP remains excluded from allocation, and a provider-held (providerRef) reservation remains held at the provider — which means it keeps incurring provider charges. That is deliberate, not a leak: retained means still yours, and an idle-but-billed address is exactly what holding an elastic IP is. The cost stops only when the reservation is actually given up, i.e. when someone deletes the IPAddress object.
• Delete removes the object, and the driver's teardown finalizer is where backend state is released before it goes: a fromClass address returns to the free range simply by its ledger entry ceasing to exist (for backends like MetalLB there is nothing else to do); a provider-side reservation the driver itself created must be released at the provider. For a reservation the driver merely adopted (it existed before the object, e.g. an admin-imported EIP), whether teardown releases it or leaves it in the provider's hands is the driver's documented policy call — the safe default is to leave what you did not create.

So the EIP question has a precise answer: Retain keeps the EIP and its bill until an admin deletes the object; Delete releases a driver-allocated EIP at the provider, via the driver's finalizer.

7. The driver contract

A per-class driver (the CSI-driver analog) plugs in with no registration step: deploying it and creating an IPAddressClass naming it is the whole integration. Its obligations:

1. Serve stamped claims. Watch IPAddressClaims whose local.sdn.cozystack.io/provisioner annotation equals the driver's name and which are not fully bound. The driver must not resolve classes or defaults itself — the stamp is the assignment.
2. Provision pre-bound. For each family the claim still misses, create one IPAddress:
   • spec.claimRef pre-set to the claim's namespace/name (UID optional — the core completes it; setting it is allowed and slightly tighter);
   • spec.className set, spec.reclaimPolicy copied from the class;
   • spec.source reflecting reality: fromClass if the driver carved the address from the class's own range, providerRef if a provider holds the reservation (whether the driver allocated it there or adopted a pre-existing one);
   • the driver's own finalizer, if teardown has any backend work. The driver determines the IP — by choosing it from a range, or by recording what its backend handed out; the core decides whether the binding stands (it is the binder).
3. Never touch core-owned state (§3): claim status, the core phases, core finalizers, or the claimRef of any existing address.
4. Tear down on deletion. When an IPAddress the driver created is deleted (reclaim Delete, or an admin action), the driver's finalizer must release backend state before letting the object go.
5. Own association (optional but standard). Attaching a bound address to a workload is a separate, reversible act, entirely driver-side: resolve the Service annotation local.sdn.cozystack.io/ip-address-claim (naming a claim in the Service's own namespace — cross-namespace sharing is not a thing), translate it into the backend's pin mechanism, maintain status.associatedTo, and enforce that one claim serves one workload at a time. Disassociation must leave the address Bound — reserved, inert.
6. Detect, and clear, conflicts. If the backend hands the reserved address to a workload the binding does not authorize, set phase Conflict (and Lost if the backing reservation disappears). The core will never override these; the driver clears them by setting the phase back to a core-owned value once the condition has passed.

Static pre-provisioning needs no driver at all: an admin may create Available addresses by hand and the core matches them (§4 step 7).

8. Interaction walkthrough (the happy path)

tenant                core                          driver
  │ create Claim        │                             │
  │────────────────────►│ finalizer, resolve class,   │
  │                     │ stamp provisioner ─────────►│ sees stamped, unbound claim
  │                     │                             │ allocates, creates IPAddress
  │                     │◄────────────────────────────│   (claimRef pre-set, own finalizer)
  │                     │ completes uid, phase Bound  │
  │◄────────────────────│ status.addresses = [ip]     │
  │ annotate Service    │                             │
  │──────────────────────────────────────────────────►│ resolves claim → ip,
  │                     │                             │ writes backend pin,
  │                     │                             │ sets associatedTo
  │ delete Service      │                             │ withdraws pin, clears associatedTo
  │                     │        (address stays Bound: reserved, inert)
  │ annotate Service B  │                             │ same ip, pinned to B

Deleting the claim, not the Service, is what releases the address — via §6, honoring the reclaim policy. This asymmetry is the entire point of the model. Auto-claims (§10) are the deliberate exception: they belong to their Service until someone detaches them.

9. Relation to the design proposal

Where community#35 is explicit, this implementation follows it. Where it is deliberately a stub, the choices made here are: PV-style annotation stamping instead of a driver-registration CRD; no modeling of driver capabilities (Allocate/Adopt/Pin) until a registration object exists to declare them on; opaque-object class parameters rather than a string map; refusal over newest-wins for multiple default classes; dual-stack as two IPAddress objects under one claim; sticky class resolution recorded in status (there is no admission webhook to default the spec); association and admission policy left entirely to drivers; and auto-claim (§10) as the one place the core watches Services. The API group is local.sdn.cozystack.io rather than the proposal's sketched ipam.cozystack.io.

10. Auto-claim for LoadBalancer Services

Plain LoadBalancer Services — a tenant's, or the host-side Services a tenant cluster's cloud provider creates — name no claim, and drivers keep their pools away from automatic assignment. Without auto-claim such a Service gets either nothing or an address from some other pool that nobody meters. Auto-claim gives it a claim instead.

A Service qualifies when it is a live type: LoadBalancer Service that:

• carries no local.sdn.cozystack.io/ip-address-claim annotation,
• does not carry local.sdn.cozystack.io/auto-claim: "false" (drivers stamp this on Services they create for their own bookkeeping),
• sets neither spec.loadBalancerClass (another implementation's) nor spec.loadBalancerIP (asks for a specific address),
• carries none of the backend's pool, pin or sharing annotations: for MetalLB, loadBalancerIPs, address-pool and allow-shared-ip under metallb.io/ or metallb.universe.tf/. Such a Service has chosen its address source. A driver's placeholders name their pool this way, so this rule excludes them even without the opt-out,
• has no status.loadBalancer.ingress yet. This is what keeps a Service addressed by other means from being moved to a new address.

The class is the one named by the Service's local.sdn.cozystack.io/ip-address-class annotation, else the single class marked is-auto-claim-class. With no marked class the Service is left alone; with several, a warning event is recorded on it and nothing is claimed.

The core then creates an IPAddressClaim named after the Service, in the Service's namespace, with that className, the family derived from the Service (Dual only for RequireDualStack), and the Service as its controller. If a claim of that name exists and is not controlled by the Service, nothing is adopted and an event says so. Finally the core sets the Service's ip-address-claim annotation to the claim; from there the driver associates the address exactly as for a tenant-made claim (§7 obligation 5).

Lifecycle: deleting the Service garbage-collects the claim, and the claim finalizer deletes its addresses regardless of reclaimPolicy (§6). To keep the address, remove the claim's controller reference; the claim then outlives the Service and can be named by another one. The core sets the reference only when creating the claim and never restores it.