Skip to content

Publish arm64 images so the operator can run on arm64 clusters #144

Description

@zarcen

Goal

Deploy the bootc-operator to an existing arm64 Kubernetes cluster (in our case one provisioned by CAPI) using the documented install path, and have both the controller and the daemon come up.

Problem

The published images are linux/amd64 only. ghcr.io/bootc-dev/bootc-operator:latest is a single application/vnd.docker.distribution.manifest.v2+json with "architecture": "amd64", with no manifest list, so there is no arm64 instance for a node to select.

That blocks both workloads, since config/manager/manager.yaml and config/daemon/daemon.yaml both reference the same ghcr.io/bootc-dev/bootc-operator:latest:

  • the controller Deployment cannot be scheduled onto an arm64 node, and
  • the daemon DaemonSet cannot run on any managed arm64 node at all. This one is structural rather than incidental: the daemon executes bootc on the host via nsenter resolved from its own filesystem (internal/bootc/executor.go), so the container has to match the node's architecture. There is no way to run an amd64 daemon against an arm64 host.

So on an all-arm64 cluster the operator is simply not installable today, and on a mixed-arch cluster the daemon would be missing from exactly the nodes it is meant to manage.

Why this should be straightforward

Nothing in the operator looks architecture specific:

  • cmd/controller and cmd/daemon are both pure Go with no cgo dependencies, so CGO_ENABLED=0 GOARCH=arm64 go build already works on both.
  • The quay.io/fedora/fedora-minimal:44 base is already published for linux/arm64 (as well as ppc64le and s390x), and provides nsenter there.

This looks like build and publish plumbing rather than a code change.

Also worth deciding

config/manager/manager.yaml still carries the kubebuilder-scaffolded nodeAffinity block commented out, listing amd64/arm64/ppc64le/s390x and referring to a docker-buildx target that does not exist in this repo. Once the image is a manifest list, kubelet picks the right instance on its own and no arch constraint is needed, so that block could either be dropped or narrowed to kubernetes.io/os: linux. Happy to follow whichever the maintainers prefer.

Out of scope

  • e2e coverage on arm64, and arm64 node disk images in bink — worth its own issue, since it depends on runner infrastructure rather than on this repo.
  • Multi-arch node images. internal/registry/resolver.go resolves a tag with remote.Get, which returns the index digest for a manifest list, while internal/daemon/reconciler.go compares that against the digest bootc status reports for the booted image. Every image in play today is a single manifest so the two cannot disagree, but that may need platform-aware resolution later.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions