Skip to content

Replace deprecated minio subchart with RustFS - #465

Draft
mgruner wants to merge 7 commits into
mainfrom
replace-minio-rustfs
Draft

Replace deprecated minio subchart with RustFS#465
mgruner wants to merge 7 commits into
mainfrom
replace-minio-rustfs

Conversation

@mgruner

@mgruner mgruner commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Closes #398. Last part of the move away from bitnami.

Important

Draft until RustFS 1.0 GA. The subchart is pinned to 1.0.0-rc.5. This branch is meant to be rebased until GA ships, then re-pinned before merge.

Why

MinIO stopped publishing community Helm charts and container images, and the MinIO operator repository was archived in March 2026. The bundled bitnami/minio subchart was already only usable through the bitnamilegacy image workaround.

RustFS is S3-compatible and has an official chart that tracks the project's releases 1:1. The alternative cloudpirates/rustfs chart was evaluated and rejected: it self-describes as ALPHA and pins rustfs:1.0.0-beta.1 from 2026-04-29 while declaring appVersion: 1.0.0, and its config vocabulary (RUSTFS_LOG_LEVEL, RUSTFS_EXTERNAL_ADDRESS, RUSTFS_CORS_*) no longer matches what the current server accepts. We only chose cloudpirates elsewhere where there was no usable official chart.

What changed

  • Chart 19.0.0 — breaking for anyone using the optional S3 storage provider.
  • zammadConfig.miniozammadConfig.rustfs; subchart values key minio:rustfs:.
  • The subchart is pinned to single-node mode (its own default is a four-pod erasure-coded cluster), its default ingress is disabled (it would otherwise be created for example.rustfs.com), logs go to stdout instead of a second PVC, and the cluster default StorageClass is used.
  • Bucket provisioning. Unlike minio's defaultBuckets, the rustfs chart cannot create buckets, and Zammad never creates one either — Store::Provider::S3 only ever does head_bucket. A new s3-init init container in the zammad-init job creates it through Zammad's own S3 provider, so it cannot drift from S3_URL. No extra image to pin. Controlled by zammadConfig.rustfs.bucketInitialisation (default true), never applied to an external S3 service.
  • README: upgrade section plus a data migration runbook. RustFS reads MinIO's on-disk layout and migrates its metadata on first start, so the attachments move as a plain file copy between the two volumes via a Job that mounts both — no S3 client, no data leaving the cluster, and the old volume is only read from, so it stays intact as a rollback path. Object-level copying is documented as a fallback for when both volumes cannot be attached to the same node.

Upgrade hazard worth reviewing closely

The bitnami minio PVC carries no helm.sh/resource-policy, and the chart's default mode is standalone (a plain PVC, not a StatefulSet template). A naive helm upgrade therefore deletes the attachment volume the moment the subchart leaves the release. Step 2 of the runbook annotates the PVC with keep before upgrading — that step is load-bearing.

Conversely, the rustfs chart does set helm.sh/resource-policy: keep, so the new volume now survives an uninstall. That behaviour change is documented.

Verification

  • Ran a real rustfs/rustfs:1.0.0-rc.5 container with the chart's exact settings and confirmed the full path via an S3 client (path-style, SigV4, region=zammad): head_bucket → 404, create_bucket, put/get/list all succeed.
  • Verified the volume-copy migration end to end: a MinIO volume holding four objects from 512 B to 12 MiB (covering both inline and out-of-line part.N storage) was copied to a pristine rustfs volume, and after adoption every object came back byte-identical with its sha256 matching the object key — which is exactly Zammad's own integrity model. Writes after the adoption work too.
  • Confirmed rustfs.replicaCount=0 renders a replicas: 0 Deployment while still creating the PVC, which is what keeps the new volume pristine during phase one.
  • helm lint and helm template clean on both CI value sets; kubeconform --strict passes (30 resources default, 40 full).
  • The rendered s3-init script passes bash -n and its Ruby body passes ruby -c.
  • Name derivation verified for a release name containing rustfs and for rustfs.fullnameOverride.

Before merge

  • Re-pin zammad/Chart.yaml to the RustFS 1.0.0 GA release
  • Perform a full manual migration test
  • Enable the commented-out rustfs source/target blocks in .github/updatecli.yaml
  • Re-check rustfs/rustfs#7014 — fixed upstream and released in 1.0.0-rc.5. The runbook now relies on this adoption behaviour, so if it ever regresses again the file copy silently produces an unreadable volume. Worth re-verifying on the GA re-pin.

Follow-up (separate PR, after this one is merged)

  • Drop the bitnami entry from .github/ct.yaml chart-repos. It cannot go in this PR: while the target branch still carries 18.x, chart-testing needs that repo to resolve the old chart's minio dependency for the upgrade test. Only once main is on 19.x is it dead weight.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Comment thread zammad/Chart.yaml Outdated
Comment on lines +26 to +30
# Optional S3 storage provider for attachments. Replaces the former bitnami/minio
# subchart: MinIO stopped publishing community Helm charts and images, and the MinIO
# operator was archived in March 2026. RustFS is an S3-compatible drop-in replacement
# with an official chart. Note that it does not create buckets on its own, so the
# zammad-init job provisions the bucket (zammadConfig.rustfs.bucketInitialisation).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is not neded.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 89ef167.

@mgruner
mgruner requested review from fliebe92 and monotek September 2, 2026 06:22
Comment thread zammad/values.yaml
Comment thread zammad/README.md Outdated
Comment thread zammad/README.md Outdated
Comment thread zammad/README.md
Comment thread zammad/templates/_helpers.tpl Outdated
Comment thread zammad/templates/_helpers.tpl Outdated
Comment thread zammad/templates/configmap-init.yaml Outdated
Comment thread zammad/README.md Outdated
Comment thread zammad/templates/_helpers.tpl Outdated
Comment thread zammad/templates/_helpers.tpl Outdated
Comment thread zammad/README.md
Comment thread zammad/README.md Outdated
mgruner and others added 7 commits September 4, 2026 10:50
MinIO stopped publishing community Helm charts and container images, and the
MinIO operator repository was archived in March 2026. The bundled bitnami/minio
subchart was already only usable through the bitnamilegacy image workaround.

Replace it with the official RustFS chart, an S3-compatible object store that
Zammad talks to exactly as it did to MinIO. Chart version 19.0.0, as this is a
breaking change for anyone using the optional S3 storage provider.

- zammadConfig.minio -> zammadConfig.rustfs, subchart values key minio -> rustfs
- Pin the subchart to single-node mode, disable its default ingress (which would
  point at example.rustfs.com), log to stdout instead of a second PVC and use
  the cluster default StorageClass
- Unlike minio's defaultBuckets, the rustfs chart cannot provision buckets, and
  Zammad only ever reads from the bucket. Add an s3-init init container to the
  zammad-init job that creates it via Zammad's own S3 provider, so it cannot
  drift from S3_URL. Controlled by zammadConfig.rustfs.bucketInitialisation.
- Document the upgrade in the README, including a data migration runbook. Note
  that the bitnami minio PVC carries no helm.sh/resource-policy, so the upgrade
  would otherwise delete the attachment volume along with the subchart.

The subchart is pinned to 1.0.0-rc.4 for now and has to be re-pinned to the
1.0.0 GA release before this is merged. The updatecli blocks stay commented out
until then, as a semver filter does not match prereleases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`kill %1` only hits the tunnel if it happens to be the first background job of
that shell. Anyone pasting the steps into a terminal that already has something
running in the background would kill the wrong process. Capture the PID via $!
instead, which also works in non-interactive shells without job control.

Also give the two "Data migration" sections distinct names, so the anchor links
resolve unambiguously.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moving the attachments through the Database or File provider means the
intermediate provider has to hold every attachment, which is not a realistic
option for an installation that already runs on S3. Replace the step-by-step
procedure with a short reference to the existing Store::File.move section.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RustFS advertises reusing the MinIO data directory as-is, which would make the
object-level copy unnecessary. It does not work with the pinned version and is
reported upstream as rustfs/rustfs#7014, so point that out to keep people from
planning a migration around the upstream blog post.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rustfs/rustfs#7014 is fixed and released in rc.5: RustFS reads MinIO's on-disk
layout again and migrates its metadata on first start. Verified against rc.5
with a MinIO volume holding four objects from 512 B to 12 MiB (both inline and
part.N storage): all of them come back byte-identical, their sha256 matching
the object key, and writes after the adoption work as well.

That turns the data migration into a plain file copy between the two volumes,
so drop the rclone procedure:

- Phase one now upgrades with rustfs.replicaCount=0, which creates the new
  volume without ever starting rustfs on it, keeping it pristine for the copy.
- A Job mounts both volumes and copies the directory across. The old volume is
  only read from, so it stays intact as a rollback path.
- The init job is disabled in phase one, as the bucket comes with the copied
  data and is then found by the s3-init container in phase two.

Object-level copying stays documented as a fallback for the case where both
volumes cannot be attached to the same node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Fail on the removed zammadConfig.minio/minio keys. Helm ignores unknown keys,
  so an upgrade with an unchanged values.yaml previously rendered no S3_URL at
  all and deleted the old attachment volume unnoticed, because the runbook step
  that protects it was never run.
- Make zammad.env.S3_URL a single exclusive branch. secrets.s3.useExisting was
  appended after the bundled branch, which ate the preceding newline and made
  `helm template` fail outright on that combination; it also emitted two S3_URL
  entries. useExisting now comes first, preserving which value won before.
- Do not run s3-init when secrets.s3.useExisting supplies a complete S3_URL,
  which points at an external service the chart must not modify.
- Rework the s3-init script: guard create_bucket against BucketAlreadyOwnedByYou
  and BucketAlreadyExists for the window where the server is still adopting a
  data directory, and move the networking rescue to an outer begin, as Ruby does
  not enter the rescue clauses of the same begin block again.
- README: the OpenShift snippet needs explicit nulls, an empty map merges
  nothing and leaves the subchart's UID/GID in place. Name the value key and the
  values file for sizing the new volume, since it cannot be shrunk afterwards.
  Drop lost+found in the copy Job and note that its root requirement conflicts
  with the restricted Pod Security Standard.
- Drop the redundant comment on the rustfs dependency.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Restore the pre-19.0.0 precedence in zammad.env.S3_URL. Making the branches
  exclusive had moved secrets.s3 ahead of externalS3Url, but up to 18.x the
  secrets.s3 block sat inside the externalS3Url else-branch and never rendered
  when that was set. Anyone carrying both keys would have silently switched
  endpoint on upgrade. Order is now externalS3Url, secrets.s3, bundled.
- Only mention protecting the attachment volume in the removed-keys failure when
  zammadConfig.minio.enabled was actually true. Whoever merely carried the old
  default of false still gets told to rename the key, but no longer gets an
  instruction that does not apply to them.
- Re-append du -sh to the copy Job, which was lost with the lost+found fix and
  left the kubectl logs step in the runbook printing nothing.
- fsGroup is a GID, and on OpenShift it is assigned per namespace, so advise
  leaving it unset there rather than setting it to the UID rustfs runs as.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mgruner
mgruner force-pushed the replace-minio-rustfs branch from da4a558 to c90fe80 Compare September 4, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate away from bitnami/minio

2 participants