Skip to content

feat(update): signed offline packages, a signed release channel, and a support-access bridge - #8

Merged
tonyctalope merged 6 commits into
mainfrom
feat/fleet-support-bridge-offline-package
Jul 30, 2026
Merged

feat(update): signed offline packages, a signed release channel, and a support-access bridge#8
tonyctalope merged 6 commits into
mainfrom
feat/fleet-support-bridge-offline-package

Conversation

@tonyctalope

Copy link
Copy Markdown
Contributor

Three changes the rented-appliance product needs, all of which belong here rather than in the private fleet layer because they are equally useful to a customer-run install.

1. A shared host↔pod directory for a support toggle

A second hostPath alongside the update one. No new port, no agent in the pod, same .path-unit idiom — it is the appliance-update contract reused verbatim.

The feature gate is the presence of a state.json with enrolled: true, not an env var. That is what lets this ship in the public chart unconditionally: a customer-run appliance mounts an empty directory and sees nothing. One values.yaml, no fork.

root:1001 0770, created before helm — kubelet's DirectoryOrCreate would otherwise create it root:root 0755 and the pod (uid 1001) could never write its triggers.

2. Signed offline update packages

Some customers cut outbound access entirely, so update.sh gains a second source. tools/build-offline-package.sh emits a self-sufficient bundle (chart, container images, scripts) from channel.json; the appliance verifies and stages it from a USB stick.

Verification is all-or-nothing: one signature over SHA256SUMS, which covers every file including the image tars. A coverage check rejects any file absent from that list — otherwise unsigned files could just be added alongside. Downgrades refused, min_from_version enforced as a floor.

Online and USB coexist. state.json moves to schema 2 with sources.{online,usb} and update_available resolved over the better of the two; fetch_manifest_online no longer dies. A network failure must never mask a valid USB package — the customers who cut the network are precisely the ones this path exists for, so that regression would defeat the feature silently.

3. A signed release channel

channel.json was signed nowhere. TLS proves we reached the right host; it says nothing about who wrote the file. Whoever controls MANIFEST_URL could tell every appliance "move to chart X, vLLM image Y". Separately, self_update() pulled update.sh — which then runs as root — over plain HTTPS.

Signing the manifest closes both with one key: channel.json gains updater_sha256, and the signature over the manifest covers it, so verifying the manifest transitively verifies the updater. Same primitive already in use (openssl pkeyutl -verify -rawin), no new dependency on the box.

Graduated so the public one-command install is untouched:

package-release.pub on the box Manifest update.sh refresh
absent (public default) TLS-only, unchanged TLS-only, unchanged
present must be signed, else refused must match the signed updater_sha256, else refused

⚠️ This PR cannot go green until the channel is signed

Both strict paths fail closed, which has a consequence worth stating plainly: forgetting to re-sign does not break the fleet, it stops it — every box quietly keeps its current version and nobody notices for weeks.

.github/workflows/channel-signature.yml exists to catch exactly that, and it will fail on this PR, because updater_sha256 is still empty and channel.json.sig does not exist. That is the workflow doing its job. To unblock, with the private key:

PACKAGE_PRIVATE_KEY=/path/to/package-release.key tools/sign-channel.sh
git add channel.json channel.json.sig && git commit

Also needed before merge: PACKAGE_PUBLIC_KEY_PEM as a repo variable (public half only, so not a secret) so the signature job actually verifies rather than warning.

Verification

tools/test-package-verify.sh23 assertions: a good package, a corrupted image tar, a foreign signature, a missing signature, a downgrade, an unmet min_from_version, a manifest tampered after signing, and the graduated no-key case. All scripts parse, including the ones generated inside heredocs (a bash -n of the parent does not cover those).

Not exercised yet: an end-to-end apply from a real USB stick on real hardware, and the reboot-with-nothing-attached gate. Both are physical.

🤖 Generated with Claude Code

tonyctalope and others added 4 commits July 30, 2026 11:49
A rented appliance needs the customer to control when we can reach in, and
the control has to live where they already are: the web UI. That means the
app must read a state file the host writes, and write trigger files the host
consumes — exactly the contract appliance-update already established.

So this adds a second hostPath alongside the update one rather than inventing
anything: no new port, no agent in the pod, same .path-unit idiom on the host
side. The feature gate is the PRESENCE of a state.json with enrolled:true, not
an env var, which is why this can ship in the public chart unconditionally —
a customer-run appliance mounts an empty directory and sees nothing at all.

The chown matters and is easy to get wrong: kubelet's DirectoryOrCreate
creates a missing hostPath as root:root 0755, and the app runs as uid/gid
1001, so the pod could never write its triggers. Both directories are
therefore created root:1001 0770 BEFORE helm runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two problems, one Ed25519 key.

1. Some customers cut outbound access entirely, so an appliance needs a
   second update source: a USB stick. tools/build-offline-package.sh emits a
   self-sufficient bundle (chart, container images, scripts) from channel.json,
   and update.sh grows pkg_verify/pkg_stage plus a `scan-usb` verb.

   The verification is all-or-nothing: one signature over SHA256SUMS, which
   covers every file in the bundle including the image tars. A coverage check
   rejects any file absent from that list — otherwise unsigned files could
   simply be added alongside. Downgrades are refused, and min_from_version is
   a floor for packages that cannot be applied from an older release.

   Crucially, online and USB COEXIST. state.json moves to schema 2 carrying
   sources.{online,usb} with update_available resolved over the better of the
   two, and fetch_manifest_online no longer dies. A network failure must never
   mask a valid USB package — the customers who cut the network are exactly
   the ones the USB path exists for, so that regression would defeat the
   feature silently.

2. channel.json was signed nowhere. TLS proves we reached the right HOST; it
   says nothing about who wrote the file. Whoever controls MANIFEST_URL could
   tell every appliance "move to chart X, vLLM image Y". Separately,
   self_update() pulled update.sh — which then runs as root — over plain
   HTTPS.

   Signing the manifest closes both: channel.json gains an updater_sha256
   field, and the signature over the manifest covers it, so verifying the
   manifest transitively verifies the updater. One key, one primitive
   (openssl pkeyutl -verify -rawin), no new dependency on the box.

   The behaviour is graduated so the public one-command install is untouched:
   with no package-release.pub present, trust stays TLS-only exactly as
   before. With the key present — which is always true on a fleet box — an
   unsigned or altered manifest is REFUSED, and an update.sh whose hash does
   not match the signed updater_sha256 is refused.

Both strict paths fail CLOSED, and that has a consequence worth stating:
forgetting to re-sign does not break the fleet, it STOPS it. Every box quietly
keeps its current version and nobody notices for weeks. Hence
.github/workflows/channel-signature.yml, which on every change verifies that
the signature validates and that updater_sha256 matches the update.sh in the
tree. It needs only the public half, so that lives in a repo variable rather
than a secret.

tools/sign-channel.sh re-verifies its own output the way an appliance will,
rather than trusting that signing succeeded.

Verified by tools/test-package-verify.sh: 23 assertions covering a good
package, a corrupted image tar, a foreign signature, a missing signature, a
downgrade, an unmet min_from_version, a manifest tampered after signing, and
the graduated no-key case.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/gen-package-key.sh takes an output basename, so running it from the
checkout drops <basename>.key straight into the working tree — which is
exactly what happened here. This repo is public and that private half is the
root of trust for every appliance: one `git add -A` would publish the ability
to sign an update for the whole fleet.

Ignore both halves. The private key belongs in the vault and in CI as a
secret; the public half is a repo VARIABLE and a file deployed to appliances,
never a tracked file here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fills updater_sha256 with the hash of this tree's update.sh and adds the
detached Ed25519 signature over channel.json, produced by
tools/sign-channel.sh.

Without these two, an appliance holding package-release.pub refuses the
channel outright — which is the correct fail-closed behaviour and is why the
channel-signature CI job was failing on its own pull request.

Verified locally the way an appliance will: the signature validates against
the public half, and the pinned updater_sha256 matches update.sh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tonyctalope
tonyctalope marked this pull request as ready for review July 30, 2026 14:24
tonyctalope and others added 2 commits July 30, 2026 17:35
Found by running this updater on real aarch64 hardware: `check` exited 1 with
NO output at all. Under `set -euo pipefail` an assignment takes its pipeline's
status, so `cur_chart="$(helm list … | sed | head -1)"` killed the script the
moment helm could not reach the cluster.

The intent was already there and already dead: the warn immediately below, the
comment saying "fall back to the pin only when the cluster is unreadable", and
the values.yaml fallback further down were all unreachable code.

Pre-existing, but it lands squarely on this branch's reason to exist: the box
this offline path serves is a DEGRADED one. k3s down, a verified USB package
sitting in the slot, and `check` dying before it can report the package — no
banner, no diagnostic, and a silent exit 1 in the daily timer that nobody
notices. Fixing it is what makes the USB source reachable in the situation it
was built for.

Verified on the Spark (aarch64, OpenSSL 3.0.13) against the real published
channel: the warn now prints, the manifest is fetched, and with the public key
installed the signature verifies on that hardware. The only remaining non-zero
exit there is chown failing as an unprivileged user, which is the test setup
rather than the code.

NOTE: this changes update.sh, so channel.json's updater_sha256 is now stale and
the channel MUST be re-signed before merge. That is the fail-closed guard doing
its job — on my own fix, which is the point of it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
updater_sha256 now pins f844c721 (was afb6eaff). Verified the way an appliance
will before committing: the signature validates against the public half the
fleet holds, and the pinned hash matches this tree's update.sh.

Nothing else in channel.json moved — chart_version, app_version and vllm_image
are untouched, so no deployed appliance will apply anything as a result of this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tonyctalope
tonyctalope merged commit 15e108b into main Jul 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant