feat(update): signed offline packages, a signed release channel, and a support-access bridge - #8
Merged
Conversation
A rented appliance needs the customer to control when we can reach in, and the control has to live where they already are: the web UI. That means the app must read a state file the host writes, and write trigger files the host consumes — exactly the contract appliance-update already established. So this adds a second hostPath alongside the update one rather than inventing anything: no new port, no agent in the pod, same .path-unit idiom on the host side. The feature gate is the PRESENCE of a state.json with enrolled:true, not an env var, which is why this can ship in the public chart unconditionally — a customer-run appliance mounts an empty directory and sees nothing at all. The chown matters and is easy to get wrong: kubelet's DirectoryOrCreate creates a missing hostPath as root:root 0755, and the app runs as uid/gid 1001, so the pod could never write its triggers. Both directories are therefore created root:1001 0770 BEFORE helm runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two problems, one Ed25519 key.
1. Some customers cut outbound access entirely, so an appliance needs a
second update source: a USB stick. tools/build-offline-package.sh emits a
self-sufficient bundle (chart, container images, scripts) from channel.json,
and update.sh grows pkg_verify/pkg_stage plus a `scan-usb` verb.
The verification is all-or-nothing: one signature over SHA256SUMS, which
covers every file in the bundle including the image tars. A coverage check
rejects any file absent from that list — otherwise unsigned files could
simply be added alongside. Downgrades are refused, and min_from_version is
a floor for packages that cannot be applied from an older release.
Crucially, online and USB COEXIST. state.json moves to schema 2 carrying
sources.{online,usb} with update_available resolved over the better of the
two, and fetch_manifest_online no longer dies. A network failure must never
mask a valid USB package — the customers who cut the network are exactly
the ones the USB path exists for, so that regression would defeat the
feature silently.
2. channel.json was signed nowhere. TLS proves we reached the right HOST; it
says nothing about who wrote the file. Whoever controls MANIFEST_URL could
tell every appliance "move to chart X, vLLM image Y". Separately,
self_update() pulled update.sh — which then runs as root — over plain
HTTPS.
Signing the manifest closes both: channel.json gains an updater_sha256
field, and the signature over the manifest covers it, so verifying the
manifest transitively verifies the updater. One key, one primitive
(openssl pkeyutl -verify -rawin), no new dependency on the box.
The behaviour is graduated so the public one-command install is untouched:
with no package-release.pub present, trust stays TLS-only exactly as
before. With the key present — which is always true on a fleet box — an
unsigned or altered manifest is REFUSED, and an update.sh whose hash does
not match the signed updater_sha256 is refused.
Both strict paths fail CLOSED, and that has a consequence worth stating:
forgetting to re-sign does not break the fleet, it STOPS it. Every box quietly
keeps its current version and nobody notices for weeks. Hence
.github/workflows/channel-signature.yml, which on every change verifies that
the signature validates and that updater_sha256 matches the update.sh in the
tree. It needs only the public half, so that lives in a repo variable rather
than a secret.
tools/sign-channel.sh re-verifies its own output the way an appliance will,
rather than trusting that signing succeeded.
Verified by tools/test-package-verify.sh: 23 assertions covering a good
package, a corrupted image tar, a foreign signature, a missing signature, a
downgrade, an unmet min_from_version, a manifest tampered after signing, and
the graduated no-key case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/gen-package-key.sh takes an output basename, so running it from the checkout drops <basename>.key straight into the working tree — which is exactly what happened here. This repo is public and that private half is the root of trust for every appliance: one `git add -A` would publish the ability to sign an update for the whole fleet. Ignore both halves. The private key belongs in the vault and in CI as a secret; the public half is a repo VARIABLE and a file deployed to appliances, never a tracked file here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fills updater_sha256 with the hash of this tree's update.sh and adds the detached Ed25519 signature over channel.json, produced by tools/sign-channel.sh. Without these two, an appliance holding package-release.pub refuses the channel outright — which is the correct fail-closed behaviour and is why the channel-signature CI job was failing on its own pull request. Verified locally the way an appliance will: the signature validates against the public half, and the pinned updater_sha256 matches update.sh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tonyctalope
marked this pull request as ready for review
July 30, 2026 14:24
Found by running this updater on real aarch64 hardware: `check` exited 1 with NO output at all. Under `set -euo pipefail` an assignment takes its pipeline's status, so `cur_chart="$(helm list … | sed | head -1)"` killed the script the moment helm could not reach the cluster. The intent was already there and already dead: the warn immediately below, the comment saying "fall back to the pin only when the cluster is unreadable", and the values.yaml fallback further down were all unreachable code. Pre-existing, but it lands squarely on this branch's reason to exist: the box this offline path serves is a DEGRADED one. k3s down, a verified USB package sitting in the slot, and `check` dying before it can report the package — no banner, no diagnostic, and a silent exit 1 in the daily timer that nobody notices. Fixing it is what makes the USB source reachable in the situation it was built for. Verified on the Spark (aarch64, OpenSSL 3.0.13) against the real published channel: the warn now prints, the manifest is fetched, and with the public key installed the signature verifies on that hardware. The only remaining non-zero exit there is chown failing as an unprivileged user, which is the test setup rather than the code. NOTE: this changes update.sh, so channel.json's updater_sha256 is now stale and the channel MUST be re-signed before merge. That is the fail-closed guard doing its job — on my own fix, which is the point of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
updater_sha256 now pins f844c721 (was afb6eaff). Verified the way an appliance will before committing: the signature validates against the public half the fleet holds, and the pinned hash matches this tree's update.sh. Nothing else in channel.json moved — chart_version, app_version and vllm_image are untouched, so no deployed appliance will apply anything as a result of this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes the rented-appliance product needs, all of which belong here rather than in the private fleet layer because they are equally useful to a customer-run install.
1. A shared host↔pod directory for a support toggle
A second hostPath alongside the update one. No new port, no agent in the pod, same
.path-unit idiom — it is theappliance-updatecontract reused verbatim.The feature gate is the presence of a
state.jsonwithenrolled: true, not an env var. That is what lets this ship in the public chart unconditionally: a customer-run appliance mounts an empty directory and sees nothing. Onevalues.yaml, no fork.root:10010770, created before helm — kubelet'sDirectoryOrCreatewould otherwise create itroot:root 0755and the pod (uid 1001) could never write its triggers.2. Signed offline update packages
Some customers cut outbound access entirely, so
update.shgains a second source.tools/build-offline-package.shemits a self-sufficient bundle (chart, container images, scripts) fromchannel.json; the appliance verifies and stages it from a USB stick.Verification is all-or-nothing: one signature over
SHA256SUMS, which covers every file including the image tars. A coverage check rejects any file absent from that list — otherwise unsigned files could just be added alongside. Downgrades refused,min_from_versionenforced as a floor.Online and USB coexist.
state.jsonmoves to schema 2 withsources.{online,usb}andupdate_availableresolved over the better of the two;fetch_manifest_onlineno longer dies. A network failure must never mask a valid USB package — the customers who cut the network are precisely the ones this path exists for, so that regression would defeat the feature silently.3. A signed release channel
channel.jsonwas signed nowhere. TLS proves we reached the right host; it says nothing about who wrote the file. Whoever controlsMANIFEST_URLcould tell every appliance "move to chart X, vLLM image Y". Separately,self_update()pulledupdate.sh— which then runs as root — over plain HTTPS.Signing the manifest closes both with one key:
channel.jsongainsupdater_sha256, and the signature over the manifest covers it, so verifying the manifest transitively verifies the updater. Same primitive already in use (openssl pkeyutl -verify -rawin), no new dependency on the box.Graduated so the public one-command install is untouched:
package-release.pubon the boxupdate.shrefreshupdater_sha256, else refusedBoth strict paths fail closed, which has a consequence worth stating plainly: forgetting to re-sign does not break the fleet, it stops it — every box quietly keeps its current version and nobody notices for weeks.
.github/workflows/channel-signature.ymlexists to catch exactly that, and it will fail on this PR, becauseupdater_sha256is still empty andchannel.json.sigdoes not exist. That is the workflow doing its job. To unblock, with the private key:PACKAGE_PRIVATE_KEY=/path/to/package-release.key tools/sign-channel.sh git add channel.json channel.json.sig && git commitAlso needed before merge:
PACKAGE_PUBLIC_KEY_PEMas a repo variable (public half only, so not a secret) so the signature job actually verifies rather than warning.Verification
tools/test-package-verify.sh— 23 assertions: a good package, a corrupted image tar, a foreign signature, a missing signature, a downgrade, an unmetmin_from_version, a manifest tampered after signing, and the graduated no-key case. All scripts parse, including the ones generated inside heredocs (abash -nof the parent does not cover those).Not exercised yet: an end-to-end apply from a real USB stick on real hardware, and the reboot-with-nothing-attached gate. Both are physical.
🤖 Generated with Claude Code