diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a931304..3f1fe2b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,7 +63,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - name: shellcheck install.sh + - name: shellcheck install.sh and contrib/pullgate # A fixed release of ShellCheck, checked by its SHA-256. Dependabot # does not update it. Update it by hand: the version and the SHA-256 # of shellcheck-VERSION.linux.x86_64.tar.xz together. @@ -80,6 +80,9 @@ jobs: tar -xJf "$dir/shellcheck.tar.xz" -C "$dir" "$dir/shellcheck-${SHELLCHECK_VERSION}/shellcheck" --version "$dir/shellcheck-${SHELLCHECK_VERSION}/shellcheck" --shell=sh install.sh + # The scripts of the pullgate recipe source lib.sh from their own + # directory. + (cd contrib/pullgate && "$dir/shellcheck-${SHELLCHECK_VERSION}/shellcheck" --shell=sh -x -- *.sh) - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: go.mod diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 46fb651..b5510e3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -97,6 +97,16 @@ jobs: tar --owner=0 --group=0 --numeric-owner --mode=go-w --mtime="$mtime" \ -C "$stage" -cf - nodkey LICENSE README.md | gzip -9n > "dist/${name}.tar.gz" done + # The pullgate recipe (contrib/pullgate, docs/setup.md). A host + # installs it from here, checked by SHA256SUMS and the + # attestation as the binaries are, never from a config repo. + name="nodkey_${VERSION}_pullgate" + stage="${RUNNER_TEMP}/${name}" + mkdir -p "$stage" + cp contrib/pullgate/* LICENSE "$stage/" + # LC_ALL=C: the same order of the names on every runner. + (cd "$stage" && export LC_ALL=C && tar --owner=0 --group=0 --numeric-owner --mode=go-w --mtime="$mtime" \ + -cf - -- *) | gzip -9n > "dist/${name}.tar.gz" cd dist sha256sum -- *.tar.gz > SHA256SUMS cat SHA256SUMS @@ -147,7 +157,7 @@ jobs: subject-path: dist/*.tar.gz # The attestation is also a file of the release. With this file, # gh attestation verify --bundle needs no GitHub login. One bundle - # covers both tarballs. + # covers all tarballs. - name: Save the attestation bundle env: BUNDLE: ${{ steps.attest.outputs.bundle-path }} diff --git a/AGENTS.md b/AGENTS.md index 1012e12..44aee31 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,6 +61,11 @@ Planned. Each v1 issue adds its part. `nodkey preapprove` (sections 8 and 15), and `nodkey agent-browser-plugin` (section 21.10), with recorded requests of agent-browser in `testdata/`. +- `contrib/pullgate/` - the pullgate recipe: the broker host pulls its + config from git, and the owner taps each change (docs/setup.md, + "Change the config from git with pullgate"; THREAT-MODEL.md T52). Its + tests are in `contrib_pullgate_linux_test.go`, at the root of the + module. The release packs it as `nodkey_VERSION_pullgate.tar.gz`. - `contrib/systemd/` - the example systemd unit of the broker, with encrypted credentials and a sandbox (DESIGN.md section 16). - `docs/` - the guides for people: diff --git a/DESIGN.md b/DESIGN.md index e36c2c1..686726e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -4585,10 +4585,42 @@ Infisical SDK in the broker (section 4, `internal/store/infisical.go`): (section 15). See docs/releasing.md. - release-please keeps a release pull request open. Merging it tags the version and creates the GitHub Release. A job then builds static Linux - binaries (`CGO_ENABLED=0`, amd64 and arm64, `-trimpath`), a - `SHA256SUMS` file, and a build provenance attestation. The attestation - is also a file of the release (`nodkey_.sigstore.json`), so it - can be checked with no GitHub login. + binaries (`CGO_ENABLED=0`, amd64 and arm64, `-trimpath`), the tarball + of the pullgate recipe (`nodkey__pullgate.tar.gz`: the files + of `contrib/pullgate/` and `LICENSE`), a `SHA256SUMS` file of all + tarballs, and a build provenance attestation of all of them. The + attestation is also a file of the release + (`nodkey_.sigstore.json`), so it can be checked with no GitHub + login. +- The pullgate recipe (`contrib/pullgate/`, issue #102) lets the broker + host pull `broker.toml` and `known_hosts` from a git repo with + pullgate, which asks the owner on its own bot and applies a commit only + after a tap (THREAT-MODEL.md T52). It is four scripts and a shared + `lib.sh`. `check.sh` (check user): both files are plain files, + `nodkey broker check --files` passes, and the commit changes no part + of the gate (`nkp_host_parts` in `lib.sh`: `listen`, `tls_key`, the + paths of `known_hosts` and `audit_log`, `[telegram]`, `[ntfy]`, a new + or changed `[store.*]`); `apply.sh` checks this again as root, also in + a rollback. `summary.sh` (check user): + `nodkey broker config-diff --short --max-chars 1200` from the applied + commit, or from the live config on the first apply, with + `⚠ needs a broker restart: all grants end` first when needed, and a + `⚠` line when the live config is not the applied commit. `apply.sh` + (root): checks the files again with the credential names, refuses to + run while a copy of the old files from an earlier failed restore is + left (`/etc/nodkey/pullgate-old`), installs + them with the owner and mode of the live files, then reloads or + restarts; on a failure it puts the old files back, reloads or restarts + again, and fails. `health.sh` (root): waits until `reload-status.json` + names the unit's main process, a result of `started` or `loaded`, and + the SHA-256 of the live `broker.toml`. The one rule for "reload or + restart" is in `lib.sh`: a reload only when every line of the full + `config-diff` is about a client, secret, project, SSH key (not a new + key or file: credentials load only at the start) or a host key of + `known_hosts`; anything else, or a failed compare, is a restart. Its + tests (`contrib_pullgate_linux_test.go`) run the scripts with `/bin/sh` + against a fake `systemctl` and a fake `nodkey broker reload`, with the + real `nodkey` for the check and the diff. CI runs shellcheck on them. - `install.sh`, at the root of the repository, installs a release on the host itself, as root: the broker host or an agent machine. It needs `sh`, `curl`, `tar`, `gzip`, `sha256sum`, `install`, `mktemp`, `stat` diff --git a/THREAT-MODEL.md b/THREAT-MODEL.md index cb18fb4..baf2bdf 100644 --- a/THREAT-MODEL.md +++ b/THREAT-MODEL.md @@ -1755,6 +1755,78 @@ What nodkey does not do: broker's memory than for one `bws secret get`. Risk that remains: low, for a broker host that is safe (section 7). +**T52. A change of the config from git, with pullgate, gives access that +the owner did not mean.** +With the recipe in `contrib/pullgate/` (docs/setup.md, "Change the +config from git with pullgate"), the broker host pulls `broker.toml` and +`known_hosts` from a private git repo. pullgate applies a commit only +after the owner taps Apply on its own Telegram bot. The new parties: who +can write the repo, the read-only deploy key, the git host, pullgate's +bot token, and the scripts that run as root. +What nodkey does: +- **The gate is the tap on the host, not CI.** A writer of the repo, + or of its CI, can push any commit, also one that widens access. CI + can check a config, but it runs where the writer has power, so it is + only a help. The broker host applies nothing without the owner's tap + on the exact commit that pullgate shows, and pullgate refuses an older + commit or a rewritten branch. +- The question shows `nodkey broker config-diff` (T42 and DESIGN.md + section 15): each change that widens access starts with `⚠`, and + these lines come first in the summary. A line about a secret names its store item by + the last 8 characters of its ID, so a commit cannot point a trusted + name at another item unseen. A change that needs a restart says so + first, because it ends every grant. A live config that differs from + the applied commit (a change by hand) is a `⚠` line too. The guide + says: tap Skip on a `⚠` line that you did not ask for. +- **A commit cannot change the gate.** The approval channel + (`[telegram]`, `[ntfy]`), `listen`, `tls_key`, the paths of + `known_hosts` and `audit_log`, and a new or changed `[store.*]` are + refused by `check.sh` and again by `apply.sh` as root, also in a + rollback. Else a commit could point the channel at its own server, or + a token file at another credential that the broker holds, and approve + its own requests. These changes stay a step on the host, as root. +- **The scripts are installed on the host, never run from the repo + tree.** They come from the release asset + `nodkey_VERSION_pullgate.tar.gz`, which is in `SHA256SUMS` and + covered by the build attestation of the release, as the binaries are. + The guide installs them only after both checks, owned by root. A + commit cannot change the scripts, the instance config or the units: + they are not in the repo. +- `check.sh` and `summary.sh` run as an unprivileged check user and read + the commit only as data, with nodkey's own parser. Both files must be + plain files, not symlinks, and pullgate refuses a symlink out of the + tree and a large tree (`refuse_symlinks`, `max_files`, + `max_tree_bytes`). A commit may not change the paths of `known_hosts` + and `audit_log`, a store's program or files, or the channel's token + file (see below). It can still name another SSH key file: that is a + `⚠` line and a restart, and the gate is the owner's tap. +- `apply.sh` checks the files again as root with `nodkey broker check`, + with the names of the credentials, before it installs them. The broker + checks the config a third time when it loads it (fail closed). On a + failed reload or restart, the old files go back, and pullgate rolls + back to the last good commit; if that fails too, it pauses and says + so. +- The repo holds no secret value: the config names secrets only by + credential paths and item IDs. A new credential is still a step on the + host, as root. +What nodkey does not do: +- It cannot protect the owner who taps Apply without reading the `⚠` + lines. A careless tap applies what a writer of the repo wrote. +- pullgate puts the summary after the commits and the changed files, and + cuts a long message from the end. A writer of the repo can push many + commits with long subjects, so that the summary, with its `⚠` lines, + is cut off. The recipe keeps the summary short (1200 characters) and + the repo small (`max_files = 20`), and the guide says: tap Skip when + the question has no complete summary, and read the full list on the + host. +- Who can read the repo learns the names of the clients, secrets, keys, + hosts and item IDs, and the approver's Telegram user ID. Keep the repo + private, and give the deploy key read access only. +- pullgate's bot token, the deploy key and pullgate itself are outside + nodkey: pullgate's own THREAT-MODEL.md covers them. +- The git host is one more outbound connection of the broker host. +Risk that remains: low, when the owner reads every `⚠` line. + ## 7. Assumptions - The broker host is patched, has no agent on it, and only the owner has diff --git a/contrib/pullgate/README.md b/contrib/pullgate/README.md new file mode 100644 index 0000000..2a67ec5 --- /dev/null +++ b/contrib/pullgate/README.md @@ -0,0 +1,21 @@ +# The pullgate recipe of nodkey + +These files let the broker host pull its config from a private git repo +with [pullgate](https://github.com/code6x/pullgate). The owner taps +Apply on Telegram for each commit. A config that fails goes back to the +last good one by itself. + +- `nodkey-broker.toml`: the pullgate instance config. +- `check.sh`: checks a new commit, as the check user. +- `summary.sh`: the change list in the question, as the check user. +- `apply.sh`: installs the two files and reloads or restarts the broker, + as root. It puts the old files back on a failure. +- `health.sh`: waits for the broker to say that it runs the new config, + as root. +- `lib.sh`: the part that the scripts share, such as the one rule for + "reload or restart". + +Install them from a release of nodkey (`nodkey_VERSION_pullgate.tar.gz`), +checked as docs/setup.md says, and never from a config repo. The steps +are in [docs/setup.md](../../docs/setup.md), "Change the config from git +with pullgate". diff --git a/contrib/pullgate/apply.sh b/contrib/pullgate/apply.sh new file mode 100755 index 0000000..307b37e --- /dev/null +++ b/contrib/pullgate/apply.sh @@ -0,0 +1,105 @@ +#!/bin/sh +# The apply of the pullgate recipe of nodkey. pullgate runs it as root +# after the owner tapped Apply, and again with the last good commit in a +# rollback. It is safe to run again: it installs two files and reloads or +# restarts the broker. +# +# On a failure it puts the old files back, reloads or restarts the broker +# again, and exits with 1. Then pullgate rolls back, or pauses. +set -u +umask 077 +. "${0%/*}/lib.sh" + +new=${PULLGATE_TREE:?PULLGATE_TREE is not set; pullgate sets it} +stage=$NKP_ETC/pullgate-new +keep=$NKP_ETC/pullgate-old + +# moved is set once the new files are in $NKP_ETC. Before that, a failure +# removes the stage and the copy: the live files did not change. +moved= +fail() { + echo "nodkey apply: $*" + [ -n "$moved" ] || rm -rf "$stage" "$keep" + exit 1 +} + +nkp_files_ok "$new" || fail "fix the config repo" +nkp_files_ok "$NKP_ETC" || fail "set up $NKP_ETC by hand first (docs/setup.md section 7)" +# known_hosts goes to $NKP_ETC/known_hosts. check.sh refuses a commit that +# moves it, so the live config must name that path. +kh=$(nkp_top_key "$NKP_ETC/broker.toml" known_hosts) || + fail "cannot read known_hosts = \"...\" at the top of $NKP_ETC/broker.toml; write it as one plain line" +[ "$kh" = "$NKP_ETC/known_hosts" ] || + fail "the live config has known_hosts = \"$kh\"; this recipe needs \"$NKP_ETC/known_hosts\"" + +# Stage the new files with the owner and mode of the live ones, and keep +# a copy of the live ones. A stage or a copy from an earlier run that was +# cut off is removed first. +# $keep is left only when an earlier apply could not put the old files +# back. It may hold the only copy of a good config: never remove it here. +[ ! -e "$keep" ] || { + moved=1 + fail "$keep is left from an earlier apply; compare it with $NKP_ETC, put back what is right, remove it, then run pullgate status \$PULLGATE_INSTANCE --resume" +} +rm -rf "$stage" || fail "cannot remove $stage; remove it by hand, as root" +mkdir -m 0700 "$stage" "$keep" || fail "cannot make $stage and $keep" +for f in $NKP_FILES; do + cur=$NKP_ETC/$f + install -m "$(stat -c %a "$cur")" -o "$(stat -c %u "$cur")" -g "$(stat -c %g "$cur")" "$new/$f" "$stage/$f" || + fail "cannot install $stage/$f; read the error above, and check the space and the modes in $NKP_ETC" + cp -p "$cur" "$keep/$f" || fail "cannot copy $cur to $keep; read the error above, and check the space in $NKP_ETC" +done + +# The checks of a start again, as root, with the names of the credentials. +if [ -d "$NKP_CREDENTIALS" ]; then + nodkey broker check --files "$stage" --credentials "$NKP_CREDENTIALS" || fail "nodkey broker check refused the new config" +else + nodkey broker check --files "$stage" || fail "nodkey broker check refused the new config" +fi + +# The same rule as check.sh, again as root: also a rollback, which runs no +# check, must not change the gate. +nkp_host_parts "$NKP_ETC" "$stage" || + fail "refused: the commit changes a part that only root on the host may change (docs/setup.md, \"What stays a step on the host\")" + +action=$(nkp_action "$NKP_ETC" "$stage") +# A broker that does not run cannot reload. So apply.sh also starts a +# broker that was stopped: pause the instance first to keep it stopped. +systemctl is-active --quiet "$NKP_UNIT" || action=restart +echo "nodkey apply: the change needs a $action" + +moved=1 +for f in $NKP_FILES; do + mv -f "$stage/$f" "$NKP_ETC/$f" || fail "cannot move $stage/$f to $NKP_ETC; the old files are in $keep" +done +rmdir "$stage" + +if nkp_run "$action"; then + rm -rf "$keep" + echo "nodkey apply: done, with a $action" + exit 0 +fi + +if [ "${PULLGATE_ROLLBACK:-}" = 1 ]; then + # The files that are live now are of the commit that failed: keep the + # last good ones in place, and try a restart. + rm -rf "$keep" + if [ "$action" = reload ] && nkp_run restart; then + echo "nodkey apply: the rollback reload failed, the restart worked" + exit 0 + fi + echo "nodkey apply: the rollback failed; the last good files are in $NKP_ETC; read journalctl -u $NKP_UNIT" + exit 1 +fi + +echo "nodkey apply: the $action failed; the old files go back" +for f in $NKP_FILES; do + mv -f "$keep/$f" "$NKP_ETC/$f" || fail "cannot put $keep/$f back; put it in $NKP_ETC by hand, as root" +done +rmdir "$keep" +if nkp_run "$action"; then + echo "nodkey apply: the old config runs again" +else + echo "nodkey apply: the $action with the old config failed too; read journalctl -u $NKP_UNIT" +fi +exit 1 diff --git a/contrib/pullgate/check.sh b/contrib/pullgate/check.sh new file mode 100755 index 0000000..4f20f85 --- /dev/null +++ b/contrib/pullgate/check.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# The check of the pullgate recipe of nodkey. pullgate runs it as the +# check user, on the new commit, before it asks the owner. It reads the +# tree only as data and runs nothing from it. A failure means no question: +# the owner gets the end of this output instead. +set -u +. "${0%/*}/lib.sh" + +new=${PULLGATE_TREE:?PULLGATE_TREE is not set; pullgate sets it} + +if ! nkp_files_ok "$new"; then + echo "Fix: put broker.toml and known_hosts at the root of the config repo, as plain files." + exit 1 +fi + +# The checks of a start, without the files of the host. +if ! nodkey broker check --files "$new"; then + echo "Fix the config in the repo; run nodkey broker check --files DIR on a copy of it first." + exit 1 +fi + +kh=$(nkp_top_key "$NKP_ETC/broker.toml" known_hosts) +if [ "$kh" != "$NKP_ETC/known_hosts" ]; then + echo "The live config in $NKP_ETC/broker.toml needs known_hosts = \"$NKP_ETC/known_hosts\" on one plain line for this recipe." + echo "Fix: change it on the host, as root (docs/setup.md, \"The config repo\")." + exit 1 +fi + +# A commit may not change the parts of the gate itself (lib.sh, +# nkp_host_parts), such as the approval channel or a store, or the paths +# of known_hosts and the audit log: apply.sh installs known_hosts in +# $NKP_ETC, and health.sh reads the status file next to the audit log. +if ! nkp_host_parts "$NKP_ETC" "$new"; then + echo "Refused: a commit may not change listen, tls_key, the paths of known_hosts or audit_log," + echo "[telegram], [ntfy] or a [store.*] (or the compare with $NKP_ETC failed)." + echo "Fix: make such a change on the host, as root (docs/setup.md, \"What stays a step on the host\")," + echo "then commit the same files. The check user must be able to read $NKP_ETC/broker.toml and known_hosts." + exit 1 +fi +echo "The config passed nodkey broker check." diff --git a/contrib/pullgate/health.sh b/contrib/pullgate/health.sh new file mode 100755 index 0000000..500d0b0 --- /dev/null +++ b/contrib/pullgate/health.sh @@ -0,0 +1,64 @@ +#!/bin/sh +# The health check of the pullgate recipe of nodkey. pullgate runs it as +# root after apply.sh. It waits until the broker itself says that it runs +# the config in $NKP_ETC/broker.toml, or until the time is up. A failure +# starts the rollback. +# +# The answer is the broker's status file, reload-status.json next to the +# audit log (DESIGN.md section 14, "Reload"). The broker writes it when it +# is ready and listens, and after each reload. It must name the process +# that systemd runs now, a result of "started" or "loaded", and the +# SHA-256 of the config file. +set -u +. "${0%/*}/lib.sh" + +audit=$(nkp_top_key "$NKP_ETC/broker.toml" audit_log) || { + echo "nodkey health: cannot read audit_log = \"...\" at the top of $NKP_ETC/broker.toml; write it as one plain line" + exit 1 +} +status=${audit%/*}/reload-status.json + +# once checks the broker one time. It prints what it found. +once() { + if ! systemctl is-active --quiet "$NKP_UNIT"; then + echo "$NKP_UNIT does not run" + return 1 + fi + pid=$(systemctl show -P MainPID "$NKP_UNIT") + want=$(sha256sum <"$NKP_ETC/broker.toml") || return 1 + want=${want%% *} + if [ ! -f "$status" ]; then + echo "$status is missing" + return 1 + fi + line=$(head -c 65536 "$status" | sed -n -E \ + 's/^\{"pid":([0-9]+),"start":[0-9]+,"reloads":[0-9]+,"result":"(started|loaded)","config_sha256":"([0-9a-f]{64})".*/\1 \3/p') + if [ -z "$line" ]; then + echo "$status says no good start or reload" + return 1 + fi + got_pid=${line% *} + got_sha=${line#* } + if [ "$got_pid" != "$pid" ]; then + echo "$status is from process $got_pid, but $NKP_UNIT runs process $pid" + return 1 + fi + if [ "$got_sha" != "$want" ]; then + echo "the broker runs another config (sha256 $got_sha), not $NKP_ETC/broker.toml (sha256 $want)" + return 1 + fi + echo "the broker (process $pid) runs $NKP_ETC/broker.toml (sha256 $want)" +} + +end=$(($(date +%s) + NKP_HEALTH_TIMEOUT)) +while :; do + if why=$(once); then + echo "nodkey health: $why" + exit 0 + fi + if [ "$(date +%s)" -ge "$end" ]; then + echo "nodkey health: no good answer in ${NKP_HEALTH_TIMEOUT}s: $why; read journalctl -u $NKP_UNIT" + exit 1 + fi + sleep 1 +done diff --git a/contrib/pullgate/lib.sh b/contrib/pullgate/lib.sh new file mode 100644 index 0000000..b227822 --- /dev/null +++ b/contrib/pullgate/lib.sh @@ -0,0 +1,123 @@ +# shellcheck shell=sh +# The shared part of the pullgate recipe of nodkey (docs/setup.md, "Change +# the config from git with pullgate"). The four scripts source it from +# their own directory. It is never run alone. +# +# The scripts are installed on the broker host from a release of nodkey, +# never run from the tree of a config repo. pullgate runs each one with an +# empty environment, so the NKP_ values below are the defaults on a host. +# Tests set them. + +NKP_ETC=${NKP_ETC:-/etc/nodkey} +NKP_UNIT=${NKP_UNIT:-nodkey.service} +NKP_CREDENTIALS=${NKP_CREDENTIALS:-/etc/nodkey/credentials} +NKP_HEALTH_TIMEOUT=${NKP_HEALTH_TIMEOUT:-60} + +# The two files of the config, in the repo and in NKP_ETC. +NKP_FILES="broker.toml known_hosts" + +# nkp_files_ok DIR: both files are in DIR, as plain files. A symlink is +# refused, even one to a plain file. +nkp_files_ok() { + for f in $NKP_FILES; do + if [ -L "$1/$f" ] || [ ! -f "$1/$f" ]; then + echo "$1/$f is missing, or is not a plain file (a symlink is refused)" + return 1 + fi + done +} + +# nkp_action OLD NEW prints "reload" when the broker can take every change +# from the config in the directory OLD to the one in NEW with a reload, +# and "restart" when it cannot. This is the one place that decides it. +# +# It reads the full list of nodkey broker config-diff. A reload can change +# the clients, the secrets, the projects, the SSH keys and the host keys +# in known_hosts (DESIGN.md section 14, "Reload"). So each line must name +# one of them; any other line, or a failed compare, means a restart. A new +# SSH key, or a key with another file, also needs a restart: systemd +# loads the credentials only at the start. +nkp_action() { + nkp_diff=$(nodkey broker config-diff "$1" "$2" 2>/dev/null) + case $? in + 0) + # The same meaning. But the reload compares [logins.*] by its exact + # form, so a new order of a list there needs a restart. + if ! cmp -s "$1/broker.toml" "$2/broker.toml" && LC_ALL=C grep -q '^[[:space:]]*\[[[:space:]]*logins\.' "$2/broker.toml"; then + echo restart + else + echo reload + fi + return 0 + ;; + 1) ;; + *) + echo restart + return 0 + ;; + esac + if printf '%s\n' "$nkp_diff" | LC_ALL=C grep -Eq '^(⚠ | )SSH key ".*": (new key|file changed)'; then + echo restart + elif printf '%s\n' "$nkp_diff" | + LC_ALL=C grep -Evq '^(⚠ | )((client|secret|project|SSH key) "|known_hosts: (new host|host) ")'; then + echo restart + else + echo reload + fi +} + +# nkp_host_parts LIVE NEW prints the changes from the live config in the +# directory LIVE to the one in NEW that only a person on the host may make, +# and fails when there is one, or when the compare fails. These parts are +# the gate itself: the approval channel and its token, the TLS key, the +# listen address, the paths of known_hosts and the audit log, and the +# stores with their programs and credentials. A commit that changes them +# could send a credential to another server, or approve its own requests. +nkp_host_parts() { + nkp_diff=$(nodkey broker config-diff "$1" "$2" 2>&1) + case $? in + 0) return 0 ;; + 1) ;; + *) + printf '%s +' "$nkp_diff" + return 1 + ;; + esac + ! printf '%s +' "$nkp_diff" | LC_ALL=C grep -E \ + '^⚠ ((listen|tls_key|audit_log|known_hosts): changed|\[(telegram|ntfy)\]|the approval channel|store ")' +} + +# nkp_top_key FILE KEY prints the string value of the top-level KEY of the +# TOML file FILE, such as audit_log. It reads only a plain line, +# KEY = "VALUE" or KEY = 'VALUE' with an optional comment, before the +# first section. Anything else, or the key twice, is a failure. +nkp_top_key() { + awk -v key="$2" ' + /^[ \t]*\[/ { exit } + { + line = $0 + sub(/^[ \t]+/, "", line) + if (index(line, key) != 1) next + rest = substr(line, length(key) + 1) + if (rest !~ /^[ \t]*=/) next + sub(/^[ \t]*=[ \t]*/, "", rest) + if (match(rest, /^"[^"\\]*"/) || match(rest, /^\047[^\047]*\047/)) { + val = substr(rest, 2, RLENGTH - 2) + if (substr(rest, RLENGTH + 1) ~ /^[ \t]*(#.*)?$/) { n++; next } + } + bad = 1 + } + END { if (bad || n != 1) exit 1; print val } + ' "$1" +} + +# nkp_run ACTION reloads or restarts the broker. +nkp_run() { + case $1 in + reload) nodkey broker reload --config "$NKP_ETC/broker.toml" ;; + restart) systemctl restart "$NKP_UNIT" ;; + *) return 1 ;; + esac +} diff --git a/contrib/pullgate/nodkey-broker.toml b/contrib/pullgate/nodkey-broker.toml new file mode 100644 index 0000000..2c57ade --- /dev/null +++ b/contrib/pullgate/nodkey-broker.toml @@ -0,0 +1,50 @@ +# The pullgate instance of the nodkey broker config (docs/setup.md, +# "Change the config from git with pullgate"). The broker host pulls +# broker.toml and known_hosts from a private git repo, asks the owner on +# Telegram, and applies a commit only after a tap. +# +# Install it as /etc/pullgate/nodkey-broker.toml (owner root, mode 0600). +# Change the host name and the [repo] section. +mode = "ask" +# A failed apply, reload, restart or health check runs apply.sh again with +# the last good commit. That is safe: apply.sh installs two files and +# reloads or restarts the broker. +on_failure = "rollback" +host = "example-host" + +[repo] +url = "ssh://git@git.example.com/owner/config-repo.git" +branch = "main" +# A read-only deploy key of the config repo. +deploy_key = "/etc/pullgate/nodkey-broker.key" +known_hosts = "/etc/pullgate/nodkey-broker.known_hosts" + +[commands] +# The scripts come from a release of nodkey (nodkey_VERSION_pullgate.tar.gz), +# never from the config repo. check and summary run as check_user and only +# read the tree as data; apply and health run as root. +check = ["/usr/local/lib/nodkey-pullgate/check.sh"] +summary = ["/usr/local/lib/nodkey-pullgate/summary.sh"] +apply = ["/usr/local/lib/nodkey-pullgate/apply.sh"] +health = ["/usr/local/lib/nodkey-pullgate/health.sh"] +check_user = "pullgate-check-nodkey-broker" +check_timeout = "2m" +# apply.sh may restart the broker, and health.sh waits up to 60s for it. +apply_timeout = "5m" + +[limits] +questions_per_day = 10 +approval_ttl = "12h" +max_failed_tries = 1 +allow_older = false +# The repo holds two small files. A symlink out of the tree stops the +# export; check.sh and apply.sh also refuse broker.toml or known_hosts as +# a symlink. +refuse_symlinks = true +# Few files: pullgate lists the changed files and the commits before the +# summary, and cuts a long message from the end. Keep the repo small. +max_files = 20 +max_tree_bytes = 1048576 + +[listener] +user = "pullgate" diff --git a/contrib/pullgate/summary.sh b/contrib/pullgate/summary.sh new file mode 100755 index 0000000..f794d14 --- /dev/null +++ b/contrib/pullgate/summary.sh @@ -0,0 +1,27 @@ +#!/bin/sh +# The summary of the pullgate recipe of nodkey. pullgate runs it as the +# check user and puts its output into the question. The lines with ⚠ come +# first; read each one before you tap Apply. +set -u +. "${0%/*}/lib.sh" + +new=${PULLGATE_TREE:?PULLGATE_TREE is not set; pullgate sets it} +old=${PULLGATE_OLD_TREE:-} +first= +if [ -z "$old" ]; then + # The first apply: compare with the config that runs now. + old=$NKP_ETC + first=1 +fi + +if [ "$(nkp_action "$old" "$new")" = restart ]; then + echo "⚠ needs a broker restart: all grants end" +fi +if [ -n "$first" ]; then + echo "First apply: the changes from the live config in $NKP_ETC." +elif ! nodkey broker config-diff "$NKP_ETC" "$old" >/dev/null 2>&1; then + echo "⚠ the live config in $NKP_ETC is not the applied commit (a change by hand?); the list is from the applied commit" +fi +nodkey broker config-diff --short --max-chars 1200 "$old" "$new" 2>&1 +# config-diff exits 1 when the configs differ: that is not a failure. +[ $? -le 1 ] diff --git a/contrib_pullgate_linux_test.go b/contrib_pullgate_linux_test.go new file mode 100644 index 0000000..e55482b --- /dev/null +++ b/contrib_pullgate_linux_test.go @@ -0,0 +1,565 @@ +package nodkey + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/code6x/nodkey/internal/config" +) + +// The tests of the pullgate recipe in contrib/pullgate run the four +// scripts with /bin/sh, as pullgate runs them, against a fake broker: +// +// - the real nodkey, built from this module, for nodkey broker check and +// nodkey broker config-diff; +// - a fake nodkey broker reload, which takes the config or refuses it, +// and writes the broker's status file, reload-status.json; +// - a fake systemctl, whose restart starts a new "process" (a new PID in +// the status file) or fails. +// +// The scripts get the paths of the test in NKP_ETC, NKP_UNIT and +// NKP_CREDENTIALS. They run as the user of the test, not as root. + +var ( + pgBuild sync.Once + pgBinary string + pgBuildErr error +) + +// pgNodkey builds nodkey once for all the tests. +func pgNodkey(t *testing.T) string { + t.Helper() + pgBuild.Do(func() { + dir, err := os.MkdirTemp("", "nodkey-pullgate-test-") + if err != nil { + pgBuildErr = err + return + } + pgBinary = filepath.Join(dir, "nodkey") + out, err := exec.CommandContext(context.Background(), "go", "build", "-o", pgBinary, "./cmd/nodkey").CombinedOutput() //nolint:gosec // G204: the test's own build. + if err != nil { + pgBuildErr = &buildError{out: string(out), err: err} + } + }) + if pgBuildErr != nil { + t.Fatalf("build nodkey: %v", pgBuildErr) + } + return pgBinary +} + +type buildError struct { + out string + err error +} + +func (e *buildError) Error() string { return e.err.Error() + ": " + e.out } + +// pgEnv is one broker host of a test. +type pgEnv struct { + dir string // the root of the test + etc string // NKP_ETC + audit string // the directory of the audit log and the status file + fake string // the state of the fakes + bin string // the fakes, first in PATH + // rollback sets PULLGATE_ROLLBACK=1. + rollback bool +} + +const pgSecretID = "00000000-0000-4000-8000-00003f9a1c2e" + +// pgConfig is a valid broker config of the env, with edits. +func (e *pgEnv) pgConfig(t *testing.T, edits ...[2]string) string { + t.Helper() + cfg := `listen = "192.0.2.10:7743" +tls_key = "${CREDENTIALS_DIRECTORY}/broker-tls-key" +audit_log = "` + e.audit + `/audit.jsonl" +known_hosts = "` + e.etc + `/known_hosts" + +[telegram] +token_file = "${CREDENTIALS_DIRECTORY}/telegram-token" +user_id = 123456789 + +[client.laptop] +pin = "` + config.Pin{1}.String() + `" + +[store.main] +type = "bws" +program = "/usr/local/bin/bws" +token_file = "${CREDENTIALS_DIRECTORY}/bws-token" + +[secret.github-token] +store = "main" +id = "` + pgSecretID + `" +clients = ["laptop"] +ask = "always" +` + for _, ed := range edits { + if !strings.Contains(cfg, ed[0]) { + t.Fatalf("the config has no %q", ed[0]) + } + cfg = strings.Replace(cfg, ed[0], ed[1], 1) + } + return cfg +} + +// pgTree writes a tree of a commit with the config cfg, and returns it. +func (e *pgEnv) pgTree(t *testing.T, name, cfg string) string { + t.Helper() + tree := filepath.Join(e.dir, "trees", name) + pgWrite(t, filepath.Join(tree, "broker.toml"), cfg) + pgWrite(t, filepath.Join(tree, "known_hosts"), "") + return tree +} + +func pgWrite(t *testing.T, path, data string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(data), 0o644); err != nil { //nolint:gosec // G306: a config file, not secret. + t.Fatal(err) + } +} + +// pgFakeStatus is a shell function that writes the status file as the +// broker does: one line of JSON, in the order of the fields of +// reloadStatus in cmd/nodkey/brokerreload.go. +const pgFakeStatus = `status() { + sha=$(sha256sum < "$NKP_ETC/broker.toml"); sha=${sha%% *} + printf '{"pid":%s,"start":7,"reloads":1,"result":"%s","config_sha256":"%s","grants_ended":0,"at":"2026-11-17T00:00:00Z"}\n' \ + "$(cat "$FAKE/pid")" "$1" "$sha" > "$AUDIT/reload-status.json" +} +# fails NAME: true, and one less, while the count in $FAKE/NAME is above 0. +fails() { + n=$(cat "$FAKE/$1" 2>/dev/null || echo 0) + [ "$n" -gt 0 ] || return 1 + echo $((n - 1)) > "$FAKE/$1" +} +` + +func newPGEnv(t *testing.T) *pgEnv { + t.Helper() + if os.Getuid() == 0 { + t.Skip("runs as root; the fakes in PATH are for a user that is not root") + } + real := pgNodkey(t) + dir := pgTestDir(t) + e := &pgEnv{dir: dir, etc: filepath.Join(dir, "etc"), audit: filepath.Join(dir, "var"), + fake: filepath.Join(dir, "fake"), bin: filepath.Join(dir, "bin")} + for _, d := range []string{e.etc, e.audit, e.fake, e.bin} { + if err := os.MkdirAll(d, 0o700); err != nil { + t.Fatal(err) + } + } + vars := "FAKE='" + e.fake + "'\nAUDIT='" + e.audit + "'\n" + pgScript(t, filepath.Join(e.bin, "nodkey"), vars+pgFakeStatus+`if [ "$1 $2" = "broker reload" ]; then + echo reload >> "$FAKE/calls" + [ -f "$FAKE/active" ] || { echo "nodkey broker reload: no broker runs" >&2; exit 1; } + if fails reload-fails; then + echo "nodkey broker reload: the broker refused the new config" >&2 + exit 1 + fi + status loaded + exit 0 +fi +exec '`+real+`' "$@" +`) + pgScript(t, filepath.Join(e.bin, "systemctl"), vars+pgFakeStatus+`case $1 in +is-active) [ -f "$FAKE/active" ] ;; +show) cat "$FAKE/pid" ;; +restart) + echo restart >> "$FAKE/calls" + if fails restart-fails; then + rm -f "$FAKE/active" + exit 1 + fi + echo $(($(cat "$FAKE/pid") + 1)) > "$FAKE/pid" + touch "$FAKE/active" + status started + ;; +*) echo "fake systemctl: $*" >&2; exit 2 ;; +esac +`) + // The broker runs the live config. + pgWrite(t, filepath.Join(e.etc, "broker.toml"), e.pgConfig(t)) + pgWrite(t, filepath.Join(e.etc, "known_hosts"), "") + pgWrite(t, filepath.Join(e.fake, "pid"), "100\n") + pgWrite(t, filepath.Join(e.fake, "active"), "") + e.runFake(t, "status", "started") + return e +} + +// pgTestDir returns a new directory for one test, on a path that passes +// the file checks of nodkey: the system temporary directory fails them +// (see privateTestDir in cmd/nodkey). It is made in $XDG_RUNTIME_DIR or +// in the user's cache directory. The test is skipped if there is no such +// place, or fails when CI is set. +func pgTestDir(t *testing.T) string { + t.Helper() + var bases []string + if d := os.Getenv("XDG_RUNTIME_DIR"); d != "" { + bases = append(bases, d) + } + if d, err := os.UserCacheDir(); err == nil { + bases = append(bases, d) + } + for _, base := range bases { + dir, err := os.MkdirTemp(base, "nodkey-pullgate-test-") + if err != nil { + continue + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) //nolint:gosec // G703: the test's own new directory. + probe := filepath.Join(dir, "probe") + ok := os.Chmod(dir, 0o700) == nil //nolint:gosec // G302: a directory, private to its owner. + if ok && os.WriteFile(probe, nil, 0o600) == nil && config.CheckPrivateFile(probe) == nil { + return dir + } + } + if os.Getenv("CI") != "" { + t.Fatalf("no directory passes the file checks of nodkey: tried %q", bases) + } + t.Skipf("no directory passes the file checks of nodkey: tried %q", bases) + return "" +} + +func pgScript(t *testing.T, path, body string) { + t.Helper() + if err := os.WriteFile(path, []byte("#!/bin/sh\n"+body), 0o700); err != nil { //nolint:gosec // G306: a fake program of the test. + t.Fatal(err) + } +} + +// runFake runs a shell function of the fakes. +func (e *pgEnv) runFake(t *testing.T, args ...string) { + t.Helper() + script := "FAKE='" + e.fake + "'\nAUDIT='" + e.audit + "'\nNKP_ETC='" + e.etc + "'\n" + pgFakeStatus + `"$@"` + cmd := exec.CommandContext(t.Context(), "/bin/sh", append([]string{"-c", script, "sh"}, args...)...) //nolint:gosec // G204: the test's own fake. + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("fake %q: %v: %s", args, err, out) + } +} + +// run runs a script of contrib/pullgate with the environment that pullgate +// gives it, and returns its exit code and output. +func (e *pgEnv) run(t *testing.T, script, tree, oldTree string) (int, string) { + t.Helper() + path, err := filepath.Abs(filepath.Join("contrib", "pullgate", script)) + if err != nil { + t.Fatal(err) + } + cmd := exec.CommandContext(t.Context(), path) //nolint:gosec // G204: a script of contrib/pullgate. + cmd.Dir = tree + cmd.Env = []string{ + "PATH=" + e.bin + ":/usr/local/bin:/usr/bin:/bin", + "PULLGATE_INSTANCE=nodkey-broker", + "PULLGATE_SHA=1111111111111111111111111111111111111111", + "PULLGATE_TREE=" + tree, + "PULLGATE_OLD_TREE=" + oldTree, + "NKP_ETC=" + e.etc, + "NKP_UNIT=nodkey.service", + "NKP_CREDENTIALS=" + filepath.Join(e.dir, "no-credentials"), + "NKP_HEALTH_TIMEOUT=2", + } + if e.rollback { + cmd.Env = append(cmd.Env, "PULLGATE_ROLLBACK=1") + } + out, err := cmd.CombinedOutput() + code := 0 + if err != nil { + var ee *exec.ExitError + if !errors.As(err, &ee) { + t.Fatalf("%s: %v", script, err) + } + code = ee.ExitCode() + } + return code, string(out) +} + +func (e *pgEnv) calls(t *testing.T) string { + t.Helper() + data, err := os.ReadFile(filepath.Join(e.fake, "calls")) + if err != nil && !os.IsNotExist(err) { + t.Fatal(err) + } + return strings.TrimSpace(strings.ReplaceAll(string(data), "\n", " ")) +} + +func (e *pgEnv) liveSum(t *testing.T) string { + t.Helper() + data, err := os.ReadFile(filepath.Join(e.etc, "broker.toml")) + if err != nil { + t.Fatal(err) + } + s := sha256.Sum256(data) + return hex.EncodeToString(s[:]) +} + +// The changes of the tests. +var ( + // A new secret: a reload takes it. + pgReloadSafe = [2]string{"[secret.github-token]", "[secret.mail-password]\nstore = \"main\"\n" + + "id = \"00000000-0000-4000-8000-000011112222\"\nclients = [\"laptop\"]\nask = \"always\"\n\n[secret.github-token]"} + // [updates] check turned off: a reload refuses it, and a commit may + // make it. + pgRestartType = [2]string{"[client.laptop]", "[updates]\ncheck = false\n\n[client.laptop]"} +) + +// pgTelegram is another timeout of [telegram]: a part of the gate. +var pgTelegram = [2]string{"user_id = 123456789", "user_id = 123456789\ntimeout = \"300s\""} + +// TestPullgateCheck: check.sh passes a good commit, and refuses a missing +// file, a symlink, a config that is not valid, and a change of the path of +// known_hosts or audit_log. +func TestPullgateCheck(t *testing.T) { + e := newPGEnv(t) + if code, out := e.run(t, "check.sh", e.pgTree(t, "good", e.pgConfig(t, pgReloadSafe)), ""); code != 0 { + t.Fatalf("a good commit: exit %d:\n%s", code, out) + } + for _, c := range []struct { + name string + tree func() string + want string + }{ + {"no known_hosts", func() string { + tree := e.pgTree(t, "no-kh", e.pgConfig(t)) + if err := os.Remove(filepath.Join(tree, "known_hosts")); err != nil { + t.Fatal(err) + } + return tree + }, "known_hosts is missing, or is not a plain file"}, + {"a symlink", func() string { + tree := e.pgTree(t, "link", e.pgConfig(t)) + p := filepath.Join(tree, "broker.toml") + if err := os.Rename(p, p+".real"); err != nil { + t.Fatal(err) + } + if err := os.Symlink("broker.toml.real", p); err != nil { + t.Fatal(err) + } + return tree + }, "a symlink is refused"}, + {"not valid", func() string { + return e.pgTree(t, "bad", e.pgConfig(t, [2]string{config.Pin{1}.String(), "not-a-pin"})) + }, "Fix the config in the repo"}, + {"audit_log moved", func() string { + return e.pgTree(t, "audit", e.pgConfig(t, [2]string{"/audit.jsonl", "/other.jsonl"})) + }, "Refused: a commit may not change listen"}, + {"another approver", func() string { + return e.pgTree(t, "approver", e.pgConfig(t, [2]string{"user_id = 123456789", "user_id = 987654321"})) + }, "Refused"}, + {"another telegram timeout", func() string { + return e.pgTree(t, "tgtimeout", e.pgConfig(t, pgTelegram)) + }, "Refused"}, + {"another store program", func() string { + return e.pgTree(t, "program", e.pgConfig(t, [2]string{"/usr/local/bin/bws", "/tmp/bws"})) + }, "Refused"}, + {"tls_key", func() string { + return e.pgTree(t, "tls", e.pgConfig(t, [2]string{"/broker-tls-key", "/bws-token"})) + }, "Refused"}, + {"known_hosts moved", func() string { + tree := e.pgTree(t, "kh", e.pgConfig(t, [2]string{"/known_hosts\"", "/known_hosts2\""})) + pgWrite(t, filepath.Join(tree, "known_hosts2"), "") + return tree + }, "Refused"}, + } { + t.Run(c.name, func(t *testing.T) { + code, out := e.run(t, "check.sh", c.tree(), "") + if code == 0 || !strings.Contains(out, c.want) { + t.Fatalf("exit %d, want a failure with %q:\n%s", code, c.want, out) + } + }) + } +} + +// TestPullgateSummary: the change list, with "⚠ needs a broker restart" +// first only when a reload cannot take the change. +func TestPullgateSummary(t *testing.T) { + e := newPGEnv(t) + old := e.pgTree(t, "old", e.pgConfig(t)) + const restart = "⚠ needs a broker restart: all grants end" + + code, out := e.run(t, "summary.sh", e.pgTree(t, "safe", e.pgConfig(t, pgReloadSafe)), old) + if code != 0 || strings.Contains(out, restart) || + !strings.Contains(out, `⚠ secret "mail-password": new secret in store "main", item …11112222`) { + t.Fatalf("a reload-safe change: exit %d:\n%s", code, out) + } + + code, out = e.run(t, "summary.sh", e.pgTree(t, "restart", e.pgConfig(t, pgRestartType)), old) + if code != 0 || !strings.HasPrefix(out, restart+"\n") || !strings.Contains(out, "[updates] check") { + t.Fatalf("a restart-type change: exit %d:\n%s", code, out) + } + + // A new SSH key needs a restart: systemd loads its credential only at + // the start. + key := [2]string{"[secret.github-token]", "[key.deploy]\nfile = \"${CREDENTIALS_DIRECTORY}/ssh-deploy\"\n" + + "clients = [\"laptop\"]\nask = \"always\"\n\n[secret.github-token]"} + code, out = e.run(t, "summary.sh", e.pgTree(t, "key", e.pgConfig(t, key)), old) + if code != 0 || !strings.HasPrefix(out, restart+"\n") { + t.Fatalf("a new SSH key: exit %d:\n%s", code, out) + } + + // Only a new host key in known_hosts: a reload takes it. + kh := e.pgTree(t, "kh", e.pgConfig(t)) + pgWrite(t, filepath.Join(kh, "known_hosts"), "web-9 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl\n") + code, out = e.run(t, "summary.sh", kh, old) + if code != 0 || strings.Contains(out, restart) || !strings.Contains(out, `known_hosts: new host "web-9"`) { + t.Fatalf("a new host key: exit %d:\n%s", code, out) + } + + // The first apply compares with the live config. + code, out = e.run(t, "summary.sh", e.pgTree(t, "first", e.pgConfig(t, pgReloadSafe)), "") + if code != 0 || !strings.Contains(out, "First apply") || !strings.Contains(out, `secret "mail-password"`) { + t.Fatalf("the first apply: exit %d:\n%s", code, out) + } + + // The live config is not the applied commit. + code, out = e.run(t, "summary.sh", e.pgTree(t, "safe2", e.pgConfig(t, pgReloadSafe)), + e.pgTree(t, "old2", e.pgConfig(t, [2]string{"ask = \"always\"", "ask = \"window\"\nwindow = \"5m\""}))) + if code != 0 || !strings.Contains(out, "⚠ the live config") { + t.Fatalf("a hand edit: exit %d:\n%s", code, out) + } +} + +// TestPullgateApply: apply.sh reloads for a reload-safe change and +// restarts for another, and health.sh then sees the new config. When the +// reload or the restart fails, the old files go back, the broker reloads +// or restarts again, and apply.sh fails. +func TestPullgateApply(t *testing.T) { + for _, c := range []struct { + name string + change [2]string + fails string // a file of the fakes with a count of failures + inactive bool // the broker does not run + rollback bool // PULLGATE_ROLLBACK=1 + wantCode int + wantCalls string + }{ + {name: "reload-safe", change: pgReloadSafe, wantCalls: "reload"}, + {name: "restart-type", change: pgRestartType, wantCalls: "restart"}, + {name: "refused reload", change: pgReloadSafe, fails: "reload-fails", wantCode: 1, wantCalls: "reload reload"}, + {name: "failed restart", change: pgRestartType, fails: "restart-fails", wantCode: 1, wantCalls: "restart restart"}, + {name: "a broker that does not run", change: pgReloadSafe, inactive: true, wantCalls: "restart"}, + // A rollback whose reload fails: the failed files are not put back; + // a restart runs the last good config. + {name: "rollback", change: pgReloadSafe, fails: "reload-fails", rollback: true, wantCalls: "reload restart"}, + } { + t.Run(c.name, func(t *testing.T) { + e := newPGEnv(t) + before := e.liveSum(t) + if c.fails != "" { + pgWrite(t, filepath.Join(e.fake, c.fails), "1\n") + } + if c.inactive { + if err := os.Remove(filepath.Join(e.fake, "active")); err != nil { + t.Fatal(err) + } + } + newCfg := e.pgConfig(t, c.change) + e.rollback = c.rollback + code, out := e.run(t, "apply.sh", e.pgTree(t, "new", newCfg), e.pgTree(t, "old", e.pgConfig(t))) + if code != c.wantCode || e.calls(t) != c.wantCalls { + t.Fatalf("exit %d, calls %q; want exit %d, calls %q:\n%s", code, e.calls(t), c.wantCode, c.wantCalls, out) + } + after := e.liveSum(t) + if c.wantCode == 0 { + want := sha256.Sum256([]byte(newCfg)) + if after != hex.EncodeToString(want[:]) { + t.Errorf("the live config is not the new one:\n%s", out) + } + } else if after != before { + t.Errorf("the old config is not back:\n%s", out) + } + for _, d := range []string{"pullgate-new", "pullgate-old"} { + if _, err := os.Stat(filepath.Join(e.etc, d)); !os.IsNotExist(err) { + t.Errorf("%s is left: %v", d, err) + } + } + // The broker runs the config that is in place now. + if code, out := e.run(t, "health.sh", e.dir, ""); code != 0 || !strings.Contains(out, "runs") { + t.Errorf("health.sh: exit %d:\n%s", code, out) + } + }) + } +} + +// TestPullgateApplyRefusesBadFiles: apply.sh installs nothing when a file +// of the commit is a symlink, or nodkey broker check refuses the config. +func TestPullgateApplyRefusesBadFiles(t *testing.T) { + e := newPGEnv(t) + before := e.liveSum(t) + bad := e.pgTree(t, "bad", e.pgConfig(t, [2]string{config.Pin{1}.String(), "not-a-pin"})) + if code, out := e.run(t, "apply.sh", bad, ""); code == 0 || !strings.Contains(out, "refused the new config") { + t.Fatalf("a config that is not valid: exit %d:\n%s", code, out) + } + link := e.pgTree(t, "link", e.pgConfig(t)) + if err := os.Remove(filepath.Join(link, "known_hosts")); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(e.etc, "known_hosts"), filepath.Join(link, "known_hosts")); err != nil { + t.Fatal(err) + } + if code, out := e.run(t, "apply.sh", link, ""); code == 0 || !strings.Contains(out, "a symlink is refused") { + t.Fatalf("a symlink: exit %d:\n%s", code, out) + } + // A part of the gate, also in a rollback, which runs no check.sh. + gate := e.pgTree(t, "gate", e.pgConfig(t, pgTelegram)) + if code, out := e.run(t, "apply.sh", gate, ""); code == 0 || !strings.Contains(out, "refused: the commit changes a part") { + t.Fatalf("a part of the gate: exit %d:\n%s", code, out) + } + // A copy of the old files that an earlier apply left: never removed. + keep := filepath.Join(e.etc, "pullgate-old") + if err := os.Mkdir(keep, 0o700); err != nil { + t.Fatal(err) + } + if code, out := e.run(t, "apply.sh", e.pgTree(t, "safe", e.pgConfig(t, pgReloadSafe)), ""); code == 0 || + !strings.Contains(out, "is left from an earlier apply") { + t.Fatalf("a left copy: exit %d:\n%s", code, out) + } + if _, err := os.Stat(keep); err != nil { + t.Errorf("the left copy is gone: %v", err) + } + if e.liveSum(t) != before || e.calls(t) != "" { + t.Errorf("the live config changed, or the broker was reloaded: calls %q", e.calls(t)) + } +} + +// TestPullgateHealth: health.sh fails, after its timeout, when the status +// file is of another process or another config, or the broker does not +// run. +func TestPullgateHealth(t *testing.T) { + for _, c := range []struct { + name string + setup func(e *pgEnv) + want string + }{ + {"another process", func(e *pgEnv) { pgWrite(t, filepath.Join(e.fake, "pid"), "101\n") }, "but nodkey.service runs process 101"}, + {"another config", func(e *pgEnv) { pgWrite(t, filepath.Join(e.etc, "broker.toml"), e.pgConfig(t, pgReloadSafe)) }, + "the broker runs another config"}, + {"not running", func(e *pgEnv) { + if err := os.Remove(filepath.Join(e.fake, "active")); err != nil { + t.Fatal(err) + } + }, "nodkey.service does not run"}, + {"a refused reload", func(e *pgEnv) { e.runFake(t, "status", "refused") }, "says no good start or reload"}, + } { + t.Run(c.name, func(t *testing.T) { + e := newPGEnv(t) + if code, out := e.run(t, "health.sh", e.dir, ""); code != 0 { + t.Fatalf("before: exit %d:\n%s", code, out) + } + c.setup(e) + code, out := e.run(t, "health.sh", e.dir, "") + if code == 0 || !strings.Contains(out, c.want) || !strings.Contains(out, "no good answer in 2s") { + t.Fatalf("exit %d, want a failure with %q:\n%s", code, c.want, out) + } + }) + } +} diff --git a/docs/releasing.md b/docs/releasing.md index 3e4c8c5..ff43f81 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -85,12 +85,14 @@ After every push to `main`, the **Release** workflow runs release-please. commit of this workflow run. Then it builds static `nodkey` binaries for Linux amd64 and arm64 (`CGO_ENABLED=0`, `-trimpath`), packs each one with `LICENSE` and `README.md` into - `nodkey__linux_.tar.gz`, and writes `SHA256SUMS`. + `nodkey__linux_.tar.gz`. It packs the files of + `contrib/pullgate/` with `LICENSE` into + `nodkey__pullgate.tar.gz`, and writes `SHA256SUMS`. - `publish` gets these files from `build`. It checks them against `SHA256SUMS` and checks the tag again. Then it makes a signed build provenance attestation for the tarballs. It saves the attestation as one more file, `nodkey_.sigstore.json` (a Sigstore bundle - for both tarballs), and checks both tarballs with it, with no login, + for all tarballs), and checks all tarballs with it, with no login, as in [Verify a download](#verify-a-download). Then it uploads the files to the draft release and publishes it. @@ -304,9 +306,12 @@ A release has these files: - `nodkey__linux_amd64.tar.gz` and `nodkey__linux_arm64.tar.gz`: the binary, `LICENSE` and `README.md`; -- `SHA256SUMS`: the SHA-256 of both tarballs; +- `nodkey__pullgate.tar.gz`: the pullgate recipe, the files of + `contrib/pullgate/` and `LICENSE` (docs/setup.md, "Change the config + from git with pullgate"). Releases before 0.7.0 do not have it; +- `SHA256SUMS`: the SHA-256 of all tarballs; - `nodkey_.sigstore.json`: the build provenance attestation of - both tarballs, as a Sigstore bundle. 0.1.0 does not have this file. + all tarballs, as a Sigstore bundle. 0.1.0 does not have this file. Set the version you want, and your machine's arch: `amd64`, or `arm64` on 64-bit ARM. diff --git a/docs/setup.md b/docs/setup.md index 6cd7b8b..3809505 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -1892,6 +1892,198 @@ the broker took has a `config_reloaded` entry in the audit log Check the setup with `sudo nodkey doctor broker` ([section 14](#14-when-something-does-not-work)). +### Change the config from git with pullgate + +With [pullgate](https://github.com/code6x/pullgate), the broker host +pulls its config from a private git repo by itself. You change the +config with a commit, not with an SSH login to the broker host. For each +new commit, pullgate asks you on Telegram, with the change list of +`nodkey broker config-diff`. It applies the commit only after you tap +**Apply**. If the new config does not work, it goes back to the last +good one by itself. + +pullgate uses its own Telegram bot, not the broker's. The broker gets no +new port and no new network path: pullgate fetches the repo with a +read-only deploy key. + +The recipe is in `contrib/pullgate/` of this repository: + +- `check.sh` runs as the check user, before the question. Both files + must be plain files (no symlinks). It runs `nodkey broker check + --files` on the commit. A commit may not change the parts of the gate + itself (see "What stays a step on the host"). A commit that fails gets + no question, only a notice. +- `summary.sh` runs as the check user. Its output is the change list in + the question: `nodkey broker config-diff --short --max-chars 1200` from + the applied commit (on the first apply, from the live config in + `/etc/nodkey`). When a reload cannot take the change, its first line + is `⚠ needs a broker restart: all grants end`. +- `apply.sh` runs as root after your tap. It checks the new files again, + with the names of the credentials, and the parts of the gate again, and + installs both in `/etc/nodkey`. + Then it runs `nodkey broker reload`, or `systemctl restart nodkey` when + a reload cannot take the change. If that fails, it puts the old files + back, reloads or restarts again, and fails. Then pullgate rolls back. + When the broker does not run, it starts it: to keep a stopped broker + stopped, pause the instance first. +- `health.sh` runs as root after `apply.sh`. It waits up to 60 seconds + for the broker to say, in its status file `reload-status.json`, that + the process that runs now uses the new `broker.toml`. A failure starts + the rollback. +- `lib.sh` holds the one rule for "reload or restart" that the scripts + share. Only changes of `[client.*]`, `[secret.*]`, `[project.*]`, + `[key.*]` (but not a new key or a new key file) and the host keys in + `known_hosts` are done with a reload. Any other change, or a failed + compare, means a restart. + +#### What stays a step on the host + +These changes still need root on the broker host. `check.sh` and +`apply.sh` refuse a commit that makes them. They are the gate itself: a +commit that changes them could send a credential to another server, or +approve its own requests. + +- `listen`, `tls_key`, and the paths of `known_hosts` and `audit_log`; +- `[telegram]` and `[ntfy]`: the approver, the bot, the server, the + topics, the token file and the timeouts; +- a new or changed `[store.*]` (a removed store is allowed); +- a new credential, such as a new SSH key file + ([Encrypted credentials](#encrypted-credentials)), and a change of + the unit; +- a new version of nodkey or of the recipe. + +Make such a change by hand (see the way back below), then commit the +same files to the repo. + +#### The config repo + +Make a **private** git repo. Put two files at its root: + +- `broker.toml`: the config, the same as `/etc/nodkey/broker.toml`; +- `known_hosts`: the broker's `known_hosts`, the same as + `/etc/nodkey/known_hosts`. + +**Never put a secret value in the repo.** The config names its secrets +only by paths in `${CREDENTIALS_DIRECTORY}` and by item IDs, and +`known_hosts` holds only public host keys. Start the repo from the live +files, so that the first change list is empty. Other files, such as a +README or a CI check with `nodkey broker check --files`, are fine: +apply.sh installs only the two files. + +The recipe needs `known_hosts = "/etc/nodkey/known_hosts"` in +`broker.toml`, and `audit_log = "..."` as one plain line at the top. +That is the form of [section 7](#7-the-config-file). + +#### Install + +You need nodkey 0.7.0 or newer, and pullgate 0.2.0 or newer. Run every +step on the broker host, as root. + +1. Install pullgate, its users and its units as its README says. Make the + check user of this instance: + + ```sh + useradd --system --no-create-home --shell /usr/sbin/nologin pullgate-check-nodkey-broker + ``` + + It must be able to read `/etc/nodkey/broker.toml` and + `/etc/nodkey/known_hosts` (mode `0644`, as in section 7), but not the + credentials. + +2. Get the recipe from the release of the nodkey version that the host + runs, and check it as [Verify a download](releasing.md#verify-a-download) + says. Never take the scripts from the config repo: they run as root. + + ```sh + VERSION=0.7.0 + cd "$(mktemp -d)" + U="https://github.com/code6x/nodkey/releases/download/v$VERSION" + curl -fLO "$U/nodkey_${VERSION}_pullgate.tar.gz" + curl -fLO "$U/SHA256SUMS" + curl -fLO "$U/nodkey_${VERSION}.sigstore.json" + sha256sum -c --ignore-missing SHA256SUMS + gh attestation verify "nodkey_${VERSION}_pullgate.tar.gz" \ + --bundle "nodkey_${VERSION}.sigstore.json" \ + --repo code6x/nodkey \ + --cert-identity https://github.com/code6x/nodkey/.github/workflows/release.yml@refs/heads/main \ + --deny-self-hosted-runners \ + --format json --jq '.[].verificationResult.statement.subject[].name' | + grep -Fx "nodkey_${VERSION}_pullgate.tar.gz" + ``` + + `sha256sum` must print `nodkey__pullgate.tar.gz: OK`, and the + last command must print the name of the file. If a check fails, stop. + Then install the files, owned by root: + + ```sh + install -d -o root -g root -m 0755 /usr/local/lib/nodkey-pullgate + tar --no-same-owner -xzf "nodkey_${VERSION}_pullgate.tar.gz" -C /usr/local/lib/nodkey-pullgate + chown -R root:root /usr/local/lib/nodkey-pullgate + chmod -R go-w /usr/local/lib/nodkey-pullgate + ls -l /usr/local/lib/nodkey-pullgate + ``` + +3. Make a read-only deploy key for the config repo, and pin the host key + of the git server, as pullgate's README says, with the names + `/etc/pullgate/nodkey-broker.key` and + `/etc/pullgate/nodkey-broker.known_hosts`. Leave write access off for + the deploy key. + +4. Write the instance config from the example. Change `host` and the + `[repo]` section: + + ```sh + install -m 0600 -o root -g root /usr/local/lib/nodkey-pullgate/nodkey-broker.toml /etc/pullgate/nodkey-broker.toml + ``` + +5. Start it: + + ```sh + systemctl daemon-reload + systemctl enable --now pullgate-apply@nodkey-broker.path pullgate-run@nodkey-broker.timer + systemctl enable --now pullgate-check@nodkey-broker.path + ``` + +6. Send `/check nodkey-broker` to pullgate's bot. The first question + compares the commit with the live config. If the repo was made from + the live files, the change list says `no change`. Tap **Apply**, and + check the result: `pullgate status nodkey-broker`, and + `journalctl -t pullgate`. + +When you update nodkey, get the recipe of the new version too, with +step 2. + +#### Read the question + +The question shows the commit, and then the change list of +`summary.sh`. The lines with `⚠` come first. Read each one before you +tap **Apply**: + +- `⚠ needs a broker restart: all grants end`: the change needs a + restart, so every open grant ends, and requests that wait for a tap + are denied. +- A line about a secret names its item by the last 8 characters of its + ID, such as `item …3f9a1c2e`. Check that it is the item you made + ([See what a config change means](#see-what-a-config-change-means)). +- `⚠ the live config in /etc/nodkey is not the applied commit`: someone + changed `/etc/nodkey` by hand. Find out why before you apply. +- **The change list must be complete.** pullgate lists the commits and + the changed files before the summary, and cuts a long message from + the end. If the question has no "Summary:" block, or says that lines + are left out, tap **Skip**. Then read the full list on the host: + `nodkey broker config-diff /etc/nodkey DIR`, with the commit's two + files in DIR. Keep commits few and their subjects short. +- **A ⚠ line that you did not ask for: tap Skip.** A writer of the repo + can add a line to a change that you expect. Each `⚠` line widens + access, or is a change that nodkey does not know. + +pullgate reports each apply, rollback and failure on its bot. After a +failed rollback, the instance is paused: the broker host needs root. +The way back by hand: `pullgate status nodkey-broker --pause` first, so +that pullgate does not undo the edit; then edit `/etc/nodkey` and run +`nodkey broker reload` or `systemctl restart nodkey`; then commit the +same files to the repo, and run `pullgate status nodkey-broker --resume`. + ## 11. A client on the agent machine ### The key and the config of the client