Repository navigation
feat(windows): MSI self-update via stable bootstrap and SYSTEM updater - #252
swarit-stepsecurity wants to merge 118 commits into
Conversation
Point(name) is an inlinable no-op unless built with selfupdate_faults. Tagged builds act on DMG_FAULT (merged with the link-time Baked value): kill, err, hang, diskfull, exit<N>, with optional @user/@System qualifiers. DMG_FAULT_TRACE records every checkpoint reached.
…ecords Layout paths rooted at an install root and a data root; strict x.y.z versions; schema-1 state.json with corrupt-as-missing Load and atomic Save (temp + fsync + MoveFileEx REPLACE_EXISTING|WRITE_THROUGH, or rename + dir fsync) with <prefix>.state-temp/-renamed fault checkpoints; non-blocking LockFileEx/flock update lock; run records capped at 10 with bad-run classification. Kill-at-checkpoint subprocess tests and fuzzers.
Windows: create the directory with, or re-assert, owner SYSTEM and a protected DACL (SYSTEM F, Administrators F, Users RX, inherited by children) via native security APIs, reporting whether drift was fixed. Rejects files and reparse points. Other platforms only create the dir.
…s and version manifest Port the latest-binary client, GitHub-then-mirror download and SSHSIG verifier from step-security#217 for the MSI updater: native arch via IsWow64Process2, env then WinHTTP machine proxy, strict x.y.z targets, signatures checked before download and sha256 after. Version dirs get a manifest so staged versions can be re-verified.
The MSI installs the launcher a second time as stepsecurity-dev-machine-guard-bootstrap.exe. Under that name it resolves the self-update state.json pointer (current -> previous -> seed), launches versions\<v>\task.exe with run-record env, picks the SYSTEM updater with a seed fallback after 3 consecutive failures, and adopts the MSI seed into versions\ without ever failing the MSI. No network or crypto code. The launcher exits 70 when started by the bootstrap and unable to start the agent, and falls back to a sibling agent.exe in version dirs. updatestate gains the frozen pointer reader, the run-chain resolver and a field-preserving state patch.
…ner with our own WinVerifyTrust (revocation off) on Windows plus a portable PKCS#7 read of the PE signature for the leaf subject and countersignature timestamp. SamePublisher pins candidates to the running updater's own signer subject; an unsigned dev build refuses updates unless a selfupdate_testkeys build pins a subject.
On an MSI bootstrap install, install points both scan tasks at `bootstrap.exe run send-telemetry` and (re)creates the SYSTEM task "StepSecurity Dev Machine Guard (Update)" from a full task XML: hourly with a 10 min random delay plus at boot after 15 min, StartWhenAvailable, battery allowed, 30 min limit, IgnoreNew. The registration is verified. Uninstall of a bootstrap install deletes all three tasks, waits up to 60 s for the scan mutex, terminates DMG processes under the install root, and removes versions\, state\ and RunningVersion. Non-bootstrap uninstalls keep their behaviour and never touch the update task.
Product.wxs installs stepsecurity-dev-machine-guard-bootstrap.exe and
seed\<version>\{agent.exe, task.exe, dmg-linux} in place of the top-level
agent and launcher. A deferred SYSTEM adopt-seed CA (Return=ignore) runs
on install, upgrade and repair; the install CA now runs on repair too
(M-7); the uninstall CA is skipped during a major upgrade and can't block
removal (M-6, M-19).
AUTOUPDATE (default 1) is passed to configure, which gains
--auto-update=<bool> and writes auto_update into the machine config.
msi-smoke asserts the layout, the versions/state ACLs, all three tasks,
a bootstrap run, a real SYSTEM update tick, repair, an N to N+1 upgrade
without downgrade, and a clean uninstall.
One tick of S0-S7: lock, sweep and GC (tolerating in-use dirs), ACL re-assert, probation health with rollback and quarantine (permanent, or seven days for an inconclusive probation), target check every 6 h or when a probation closes, verified staging into versions\<v>, SYSTEM smoke test, mutex-guarded switch with deferral reporting. Events are written to a state.json outbox in the same atomic write as the change they report and retried until the backend takes them.
Update events go to run-status as status=update with retries; 4xx drops the event, anything else stays in the updater's outbox. The SYSTEM updater resolves the device id from the hardware and rewrites the device-id file. Scans mark the bootstrap's run record once run-status started is accepted and report update_pointer_invalid when the bootstrap fell back. Adds the agent.prestart checkpoint, configure --auto-update and the auto_update/update_mirror_url/lag/cooldown config fields.
Overlapping launches (hourly and logon tasks on wake) updated whichever record was newest, so the run that scanned could stay open forever and another run could take its verdict. Records now carry the bootstrap's run_id and MarkRunStarted/FinalizeRun match it; the agent reads STEPSEC_RUN_ID.
The previous updater runs during every probation and rewrote state.json from its own struct, erasing whatever a newer version had added. State, probation, quarantine entries and updater now carry unknown keys through Load and Save verbatim.
A valid state that omitted or nulled them loaded with nil maps, so the first write into them would panic.
An unknown @qualifier, an unknown action or a colon-bearing checkpoint without an action left the rule silently unmatched, so a typo in a fault test passed without injecting anything. Only in selfupdate_faults builds.
…ndows CreateFile without FILE_FLAG_BACKUP_SEMANTICS fails on a directory with access denied, so Load returned an I/O error instead of corrupt and the updater skipped its rebuild path.
…vent fields Each update event keeps the execution_id minted when it was queued, so only its own retries reuse it. Events agent-api would refuse (no install_channel, rollback without update_from) are not posted.
A clean run of each scenario (update, rollback) under DMG_FAULT_TRACE lists the checkpoints of every tick; each traced checkpoint is then hit with kill, err, hang and diskfull in a subprocess, followed by clean ticks, asserting convergence and its event, hygiene, no flip-flop and the frozen bootstrap read against an in-process fake API. A state corruption matrix covers the updater side of the bootstrap cases. The matrix found that a rolled-back version's dir survived until the next tick, so GC now also runs right after a rollback. Runs on Linux and windows-latest.
A second os=linux call could resolve a different version if the release gate moved between calls. dmg-linux now uses linux_checksum and signed_linux_checksum from the windows response; missing fields fail verification.
… backend During an outage the primary updater exits 1 three times and the fallback would then report a broken updater that was never broken. The fallback now reports it only on a tick where its own latest-binary call succeeded.
…/win-msi-self-update-boot Brings in the core updater (C1-C4) so the bootstrap builds against its updatestate API. Conflicts in cli/config (--auto-update) and tests.yml resolved in favour of core; the bootstrap is adapted in the merge so it compiles: release file names (updatestate.AgentFile/LauncherFile), run records keyed by run_id (STEPSEC_RUN_ID), unknown-field-preserving Save instead of the local patch helper, and a manifest for the adopted seed.
…te=false The seed holds stepsecurity-dev-machine-guard.exe, stepsecurity-dev-machine-guard-task.exe and dmg-linux, the same names as versions\<v>\. AUTOUPDATE=0 runs configure --auto-update=false on install, upgrade and repair; the default writes nothing. msi-smoke waits only for msiexec itself with a bound, dumps the log and DMG processes on a hang, checks the adopted seed's manifest, and uploads logs when cancelled too.
…ate specs in drivers A malformed DMG_FAULT/Baked panicked with exit 2, the same code the exit2 action and a real Go panic use, so a typo could pass for a crashing version (a Bad run). It now exits 64 after writing 'faultinject: malformed spec:' to stderr. Validate (all builds) lets test drivers reject a spec before launching; the fault drivers and the kill matrix use it and fail on the marker.
A record with no run_id, or one already in the file, could never be matched by its own launch and would stay open (or be closed by another).
Parallel cases named their Windows mutex from the clock; two created in one clock tick shared it, so one saw a held mutex and deferred its switch.
…heck The case cleared the signature first, so the empty-signature check rejected it and the sha256 shape check was never exercised.
A mirror or release URL answering 302 to http:// was followed and the asset fetched in cleartext. Every updater request (latest-binary, GitHub, mirror, events) now fails on a downgrade redirect.
ProxyFromEnvironment returns nil both for 'no env proxy' and for a host excluded by NO_PROXY, and both fell through to the machine proxy. An env proxy config now decides alone; without one, NO_PROXY hosts go direct before the machine proxy and its bypass list apply.
…elease that ships it
|
Ready for review. Pre-merge checklist:
Testing summary: unit + trace-driven kill matrix (Linux + windows-latest), msi-smoke (install/repair/upgrade/forced-failure rollback/uninstall, no-reboot assertions), and the real-Windows upgrade-safety catalogue in integration-test (all shards green, nothing masked). |
|
Additional release prerequisite: the updater verifies |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved output-bounding, process-containment, installation-path, test-isolation, and CI failure-propagation issues affect update reliability.
5 open issues
What changed in this PR
This PR adds self-update support for Windows MSI installations, allowing agents to follow tenant-selected releases through a stable bootstrap and SYSTEM updater.
Changes:
- Adds versioned installation state, seed adoption, signature verification, and update-event reporting.
- Integrates bootstrap-based scheduled scans, process cleanup, and sleep-aware watchdogs.
- Adds fault-injection, concurrency, platform, and fuzz coverage, plus Intune self-update configuration.
| File | Description |
|---|---|
| tests/test_fuzz_go.sh | Includes updater-state fuzz targets. |
| packaging/windows/intune/uninstall.cmd | Prevents automatic uninstall reboots. |
| packaging/windows/intune/install.cmd | Prevents reboots and documents AUTOUPDATE forwarding. |
| internal/winproc/release.go | Polls for file-handle release. |
| internal/winproc/release_windows.go | Stops installation processes and checks held files. |
| internal/winproc/release_windows_test.go | Tests termination and file release. |
| internal/winproc/release_other.go | Provides non-Windows stubs. |
| internal/updatestate/version.go | Parses and compares strict release versions. |
| internal/updatestate/version_test.go | Tests version validation and ordering. |
| internal/updatestate/unknownfields.go | Preserves unknown state fields. |
| internal/updatestate/seed.go | Copies, verifies, and adopts MSI seeds. |
| internal/updatestate/seed_test.go | Tests publisher checks and copy integrity. |
| internal/updatestate/runlock_test.go | Tests concurrent run-record updates. |
| internal/updatestate/owner.go | Records bootstrap process ownership. |
| internal/updatestate/owner_windows.go | Checks Windows process identity and liveness. |
| internal/updatestate/owner_other.go | Provides Unix liveness checks. |
| internal/updatestate/open_windows.go | Opens shared files without following reparse points. |
| internal/updatestate/open_other.go | Opens files without following final symlinks. |
| internal/updatestate/msiowned.go | Identifies MSI-owned files and helper location. |
| internal/updatestate/manifest.go | Writes manifests and verifies version contents. |
| internal/updatestate/manifest_test.go | Tests integrity checks and MSI file ownership. |
| internal/updatestate/lock.go | Defines persistent exclusive file locks. |
| internal/updatestate/lock_windows.go | Implements Windows locking. |
| internal/updatestate/lock_test.go | Tests exclusivity and process-death release. |
| internal/updatestate/lock_other.go | Implements Unix locking. |
| internal/updatestate/layout.go | Defines self-update paths. |
| internal/updatestate/layout_test.go | Tests paths and staging directories. |
| internal/updatestate/fuzz_test.go | Fuzzes state, records, and versions. |
| internal/updatestate/crash_test.go | Tests atomic state writes under termination. |
| internal/updatestate/crash_faults_test.go | Exercises injected state-write failures. |
| internal/updatestate/atomic.go | Implements atomic writes and temporary-file cleanup. |
| internal/updatestate/atomic_windows.go | Replaces files using MoveFileEx. |
| internal/updatestate/atomic_other.go | Renames files and syncs parent directories. |
| internal/updatestate/adopt_snapshot.go | Stores MSI adoption rollback snapshots. |
| internal/updatestate/acl.go | Rejects redirected or non-directory ACL targets. |
| internal/updatestate/acl_windows_test.go | Tests protected directory permissions. |
| internal/updatestate/acl_other.go | Provides portable directory creation. |
| internal/updatestate/acl_other_test.go | Tests directory validation. |
| internal/telemetry/update_event.go | Posts update events and marks accepted scans. |
| internal/telemetry/telemetry.go | Connects scan health and panic reporting. |
| internal/telemetry/run_status.go | Returns whether the backend accepted status. |
| internal/telemetry/panic_faults_test.go | Exercises panic and hang health classification. |
| internal/telemetry/exitcode.go | Maps recovered panics to crash exits. |
| internal/selfupdate/testkeys_on.go | Enables test signing and timing overrides. |
| internal/selfupdate/testkeys_off.go | Disables test overrides in release builds. |
| internal/selfupdate/system_windows_test.go | Tests config permissions and free-space queries. |
| internal/selfupdate/system_other.go | Provides portable system helpers. |
| internal/selfupdate/sshsig_test.go | Tests SSH signature verification. |
| internal/selfupdate/sshsig_sign_test.go | Generates test SSH signatures. |
| internal/selfupdate/smoke.go | Runs and validates candidate self-checks. |
| internal/selfupdate/smoke_windows_test.go | Tests smoke-process tree termination. |
| internal/selfupdate/smoke_test.go | Tests self-check output and deadlines. |
| internal/selfupdate/proxy_windows.go | Reads the Windows machine proxy. |
| internal/selfupdate/proxy_other.go | Provides a non-Windows proxy stub. |
| internal/selfupdate/main_test.go | Supplies staged-agent subprocess fixtures. |
| internal/selfupdate/knobs.go | Defines updater timing constants. |
| internal/selfupdate/fakeapi_test.go | Supplies fake release and telemetry endpoints. |
| internal/selfupdate/arch.go | Exposes native architecture detection. |
| internal/selfupdate/arch_windows.go | Detects native Windows architecture. |
| internal/selfupdate/arch_windows_test.go | Tests architecture mapping and proxy access. |
| internal/selfupdate/arch_other.go | Uses runtime architecture off Windows. |
| internal/schtasks/selfupdate_windows.go | Adds process cleanup and registry removal. |
| internal/schtasks/selfupdate_windows_test.go | Tests cleanup and scan-mutex waiting. |
| internal/schtasks/selfupdate_other.go | Provides non-Windows scheduler stubs. |
| internal/schtasks/schtasks.go | Routes MSI tasks through the bootstrap. |
| internal/launcher/stopformsi_test.go | Tests MSI helper refresh and stop dispatch. |
| internal/launcher/spawn.go | Adds child results and awake-time deadlines. |
| internal/launcher/spawn_windows.go | Adds hidden spawning and job containment. |
| internal/launcher/spawn_windows_test.go | Tests deadline-driven process-tree cleanup. |
| internal/launcher/spawn_other.go | Provides portable child spawning. |
| internal/launcher/spawn_other_test.go | Tests launch errors and deadlines. |
| internal/launcher/runs_concurrency_test.go | Tests concurrent bootstrap record writes. |
| internal/launcher/runningversion_windows.go | Writes running-version registry inventory. |
| internal/launcher/runningversion_windows_test.go | Tests running-version registry writes. |
| internal/launcher/runningversion_other.go | Provides a registry no-op. |
| internal/launcher/job_windows_test.go | Adapts job tests to shared launcher APIs. |
| internal/launcher/bootstrap_faults_test.go | Tests adoption and launch fault checkpoints. |
| internal/launcher/adopt_rollback_test.go | Tests MSI adoption rollback behavior. |
| internal/faultinject/spec.go | Parses and validates fault specifications. |
| internal/faultinject/spec_test.go | Tests fault-specification validation. |
| internal/faultinject/point_on.go | Implements tagged fault injection and tracing. |
| internal/faultinject/point_off.go | Makes release checkpoints no-ops. |
| internal/faultinject/point_off_test.go | Tests disabled fault injection. |
| internal/faultinject/os_windows.go | Adds Windows fault helpers. |
| internal/faultinject/os_other.go | Adds Unix fault helpers. |
| internal/faultinject/faultinject.go | Documents fault injection and shared errors. |
| internal/faultinject/diskfull_test.go | Tests disk-full error wrapping. |
| internal/device/system_id.go | Resolves and caches SYSTEM device identity. |
| internal/device/system_id_test.go | Tests identity trust and fallback behavior. |
| internal/config/config.go | Adds self-update configuration fields. |
| internal/config/config_test.go | Tests update configuration decoding. |
| internal/cli/cli.go | Parses and documents auto-update configuration. |
| internal/cli/cli_test.go | Tests auto-update flag parsing. |
| internal/awakeclock/awakeclock.go | Implements awake-time deadlines. |
| internal/awakeclock/awakeclock_windows.go | Reads Windows time excluding sleep. |
| internal/awakeclock/awakeclock_test.go | Tests deadline and sleep behavior. |
| internal/awakeclock/awakeclock_other.go | Uses portable monotonic elapsed time. |
| internal/authenticode/verify_windows.go | Reads signer identity from validated trust data. |
| internal/authenticode/verify_other.go | Reports unsupported verification off Windows. |
| internal/authenticode/verify_other_test.go | Tests unsupported-platform behavior. |
| internal/authenticode/testkeys_on.go | Enables test publisher pinning. |
| internal/authenticode/testkeys_off.go | Disables publisher overrides in release builds. |
| internal/authenticode/authenticode.go | Enforces signature and publisher requirements. |
| cmd/stepsecurity-dev-machine-guard/watchdog_windows.go | Excludes sleep from scan watchdog timing. |
| cmd/stepsecurity-dev-machine-guard/main.go | Integrates updater verbs and crash exit codes. |
| cmd/stepsecurity-dev-machine-guard-task/main.go | Dispatches bootstrap mode and shared job helpers. |
| CHANGELOG.md | Documents Windows MSI self-update support. |
| .github/workflows/tests.yml | Adds platform and fault-injection test jobs. |
| .github/workflows/fuzz.yml | Adds updater-state fuzz coverage. |
Review effort: Balanced
| cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: CreateNoWindow} | ||
| if err := cmd.Start(); err != nil { | ||
| return SpawnResult{Err: err} | ||
| } | ||
| job, jerr := NewKillOnCloseJob() |
| type limitedBuffer struct { | ||
| bytes.Buffer | ||
| max int | ||
| } | ||
|
|
||
| func (b *limitedBuffer) Write(p []byte) (int, error) { | ||
| if room := b.max - b.Len(); room > 0 { | ||
| if len(p) > room { | ||
| b.Buffer.Write(p[:room]) | ||
| } else { | ||
| b.Buffer.Write(p) | ||
| } | ||
| } | ||
| return len(p), nil | ||
| } |
| go test -race -count=1 -tags selfupdate_faults,selfupdate_testkeys ./internal/faultinject/ ./internal/updatestate/ ./internal/selfupdate/ ./internal/authenticode/ | ||
| go test -race -count=1 -tags selfupdate_faults,selfupdate_testkeys -run 'Test(ScanPanicIsABadRun|ScanHangAfterStartedIsABadRun)' ./internal/telemetry/ | ||
| - name: Authenticode against test-signed binaries (verbose, shows skips) | ||
| run: go test -count=1 -v -run 'TestVerify|TestForgedBag|TestTrustData|TestSameSeedPublisher' ./internal/authenticode/ | ||
| - name: Test launcher bootstrap mode and self-update tasks | ||
| run: | | ||
| go test -race -count=1 ./internal/launcher/ | ||
| go test -race -count=1 -tags selfupdate_faults ./internal/launcher/ |
| go func() { | ||
| _ = Run(mock, progress.NewLogger(progress.LevelError), &cli.Config{SearchDirs: []string{t.TempDir()}}) | ||
| }() |
| func MSILayout() Layout { | ||
| return Layout{InstallRoot: DefaultInstallRoot, DataRoot: DefaultDataRoot} |
The embedded bytes.Buffer exposed ReadFrom, so os/exec's io.Copy bypassed the bounded Write.
The default pwsh shell fails a step only on the last native command's exit code.
Fails on the current SpawnHidden: a slow job setup leaves the child (and a grandchild it spawns) outside the job, and a setup failure runs the child uncontained.
SpawnHidden now creates the job first and starts the child suspended, assigns it, then resumes it (winproc.StartInJob, shared with the smoke runner). A setup failure terminates and reaps the child and returns an error instead of running it uncontained.
The hang test left a live Run (scan lock, stderr capture, signal handler) in the test process; it now asserts the lock is free after the watchdog kill.
The updater, self-check device-id cache and fallback report hard-coded C:\Program Files\StepSecurity. They now resolve the root from their own path by the bootstrap's rule (updatestate.BootstrapRoot, shared with schtasks), refuse a root without the bootstrap or one users can write (owner and DACL of the root and its ancestors), and put runs under the ProgramData known folder.
C:\ and C:\Program Files are owned by TrustedInstaller; the wrong SID refused every MSI root (TestResolveInstallRoot_ProtectedCustomRoot, red on CI at 08bc9a7).
…s if it gives up A scan the failed install started can still run from versions\<seed> when the rollback runs; the rename failed and the dir was silently kept (msi-smoke run 39915353203). The rollback now stops processes under the dir and retries for up to 30s, logging loudly if it gives up.
An MSI upgrade with BOOTSTRAPFILE replaced config.json, losing an AUTOUPDATE=0 opt-out (SCCM e2e B1). Keys the file carries override, keys it omits are kept, and unknown keys survive every save.
…helper adopt-seed keeps the previous state\msi helper as .prev until the MSI commits; adopt-seed --rollback puts it back (SCCM e2e B2).
The normal and the forced-failure upgrade now ship a bootstrap file without auto_update: the opt-out survives both, the key rotates on the good one, and the failed one's rollback restores the MSI helper.
…ade/uninstall behaviour
Helpers dispatched from TestMain never arm -test.timeout and -test.run helpers start without it, so a helper whose test died slept or hung on (up to an hour, or forever for the hang fault). internal/proctest: Command ties a child to its test (process-group kill on end/deadline), Guard makes a helper exit at that deadline or when its parent exits, and Main kills children still running just before -test.timeout.
TestStress_ConcurrentHookInvocations ran 64 RunHook calls that each forked real host probes. Under -race on macOS a fork child can spin in the race runtime before exec; the hook hung, the test timed out and the child was orphaned at ~100% CPU (two such cli.test ran for 5 days). Tests now get a mock executor and a temp HOME/USERPROFILE.


What
Windows MSI installs (Intune, SCCM) now keep themselves on the release the tenant selects. Today they only move when IT pushes a new MSI, so fixes reach them last.
state.jsonand runsversions\<current>\, falling back to previous, then to the seed.agent.exe self-update --systemfrom the confirmed version. Update logic lives in the versioned agent, so updater bugs are fixed by self-update too. A broken updater falls back to the previous version's or the seed's updater.Layout
versions\andstate\are SYSTEM-owned with a protected ACL, re-applied on every update run. The updater never touches the MSI's files (bootstrap, seed,AgentVersion); the MSI touchesversions\/state\only throughadopt-seed(copy the seed in, movecurrentforward, never back) and a real uninstall (remove both).AgentVersion(the Intune detection value) stays the MSI version.Safety properties
.staging-*; only a verified, manifest-stamped directory becomesversions\<v>.state.jsonis written atomically with a generation counter, unknown fields are preserved for older and newer readers, and every event is stored in the same write as the change it reports.update_*events go torun-status(status: "update"). Each is saved with the state change it reports; a validation reject (400/422) is dropped, any other failure is retried with a 1 h → 24 h backoff, and the outbox keeps up to 64 events (oldest dropped and counted).current; uninstall removes all three tasks,versions\,state\andRunningVersion.Testing
previous, skipping the re-read after a failed write, dedupe ignoring the event class) turn it red.AUTOUPDATEhandling, and uninstall cleanup.Rollout notes
minWindowsSelfUpdateVersionis set to 1.18.0 (the first release with this updater). A target below it is refused, because it would have no updater.AUTOUPDATE=0: stops taking new targets; the device stays on its current version (possibly a self-updated one) and reportsupdate_disabledwith the target it isn't taking.docs/windows-self-update.mdandpackaging/windows/README.mddescribe the design,AUTOUPDATEand the Intune detection rule.