usb-ready releases the drive before ssh-keygen-boot has run, so Reflash mounts sda2 mid-mkfs

[eliasbakken]

usb-ready reports the drive as ready in the window before
ssh-keygen-boot.service has been evaluated, so Reflash mounts /dev/sda2
while expand-usb's mkfs.ext4 is still running. The mount fails with
bad superblock, storage is marked FAILED for that boot, no host keys are
written, and sshd never starts.

Seen on v1.1.0-RC8, on a8 (serial 0482), 2026-09-16 20:41.

What the board logged

Reflash's own log, in order:

[info]  Running expand USB script
[info]  Creating partition on unused space
[info]  -- Server started at 18:40:11 --
[info]  Waiting for /dev/sda2 to appear
[info]  Creating ext4 filesystem            <- mkfs starts
[info]  boot: wait-for-usb       7.36s      <- the wait ends here
[info]  Mounting /dev/sda2 on /mnt/usb as ro
[error] [mount-unmount-usb mounted ro]: exit status 32: mount: /mnt/usb: wrong fs type,
        bad option, bad superblock on /dev/sda2, missing codepage or helper program
[error] USB storage unavailable: exit status 32

The serial console confirms the unit was still running when that happened -
Starting ssh-keygen-boot.service at 20:41:24, with no Finished or Failed
line after it:

20:41:24  Starting ssh-keygen-boot.service - ... keys from USB storage (see #80)...
(no Finished/Failed within the next two minutes)

The superblock was not bad. It did not exist yet.

Why usb-ready said yes

state=$(systemctl show -p ActiveState --value "$UNIT")
cond=$(systemctl show -p ConditionResult --value "$UNIT")
case "$state" in
    active|failed) echo "true" ;;
    *) [ "$cond" = "no" ] && echo "true" || echo "false" ;;
esac

ConditionResult is no both for a unit that was skipped and for one that
has not been evaluated yet, so the fallback written for the skipped case
also fires before the unit starts. Measured with a probe unit
(Type=oneshot, RemainAfterExit=yes, condition that passes):

unit state	ActiveState	ConditionResult	usb-ready says
never started	inactive	no	true (wrong)
ExecStart running	activating	yes	false
finished	active	yes	true

This is the same hazard the comment above that case says was already fixed:

A Type=oneshot unit reads "inactive" both before it runs and after a
Condition skips it, so treating inactive as ready let Reflash mount in the
gap before the unit started - and it would then steal the mount.

Moving the test from ActiveState to ConditionResult did not close it,
because the two states are indistinguishable in both properties.

Suggested discriminator

ExecMainStartTimestamp (or ActiveEnterTimestamp) is empty for a unit that
has never run and set once it has, so "inactive and never started" can be told
from "inactive because skipped":

started=$(systemctl show -p ExecMainStartTimestampMonotonic --value "$UNIT")
case "$state" in
    active|failed) echo "true" ;;
    activating)    echo "false" ;;
    *) if [ "$cond" = "no" ] && [ "${started:-0}" != 0 ]; then echo "true"   # skipped, evaluated
       elif [ "$cond" = "no" ] && [ "${started:-0}" = 0 ]; then echo "false" # not evaluated yet
       else echo "false"; fi ;;
esac

Worth checking against the skip path too: a unit skipped by its condition may
never set that timestamp, in which case the honest test is "has systemd
finished deciding", e.g. waiting for the unit to leave inactive at all with a
bounded timeout before falling back.

Why it matters beyond one boot

The board comes up serving HTTP but with no sshd, which does not look like a
storage fault from the outside. On the CI rig it failed a test run as "did not
come up in Reflash" while /api/get_serial_number was answering 0482
correctly the whole time. The same shape cost an afternoon on a6 earlier that
day before the cause was understood.

Unclear and worth confirming: whether mkfs completes after Reflash gives up
(so the next boot is clean) or is disturbed by the concurrent mount and leaves
/dev/sda2 permanently unformatted. If the latter, note that expand-usb
early-exits on [ -b /dev/sda2 ], so a partition that exists without a
filesystem is never repaired on subsequent boots.

[eliasbakken]

Fixed By MR #148