[plotnick]

With a YubiKey for authentication, require just one touch per login instead of (up to) one touch per sled per login. Running distributed jobs on Dublin (4 sleds) sometimes required multiple (up to 4) touches. This will matter more as we run on bigger racks (e.g., Dogfood & Colo).

The problem turned out to be a race between client polls and gossiped IdentityRequest::Login messages. It could bite any time the client sent requests to two or more sleds in quick succession, such as sush job start -w -T '*' (run a job on every sled and poll for status). If a Login was received via gossip before the client request arrived, the credentials were accepted and everything was fine. But if the request arrived first, the server didn't recognize the login, and so returned 401, forcing the client to re-authenticate:

Before:

sequenceDiagram
    participant C as Client
    participant A as Sled A
    participant B as Sled B
    C->>A: request
    A-->>C: 401, nonce
    Note over C: touch
    C->>A: login
    A--)B: gossip login
    C->>B: request
    B-->>C: 401, nonce
    Note over C: touch again
    C->>B: login

Loading

Our solution here is to use login receipts, which are verifiable signatures over login requests. They allow one server to trust credentials that don't match its nonce, but which are accompanied by proof that they did match a peer's. Receipts are signed by a vouched, ephemeral server key, very much like the proxy TLS key already in use. Servers can verify each other's receipts by checking signatures, which means that the user only has to sign a login challenge once:

After:

sequenceDiagram
    participant C as Client
    participant A as Sled A
    participant B as Sled B
    C->>A: request
    A-->>C: 401, nonce
    Note over C: touch
    C->>A: login
    A-->>C: receipt
    C->>B: request
    B-->>C: 401, nonce
    C->>B: login, receipt

Loading

Receipts are never written to disk or gossiped among peers; they are only returned by the /iam endpoint of the HTTP API, and may optionally be included in a client's POST to the same endpoint. They are not secret or sensitive. They carry complete cert chains, which each server must verify against the platform roots.

The constraints that led to this design were:

• The gossip network must stay quiet when no jobs are running.
• We must not depend on wall-clock time (but local monotonic timers are fine).
• No call-and-return gossip requests between sleds.
• No waiting for gossip to arrive.
• The RoT must not sign anything at every login (because IPCC is slow and shared).

Receipts as implemented here are a somewhat heavyweight mechanism, with a new server key and a cert chain on every receipt). However, with them we no longer need to gossip logins (6dd786d), and if we were (in a follow-up) to gossip server keys and certs, we could shrink receipts down to just a signature. Signing gossip messages and job output also become possible.

[plotnick]

Superseded by #108.