Skip to content

CORE (signing sitting): chain checkpoints and inclusion proofs #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
githubscum merged 1 commit into from
Oct 28, 2026
Merged

CORE (signing sitting): chain checkpoints and inclusion proofs #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions proposals/chain-checkpoints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Chain checkpoints and inclusion proofs (WO-CARTRIDGE-SEAL part B)

Status: CORE, queued for the signing sitting. Patch: `proposals/chain-checkpoints.patch`,
generated by `node proposals/make-chain-checkpoints-patch.mjs` (never typed, KNOWN-LIMITS 80).
Sources of the three new core files live in `proposals/chain-checkpoints/` until the sitting.
Rehearsal: `test/chain-checkpoints.test.js`.

## What it adds

Today a cartridge carries the chain head, and checking it needs the whole chain. `verify`,
`loadChain` and queries read from genesis every time. This patch adds:

- **Segments.** Every 1024 entries (measured below) the store can seal a segment: a signed
checkpoint over the RFC 9162 Merkle tree of every entry so far. `chain.jsonl` is never
rewritten. Checkpoints live in a sidecar, `receipts/checkpoints.jsonl`.
- **Signed checkpoints in the published format.** C2SP `tlog-checkpoint` notes (origin, tree
size, base64 root), signed as C2SP `signed-note` Ed25519 (type 0x01) with the chain key. Key
name = origin = `lotor.local/chain/<first 16 hex of SHA-256(pubkey)>`. No extension lines.
- **Linking.** Each checkpoint is linked to the previous one the standard way: the newer tree
extends the older one, and `proveConsistency()` produces the RFC 9162 consistency proof a
stranger checks with `verifyConsistencyProof()`.
- **Incremental verify.** `store.verifyIncremental()` starts from the newest checkpoint, or from
one the caller already trusts (`{ trusted: <note> }`), and reads only the entries after it.
It also checks every later checkpoint against the tree it recomputes, so a fork after a kept
checkpoint shows. `store.verify()` (full chain) is unchanged; `store.verifySealed()` is the full
read of the chain plus every checkpoint.
- **Inclusion proofs.** `store.proveInclusion(seq) -> { leaf, index, size, path, checkpoint }`
and the pure `verifyInclusionProof(proof, publicKey)` in `src/chain/checkpoint.js`, which needs
only the proof and the chain's public key.
- **Head checkpoint on demand.** `store.seal({ atHead: true })` also seals the partial segment at
the head, so a receipt can be proven right away (a prediction signed before its outcome needs
this). The next segment seal still ends on a multiple of 1024.
- **verifyRange.** `src/chain/index.js` gains `verifyRange(entries, key, { startSeq, prevHash })`;
`verifyChain` becomes `verifyRange` from genesis. Same checks, same order, same reason strings;
the rehearsal asserts deep equality of results against today's live `verifyChain` on seven
chains (clean, payload tamper, sig flip, broken link, seq skip, genesis offset, empty).

Leaves are the existing entry hashes (`SHA-256(0x00 || 32-byte entry hash)`). The entry hash
already commits to seq, timestamp, nonce, prevHash and payload, so a verifier needs no JSON
canonicalisation and a cartridge can cite a receipt by hash.

## Preflight (existing solutions)

- **Formats:** C2SP tlog-checkpoint and signed-note (read 2026-10-26, editor's copies dated
2026-10-07), RFC 9162 section 2.1 hashing, inclusion (2.1.3) and consistency (2.1.4) proofs.
Nothing invented. The C2SP signature line starts with U+2014, which this repo bans from its
text; the code builds it with `String.fromCharCode(0x2014)`, and the rehearsal asserts the
patch carries no U+2014 byte.
- **tlog-tiles / Trillian Tessera:** tiles is an HTTP layout for serving a log's tree to the
public, and Tessera is the Go library that writes it. Lotor has no server and its log is a
local file, so neither is adopted here. The tree and checkpoint formats are the ones tiles
serves, so a tiles export can be added later without changing anything signed.
- **npm libraries:** none adopted. Candidates found on 2026-10-26: `ctjs` (last published 2018),
and several 2026 release-candidate packages with no track record. The algorithms are about 150
lines of RFC text, and this code lands in protected core, where a dependency is supply-chain
surface the gate cannot see. Correctness is pinned instead: the RFC 6962 reference roots for
sizes 1..8, the C2SP signed-note spec example (vkey `example.com/foo+530d903a+...`), and an
exhaustive round-trip of every inclusion and consistency proof up to tree size 40.

## Choosing N (measured 2026-10-26, node 24.18, Windows 11, this machine)

`node proposals/chain-checkpoints/measure-n.mjs` over a 20,000-entry chain with ~640-byte
lines, two runs (a third run while writing this: 33.4 / 80.0 / 2.9 / 0.75 us):

| cost per entry | run 1 | run 2 |
|---|---|---|
| append (sign) | 33.8 us | 34.0 us |
| verifyChain (hash + Ed25519 verify + links) | 81.2 us | 78.1 us |
| Merkle leaf + tree hashing | 2.9 us | 2.8 us |
| JSON.parse of a line | 0.75 us | 0.73 us |

Verification dominates by ~30x, so N sets the worst-case work an incremental verify does past the
newest checkpoint: N - 1 entries at ~80 us. A budget of about 100 ms gives N <= 1250. N = 1024
(power of two, so a segment ending on a multiple of N is a perfect subtree of the RFC tree) puts
the worst case at ~82 ms. A full verify of 40,000 entries is ~3.2 s by comparison. Sealing one
segment holds the chain lock for about that 82 ms; the lock is taken per segment, never across a
migration, because the lock goes stale after 30 s and appends give up after 5 s (KNOWN-LIMITS 7).

## Measured: inclusion proofs in a cartridge (NOT built here)

The non-core follow-on (the cartridge carries proofs for the receipts it cites) is not in this
patch. Measured on the 40,000-entry rehearsal chain: path 16 hashes; the signed note is 219 B.
A compact binary encoding (index u32, size u32, path, root 32, signature 64; the origin and key
are derivable from the pubkey the manifest already carries) is 616 B for the first proof and
516 B for each further proof against the same checkpoint. Against the 2,808 B useful PAP budget
that is at most 5 proofs with an empty spine, about 3 next to a 1 KB compressed spine, before
sealed-mode overhead. The path grows by 32 B each time the chain doubles (20 hashes at ~1M
entries). Proofs for nearby receipts share upper nodes; deduplicating them is not done.

## Sitting instructions

1. `git apply --check proposals/chain-checkpoints.patch` (clean on `337358f`).
2. Apply with every target named on the command line (KNOWN-LIMITS 82), for example one
signature for one command:
`git apply --include=src/chain/index.js --include=src/chain/merkle.js --include=src/chain/checkpoint.js --include=src/store/index.js --include=src/store/checkpoints.js proposals/chain-checkpoints.patch`
3. `npm run limits-pin -- --stamp`, then `npm test`. In the rehearsal the patched tree ran the
full suite with the cases un-todo'd; see the PR body for the count.
4. Append the KNOWN-LIMITS entries below with the next free numbers, then re-stamp.

## Draft KNOWN-LIMITS entries (numbered at landing)

**Checkpoints are signed by the chain key, so they bind a history only for someone who kept
one.** The key that signs entries also signs checkpoints, and it is plaintext at rest (limits 8
and 48). Its holder can rewrite the chain, re-seal, and `verifySealed()` passes on the result.
What catches the rewrite is an earlier checkpoint kept elsewhere: `verifyIncremental({ trusted })`
reports a fork, and no consistency proof from the kept checkpoint exists. A cosigner or witness
(C2SP tlog-cosignature) or an external anchor would remove the need to keep one; neither is in
this patch, and OpenTimestamps anchoring is out of scope until the notary is back.

**Incremental verify does not read sealed segments.** From the newest checkpoint, an edit inside
an older sealed segment is out of sight until a full read (`verifySealed()` or `store.verify()`).
That is the trade that makes it incremental, and the rehearsal pins it as documented behavior.

**Nothing seals automatically.** No hook or CLI calls `store.seal()` in this patch. Until one does
(session-end hook, which is core, or `pap-export`, which is not), a chain has checkpoints only when
something asks for them. Sealing inside the append path was rejected: it would add work and a new
failure mode to the unattended hook path.

Not limits, recorded so nobody rediscovers them: sealing refuses a chain line that does not parse,
where `loadChain` silently drops it, so a chain with an inserted garbage line can pass
`store.verify()` and still refuse to seal. Re-sealing from scratch yields byte-identical notes
(Ed25519 is deterministic and the note text is fixed by size and root), so losing the sidecar
loses nothing a stranger kept.
Loading