Extend #12 so that the entire protocol is CBOR (proposal)

[plaidfinch]

In the course of implementing #12, we converted the internal serialized representation of Rumors messages to CBOR. However, we still have a bespoke wire format which can only be parsed by Rumors; what if, we didn't have that?

It would be nice if off-the-shelf tools could be bolted onto the Rumors wire format to debug and inspect a gossip session; for example, one could imagine a tracing adapter which parses protocol messages and translates them into tracing's own vocabulary. More pedestrian, our own protocol snapshot tests (of which there are a great many!) could cease to be opaque hexadecimal, and be legible, and human-auditable!

This and other protocol introspection tooling would be much easier to create if the entire wire protocol was a uniform, standardized thing, rather than the set of specific and somewhat arbitrary choices I've made thus far.

What changes

We're almost there! Here's what it would look like to fully transform the protocol (per stream) into a CBOR sequence (table estimates by Claude, checked by me — I think this seems accurate):

Layer	Today	With CBOR	Recurring cost
Frame signal	one dense byte (stream × state, 17 × 10 codes)	unsigned int item	+1 byte for codes ≥ 24 (most)
Frame	signal ‖ raw body	small array [signal, body…]	+1 byte array header
Record framing	u32 BE record header	tag 63 ("embedded CBOR sequence in a byte string"): 63(bstr(version ‖ payload))	≈ 0 (tag 2B + bstr header 1–5B vs flat 4B; often equal, −1 for small records)
Record body	CBOR bstr(version) ‖ CBOR(payload) — already a sequence	unchanged, now inside the tag-63 bstr	0
Run length	u32 BE	bstr header arithmetic	≈ 0
Query child listing	raw (radix ‖ 24-byte hash)*	alternating array [radix, h'…', …] (+2 B/child) or map {radix: hash} (+3 B/child)	the one hot cost: ballpark +3–4% on digest-dominated dispute traffic; note that CBOR maps are mandated ascending order, which coincides elegantly with our representation invariant
Greeting	fixed-offset block + frames	text-keyed map ({"network": …, "version": …, "listing": …, "set_len": …, "max_version_bytes": …})	few dozen bytes, once per session
Preamble magic	6 raw bytes	CBOR self-described tag 55799 (0xd9d9f7) opening the control stream, then protocol version/intent as ints, network as bstr	once per session
Stream open label	epoch byte ‖ index byte	two leading int items	+0–2 bytes per stream
Epilogue marker	one byte	int item	0

If we're using the CBOR magic for our format, we should also embed a new protocol-magic field in the greeting, to mark this as a rumors CBOR stream.

I would call this format diff "negligible" relative to payload sizes; we just approved a shift from 16 to 24 byte Merkle hashes, and that moves the needle far more than this — and the whole founding principle of Rumors is "you're not hyper-bandwidth-constrained". We should be frugal, but not to the point of silliness.

Opaque atoms

In this plan, Version and Party atoms stay opaque, but could optionally be given a CBOR Tag, which would allow us to uniquely pick them out and apply specialized semantic rendering to them, without needing to parse an entire bespoke protocol. We might even consider registering our Version and Party types with the First-Come, First-Served CBOR Tag Registry, to avoid future conflict — this appears to be a somewhat lightweight process.

Bookmarks

In this changeover, we might consider making the stored bookmark format CBOR as well, so the whole file parses as CBOR, not just its payload. Sketch: the file opens with the self-described tag and carries the format version, the integrity hash, and the payload as items, with the hashed region an embedded byte string (tag 24, "encoded CBOR data item"), so "the bytes the hash covers" is a well-defined CBOR-visible region rather than an offset convention:

55799( [ format_version: int, integrity: bstr, payload: 24(bstr(map)) ] )

FormatError can be unaltered here, I think: BadMagic becomes "not self-described CBOR / wrong shape", VersionMismatch and HashMismatch are unchanged in meaning, Truncated becomes a sequence truncation. This is a format-version bump, and because the CBOR self-describing tag is not the current bookmark magic number, we don't have to worry about backcompat.

Introspection hook on Peer

Capturing the messages during gossip could be done by third-party tooling which instruments the established streams; this is part of the point here. However, we might also like to be able to install a hook to a Peer which instruments its protocol messages. The leading use cases in mind are those above: legible snapshot tests, and tracing support. The idea would be that you install a handler once for the Peer, and the handler satisfies some trait which says how to create a per-gossip-session scoped handler, which itself knows how to create a per-stream scoped handler, which itself finally knows how to take a single protocol message (as CBOR bytes) and do ... well, whatever with it. This would mean that a tracing adapter reduces to a CBOR-to-tracing translator, plus appropriate scopes; a snapshot capture mechanism is pretty much the same, except that it substitutes generic human-readable CBOR rendering.