Skip to content

format-compatibility.md

Latest commit

 

History

History
203 lines (168 loc) · 10.7 KB

format-compatibility.md

File metadata and controls

203 lines (168 loc) · 10.7 KB
Raw

Format Compatibility Policy

This document defines the rules Midge will use to freeze and verify on-disk compatibility on the path to 1.0.

Covered Formats

The compatibility policy applies to:

  • WAL frames and records
  • manifest files and manifest journal
  • intent-log persistence format
  • SST file layout and footer/version identifiers
  • cloud WAL publication catalog and epoch-scoped object-key layout
  • cloud lease metadata authority and immutable control-metadata generations

Current Local Format

Database FORMAT version 7 is current. Version 5 named the WAL's history with a segment ledger, wal/LEDGER (#842): a checksummed journal that records which wal/{id:020}.wal files are history, their order, the next segment id and the retirement point. Segments carry their final name from creation; there is no active wal.log and no rename on rotation, and recovery never lists wal/ to decide what to replay.

Version 6 (#843) links the segments themselves and moves the writer epoch out of the records:

  • Segment format v2. Every segment starts with a 128-byte header (magic MWSEGHDR, version 2, segment id, writer epoch, holder, first sequence, the link of the segment before it, and a CRC32C). A sealed segment ends with a footer frame naming its own link: its id, last sequence and an xxh3-128 digest of every byte before the footer, with its record count and per-family maximum sequences. Nothing may follow the footer. Recovery admits a segment only when its header names the segment the ledger (or, for cloud objects, the catalog) expects, its link matches the segment before it and its epoch does not go backwards. A cloud object must be a sealed v2 segment whose header id and epoch match its catalog entry.
  • Record format v2. A WAL record no longer carries a writer epoch; the segment header does. Transaction frames carry their transaction's column family, and every operation in a transaction batch must belong to it.
  • LEDGER v2. Members record their header facts and, once sealed, their footer facts; the ledger keeps the link of the last retired segment as its anchor and a durable sequence reservation, so a sequence the engine handed out is never handed out again after a crash.

Version 7 (#844) chains the cloud WAL publication catalog and drops the metadata compatibility paths older databases needed:

  • Catalog v2. The cloud WAL publication catalog is a chain like the ledger, with an anchor, a next segment id and a sequence floor. Its document is format version 2; see the catalog description below.
  • No compatibility encodings. Every manifest journal record must carry its edit envelope; a record without one is corruption. Manifest key bounds decode only as hex strings; the byte-array form written before FORMAT 4 is rejected.
  • Removed metadata. The manifest no longer has next_wal_seq or cloud_checkpoint, and the SetCloudCheckpoint journal record is gone. Maintenance no longer re-summarizes SSTs whose manifest entry lacks complete key bounds.

Version 7 is an intentional pre-1.0 break with no in-place upgrade and no migration: current binaries reject database FORMAT versions 1 through 6 with CompatibilityError, and leave the marker untouched. Cloud databases are rejected the same way, because their hydrated metadata carries the marker, and a publication catalog of any other format version fails with CompatibilityError and neither catalog copy changes. FORMAT 7 requires SST V4; current binaries also reject SST versions 1 through 3 and unknown future versions. There is no legacy decoder or best-effort fallback in the V4 reader.

SST V4 has the following integrity and compatibility boundaries:

  • an 84-byte fixed footer containing its version, encoded footer length, self-identifying magic, and CRC32C
  • exact, non-overlapping block handles validated against the file length
  • a mandatory five-byte compression trailer (codec plus CRC32C) on every persisted block
  • explicit TTL presence, so None and Some(u64::MAX) remain distinct
  • codec identifiers 0 through 3 only: raw, LZ4, Zstd level 3, and Zstd level 9

Removed codec identifiers and malformed or unknown identifiers fail with corruption or compatibility errors. The reader never retries such bytes as an uncompressed block.

New writes enforce a 64 MiB encoded-entry admission ceiling, including key and header bytes, and validate range-tombstone endpoints before staging. This does not change SST V4. Historical V4 raw blocks larger than that ceiling remain readable and can be rewritten by budgeted compaction; their output stays raw even if compression is configured. Compaction still requires enough memory for its reserved working buffers. Oversized compressed blocks remain corruption errors under the existing decoded-size limit. See the PR #278 compatibility regressions.

Required Rules For 1.0

  • every persistent format must carry an explicit version identifier or versioned decoding path
  • current builds must distinguish:
    • supported historical format
    • current format
    • unsupported future format
  • unsupported future format must return CompatibilityError
  • corruption and incompatibility must remain distinct error classes

Compatibility Guarantees

Before 1.0

  • no blanket on-disk compatibility promise
  • any format movement must be called out in the changelog and migration guide

The current cloud WAL layout is publication catalog format version 2 (#844). Mere object presence is not authority: recovery reads only entries in the validated publication catalog. Midge stores identical primary and recovery-mirror copies at wal/publication-catalog.v1.json and wal/publication-catalog.v1.mirror.json; the v1 in the object names is not the document's format version. The mirror is used only when the primary is missing or malformed. The pre-v1 segment-only cloud layout is not auto-migrated because it cannot prove whether a late stale-writer upload was accepted before fencing.

The version 2 JSON document is a chain, like the WAL ledger. It contains format_version, the current fencing_epoch, an anchor, a next_segment_id, a seq_floor and segments, oldest first. The anchor is the link of the last segment the catalog retired, or the genesis link. Each entry records its writer epoch, the link of the segment before it (prev), its own link (link: segment id, last sequence and the digest its footer seals), and its object's exact byte length and CRC32C; the epoch-scoped object key derives from the id and epoch. The first entry links the anchor and each later one links the entry before it. Epochs never fall and none passes the fencing epoch. next_segment_id passes the head and never falls, so retiring entries never loses the chain or hands out a published id again. Retirement removes only the oldest entries and moves the anchor; salvage removes only a suffix and raises seq_floor over what it set aside. A cloud object must be the sealed segment its entry names: the same size and CRC32C, and a header and footer naming the entry's epoch, prev and link.

A catalog of any other format version fails startup with CompatibilityError; Midge neither repairs it from the mirror nor converts it, and older binaries cannot read a version 2 catalog. Malformed entries, a broken chain, a future writer epoch, or WAL objects without the required catalog fail startup explicitly; salvage mode does not invent publication authority. If both catalog copies are invalid, startup fails closed. Metadata bootstrap refuses any catalog that is not pristine: a catalog whose entries have all retired still holds history.

Provider-backed cloud metadata now uses lease document version 2. Its committed descriptor names the complete set of mirrored local control files under metadata/generations/, with exact lengths and CRC32C checksums. The lease CAS publishes the descriptor; mutable metadata/ copies and unreferenced uploads do not grant manifest recovery authority. The cloud DDL registry remains a separate conditional authority. Its version 2 wire format records the active lease epoch and omits the legacy required epoch field, so older writers cannot parse or rewrite it. A successor conditionally fences the registry before recovery; a predecessor write that wins first is incorporated before serving. The metadata/authority-initialized.v1 marker starts pending and becomes active only after the first lease is stored. Loss of a lease with an active marker is a fail-closed condition. A version 2 lease document intentionally cannot be parsed by older binaries, which prevents an old writer from renewing the new lease.

Legacy provider-backed cloud databases require offline logical export and import into a new empty prefix as described in the cloud setup guide. Current binaries reject legacy lease documents and uncommitted legacy metadata. Binary rollback against the new prefix is unsupported; keep the original prefix as the rollback copy.

At 1.0 and Within 1.x

  • patch upgrades must preserve supported on-disk compatibility
  • minor upgrades must preserve supported on-disk compatibility unless a verified migration step is explicitly required and documented
  • incompatible format changes require a major release

Golden Fixture Requirements

Before 1.0, Midge should maintain fixtures for:

  • opening current code on prior released data
  • verifying prior released data in strict mode
  • recovering from prior released WAL/manifests
  • rejecting synthetic future-version data with CompatibilityError

Upgrade and Rollback Rules

  • upgrade support must be explicit per release
  • rollback support must be explicit per release
  • if rollback is unsupported, docs must say so and provide export/import or backup/restore guidance
  • midge verify should be part of the documented pre-upgrade and post-upgrade workflow

The FORMAT 3/SST V4 transition has no in-place upgrade and no binary rollback. Before installing a FORMAT 3 binary, use the old binary to open the old database and export its logical column-family key/value contents through the public API. Import that logical export into a newly created FORMAT 3 database. Applications that use TTL must also preserve enough application metadata to reconstruct each remaining expiration; the public scan result intentionally does not expose internal persisted expiration timestamps. Keep the original database as the rollback copy. Once a FORMAT 3 binary creates or writes the new database, an older binary must not open it.

Fixture Coverage

Repository gates reject the checked-in populated FORMAT 3/SST V4 fixture, a prior FORMAT 2 fixture and a synthetic future FORMAT 8 fixture with CompatibilityError, without rewriting their markers. Format changes must update these fixtures and their CI assertions together.