Skip to content

MST/2 hydration: stream large files into the CAS within a memory budget instead of whole-file buffers #61

Description

@Ivanbeethoven

Context

Hydration currently materialises every file as a whole Vec<u8> in memory
before writing it to the content-addressed store:

  • FetchCoordinator::lead (src/snapshot/coordinator.rs) returns
    Arc<Vec<u8>> for a whole file;
  • DurableStore::hydrate_concurrent (src/snapshot/durable.rs) verifies the
    whole buffer with SHA-256, then writes it atomically;
  • the frame path for large files (SnapshotReader::read_file_frames) assembles
    the complete file from chunks and re-hashes it as one buffer.

For a multi-gigabyte file this means the resident set grows with the file size,
and a single large file can exhaust the process even though the transfer itself
is chunked. Spec 11 §9 sets an explicit memory budget that must cover wire and
raw buffers, proof pages, and pending requests — the current design has no way to
respect it. Spec 07 §7 also separates per-chunk verification from the whole-file
DURABLE_COMPLETE gate, which is exactly what a streaming implementation can do
without holding the file.

Scope

  1. Streaming hydration for large files. Fetch chunks in order, verify each
    against its authenticated leaf digest, and append to the store's temporaries
    while updating a running whole-file SHA-256; only then rename into the CAS and
    journal the file. Memory stays bounded by the chunk size plus the digest
    state, independent of file size.
  2. Keep the two gates distinct and honest. Per-chunk verification is what a
    range read can prove; the whole-file digest remains the DURABLE_COMPLETE
    gate. An interrupted streaming hydration must leave the marker absent and be
    resumable (the existing journal/resume rules apply unchanged).
  3. Bound the other buffers too: object batches (<=8 MiB by protocol) and
    proof pages must be accounted against the same budget; exceeding it must
    pause or fail with a typed error, never silently grow.
  4. No behavioural change for small files: they keep the whole-object path
    (<=256 KiB), where a single buffer is the right shape.

Acceptance

  • A test hydrating a file much larger than the configured memory budget with the
    peak resident bytes asserted to stay under the budget (measure the process, not
    the file size).
  • A test where the stream is interrupted mid-file: the store has no
    DURABLE_COMPLETE, the partial temporary is not exposed as content, and a
    resume fetches only what is missing.
  • A test that a corrupted chunk aborts the hydration with a typed digest error
    and leaves previously hydrated files intact.
  • The range-read path (src/snapshot/range.rs) and the compose acceptance
    (tests/mst2-e2e) still pass unchanged.

References

  • Spec 11 §§7–9 (hydration queue, durable commit, resource isolation),
    spec 07 §§6–7 (range reads, whole-file promotion)
  • src/snapshot/{coordinator,durable,reader,range}.rs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions