Context
Hydration currently materialises every file as a whole Vec<u8> in memory
before writing it to the content-addressed store:
FetchCoordinator::lead (src/snapshot/coordinator.rs) returns
Arc<Vec<u8>> for a whole file;
DurableStore::hydrate_concurrent (src/snapshot/durable.rs) verifies the
whole buffer with SHA-256, then writes it atomically;
- the frame path for large files (
SnapshotReader::read_file_frames) assembles
the complete file from chunks and re-hashes it as one buffer.
For a multi-gigabyte file this means the resident set grows with the file size,
and a single large file can exhaust the process even though the transfer itself
is chunked. Spec 11 §9 sets an explicit memory budget that must cover wire and
raw buffers, proof pages, and pending requests — the current design has no way to
respect it. Spec 07 §7 also separates per-chunk verification from the whole-file
DURABLE_COMPLETE gate, which is exactly what a streaming implementation can do
without holding the file.
Scope
- Streaming hydration for large files. Fetch chunks in order, verify each
against its authenticated leaf digest, and append to the store's temporaries
while updating a running whole-file SHA-256; only then rename into the CAS and
journal the file. Memory stays bounded by the chunk size plus the digest
state, independent of file size.
- Keep the two gates distinct and honest. Per-chunk verification is what a
range read can prove; the whole-file digest remains the DURABLE_COMPLETE
gate. An interrupted streaming hydration must leave the marker absent and be
resumable (the existing journal/resume rules apply unchanged).
- Bound the other buffers too: object batches (<=8 MiB by protocol) and
proof pages must be accounted against the same budget; exceeding it must
pause or fail with a typed error, never silently grow.
- No behavioural change for small files: they keep the whole-object path
(<=256 KiB), where a single buffer is the right shape.
Acceptance
- A test hydrating a file much larger than the configured memory budget with the
peak resident bytes asserted to stay under the budget (measure the process, not
the file size).
- A test where the stream is interrupted mid-file: the store has no
DURABLE_COMPLETE, the partial temporary is not exposed as content, and a
resume fetches only what is missing.
- A test that a corrupted chunk aborts the hydration with a typed digest error
and leaves previously hydrated files intact.
- The range-read path (
src/snapshot/range.rs) and the compose acceptance
(tests/mst2-e2e) still pass unchanged.
References
- Spec 11 §§7–9 (hydration queue, durable commit, resource isolation),
spec 07 §§6–7 (range reads, whole-file promotion)
src/snapshot/{coordinator,durable,reader,range}.rs
Context
Hydration currently materialises every file as a whole
Vec<u8>in memorybefore writing it to the content-addressed store:
FetchCoordinator::lead(src/snapshot/coordinator.rs) returnsArc<Vec<u8>>for a whole file;DurableStore::hydrate_concurrent(src/snapshot/durable.rs) verifies thewhole buffer with SHA-256, then writes it atomically;
SnapshotReader::read_file_frames) assemblesthe complete file from chunks and re-hashes it as one buffer.
For a multi-gigabyte file this means the resident set grows with the file size,
and a single large file can exhaust the process even though the transfer itself
is chunked. Spec 11 §9 sets an explicit memory budget that must cover wire and
raw buffers, proof pages, and pending requests — the current design has no way to
respect it. Spec 07 §7 also separates per-chunk verification from the whole-file
DURABLE_COMPLETEgate, which is exactly what a streaming implementation can dowithout holding the file.
Scope
against its authenticated leaf digest, and append to the store's temporaries
while updating a running whole-file SHA-256; only then rename into the CAS and
journal the file. Memory stays bounded by the chunk size plus the digest
state, independent of file size.
range read can prove; the whole-file digest remains the
DURABLE_COMPLETEgate. An interrupted streaming hydration must leave the marker absent and be
resumable (the existing journal/resume rules apply unchanged).
proof pages must be accounted against the same budget; exceeding it must
pause or fail with a typed error, never silently grow.
(<=256 KiB), where a single buffer is the right shape.
Acceptance
peak resident bytes asserted to stay under the budget (measure the process, not
the file size).
DURABLE_COMPLETE, the partial temporary is not exposed as content, and aresume fetches only what is missing.
and leaves previously hydrated files intact.
src/snapshot/range.rs) and the compose acceptance(
tests/mst2-e2e) still pass unchanged.References
spec 07 §§6–7 (range reads, whole-file promotion)
src/snapshot/{coordinator,durable,reader,range}.rs