Problem or outcome
record_export::export_at (#46) reads every finalized version of the scope with fetch_all and then assembles the ZIP archive in a Vec<u8> before the HTTP handler returns it as one body. For the installation scope this holds the corpus in memory while the export is produced. #46 already releases each version's bytes as they are written (rows are consumed by build_archive), so the peak is roughly one copy of the exported bytes plus one row rather than two copies, but a very large finalized history still costs memory in proportion, and the process can be pressured before the download completes.
The outcome is an installation export whose memory use is bounded by a few units, not by the history: rows are read incrementally and the archive is written to the response as it is produced, with the same documented bytes.
Scope
In scope:
- Stream the query (a cursor over
evaluation_version in archive order) into a ZIP writer that writes to the response body as it goes, for every scope or for the installation scope alone.
- Keep the format byte-identical for a given scope and export instant (
docs/formats/record-export.md): stored entries, fixed entry order and metadata, canonical manifests. Note that a non-seekable writer emits data descriptors, which the format document must either permit explicitly or the implementation must avoid (for example by computing sizes and CRCs first, since stored entries have known lengths).
- Decide how a failure part-way through a streamed body is surfaced to the operator (a truncated ZIP does not verify, which is the right failure, but the interface should say what happened).
- Update ADR 0014's cost note once exports stream.
Out of scope:
Acceptance criteria
Expected proof
cargo test --workspace with a test producing a large invented corpus and comparing streamed and buffered bytes.- The usual gates:
cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, web/ check and build if the interface changes.
Related roadmap, decisions, or issues
Problem or outcome
record_export::export_at(#46) reads every finalized version of the scope withfetch_alland then assembles the ZIP archive in aVec<u8>before the HTTP handler returns it as one body. For the installation scope this holds the corpus in memory while the export is produced. #46 already releases each version's bytes as they are written (rows are consumed bybuild_archive), so the peak is roughly one copy of the exported bytes plus one row rather than two copies, but a very large finalized history still costs memory in proportion, and the process can be pressured before the download completes.The outcome is an installation export whose memory use is bounded by a few units, not by the history: rows are read incrementally and the archive is written to the response as it is produced, with the same documented bytes.
Scope
In scope:
evaluation_versionin archive order) into a ZIP writer that writes to the response body as it goes, for every scope or for the installation scope alone.docs/formats/record-export.md): stored entries, fixed entry order and metadata, canonical manifests. Note that a non-seekable writer emits data descriptors, which the format document must either permit explicitly or the implementation must avoid (for example by computing sizes and CRCs first, since stored entries have known lengths).Out of scope:
data/exports/(ADR 0014 records why nothing is retained on disk).Acceptance criteria
export verifyaccepts the streamed archive, and the format document says exactly what changed in the container, if anything.Expected proof
cargo test --workspacewith a test producing a large invented corpus and comparing streamed and buffered bytes.cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,web/check and build if the interface changes.Related roadmap, decisions, or issues
record_export.rs,collectandbuild_archive).