records(export): structured record exports and the export format - #46
Conversation
Deliver Milestone 5 slice 1 (#45; #44 decision 1): finalized versions leave the installation as verifiable archives whose record bytes are the stored canonical bytes, byte for byte. - record_export: units (stored bytes beside a canonical-JSON unit manifest), ZIP archives with stored entries in a fixed order and metadata (deterministic per scope and instant), four scopes (version, record, enrollment, installation), read-rule authorization with export_records for the installation, typed refusals including nothing_to_export, record_exported audit events, and verification from the archive alone with typed findings and predecessor linking. - exports_http: downloads for every scope and the export summary; main: `export verify <archive>` opens no data directory. - web: export buttons on the record page (this version, all versions), the enrollment page, and an installation-export card on Home; downloads surface refusals as errors instead of saved error bodies. - docs: docs/formats/record-export.md (normative), ADR 0014, the RecordExport domain-model entry; zip added without default features. - tests: verbatim bytes, determinism, every verification finding, scope authorization, API delivery, CLI verification; the drafts e2e downloads a version export from the browser and verifies it with the CLI. Refs #44 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e0d2770f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two verifier gaps from review: the declared scope was copied into the report without checking the listed units against it, so a version scope listing another version, several units, or none at all still verified; and the zip reader indexes entries by name, so a name written twice — where extraction tools disagree on which copy they take — was invisible. - scope_findings: no_units, scope_cardinality (a version scope is exactly one unit), unit_outside_scope (version and record scopes); enrollment and installation scopes state nothing the bytes confirm. - duplicate_entry_findings walks the central directory itself (APPNOTE 6.3, ZIP64 fields included) and reports every name written more than once; an unreadable directory is its own finding. - docs/formats/record-export.md and ADR 0014 name both checks. - tests: empty units, mislabeled version scope, foreign record scope, and a hand-assembled container with a duplicated record entry whose genuine copy the reader resolves. Refs #45, #44 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37efc28ec6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
record_export crossed the 1,000-line reorganization threshold (AGENTS.md) once it owned both production and the independent check. Ownership now splits along the format's two sides: - record_export keeps the format vocabulary (constants, Scope, both manifests, unit_path, canonical_json) and production: authorization, row collection, the deterministic archive writer, audit, summary. - export_verify owns verification from the archive alone: findings, per-unit and per-archive reports, the unit and scope checks, the container walk including the central directory. No public behavior changes; the CLI, tests, and format document point at the new owner. Nothing persisted changes. Refs #45, #44 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2571d0a39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The verifier parsed record.json as an arbitrary document and checked six identity members, so a canonical, correctly hashed document that was not a record — only those members, or a mistyped or missing presentation member — still verified. - record_envelope: the typed reading side of ADR 0011's shape (schema 1) and ADR 0013's (schema 2): every named member with its type, no unnamed member anywhere, every nullable member present, and daily_reports present exactly for schema 2. - export_verify reads record.json through it before the identity cross-check; a document that is not an envelope of a known schema is a typed `envelope_invalid` finding. - docs/formats/record-export.md names the check (unit check 4); ADR 0014's verification bullet points at the reader. - tests: a six-member counterfeit with matching hashes, a schema-2 envelope without daily_reports, an unnamed member, a mistyped member, an absent nullable member, an unknown schema — each invalid and nothing else; the same content shaped as schema 1 verifies; the genuine bytes read as the typed envelope they are. Refs #45, #44 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b02e19e698
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two gaps in the new validation surface from review:
- The typed envelope reader accepted "attachments":[{}] because the
element type was an empty struct; both known schemas require the
array empty. The element type is now uninhabited, so no value of any
shape deserializes into it.
- The lineage rule (predecessor null exactly for version 1) is
satisfied trivially by version 0 or a negative number with a
predecessor, and the predecessor lookup then reports "not in export"
without objecting. Identity is positive by the format; a zero or
negative record_id or version_number is the typed finding
identity_out_of_range.
docs/formats/record-export.md names both (unit checks 4 and 7);
tests cover an attachment element and a renumbered version 0 unit whose
only objection is the range check.
Refs #45, #44
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fe27df352
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- The chain recomputation decodes hex case-insensitively, so an uppercase predecessor hash carried consistently through both manifests and the envelope verified for a lone successor and would then fail to link beside its lowercase predecessor. Hash members are 64 lowercase hex characters by the format; a deviation is the typed finding hash_not_canonical (docs: unit check 2). - build_archive consumes its rows so each version's bytes are released once written: the assembled archive is the one copy held to the end, bounding the peak near one copy of the corpus instead of two. Streaming the response itself is #47; ADR 0014's cost note now says what an installation export costs until then. Refs #45, #44, #47 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1c7aa3e81
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Entry order, modification times, and permissions are production rules that make a fresh export deterministic; they carry no record content, and an intact archive stays verified after a tool repacks it. The format document now states this so the verifier's scope is explicit. Refs #45, #44 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
2 participants
Primary Issue
Closes #45
Refs #44 (Milestone 5 design; decision 1 and the slice plan)
Problem And Outcome
Finalized records existed only inside the installation's database: readable and hash-verifiable over the API, but nothing produced a portable artifact whose integrity could be checked anywhere else. After merge, a version, a record (every retained version), an enrollment, or the whole installation exports as a ZIP archive whose
record.jsonentries are the stored canonical bytes, byte for byte, beside canonical-JSON manifests; the same scope exported at the same instant is byte-identical; andconsolebook-server export verify <archive>proves an export consistent with its stated fingerprints from the file alone, opening no data directory. The format is normative indocs/formats/record-export.md, the decisions are ADR 0014.Changes
record_export.rs(production and the format vocabulary):Scope(version, record, enrollment, installation), typedExportRefusal(includingnothing_to_export— an empty scope is refused, never exported as a complete-looking archive), deterministic ZIP writer (stored entries, fixed order, export instant as entry mtime,0644; rows are consumed so each version's bytes are released once written), archive and unit manifests as canonical JSON,record_exportedaudit events with actor and subject and never content.record_envelope.rs: the typed reading side of the record envelope (ADR 0011 schema 1, ADR 0013 schema 2) — every named member with its type,deny_unknown_fieldsthroughout, nullable members required to be present,daily_reportspresent exactly for schema 2,attachmentsan array of an uninhabited element type (no known schema carries attachments). Verification reads through it; packets and rendering will too.export_verify.rs(verification from the archive alone, split out when the module crossed the 1,000-line threshold): typedFindings covering content hash over the bytes, canonical re-serialization, the bytes being a valid envelope of their declared schema, envelope identity cross-check (record.*,instance,canonicalization), lowercase 64-hex hash members, chain hash from the carried predecessor hash, positive identity, lineage shape, predecessor linking within the archive, declared scope against the listed units (no_units,scope_cardinality,unit_outside_scope), unlisted and missing entries, non-canonical manifests, unsupported format versions, and duplicate entry names found by walking the central directory itself (thezipreader collapses duplicates by name).export_recordscapability for the installation.exports_http.rs: downloads for all four scopes plusGET /api/exports/summaryforexport_recordsholders;main.rs:export verify <archive>.export_recordsholders (counts, one-click installation export). Downloads go throughfetchso a refusal surfaces as an error rather than a saved error document.docs/formats/record-export.md(normative),docs/decisions/0014-record-export-format.md, aRecordExportentry indocs/domain-model.md.zip8.6 with default features off (stored entries only; addscrc32fastandtyped-path).Scope
data/exports/(this slice streams and retains nothing, recorded in ADR 0014); importing record exports; streaming the export body instead of assembling it in memory (Stream installation exports instead of assembling the archive in memory #47, raised in review — the peak is bounded near one copy of the corpus here).Verification
tests/record_export.rsproves:record.jsonequals the storedcanonical_bytes; both manifests carry the documented fields and are canonical; two exports at one instant are byte-identical; the record scope links v2 to v1 while a lone v2 reports its predecessor as not in the export; a repacked archive with altered bytes, a swapped manifest hash, a replaced record, a missing entry, an unlisted entry, a pretty-printed manifest, an unknown format version, reordered units, a bad path, a broken lineage, an emptied unit list, a mislabeled version scope, a foreign record scope, and a hand-assembled container with a duplicated entry each produce the named finding; a correctly hashed counterfeit carrying only the identity members, a schema-2 envelope withoutdaily_reports, an unnamed member, a mistyped member, an absent nullable member, an attachment element, and an unknown schema are eachenvelope_invalidwhile the same content shaped as schema 1 verifies; a renumbered version 0 fails on identity alone and an uppercase predecessor hash on hash shape alone; a raw byte flip fails; every scope's authorization and typed refusals; the API's headers, bodies, and error codes; and the CLI's exit codes and output with a never-created--data-dirleft untouched.Review And Merge Notes
finalization::envelope(same member set, same types); the authorization line (read rules for scoped exports,export_recordsfor breadth) recorded in ADR 0014's rejected alternatives; determinism assumptions on thezipwriter (no extended-timestamp extra fields without thetimefeature).record_exportedaudit kind. No migration.🤖 Generated with Claude Code
https://claude.ai/code/session_01DNP3dkn9c84v2YoNwx7KTg
Generated by Claude Code