Skip to content

MST/2 reader: use the binary META page surface for large-directory navigation #58

Description

@Ivanbeethoven

Context

The MST/2 client reads directory listings only through the JSON directory
endpoint with cursor pagination
(SnapshotReader::directory_page / walk_dir in src/snapshot/reader.rs).

The binary metadata surface exists and is already consumed by the client —
Mst2Client::metadata_pages in src/snapshot/frames.rs decodes META frames and
verifies every page id — but nothing in the reader uses it. The server side has
been shown to be equivalent for both representations (a directory's
directory_root page id and the JSON listing describe the same set), so this is
a pure efficiency and fidelity question, not a correctness gap.

Spec 11 §6 asks for: JSON directory for small/aggregate nodes, lookup_many
for deep paths, and per-page or batched META frames for very large directories
or when the proof budget is short — all writing into one metadata store so
JSON and binary cannot disagree.

Scope

  1. Add a page-based enumeration path to SnapshotReader that walks MTP2 pages
    through metadata_pages (route-based descent), producing the same
    SnapshotFile list as the JSON walk.
  2. Choose the representation per directory (spec 11 §6): small directories keep
    the JSON page (fewer round trips, already cached), large or cursor-heavy ones
    use META pages. The threshold must be a local policy constant, not a protocol
    change.
  3. Single metadata store: both representations must populate the same cache
    (pages by page_id, entries by path) so a directory read one way is not
    re-read the other way, and so no two attribute views can diverge.
  4. Keep the change invisible to FUSE: the mounted tree must be byte-identical
    and its listing identical (including ordering and ../. semantics).

Acceptance

  • A test asserting the two paths produce identical manifests for the same
    view: same set, same order, same digests, same sizes (this is the equivalence
    the server side already documents, now asserted from the client).
  • A test on a directory with more than 128 entries (so the page tree is a
    branch, not a leaf) exercising route descent, with the same equality
    assertion.
  • A large-directory benchmark showing the page path issues fewer round trips
    than the cursor walk, or an explicit note saying it does not and why.
  • With metadata_pages absent from capabilities, the reader must fall back to
    the JSON walk without error (capability gating stays honest).

References

  • Spec 11 §6, spec 04 §§5–8, spec 05 §5 (route/branch semantics)
  • src/snapshot/reader.rs, src/snapshot/frames.rs (metadata_pages),
    src/snapshot/types.rs (DirectoryResponse)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions