Skip to content

Manifests B: engine manifest file lists as objects, collection in batches #3143

Closed

Description

@timohueser

Status: done
Part of #3141 (workstream 3 of #3000).

Depends on #3142.

TL;DR

Goal: The engine manifest releases/<id>.json of obc data grows with the number of steps, not with the number of files, and collection deletes in batches.
Problem: The engine manifest lists every output file, input file, source file and named file (6.5 MB today, 150 MB or more at the target coverage, estimated). The follower, the review, rollback and collection parse it whole. Collection deletes one key per request.
Decision: See #3141, decisions 4 and 5. Engine manifest format 2: each list is an object of its own, and the manifest holds its size and SHA-256. Release::load reads the lists where they are needed; the follower reads only the list of the named files. Collection deletes up to 1,000 keys per request, the lists of a dropped release last.
Systems: data-engine, planner-map-baking (catalog.rs verify)
Contracts: specs/obc-data.md
Size: L (a breaking change of a stored manifest format)

Plan

The format. Engine manifest format 2. A list is an object of its own: compact JSON, the keys of each object in byte order, the same items as today, at objects/<sha256> of the target and in the store. The manifest names each list as {"size", "sha256"}:

  • an output has list instead of files and inputs: the list object is {"files", "inputs"} with the items of today;
  • a source has list instead of files;
  • named is the reference of the list of the named files.

pointers stay in the manifest. The ID is the SHA-256 of the manifest, as today.

Steps.

  1. host/obc-data/src/release.rs: FORMAT = 2. Output, Fetched and Release hold a List reference {size, sha256} instead of the inline lists. Release::parse checks only the manifest. A new type Files holds the loaded lists: Release::load(&self, read: impl Fn(&List) -> Result<Vec<u8>, String>) -> Result<Files, String> reads each list, checks its size and SHA-256, and runs the checks of Release::check that need the lists. files, objects and client_files move to Files. A manifest of format 1 parses into the same types: its inline lists are encoded as list objects in memory, and load reads them from there. Add tests/manifests/format-2.json.
  2. Publish (release::publish): write the lists to the store when the manifest is made, and upload them with the other objects, before the manifest.
  3. host/obc-data/src/serve.rs served: read only the list of the named files, and find the planner release.json in it. The follower does not load the other lists.
  4. The review, rollback, overview and catalog.rs verify (output.files, release.objects()): use Release::load.
  5. host/obc-data/src/collection.rs: keep the lists of the kept releases and the files that they name. Delete the other files in batches of up to 1,000 keys, then the manifests of the dropped releases, then their lists. So a collection that stops can always load the manifests that remain. Add Target::remove_many(&[String]): on R2 it calls Bucket::delete, with the reasons of today in removed.jsonl; on a folder it removes each key. Check the lock before each batch.
  6. specs/obc-data.md: Release manifest format 2, the lists, and the collection in batches.
  7. Tests: release.rs: a manifest of format 2 names its lists, the ID changes when a list changes, and a manifest of format 1 still reads. collection.rs: a collection keeps the lists of the kept releases, deletes in batches, and resumes after it stops between the files and the lists.

Done when


Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions