Repository navigation
Description
Status: done
Part of #3141 (workstream 3 of #3000).
Depends on #3142.
TL;DR
Goal: The engine manifest releases/<id>.json of obc data grows with the number of steps, not with the number of files, and collection deletes in batches.
Problem: The engine manifest lists every output file, input file, source file and named file (6.5 MB today, 150 MB or more at the target coverage, estimated). The follower, the review, rollback and collection parse it whole. Collection deletes one key per request.
Decision: See #3141, decisions 4 and 5. Engine manifest format 2: each list is an object of its own, and the manifest holds its size and SHA-256. Release::load reads the lists where they are needed; the follower reads only the list of the named files. Collection deletes up to 1,000 keys per request, the lists of a dropped release last.
Systems: data-engine, planner-map-baking (catalog.rs verify)
Contracts: specs/obc-data.md
Size: L (a breaking change of a stored manifest format)
Plan
The format. Engine manifest format 2. A list is an object of its own: compact JSON, the keys of each object in byte order, the same items as today, at objects/<sha256> of the target and in the store. The manifest names each list as {"size", "sha256"}:
- an output has
listinstead offilesandinputs: the list object is{"files", "inputs"}with the items of today; - a source has
listinstead offiles; namedis the reference of the list of the named files.
pointers stay in the manifest. The ID is the SHA-256 of the manifest, as today.
Steps.
host/obc-data/src/release.rs:FORMAT = 2.Output,FetchedandReleasehold aListreference{size, sha256}instead of the inline lists.Release::parsechecks only the manifest. A new typeFilesholds the loaded lists:Release::load(&self, read: impl Fn(&List) -> Result<Vec<u8>, String>) -> Result<Files, String>reads each list, checks its size and SHA-256, and runs the checks ofRelease::checkthat need the lists.files,objectsandclient_filesmove toFiles. A manifest of format 1 parses into the same types: its inline lists are encoded as list objects in memory, andloadreads them from there. Addtests/manifests/format-2.json.- Publish (
release::publish): write the lists to the store when the manifest is made, and upload them with the other objects, before the manifest. host/obc-data/src/serve.rsserved: read only the list of the named files, and find the plannerrelease.jsonin it. The follower does not load the other lists.- The review, rollback, overview and
catalog.rsverify(output.files,release.objects()): useRelease::load. host/obc-data/src/collection.rs: keep the lists of the kept releases and the files that they name. Delete the other files in batches of up to 1,000 keys, then the manifests of the dropped releases, then their lists. So a collection that stops can always load the manifests that remain. AddTarget::remove_many(&[String]): on R2 it callsBucket::delete, with the reasons of today inremoved.jsonl; on a folder it removes each key. Check the lock before each batch.specs/obc-data.md: Release manifest format 2, the lists, and the collection in batches.- Tests:
release.rs: a manifest of format 2 names its lists, the ID changes when a list changes, and a manifest of format 1 still reads.collection.rs: a collection keeps the lists of the kept releases, deletes in batches, and resumes after it stops between the files and the lists.
Done when
- The pull request passes the suites that
obc test affected --base origin/developselects, andobc exec cargo clippy -p obc-data -p obc-data-steps --all-targets -- -D warnings. - The measurements and checks of "Done when" in Manifests: small release manifests, per-cell offline catalogs and file lists as objects #3141 run on the one re-bake, after A and B merge.
Generated by Claude Code
Activity