Repository navigation
Keep raw and typed objects from sharing a storage address - #151
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87f6e089ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Typed objects are stored as JSON envelopes and raw objects (memory segments and manifests) as bare bytes, but both were addressed in one shared namespace. A blob whose content equals a raw object's bytes could therefore occupy that address first, after which the honest raw write was skipped and the later read failed its integrity check, leaving a commit that could not be restored. On GCS a later upload could also overwrite an existing object. Raw objects now live under a raw/ sub-namespace in both stores, with a fallback read of the legacy shared path. Blobs that are the canonical encoding of a tree or commit are rejected with a typed AmbiguousBlob error, mapped to a validation error on the wire. GCS uploads are conditional on the object not existing (HTTP 412 counts as already stored) and the local cache is written only after the upload succeeds.
The first fix stopped new collisions but still accepted an occupied address without looking at it, so an object squatted before the fix could win forever and the local cache would hide it. A put now reads what is already stored and compares it with what is being written. The same content is a dedup hit, anything else is replaced: atomically on disk, and on GCS with a precondition on the generation that was just observed, where losing that race is an error. The GCS cache is written only once the bucket is known to hold the verified object. Legacy raw reads now accept a blob envelope of the same bytes, so replacing a squatter cannot destroy a legacy segment. GCS raw objects get a distinct .raw.zst suffix so a store with prefix p and one with prefix p/raw cannot share a name, and get_raw consults the verified local cache under both names before any network request.
The prefix was derived from the clock alone, and parallel tests that start within the same clock tick shared one. Two tests that store the same payload then raced on one object, and one of them failed intermittently. A per-call counter keeps each test's objects separate.
Replacing every occupant that differs from the object being written could still destroy a valid legacy object, for example a blob whose bytes are the canonical encoding of a tree. Both stores now classify the occupant as the matching read would: the same content is a dedup hit, a valid object of another kind fails the put with a typed AddressOccupied error (a validation error on the wire) and nothing is written or cached, and only an undecodable or integrity-failing occupant is replaced. Filesystem raw objects now use a distinct .raw suffix so stores rooted at p and p/raw cannot share a file. The GCS metadata lookup for a missing generation is removed: replacing a corrupt object requires the generation reported by the download, and fails clearly without it.
…wn temp file The filesystem store replaced any occupant it failed to read, so a permission error on an existing object was handled like corruption and the object was overwritten. Only bytes that were read but fail to decode or verify count as corrupt now; any other read failure fails the put and leaves the file alone. Writes also shared one temp name per object path, which two stores with nested roots could collide on. Each write now creates its own temp file in the target directory, named with the process id and a counter, and removes it if the write fails.
An AlreadyExists on the temp file belonged to another writer, and the cleanup removed it. Retry with the next name and clean up only a file this call created.
… the namespace decision A store written before this change can hold a prompt, tool or model blob whose bytes are canonical Tree or Commit JSON. Rollback re-stages those blobs when it forward-records, and the new ambiguity check rejects them, but only after migrations, memory restore and prompt writes had changed live state. Rollback now loads and checks everything it will re-stage right after validating the target, dry runs included, and refuses with AmbiguousBlob before any side effect. The design is recorded in ADR-0018. GcsObjectStore::has now checks both cache namespaces before any network request. The temp-name collision test for FsObjectStore::write_at takes an injectable name counter, so it forces the collision instead of depending on the shared counter.
66250f2 to
962a516
Compare
The object store kept typed objects (blobs, trees, commits) and raw objects (memory segments and manifests) under the same address scheme. A socket client controls prompt bytes and can predict segment bytes, so it could store a prompt blob at the address a segment would use. On the filesystem store, the honest segment write then saw an occupied path and did nothing. On GCS, which uploaded unconditionally, a blob write could overwrite an existing segment. Either way a commit landed that could never be restored.
Now:
raw/<a>/<b>.rawon disk, and<prefix>/raw/<a>/<b>.raw.zston GCS and in its cache. Reads fall back to the old shared path, with integrity checks, so existing stores keep working.ambiguous_blob), since that is the only way two typed objects can share an address.address_occupied) and nothing is written. Only a corrupt occupant is replaced. A read error other than not-found fails the put.ifGenerationMatch=0. A 412 triggers a download and the same check. A replacement is conditional on the observed generation, and the local cache is written only once the bucket holds the verified object.get_rawchecks the hash-verified local cache before the network, so a cached snapshot restores while GCS is unreachable.Cost: a GCS dedup hit is now two requests (412, then a GET) instead of one upload.
Residual: a store written before this change can hold a prompt blob whose bytes are exactly canonical Tree JSON. A later put of that Tree now fails loudly with
address_occupied. A rollback to a commit that references such a blob is refused before anything changes. ADR-0018 records the layout, this residual and the repair step.Proof: new filesystem and GCS tests (a multi-request mock server, since fake-gcs ignores
ifGenerationMatch) cover the namespace split, dedup, refusal of valid occupants, healing of corrupt ones, the second-412 error, legacy reads, cache-before-network, and private temp files. They fail without their fixes.cargo test --workspace, the fake-gcs integration suite and the Postgres suites pass.Reviewed with sol-loop: 5 rounds with gpt-6.2-sol; the remaining finding (kind-specific storage for legacy data) was declined as needing its own layout ADR. Mythos 5.2 security review: approved (round 3).
Created with Claude Opus 5.5 and Claude Sonnet 5.5 in Claude Code.