Skip to content

Keep raw and typed objects from sharing a storage address - #151

Merged
tonibergholm-codento merged 8 commits into
mainfrom
fix-object-address-collision
Nov 20, 2026
Merged

tonibergholm-codento merged 8 commits into
mainfrom
fix-object-address-collision

Conversation

@tonibergholm-codento

@tonibergholm-codento tonibergholm-codento commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

The object store kept typed objects (blobs, trees, commits) and raw objects (memory segments and manifests) under the same address scheme. A socket client controls prompt bytes and can predict segment bytes, so it could store a prompt blob at the address a segment would use. On the filesystem store, the honest segment write then saw an occupied path and did nothing. On GCS, which uploaded unconditionally, a blob write could overwrite an existing segment. Either way a commit landed that could never be restored.

Now:

  • Raw objects live in their own namespace: raw/<a>/<b>.raw on disk, and <prefix>/raw/<a>/<b>.raw.zst on GCS and in its cache. Reads fall back to the old shared path, with integrity checks, so existing stores keep working.
  • A blob whose bytes are exactly the canonical JSON of a Tree or Commit is rejected (ambiguous_blob), since that is the only way two typed objects can share an address.
  • An occupied address is checked before it is accepted. Equal content is a dedup. A valid object of another kind is refused (address_occupied) and nothing is written. Only a corrupt occupant is replaced. A read error other than not-found fails the put.
  • GCS uploads use ifGenerationMatch=0. A 412 triggers a download and the same check. A replacement is conditional on the observed generation, and the local cache is written only once the bucket holds the verified object.
  • get_raw checks the hash-verified local cache before the network, so a cached snapshot restores while GCS is unreachable.

Cost: a GCS dedup hit is now two requests (412, then a GET) instead of one upload.

Residual: a store written before this change can hold a prompt blob whose bytes are exactly canonical Tree JSON. A later put of that Tree now fails loudly with address_occupied. A rollback to a commit that references such a blob is refused before anything changes. ADR-0018 records the layout, this residual and the repair step.

Proof: new filesystem and GCS tests (a multi-request mock server, since fake-gcs ignores ifGenerationMatch) cover the namespace split, dedup, refusal of valid occupants, healing of corrupt ones, the second-412 error, legacy reads, cache-before-network, and private temp files. They fail without their fixes. cargo test --workspace, the fake-gcs integration suite and the Postgres suites pass.

Reviewed with sol-loop: 5 rounds with gpt-6.2-sol; the remaining finding (kind-specific storage for legacy data) was declined as needing its own layout ADR. Mythos 5.2 security review: approved (round 3).

Created with Claude Opus 5.5 and Claude Sonnet 5.5 in Claude Code.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7d9a7d82-b86e-40ba-b375-53c9a83cc3da
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T08:06:00.976883Z 91ef7dc New commits
🔒 Security Review ✅ Completed 2026-11-20T08:09:33.826395Z 91ef7dc New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87f6e089ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/agentic-core/src/store.rs Resolved
Comment thread crates/agentic-core/src/store.rs Resolved
Comment thread crates/agentic-core/src/gcs_store.rs Resolved
Typed objects are stored as JSON envelopes and raw objects (memory segments and manifests) as bare bytes, but both were addressed in one shared namespace. A blob whose content equals a raw object's bytes could therefore occupy that address first, after which the honest raw write was skipped and the later read failed its integrity check, leaving a commit that could not be restored. On GCS a later upload could also overwrite an existing object.

Raw objects now live under a raw/ sub-namespace in both stores, with a fallback read of the legacy shared path. Blobs that are the canonical encoding of a tree or commit are rejected with a typed AmbiguousBlob error, mapped to a validation error on the wire. GCS uploads are conditional on the object not existing (HTTP 412 counts as already stored) and the local cache is written only after the upload succeeds.
The first fix stopped new collisions but still accepted an occupied address without looking at it, so an object squatted before the fix could win forever and the local cache would hide it. A put now reads what is already stored and compares it with what is being written. The same content is a dedup hit, anything else is replaced: atomically on disk, and on GCS with a precondition on the generation that was just observed, where losing that race is an error. The GCS cache is written only once the bucket is known to hold the verified object.

Legacy raw reads now accept a blob envelope of the same bytes, so replacing a squatter cannot destroy a legacy segment. GCS raw objects get a distinct .raw.zst suffix so a store with prefix p and one with prefix p/raw cannot share a name, and get_raw consults the verified local cache under both names before any network request.
The prefix was derived from the clock alone, and parallel tests that start within the same clock tick shared one. Two tests that store the same payload then raced on one object, and one of them failed intermittently. A per-call counter keeps each test's objects separate.
Replacing every occupant that differs from the object being written could still destroy a valid legacy object, for example a blob whose bytes are the canonical encoding of a tree. Both stores now classify the occupant as the matching read would: the same content is a dedup hit, a valid object of another kind fails the put with a typed AddressOccupied error (a validation error on the wire) and nothing is written or cached, and only an undecodable or integrity-failing occupant is replaced.

Filesystem raw objects now use a distinct .raw suffix so stores rooted at p and p/raw cannot share a file. The GCS metadata lookup for a missing generation is removed: replacing a corrupt object requires the generation reported by the download, and fails clearly without it.
…wn temp file

The filesystem store replaced any occupant it failed to read, so a permission error on an existing object was handled like corruption and the object was overwritten. Only bytes that were read but fail to decode or verify count as corrupt now; any other read failure fails the put and leaves the file alone.

Writes also shared one temp name per object path, which two stores with nested roots could collide on. Each write now creates its own temp file in the target directory, named with the process id and a counter, and removes it if the write fails.
An AlreadyExists on the temp file belonged to another writer, and the
cleanup removed it. Retry with the next name and clean up only a file
this call created.
… the namespace decision

A store written before this change can hold a prompt, tool or model blob whose bytes are canonical Tree or Commit JSON. Rollback re-stages those blobs when it forward-records, and the new ambiguity check rejects them, but only after migrations, memory restore and prompt writes had changed live state. Rollback now loads and checks everything it will re-stage right after validating the target, dry runs included, and refuses with AmbiguousBlob before any side effect.

The design is recorded in ADR-0018. GcsObjectStore::has now checks both cache namespaces before any network request. The temp-name collision test for FsObjectStore::write_at takes an injectable name counter, so it forces the collision instead of depending on the shared counter.
@tonibergholm-codento
tonibergholm-codento force-pushed the fix-object-address-collision branch from 66250f2 to 962a516 Compare November 20, 2026 08:03
@tonibergholm-codento
tonibergholm-codento merged commit 1e6bd9f into main Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant