Skip to content

fix(graph-db): keep a serving code-shard head through the replay sweep - #3072

Merged
ScriptedAlchemy merged 12 commits into
masterfrom
devin/1791128059-code-shard-head-retention
Oct 5, 2026
Merged

ScriptedAlchemy merged 12 commits into
masterfrom
devin/1791128059-code-shard-head-retention

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • During the superseded-generation replay sweep, retain every relational verified head in a code-shard: namespace. A code-shard head remains the serving authority until a successor publish supersedes it, whether or not an in-memory lease is installed. The sweep therefore returns Retained and never reaches retire_verified_head_replay.
  • Add a contract test for both mid-publish paths: one with an installed lease and one after a remount without a lease. The test also checks that the ordinary path retires the head after the successor publishes.

Motivation

Fixes #1226. After a SIGTERM mid-refresh, the next daemon restart replayed the entire sealed generation instead of resuming publication.

publish_atomically moves the durable pointer at seal, but the graph head CAS lands later. During the text-artifact build, the previous generation's replay remains the projection's relational verified head. If maintenance deletes that superseded generation's file and runs the release reconcile during this gap, the sweep treats the shared code-shard head as disposable. The per-generation retention logic excludes candidate heads from retained and routes the selection through retire_verified_head_replay, which deletes the head row.

The restart then finds verified_head = None and logs code_index_graph_head_recovery_degraded with "replay the exact sealed generation to repair its quarantined graph projection". It re-climbs every file. The race between the head CAS and the maintenance tick caused the flaky pass rate.

Under the canonical code-shard: namespace, the code must never retire a head that is still installed. The head is the serving projection and may move only through a successor CAS. After that CAS, the release retries and reclaims the old replay through the ordinary path. The per-generation head-retirement exemption remains unchanged for the legacy layout. In that layout, each generation is the permanent head of a one-member projection, and head retirement is the drain mechanism.

Changes

  • crates/tracedecay-graph-db/src/registry/publication.rs: the heads census in retire_one_code_generation_replay retains code-shard head locators even when they match a deletion candidate, regardless of installed-lease state.
  • crates/tracedecay-graph-db/tests/graph_db_suite/verified_generation_contract/code_graph_layout.rs: sweep_keeps_the_shared_head_while_the_deleted_generation_still_serves asserts Retained and preserves the head while the deleted generation still serves. It covers both an installed lease and a remount without one, then checks for Retired through the ordinary path after the successor publishes.

Test plan

  • cargo nextest run --workspace --no-fail-fast passes (ran cargo test -p tracedecay-graph-db --features test-helpers --test graph_db_suite: 157 passed, 0 failed)
  • cargo clippy has no new warnings (cargo clippy -p tracedecay-graph-db --all-targets --features test-helpers -- -D warnings clean)
  • New regression test verified red on master (returns Ok(Retired), head deleted, including the no-lease remount path) and green on this branch; run via bash scripts/require-exact-test.sh cargo test -p tracedecay-graph-db --features test-helpers --test graph_db_suite verified_generation_contract::code_graph_layout::sweep_keeps_the_shared_head_while_the_deleted_generation_still_serves -- --exact
  • cargo fmt --all -- --check clean; python3 scripts/linux-test-partitions.py check clean (no new test target)

Checklist

  • CHANGELOG.md updated (under [Unreleased] if no version bump)
  • No secrets, credentials, or .env files included
  • Breaking changes documented (if any)

Link to Devin session: https://app.devin.ai/sessions/3c49720ba0f64d208d7aae99d4851be6
Open in Devin Desktop: https://app.devin.ai/desktop/session/3c49720ba0f64d208d7aae99d4851be6?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

The superseded-generation replay sweep treated an installed verified head
as disposable whenever the code index had deleted its generation file and
no snapshot reader pinned it. That holds only for per-generation
namespaces, where each generation is the permanent head of a one-member
projection. Under the canonical code-shard namespace every generation of
one scope publishes into a single shared projection, so the installed
head is the serving authority until a successor publish supersedes it.

When maintenance deleted a superseded generation before its successor's
graph had published (the normal mid-refresh window, since the durable
pointer moves at seal while the graph CAS lands later), the sweep
retired the projection's live head. The next daemon restart found no
verified head, logged code_index_graph_head_recovery_degraded, and
replayed the entire sealed generation to repair a projection it had
made headless itself — the full-file rebuild issue #1226 reports.

Gate the head bypass to non-code-shard namespaces so a serving shared
head stays retained like any other installed head: the release answers
Retained, retries on the next tick, and reclaims the replay through the
ordinary path once the successor's publish supersedes it.

Fixes #1226

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a417e8f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

ScriptedAlchemy and others added 11 commits October 4, 2026 15:42
The serving-head retention must not depend on in-memory lease state:
after a restart the registry mounts with no installed lease, and the
relational verified head is the only authority left. Retain every
relational code-shard head outright instead of gating on the lease
census, and drop the now-subsumed namespace check in the lease loop.
The regression test remounts before the second sweep to pin it.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@ScriptedAlchemy
ScriptedAlchemy merged commit 51f9c77 into master Oct 5, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code-index: restart after SIGTERM mid-refresh rebuilds every file; 772-file batch misses its receipt

1 participant