fix(graph-db): keep a serving code-shard head through the replay sweep - #3072
Merged
ScriptedAlchemy merged 12 commits intoOct 5, 2026
Merged
Conversation
The superseded-generation replay sweep treated an installed verified head as disposable whenever the code index had deleted its generation file and no snapshot reader pinned it. That holds only for per-generation namespaces, where each generation is the permanent head of a one-member projection. Under the canonical code-shard namespace every generation of one scope publishes into a single shared projection, so the installed head is the serving authority until a successor publish supersedes it. When maintenance deleted a superseded generation before its successor's graph had published (the normal mid-refresh window, since the durable pointer moves at seal while the graph CAS lands later), the sweep retired the projection's live head. The next daemon restart found no verified head, logged code_index_graph_head_recovery_degraded, and replayed the entire sealed generation to repair a projection it had made headless itself — the full-file rebuild issue #1226 reports. Gate the head bypass to non-code-shard namespaces so a serving shared head stays retained like any other installed head: the release answers Retained, retries on the next tick, and reclaims the replay through the ordinary path once the successor's publish supersedes it. Fixes #1226 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
The serving-head retention must not depend on in-memory lease state: after a restart the registry mounts with no installed lease, and the relational verified head is the only authority left. Retain every relational code-shard head outright instead of gating on the lease census, and drop the now-subsumed namespace check in the lease loop. The regression test remounts before the second sweep to pin it. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…-retention' into repair/pr-3072-ci
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
code-shard:namespace. A code-shard head remains the serving authority until a successor publish supersedes it, whether or not an in-memory lease is installed. The sweep therefore returnsRetainedand never reachesretire_verified_head_replay.Motivation
Fixes #1226. After a SIGTERM mid-refresh, the next daemon restart replayed the entire sealed generation instead of resuming publication.
publish_atomicallymoves the durable pointer at seal, but the graph head CAS lands later. During the text-artifact build, the previous generation's replay remains the projection's relational verified head. If maintenance deletes that superseded generation's file and runs the release reconcile during this gap, the sweep treats the shared code-shard head as disposable. The per-generation retention logic excludes candidate heads fromretainedand routes the selection throughretire_verified_head_replay, which deletes the head row.The restart then finds
verified_head = Noneand logscode_index_graph_head_recovery_degradedwith "replay the exact sealed generation to repair its quarantined graph projection". It re-climbs every file. The race between the head CAS and the maintenance tick caused the flaky pass rate.Under the canonical
code-shard:namespace, the code must never retire a head that is still installed. The head is the serving projection and may move only through a successor CAS. After that CAS, the release retries and reclaims the old replay through the ordinary path. The per-generation head-retirement exemption remains unchanged for the legacy layout. In that layout, each generation is the permanent head of a one-member projection, and head retirement is the drain mechanism.Changes
crates/tracedecay-graph-db/src/registry/publication.rs: the heads census inretire_one_code_generation_replayretains code-shard head locators even when they match a deletion candidate, regardless of installed-lease state.crates/tracedecay-graph-db/tests/graph_db_suite/verified_generation_contract/code_graph_layout.rs:sweep_keeps_the_shared_head_while_the_deleted_generation_still_servesassertsRetainedand preserves the head while the deleted generation still serves. It covers both an installed lease and a remount without one, then checks forRetiredthrough the ordinary path after the successor publishes.Test plan
cargo nextest run --workspace --no-fail-fastpasses (rancargo test -p tracedecay-graph-db --features test-helpers --test graph_db_suite: 157 passed, 0 failed)cargo clippyhas no new warnings (cargo clippy -p tracedecay-graph-db --all-targets --features test-helpers -- -D warningsclean)Ok(Retired), head deleted, including the no-lease remount path) and green on this branch; run viabash scripts/require-exact-test.sh cargo test -p tracedecay-graph-db --features test-helpers --test graph_db_suite verified_generation_contract::code_graph_layout::sweep_keeps_the_shared_head_while_the_deleted_generation_still_serves -- --exactcargo fmt --all -- --checkclean;python3 scripts/linux-test-partitions.py checkclean (no new test target)Checklist
CHANGELOG.mdupdated (under[Unreleased]if no version bump).envfiles includedLink to Devin session: https://app.devin.ai/sessions/3c49720ba0f64d208d7aae99d4851be6
Open in Devin Desktop: https://app.devin.ai/desktop/session/3c49720ba0f64d208d7aae99d4851be6?variant=devin
Requested by: @ScriptedAlchemy