Path-scoped context blocks for minutes on a cold index and rescans full history in fallback

[MongLong0214]

What happens

A path-scoped context query can block for minutes when the derived index is absent, and the documented --no-index fallback performs more than one full-history parse for the same query.

Reproduced from current main at 12379345a3d57117d9b2f841754c9c675624925c against a clean checkout of NousResearch/hermes-agent at 4b601931be1cb10094c353e0d344e39f8e2d3b0c:

• repository history: 21,446 commits;
• commits touching the requested path: 1,492;
• cold context --json --limit 1 gateway/run.py: did not finish within 30 seconds;
• explicit cold index --rebuild: 186.10 seconds;
• the same context call with the warm index: 0.43 seconds;
• context --json --no-index --limit 1 gateway/run.py: did not finish within 30 seconds;
• Git's path-scoped trailer projection over the same path: 0.23 seconds.

The warm query returned the expected notes-mirror diagnostic because the clean checkout intentionally had not fetched the notes ref. That diagnostic is unrelated to the latency difference.

Reproduction

From a clean checkout of a repository with a substantial history and no existing CommitLore index:

# Build CommitLore main first, then run these from the target repository.
node <commitlore-checkout>/dist/commitlore.mjs context --json --limit 1 gateway/run.py
node <commitlore-checkout>/dist/commitlore.mjs context --json --no-index --limit 1 gateway/run.py
node <commitlore-checkout>/dist/commitlore.mjs index --rebuild
node <commitlore-checkout>/dist/commitlore.mjs context --json --limit 1 gateway/run.py

The first command takes the cold rebuild path synchronously. The second takes the full-scan fallback. The final command demonstrates that the path query itself is fast once the index exists.

Root cause in current source

src/core/query.ts opens the row source before answering the path query. The indexed branch calls ensureIndex() synchronously, so a missing or invalid derived index rebuilds the entire repository before one path can be answered.

The fallback has a separate amplification:

1. scanTrailers() in src/core/index-db.ts always runs revList(cwd, 'HEAD') and reads commit records for every SHA.
2. It applies matchesQuery() only after all records have been parsed.
3. runQuery() first calls foldStates(source, ...), which fetches lifecycle keys repository-wide so off-path supersessions remain authoritative.
4. It then calls collectRows(source, aliases) for the requested path.
5. With a scan source, each fetch() invokes scanTrailers() independently, so one single-alias path query parses the whole history at least twice.
6. --limit 1 is applied only after lifecycle folding, grouping, merging, filtering, and ordering; it does not bound the cold work.

The lifecycle-wide read is semantically necessary under SPEC §5. The duplicate full-history parsing and the synchronous unbounded rebuild on an edit-time query are the issue.

Impact

context, injection, and commitlore_before_change sit on the before-edit path. A missing, retired, or unusable derived index turns that path into a multi-minute synchronous operation. If the caller enforces a shorter timeout, the rebuild is terminated and the next edit retries from a cold state, so the index may never become warm.

The fallback remains correctness-preserving, but its current cost makes it unsuitable as an automatic recovery path on a repository of this size.

Prior art and distinct scope

• No-index query at 100k regressed from 7.7s to 90s — cause not yet identified #163 established that the complete no-index parser is much slower than the earlier incomplete parser and closed the apparent regression as diagnosed. This report does not dispute that parser accounting.
• Concurrent hooks make half the injections fall back to a full scan, because no SQLite busy timeout is set #420 added a SQLite busy timeout for concurrent cold-start hooks. This reproduction uses one clean process with no pre-existing index or lock contention.

This issue is the remaining cold-path contract: a path-scoped before-change query synchronously rebuilds the complete index, while its fallback parses the complete history more than once.

Expected contract

Keep path scoping, rename following, notes, trust grading, and repository-wide lifecycle semantics unchanged, while making the cold path bounded and non-duplicative:

• one path-scoped query must not independently parse the complete commit corpus once for lifecycle state and again for each path alias;
• a missing or invalid derived index must not make a before-change call block on an unbounded full rebuild;
• interruption or fallback must remain fail-closed and must never turn unknown history into an empty successful answer;
• indexed and fallback answers must remain byte-equivalent for the existing correctness matrix;
• a regression fixture should exercise a substantial history with few CommitLore records and prove both correctness and the cold-path work bound structurally, without a fragile wall-clock assertion.

[MongLong0214]

Fixed on dev in #526. Merged to dev, unreleased — this stays open until a release carries it, because closing against a tag that does not contain the fix is what #525 is about.

What landed:

• one path-scoped query no longer parses the corpus once for lifecycle state and again per alias; the corpus is materialized once and each predicate applied in memory
• a consumer query may catch the index up but never rebuild it. The incremental range is last_indexed_sha..HEAD and is always a subset of what the fallback would read, so taking it is never the slower choice; only the unbounded full rebuild is refused
• opening the index no longer creates the file, so a query on a repository without one leaves nothing behind

Limit: a genuinely cold fallback still reads the whole history once — repository-wide lifecycle folding cannot be scoped to a path without changing what the answer means.

Limit: nothing outside index and init builds the index now, so a repository whose derived file was deleted stays on the scan path until one of them runs.

A 1,024-commit fixture pins both properties structurally — one corpus pass, no index file created, and an index one commit behind caught up rather than abandoned. No wall-clock assertion: a timing test on CI is a flake that gets deleted six weeks later and takes the guarantee with it.