Skip to content

Code lookup preserves source coverage and identifier resolution - #1115

Merged
zlareb1 merged 1 commit into
mainfrom
fix/code-retrieval-correctness
Nov 21, 2026
Merged

zlareb1 merged 1 commit into
mainfrom
fix/code-retrieval-correctness

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Related: #1107

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Code lookup resolves Python import spellings, retains class documentation and data for physical source reads, and finds complete underscore identifiers through the lexical text-index gate.

Why

Retrieval can miss an existing definition when its import spelling omits the physical package module, when compacted class summaries hide the original source, or when the text-index tokenizer splits a stored identifier differently from the query. These changes preserve stored symbol identities and match against retained source and exact lexemes.

Scope and risk

  • Python import forms use the existing bounded short-name candidate read after an exact miss. Only Python source rows participate; complete module matches precede suffixes, while ties and incomplete candidate sets decline. Responses disclose resolved_by: python_module.
  • The chunker retains class and struct source outside method chunks through its existing raw-source chunks, including short documentation and data runs. Class summaries and conservative partial-coverage warnings remain available. Additional chunks can increase embedding work.
  • Lexical retrieval passes a String query to the text-index tokenizer and requires a positive exact score. A query containing an underscore-only token uses the scoped exact-score predicate without the coarse text gate. That case can scan more rows; tenant, namespace, model, path and pool bounds remain intact. The scoring expression is unchanged.

The source-coordinate helpers come from the linked merged fix. This pull request contains the three additional retrieval fixes and their regression coverage. It makes no measured time, cost or task-success claim.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change. The existing string field resolved_by can additionally report python_module.
  • Operator steps: the shared chunk-stage stamp changes across language families. Existing indexes need that stage refreshed before previously omitted class source is readable. Python name forms and lexical token matching work against existing rows. No backfill is performed by this change.
  • Rollback: revert the changes and let the chunk stage refresh. No schema migration is required; stored symbol identities remain intact.

Tests

Ready for independent review. The complete local unit gate and hosted checks are not claimed as passing.

  • Regressions reproduce missing Python package forms, physical class documentation/data, and underscore identifier hits before their respective fixes.
  • Combined ClickHouse analyzer corpus: 12 passed.
  • Combined source-coordinate, lexical and code-index integration checks: 17 passed.
  • Five HTTP probes after a two-file ingest with a fake embedder verify package aliases, an existing definition, cross-file callers, a physical grep line and index identity.
  • make format-check, make lint and make typecheck pass on the combined source.
  • The complete Linux make test-unit target is unverified. A complete unit run had an interpreter-path guard failure and a Lua scaling guard failure. The correct checkout virtualenv passes all 62 creator checks; all seven Lua scaling cases pass in isolation, and the previously failing Lua case also passed in the full rerun. That full rerun was stopped before completion, so no passing full-suite result is claimed.
  • Fresh independent review found no issues on the final head. Its independent Linux run passed 46 focused checks in Python name forms, class source, lexical statements and the actual text-index gate. Hosted checks have not been waited for.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Codex implemented the retrieval fixes and regressions. The validation results state the checks completed on the final source.

  • I can explain every line of this diff and its interaction with retrieval and the extractor stamp.

Checklist

  • Formatting, lint, typing and generated-file checks pass locally.
  • The complete Linux unit and benchmark-test target passes.
  • Combined analyzer and applicable integration checks pass.
  • Fresh independent review passes on the final head.
  • Architecture and operation documentation describe the response value and chunk-stage refresh.
  • No schema, settings, plugin or frontend behavior changes.
  • Public text contains no private benchmark artifacts or deployment details.
  • Commits are signed off.

Retain physical class source outside compacted summaries, bind Python import
spellings to stored definitions, and admit exact underscore lexemes through
the lexical gate without filtering on a SELECT alias. Existing identities and
scope predicates remain intact. Retaining class source refreshes the chunk stage.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 2:11pm UTC

Request Review

@zlareb1
zlareb1 merged commit 75809aa into main Nov 21, 2026
55 of 56 checks passed
@zlareb1
zlareb1 deleted the fix/code-retrieval-correctness branch November 21, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant