Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/architecture/code-index-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,15 @@ their symbol spans (`lane_units.span_chunks`). A chunk id is
an unchanged chunk keeps its id and hash and the embed step skips it without a lookup.
Identical chunks in one file get an ordinal suffix.

Class skeletons carry compacted presentation text, not physical source coordinates.
The chunker retains class source outside method chunks as `other` chunks, including
documentation, declarations and data members. These source runs keep their original
line spans even when they are shorter than the general leftover-chunk threshold.
Skeletons remain available for class summaries, and unmappable chunks still disclose
partial grep coverage. Retaining source adds chunk rows and can add embedding work.
The shared chunker's stamp moves only the chunk stage, across language families;
existing indexes need that stage re-derived before missing source becomes readable.

### Symbols and edges

A symbol id is `treesitter.symbol_id(qualified_name, signature)`: the first 128 bits of
Expand Down Expand Up @@ -471,6 +480,12 @@ Every column is in the generated [schema reference](../reference/schema.md).

## Query patterns

Python import spellings use the bounded short-name candidate read after an exact
symbol miss. Read-side matching preserves dotted module scopes and treats a package's
`__init__.py` as implicit. Only Python source rows participate; ambiguous or incomplete
candidate sets decline. The response discloses `resolved_by: python_module` while
retaining stored names and ids. Extraction and its stamp do not change.

| Op | Path | How |
| -------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `code.search` | `code/app/search/pipeline.py` | Routes first (pattern, path, history question), then runs legs under the deadline (`gather.py`): lexical (idf-weighted token count), vector (a scan with no vector index, or twin shortlist), PR history, fuzzy symbol vote. `fusion.py` fuses by reciprocal rank (k = 60); `serve.py` pages into the token budget or abstains. A repeat within the memo window is one read of `code_search_memo`. |
Expand All @@ -479,6 +494,11 @@ Every column is in the generated [schema reference](../reference/schema.md).
| `code.callers`, `code.impact` | `code/store/index/edges.py` | Callers are the `to_symbol_id` range of `code_edges`, compiler rows preferred, with the unresolved count. `code.impact` maps diff hunks to seed symbols and walks outward; an empty walk answers `no_indexed_paths`, `no_seeds_in_hunks` or `no_static_callers`. |
| `code.outline`, `code.enclosing` | — | Per-path reads of `code_symbols` and `code_chunks` spans. |

The lexical text-index gate tokenizes the query with `splitByNonAlpha`, matching the
index tokenizer. The score counts only whole app lexemes and rejects partial matches
from the gate. Underscore-only lexemes use the exact score within the same tenant,
model and path scope because the text index has no token for them.

### Symbol bodies

Literal hits, rendered-message matches and symbol mentions use the same retained-source
Expand Down
6 changes: 6 additions & 0 deletions docs/features/code-index/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,12 @@ unless `LOOM_MCP_CODE_TOOLS=true`; `?toolset=coding` always does.

Every op, grouped by task, is on [operations.md](operations.md).

Python symbol lookups accept dotted import names such as `package.encoder.dumps`
and package names such as `package::dumps` for a definition in
`package/__init__.py`. A unique match reports `resolved_by: python_module` and
retains the stored qualified name and symbol id. Ambiguous matches remain unresolved.
This name matching reads the existing index and requires no reindexing.

## One namespace is one branch

A code namespace tracks one repository branch. Each index tier advances its own commit
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/operations/code.md
Original file line number Diff line number Diff line change
Expand Up @@ -835,7 +835,7 @@ Symbol-shaped (name/symbol_id) or diff-shaped (files/diff) blast radius.
| `causes` | `list[str]` | Why `epistemic` is `lower_bound`, from the set above; empty when it is `exact`. |
| `by_hop_counts` | `dict[int, int]` | Distinct symbols reached per hop BEFORE any slicing: {1: n, 2: m}. |
| `truncated` | `bool` | True when `impacted_symbols` is a page of the radius (per_hop_limit / offset sliced it, or the per-hop edge read hit its cap). |
| `resolved_by` | `str` | `anonymous_namespace` \| `tail` when the symbol shape's name bound through symbol-form tolerance; "" and off the wire otherwise. |
| `resolved_by` | `str` | `anonymous_namespace` \| `tail` \| `python_module` \| `registration` when a read-side name form bound; "" and off the wire for an exact hit. |
| `verdict` | `str` | Why the radius is empty, '' when it is not. |
| `paths_seen` | `list[dict[str, str]]` | Diff shape: every path the request named, with its class — `test` \| `doc` \| `config` \| `unindexed` (outside the indexed prefixes, or a source path the index holds no file row for) \| `src` — so the verdict is actionable without re-deriving… |
| `overlay` | `bool` | On the wire only when `pr_number` was given — see CodeCallersOutput. |
Expand Down Expand Up @@ -1579,7 +1579,7 @@ Exact symbol lookup: definition, signature, doc comment, and body of a function/
| `uses_truncated` | `bool` | True when the `uses` list stopped at `uses_limit`. |
| `uses_tokens` | `int` | Estimated tokens the compact `uses` block cost out of `token_budget`. |
| `picked_by` | `str` | Which context key ranked the definitions when `near_paths` / `pr_number` was given — same_file \| same_dir \| co_change \| path_distance \| none. "" and off the wire when no context was given. |
| `resolved_by` | `str` | `anonymous_namespace` \| `tail` when an over-qualified name bound through symbol-form tolerance; "" and off the wire otherwise. |
| `resolved_by` | `str` | `anonymous_namespace` \| `tail` \| `python_module` \| `registration` when a read-side name form bound; "" and off the wire for an exact hit. |
| `documented_in` | `list[dict[str, Any]]` | The document sections that cite the definition (the Markdown lane's `reads` edges) — section qualified name, path, line; at most five distinct sections, first by confidence. |
| `neighbours` | `list[dict[str, Any]]` | The neighbours graph block on the definitions (one entry per definition anchor); empty unless `neighbours` ran. |
| `anchors` | `int` | How many definitions the `neighbours` block was built for; 0 when it is off. |
Expand Down
13 changes: 7 additions & 6 deletions server/code/api/models_symbol.py
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,8 @@ def _drop_empty_presence(self, handler: SerializerFunctionWrapHandler) -> dict[s
# — same_file | same_dir | co_change | path_distance | none. "" and off the wire
# when no context was given.
picked_by: str = ""
# `anonymous_namespace` | `tail` when an over-qualified name bound through
# symbol-form tolerance; "" and off the wire otherwise.
# `anonymous_namespace` | `tail` | `python_module` | `registration` when a
# read-side name form bound; "" and off the wire for an exact hit.
resolved_by: str = ""
# The document sections that cite the definition (the Markdown lane's `reads` edges)
# — section qualified name, path, line; at most five distinct sections, first by
Expand Down Expand Up @@ -372,8 +372,9 @@ def _drop_unrequested_candidates(
# `anonymous_namespace` (`(anonymous namespace)::` segments stripped) or `tail` (a
# namespace the definition does not carry dropped; one definition fit), or
# `registration` when a SQL-level name (`sumMap`, `arrayFold::executeImpl`) bound to
# the implementing class's methods through its registration statement. "" and off
# the wire for an exact hit.
# the implementing class's methods through its registration statement, or
# `python_module` for an import spelling of a retained Python name. "" and
# off the wire for an exact hit.
resolved_by: str = ""
# A bare short name that resolves to many unrelated definitions (setColumns
# on 60 classes) is not walked: `ambiguous` is set, `candidates` lists the
Expand Down Expand Up @@ -550,8 +551,8 @@ def _drop_unset(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:
# True when `impacted_symbols` is a page of the radius (per_hop_limit /
# offset sliced it, or the per-hop edge read hit its cap).
truncated: bool = False
# `anonymous_namespace` | `tail` when the symbol shape's name bound through
# symbol-form tolerance; "" and off the wire otherwise.
# `anonymous_namespace` | `tail` | `python_module` | `registration` when a
# read-side name form bound; "" and off the wire for an exact hit.
resolved_by: str = ""
# Why the radius is empty, '' when it is not. Answers:
# `no_indexed_paths` (the change names no path the symbol table holds
Expand Down
61 changes: 51 additions & 10 deletions server/code/app/index/symbol_forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@
after rules 1 and 2, for bare names too: a bare SQL name is not an
over-qualification but a different vocabulary, and the class it names
is one class — the ambiguity guard is skipped for these rows.

Python import spellings bind through the same bounded bare-name read. A dotted
module remains one stored scope segment, and a package's `__init__.py` is implicit
in its import name. Only Python rows participate; ties and incomplete candidate
sets decline (`resolved_by: python_module`). Stored names and ids stay intact.
"""

from __future__ import annotations
Expand All @@ -47,6 +52,7 @@
RESOLVED_ANON = "anonymous_namespace"
RESOLVED_TAIL = "tail"
RESOLVED_REGISTRATION = "registration"
RESOLVED_PYTHON = "python_module"
# Bare-name definitions read for the tail rule; a hub name past this many
# rows is not bound (the rule needs the whole candidate set to call a match unique).
TAIL_READ_LIMIT = 400
Expand Down Expand Up @@ -167,39 +173,74 @@ def dropped_segments(written: str, rows: list[dict[str, Any]]) -> set[str]:
return out


def python_bind(written: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Bind an import spelling to one retained Python definition name."""
spelling = ".".join(segments(written))
if "." not in spelling:
return []
best: dict[str, list[dict[str, Any]]] = {}
best_rank = 0
for row in rows:
path = str(row.get("path") or "")
if not path.endswith(".py"):
continue
qualified = str(row.get("qualified_name") or "")
parts = segments(qualified)
if len(parts) < 2:
continue
forms = [".".join(parts)]
# Only the file's module scope is implicit in a package import. A class's
# constructor and a directory named __init__ keep their names.
if path.endswith("/__init__.py") and parts[0].endswith(".__init__"):
forms.append(".".join([parts[0].removesuffix(".__init__"), *parts[1:]]))
rank = max(
(2 if form == spelling else int(form.endswith("." + spelling))) for form in forms
)
if rank > best_rank:
best_rank, best = rank, {}
if rank and rank == best_rank:
best.setdefault(qualified, []).append(row)
return next(iter(best.values())) if len(best) == 1 else []


async def resolve_tolerant(
repo: Any,
org: str,
ns: str,
name: str,
) -> tuple[list[dict[str, Any]], str]:
"""The two rules over the repository, after an exact miss: (rows,
resolved_by). ([], "") when neither binds. Rule 1 runs for any name with
an anonymous segment (the rest may be bare and is looked up as such);
rule 2 only for a qualified name."""
"""Read-side name forms after an exact miss: (rows, resolved_by).
Anonymous scopes, Python imports and namespace tails precede registration;
every binding retains the stored identity."""
stripped = strip_anonymous(name)
if stripped:
rows = await repo.symbols_by_name(org, ns, stripped)
if rows:
return rows, RESOLVED_ANON
if is_qualified(name):
bound = await _tail(repo, org, ns, name)
if is_qualified(name) or "." in name:
bound, resolved_by = await _qualified(repo, org, ns, name)
if bound:
return bound, RESOLVED_TAIL
return bound, resolved_by
return await _registration(repo, org, ns, name)


async def _tail(repo: Any, org: str, ns: str, name: str) -> list[dict[str, Any]]:
async def _qualified(repo: Any, org: str, ns: str, name: str) -> tuple[list[dict[str, Any]], str]:
bare = _named(name)[-1]
if "::" not in name:
bare = bare.rsplit(".", 1)[-1]
rows = await repo.symbols_by_name(org, ns, bare, limit=TAIL_READ_LIMIT)
if not rows or len(rows) >= TAIL_READ_LIMIT:
return []
return [], ""
bound = python_bind(name, rows)
if bound:
return bound, RESOLVED_PYTHON
classes: set[str] = set()
for seg in sorted(dropped_segments(name, rows)):
hits = await repo.symbols_by_name(org, ns, seg, limit=5)
if any(str(h.get("kind") or "") in {"class", "struct"} for h in hits):
classes.add(seg)
return tail_bind(name, rows, class_names=classes)
bound = tail_bind(name, rows, class_names=classes)
return bound, RESOLVED_TAIL if bound else ""


async def _registration(
Expand Down
16 changes: 12 additions & 4 deletions server/code/domain/extract/chunker.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
* every class/struct -> one `class_skeleton` chunk: the class text
with method BODIES elided to `{ ... }` (methods get their own chunks)
* leading comments+includes -> one `file_header` chunk
* anything left at top level -> grouped `other` chunks
* uncovered source, including class documentation and data -> `other` chunks
* a function larger than max_chars splits at top-level statement
boundaries inside its body (the AST's fresh-line child starts), each
piece re-prefixed with the signature header so no piece ever floats
Expand Down Expand Up @@ -507,6 +507,7 @@ def chunk_file(
cls_syms = [s for s in pf.symbols if s.kind in kinds]

covered = [False] * (len(src_lines) + 2)
container_source = [False] * len(covered)

# function chunks (whole definitions, with the doc comment attached)
for fn in sorted(fn_syms, key=lambda s: (s.start_line, -s.end_line)):
Expand Down Expand Up @@ -608,14 +609,19 @@ def chunk_file(
)
for i in range(cls.start_line, cls.end_line + 1):
if i < len(covered):
covered[i] = True
if cls.kind in _BASE_CONTAINER_KINDS:
# Skeletons cannot establish physical source coordinates.
# Keep source outside the method chunks for exact reads.
container_source[i] = True
else:
covered[i] = True

# file header: leading run of comments/includes/imports/blank lines
header_chunk = _file_header_chunk(pf, src_lines, covered, max_chars)
if header_chunk is not None:
chunks.append(header_chunk)

# leftovers: uncovered top-level runs (blank lines and covered lines end
# leftovers: uncovered source runs (blank lines and covered lines end
# a run), optionally merged across blank lines up to merge_target_chars
runs: list[tuple[int, int]] = []
run_start = None
Expand All @@ -630,7 +636,9 @@ def chunk_file(
run_start = None
for run_start, run_end in _merge_runs(runs, src_lines, covered, merge_target_chars):
content = "".join(src_lines[run_start - 1 : run_end])
if len(content.strip()) < _MIN_CHUNK_CHARS:
if len(content.strip()) < _MIN_CHUNK_CHARS and not any(
container_source[run_start : run_end + 1]
):
continue
for piece, s_ln, e_ln in (
[(content, run_start, run_end)]
Expand Down
38 changes: 23 additions & 15 deletions server/code/store/index/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,32 +41,39 @@ async def lexical_search_chunks(
`scope` adds the pattern / path predicates to the plain WHERE after
the identity keys; unset, the statement is unchanged byte for byte.

The gate is `hasAnyTokens(lexemes, tokens)`, the predicate the `text`
index on `lexemes` (tokenizer splitByNonAlpha — the
writer's own token boundaries) answers per granule;
`multiSearchAny(concat(' ', lexemes, ' '), …)` reads every chunk of
the namespace (293 MB and 462 ms per call on the ClickHouse
repository)."""
The String query form of `hasAnyTokens` tokenizes the query with
the text index's splitByNonAlpha tokenizer. Its Array form takes
pre-tokenized terms, so whole underscore identifiers cannot gate
on it. The exact score rejects the coarse gate's partial matches.
Underscore-only lexemes have no text-index token and use only the
exact score predicate within the same scope.

The text-index gate avoids reading every chunk in the namespace.
"""
if not tokens:
return []
where_prefix = "AND startsWith(path, {prefix:String})" if path_prefix else ""
gate = (
"AND hasAnyTokens(lexemes, {gate_query:String})"
if all(tok.strip("_") for tok in tokens)
else ""
)
score = (
"arraySum(arrayMap((tok, idf) -> idf * countSubstrings("
"concat(' ', lexemes, ' '), concat(' ', tok, ' ')), "
"{tokens:Array(String)}, {idfs:Array(Float32)}))"
)
scope_sql, scope_params = _scope_sql(scope)
rows = await self._ch.query_all(
f"""
SELECT chunk_id, path, scope, kind, start_line, end_line,
content, symbol_ids, context_description,
arraySum(
arrayMap(
(tok, idf) -> idf * countSubstrings(
concat(' ', lexemes, ' '), concat(' ', tok, ' ')
),
{{tokens:Array(String)}}, {{idfs:Array(Float32)}}
)
) AS lex_score
{score} AS lex_score
FROM loom.code_chunks FINAL
WHERE org = {{org:String}} AND namespace = {{ns:String}}
AND embedding_model = {{model:String}} AND tombstoned_at IS NULL
AND hasAnyTokens(lexemes, {{tokens:Array(String)}})
{gate}
AND {score} > 0
{where_prefix}
{scope_sql}
ORDER BY lex_score DESC
Expand All @@ -79,6 +86,7 @@ async def lexical_search_chunks(
"tokens": tokens,
"idfs": idfs,
"pool": pool,
**({"gate_query": " ".join(tokens)} if gate else {}),
**({"prefix": path_prefix} if path_prefix else {}),
**scope_params,
},
Expand Down
Loading
Loading