Merge pull request #97 from GoodbyePlanet/fix/bm25-sparse-text-metadata

fix: include annotations, http route, package, and name in BM25 text

Author: GoodbyePlanet

docs/ingestion.md

@@ -99,10 +99,12 @@ The whole embedding text (preamble + signature + docstring + source) is budgeted
 
 #### Sparse: `_build_bm25_text`
 
-Simpler — only the functional code text, no metadata preamble:
+No preamble sentence, but folds in the same high-signal identifiers as the dense
+preamble so keyword search on an annotation, route, package, or symbol name still
+gets sparse matches:
 
 ```
-signature + docstring + source
+name + package + annotations (@-prefixed) + HTTP method/route + signature + docstring + source
 ```
 
 This text is then pre-processed by `split_code_identifiers` (see [sparse-vectors.md](sparse-vectors.md)) before BM25 encoding.
@@ -189,7 +191,7 @@ All `CodeSymbol` fields are stored verbatim, plus:
 
 **No embedding retry** — a transient API error on either embedding call causes the file to be silently skipped, leaving its existing index stale indefinitely. There is no exponential backoff or retry queue. Reindexing requires either a force reindex or waiting for the file's content to change.
 
-**BM25 text excludes metadata** — `_build_bm25_text` produces only `signature + docstring + source`. Metadata present in the dense text (service name, language, symbol type) is absent. A BM25 query for "Python method" will not match unless the word "Python" or "method" appears in the source code itself.
+**BM25 text still omits some dense-only metadata** — `_build_bm25_text` folds in name, package, annotations, and HTTP method/route, but the dense preamble's service name, language, and symbol-type phrasing (e.g. "Java method") are still dense-only. A BM25 query for "Python method" will not match unless the word "Python" or "method" appears elsewhere in the folded-in fields or the source code itself.
 
 **delete-before-upsert gap** — The pipeline deletes all entries for a file before upserting the new ones. If the process is interrupted between delete and upsert, the file has no index entries. The next incremental run will redownload and reindex the file correctly — but until then, queries miss the file entirely.
 

docs/sparse-vectors.md

@@ -104,6 +104,6 @@ SparseVectorParams(index=SparseIndexParams(on_disk=False))
 
 **In-memory sparse index** — `on_disk=False` is not configurable. On very large codebases, the sparse index memory footprint may become a concern. Qdrant supports `on_disk=True` for sparse vectors, but switching requires dropping and recreating the collection.
 
-**BM25 text excludes metadata** — the text passed to BM25 (`_build_bm25_text`) contains only `signature + docstring + source`. The rich metadata preamble used for dense embeddings (service name, language, symbol type, HTTP routes) is absent. A keyword search for "POST /orders" or "Java method" will not match via the sparse path unless those strings appear literally in the source code.
+**BM25 text still omits some dense-only metadata** — the text passed to BM25 (`_build_bm25_text`) folds in name, package, annotations, and HTTP method/route alongside `signature + docstring + source`, but the dense preamble's service name, language, and symbol-type phrasing (e.g. "Java method") remain dense-only. A keyword search for "POST /orders" now matches via the sparse path; a search for "Java method" still will not, unless those words appear literally in the source code.
 
 **Expanded form affects IDF statistics** — `split_code_identifiers` appends the expanded form, making each document approximately twice as long as the raw source. BM25's document length normalization (the `b` parameter in the BM25 formula) is computed over this expanded length, which may reduce scores for long symbols relative to what they would be with raw text.

server/indexer/pipeline.py

@@ -111,7 +111,17 @@ def _build_embedding_text(
 
 
 def _build_bm25_text(symbol: CodeSymbol) -> str:
-    parts = []
+    extras = symbol.extras or {}
+    parts = [symbol.name]
+    if symbol.package:
+        parts.append(symbol.package)
+    if symbol.annotations:
+        parts.extend(
+            f"@{a}" if not a.startswith("@") else a for a in symbol.annotations
+        )
+    if http_method := extras.get("http_method"):
+        route = extras.get("http_route") or ""
+        parts.append(f"{http_method} {route}".strip())
     if symbol.signature:
         parts.append(symbol.signature)
     if symbol.docstring:

tests/test_pipeline.py

@@ -135,6 +135,45 @@ def test_bm25_text_contains_signature_and_source() -> None:
     assert "void placeOrder(PlaceOrderRequest req) {}" in text
 
 
+def test_bm25_text_includes_name_package_annotations_and_http_route() -> None:
+    sym = CodeSymbol(
+        name="placeOrder",
+        symbol_type="method",
+        language="java",
+        source="void placeOrder(PlaceOrderRequest req) {}",
+        file_path="svc/OrderController.java",
+        start_line=10,
+        end_line=12,
+        package="com.example.orders",
+        annotations=["RestController", "PostMapping"],
+        signature="void placeOrder(PlaceOrderRequest req)",
+        docstring="Places an order.",
+        extras={"http_method": "POST", "http_route": "/orders"},
+    )
+    text = _build_bm25_text(sym)
+    assert "placeOrder" in text
+    assert "com.example.orders" in text
+    assert "@RestController" in text
+    assert "@PostMapping" in text
+    assert "POST /orders" in text
+
+
+def test_bm25_text_annotation_already_prefixed_not_double_prefixed() -> None:
+    sym = CodeSymbol(
+        name="placeOrder",
+        symbol_type="method",
+        language="java",
+        source="void placeOrder() {}",
+        file_path="svc/Order.java",
+        start_line=1,
+        end_line=1,
+        annotations=["@Deprecated"],
+    )
+    text = _build_bm25_text(sym)
+    assert "@@Deprecated" not in text
+    assert "@Deprecated" in text
+
+
 def test_bm25_text_excludes_preamble() -> None:
     sym = CodeSymbol(
         name="placeOrder",