Skip to content

Return the relevance score that search already computes - #64

Open
a-optic wants to merge 1 commit into
benclawbot:masterfrom
a-optic:fix/search-score-in-response
Open

Return the relevance score that search already computes#64
a-optic wants to merge 1 commit into
benclawbot:masterfrom
a-optic:fix/search-score-in-response

Conversation

@a-optic

@a-optic a-optic commented Sep 9, 2026

Copy link
Copy Markdown

The bug

search_memories computes a score for every row it returns — (embedding <=> %s::vector) as score for a semantic search, and a constant for the text and filter-only fallbacks — and its docstring says it returns "memory records with scores".

The field is never declared on MemoryResponse, so FastAPI's response_model filters it out before it reaches the caller.

Why it matters

POST /memories/search currently gives an ordering but no magnitude. A caller can rank results, but cannot tell a strong match from a weak one — and ranking alone cannot answer "is the best hit actually relevant?", which is the question that matters when the alternative is acting on an irrelevant memory. Today the only way to get the number is to bypass the API and query Postgres directly.

The change

One field, Optional[float] = None. GET /memories and GET /memories/{memory_id} share this model and produce no score, so their responses are unchanged.

Verification

Against a populated database — 53 embedded memories, snowflake-arctic-embed2 at 1024 dimensions:

POST /memories/search  "which books changed how I see the world"
  before:  score absent
  after:   score = 0.3666634909639195
           score = 0.38928308634373443

GET /memories   -> score: null   (unchanged, as intended)
GET /stats      -> 200           (unaffected)

The returned values match the cosine distance recomputed independently from the stored vectors to six decimal places, so the number reaching the client is the one pgvector computed.

`search_memories` computes a score for every row it returns -- the cosine
distance `(embedding <=> %s::vector) as score` for a semantic search, and a
constant for the text and filter-only fallbacks -- and its docstring says it
returns "memory records with scores". The field was never declared on
`MemoryResponse`, so FastAPI's `response_model` filtered it out before it
reached the caller.

The effect is that `POST /memories/search` gives you an ordering but no
magnitude, so a caller cannot tell a strong match from a weak one. Ranking
alone cannot answer "is the best hit actually relevant?", which is the
question that matters when the alternative is acting on an irrelevant memory.

Optional, defaulting to None, because `GET /memories` and
`GET /memories/{memory_id}` share this model and produce no score -- their
responses are unchanged.

Verified against a populated database (53 embedded memories,
snowflake-arctic-embed2): POST /memories/search now returns
score=0.3666634909639195 for the closest hit and 0.38928308634373443 for the
next, matching the value recomputed independently from the stored vector to
six decimal places. GET /memories and GET /stats are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant