Return the relevance score that search already computes - #64
Open
a-optic wants to merge 1 commit into
Open
Conversation
`search_memories` computes a score for every row it returns -- the cosine
distance `(embedding <=> %s::vector) as score` for a semantic search, and a
constant for the text and filter-only fallbacks -- and its docstring says it
returns "memory records with scores". The field was never declared on
`MemoryResponse`, so FastAPI's `response_model` filtered it out before it
reached the caller.
The effect is that `POST /memories/search` gives you an ordering but no
magnitude, so a caller cannot tell a strong match from a weak one. Ranking
alone cannot answer "is the best hit actually relevant?", which is the
question that matters when the alternative is acting on an irrelevant memory.
Optional, defaulting to None, because `GET /memories` and
`GET /memories/{memory_id}` share this model and produce no score -- their
responses are unchanged.
Verified against a populated database (53 embedded memories,
snowflake-arctic-embed2): POST /memories/search now returns
score=0.3666634909639195 for the closest hit and 0.38928308634373443 for the
next, matching the value recomputed independently from the stored vector to
six decimal places. GET /memories and GET /stats are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
search_memoriescomputes a score for every row it returns —(embedding <=> %s::vector) as scorefor a semantic search, and a constant for the text and filter-only fallbacks — and its docstring says it returns "memory records with scores".The field is never declared on
MemoryResponse, so FastAPI'sresponse_modelfilters it out before it reaches the caller.Why it matters
POST /memories/searchcurrently gives an ordering but no magnitude. A caller can rank results, but cannot tell a strong match from a weak one — and ranking alone cannot answer "is the best hit actually relevant?", which is the question that matters when the alternative is acting on an irrelevant memory. Today the only way to get the number is to bypass the API and query Postgres directly.The change
One field,
Optional[float] = None.GET /memoriesandGET /memories/{memory_id}share this model and produce no score, so their responses are unchanged.Verification
Against a populated database — 53 embedded memories,
snowflake-arctic-embed2at 1024 dimensions:The returned values match the cosine distance recomputed independently from the stored vectors to six decimal places, so the number reaching the client is the one pgvector computed.