Repository navigation
LongMemEval v2 runner: turn-pair chunking buries relevant memories under noise #12
Description
Problem
The v2 LongMemEval runner (run_longmemeval_v2.py) stores individual turn-pairs as separate memories (~250 embeddings/question). This creates so much retrieval noise that relevant memories don't rank in the top-15 results.
Evidence
| Runner | Storage strategy | Accuracy (50q) | Refusal rate |
|---|---|---|---|
v1 (run_longmemeval.py, March) |
Full sessions (1 block/session, ~5 embeddings/question) | 22/50 (44%) | 24% |
v2 (run_longmemeval_v2.py, April 8) |
Turn-pairs (~250 blocks/question) | ~21/50 (42%) | 24% |
While final accuracy is similar, v2 has much higher retrieval noise. The bottleneck is the reader LLM (gemma3:1b), not ShrimPK retrieval — echo hit rate is ~80% on LME questions, but the LLM fails to extract the answer from retrieved context.
Root Cause
250 embeddings per question all compete in cosine similarity space. The relevant memory (containing the specific fact asked about) gets a lower rank than irrelevant turn-pairs that happen to be semantically closer to the query. With max_results=15, the relevant memory is often excluded.
Fix Directions
| Approach | Complexity | Notes |
|---|---|---|
| v3 hybrid chunking — store full session + extracted key facts | M | Best of both: dense context + precision retrieval |
| Reduce max_results — top-5 instead of top-15 | XS | Higher precision, lower recall |
| BM25 re-ranking — keyword filter on top of vector results | M | Addresses lexical gap |
| Better reader LLM — replace gemma3:1b with larger model | XS config | Addresses actual bottleneck |
For KS74: Use v1 runner for 500q full benchmark. Build v3 hybrid chunking. Validate that reader LLM (gemma3:1b) is the bottleneck by testing with gemma3:4b or qwen2.5:3b.
Discovered during KS73 overnight benchmark session (2026-04-08).
Activity
Liorrr commentedon Jun 19, 2026
Review (2026-06-19): partially addressed + one XS fix shipping
Both runners still hardcode the reader model
gemma3:1b(run_longmemeval.py:318,run_longmemeval_v2.py:357) — never synced to the project's qwen2.5 default, even though this issue names the reader LLM as the actual bottleneck. Shipping a runner-only fix (bump the default reader model + top-5 on v2; purebenchmarks/Python, no GPU). v3 hybrid chunking (option A) remains the real fix — keeping this issue open for it.Note: the 42–44% keyword-match figures here are superseded by the official 24.2% GPT-4o-judge baseline (ROADMAP).
Default benchmark reader model to qwen2.5:1.5b, tighten v2 depth (#33)
Liorrr commentedon Jun 20, 2026
Validation (2026-06-20) — deferred to the GPU run (#19)
Reader-model fix is merged (#33: both runners default to qwen2.5:1.5b; v2 retrieval depth 15→5). A CPU validation sample was attempted but deferred because (a) the runner's
clear_memories()deletes the live~/.shrimpk-kernel/echo_store.shrmper question, and (b) no GPT-4o judge is available here (keyword-match is inflated per our eval policy).The reader-model effect will be measured properly as part of the full 500q GPT-4o-judged run on the GPU workstation (#19). The v2 turn-pair chunking-noise root cause (v3 hybrid chunking) remains the real fix and stays open here.
Harness note for the GPU run: the runner wipes the default data dir — isolate via
SHRIMPK_DATA_DIR(the hardcoded delete path atrun_longmemeval_v2.py:76should be made env-aware so it doesn't clobber a real store).