Repository navigation
feat(recall): explore file-aware automatic recall - #490
Phototonic wants to merge 3 commits into
Conversation
|
thanks for the draft, and for writing the failed runs up honestly instead of hiding them. my call is no-go for now. not about the code, about the evidence. in your own numbers both arms got the same semantic memories incl. the target lesson, no file memory was delivered in those comparison runs, and all six sessions hit the step limit. so nothing shows it beats normal recall. and it's not free: ~800 lines in src (v2/adapter.ts, index.ts), on by default, so every V2 database user pays for extra lookups and maybe embedding calls for a gain we haven't measured. the idea is fine, i'm not against it. open it again with:
then i'll look again. closing this for now, thanks again for the careful work. |
|
Thanks for the clear feedback, and for taking the time to look through the draft. I tried another local testing round along the lines you suggested before deciding whether to take this any further. I changed the experiment to default-off and ran eight sessions across two scenarios, with each scenario repeated twice in both modes. All eight finished within the step limit and ran focused tests. There was a small success with the labels scenario. In both repetitions, the lesson was missing from the usual top-three semantic memories, file recall brought it in after the file was read, and the enabled version preserved capitalization. It passed all ten independent behavioral checks, while the disabled version passed nine and missed that constraint. The cache scenarios were ties because semantic recall already supplied the relevant lesson. The more telling check was increasing ordinary semantic recall from three results to four: that also included the labels lesson. I didn't run another answer-quality trial at four slots, so I can't claim equivalent outcomes, but it shows the lesson was already available through the existing system. The improvement we saw was selecting it within a smaller context budget, rather than finding something semantic retrieval couldn't find. Given how narrow that result is, I don't think it justifies the extra code and lookup complexity. I'm going to set this implementation aside and leave the PR closed. Thanks again for the thoughtful review; it was useful to test the idea properly before adding something the project would have to maintain. |
This is an experiment to see whether memories linked to files can help when OpenCode starts working on those files. The capture and recall paths work, but I haven't shown that this improves results over ordinary semantic recall. I don't think this is ready to merge. I'm opening it early to ask whether the idea is worth pursuing, or whether we should close it here.
@tickernelz, would you be willing to give a go/no-go on continuing this direction versus closing the draft? In particular, I'd appreciate your view on whether the extra lifecycle and retrieval logic is worth having before spending more time on it.
Summary
Adds experimental file-aware automatic recall for local OpenCode V2 sessions using the database backend. The aim is to keep memory capture effortless and tied to what OpenCode actually does, without introducing another model or a general-purpose indexing system.
read,editandwriteactivity supplies bounded path observations. Directory reads, shell commands, external tools and IDE active-file guesses do not.Default in this draft: enabled for the supported local V2/database path.
chatMessage.fileAwareRecall: falseopts out through configuration; no dashboard is required. A valid inherited global setting survives unrelated project settings, and an explicit project boolean overrides it. This default is a design choice for review, not a recommendation to release the experiment yet. V1, Markdown storage and managed/remote workspaces retain their existing behavior.What the investigation found
An early version attached every observed file to the whole task memory. That recalled a TTL lesson for an incidentally inspected static vendor helper. The exact-path summary filter fixes that specific failure, including query-time filtering of existing broad metadata. A fresh natural capture also inspected
package.jsonwithout naming it in the lesson; a later read correctly selected no associated memory.A second issue was replacement: the newest matching episode could already be present semantically, yet displace the additional file lesson. The shortlist/novelty selection change addresses that case without adding sticky session memory state. Regression tests cover both the recall service and the shared plugin bridge, including an overlap-only result preserving cached bytes/profile state.
An earlier four-memory experiment at the default threshold/count delivered a TTL lesson absent from the initial three semantic memories. It stayed for one model request, then was displaced. That was additional delivery, not an answer-quality win: the enabled answer omitted a key TTL distinction that the disabled answer explained from source.
With the final implementation, repeating the broad inspection prompt produced no extra TTL delivery: the agent reached that file after exhausting the four-refresh budget. Early unmatched reads can consume that budget. The cost bound remains in this draft rather than increasing retrieval until an example succeeds.
Validation
Local validation against current
mainat5b05e1f(the subsequent CI fixes only wire the documentation and canonicalize test fixture roots):bun run buildpassed.bun run buildpassed (148 pages). The initial Site CI failure exposed the missing README heading map; the follow-up commit maps the new section and explicitly marks it untranslated.bun run checkpassed (Prettier, ESLint and TypeScript).bun test --timeout 15000with networking disabled and live config/store/cache masked. The longer timeout accommodates an existing large-tool-log stress test near its default five-second limit; the ordinary default-timeout invocation is not being claimed here.mistral-small-latestagent/summary calls andmistral-embedembeddings. Four sessions naturally generated memories, including a read-only investigation; no memory text, associations or vectors were fabricated.Outcome comparisons: benefit remains unproven
Three constructed historical-regression scenarios were declared before running: reintroduce the cache truthiness bug, label lowercasing and pagination slice-end bug; remove the matching visible regression test; ask the agent to repair the module and add tests. Both arms used identical frozen source/memory copies, normal threshold 0.6, count 3, and
injectOn: first. Each pair ran once, enabled then disabled, with equal 16-step/180-second limits. The independent checks below cover ten behaviors across the generated project.All six comparison sessions hit the agent step limit, despite client exit zero. Harness validity, behavioral correctness, and completing the requested task are separate outcomes. Every pair already received identical semantic memory IDs/scores, including its target lesson, and no file-selected memory. These mixed results therefore do not demonstrate a benefit from file recall.
The disabled pagination run attempted one manual memory search. It failed with empty output; no successful extra retrieval was observed. The native-hook observer missed the attempt. Under the strict no-manual-memory protocol, that comparison is invalid. The audit was corrected to include client-visible attempts, including failed ones; the original report is preserved rather than retrospectively calling it a clean comparison.
Per-arm snapshots substantiate source-edit scope for labels and pagination. Cache was run before that evidence gap was noticed; its saved tool operations give partial evidence, but comprehensive unrelated-edit exclusion is unverified. Independent inspection of all 90 saved repair-tool inputs found no access to evaluation infrastructure. Oracles were outside the task/project but accessible in the isolated filesystem, so they are not claimed to have been inaccessible to the agent.
The latest four phases made 146 provider requests, all HTTP 200; every broker stopped with zero in flight. Evaluation used fresh copies in isolated filesystem/network/process namespaces. Live memory, configuration and installation were untouched. Deliberate host shutdown still exited 1 with repeated-SIGTERM diagnostics; hosts were reaped without forced kills and database integrity passed. That is not a claim of clean-zero server shutdown. The local evaluation harness/stores are not included in this patch; the automated regression tests are.
Known limitations / reasons to keep this draft unmerged
My current recommendation is to hold merging. The narrow association and novelty fixes are justified, but more retrieval machinery is not justified by these results. A maintainer go/no-go would help decide whether to pursue stronger outcome evidence, narrow the proposal, or stop here.
Inspiration
The idea was informed by AgentMemory's OpenCode integration and Codemem's file-context ranking. This draft uses opencode-mem's own capture, storage and native V2 lifecycle rather than adopting their broader cross-agent architecture.
Checklist
main.