Skip to content

feat(recall): explore file-aware automatic recall - #490

Closed
Phototonic wants to merge 3 commits into
tickernelz:mainfrom
Phototonic:feat/file-aware-recall-exploration
Closed

Phototonic wants to merge 3 commits into
tickernelz:mainfrom
Phototonic:feat/file-aware-recall-exploration

Conversation

@Phototonic

@Phototonic Phototonic commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

This is an experiment to see whether memories linked to files can help when OpenCode starts working on those files. The capture and recall paths work, but I haven't shown that this improves results over ordinary semantic recall. I don't think this is ready to merge. I'm opening it early to ask whether the idea is worth pursuing, or whether we should close it here.

@tickernelz, would you be willing to give a go/no-go on continuing this direction versus closing the draft? In particular, I'd appreciate your view on whether the extra lifecycle and retrieval logic is worth having before spending more time on it.

Summary

Adds experimental file-aware automatic recall for local OpenCode V2 sessions using the database backend. The aim is to keep memory capture effortless and tied to what OpenCode actually does, without introducing another model or a general-purpose indexing system.

  • Successful native read, edit and write activity supplies bounded path observations. Directory reads, shell commands, external tools and IDE active-file guesses do not.
  • New automatic captures associate a file only if it was observed in the authored request and its exact relative path is named in the sanitized saved summary. Observation alone is insufficient; edits are neither necessary nor sufficient. Read-only investigations can qualify.
  • Lookup repeats that summary-reference check for older broad associations. A digest ties the association to the saved content; rewrites must bind against their final text or lose the old association. Full memory text is preserved.
  • Matches require the same project, canonical checkout/workspace and exact case-preserving path. Separate worktrees, basename collisions, outside-root paths, missing files and symlink aliases do not match. Existing session, age and deletion filters still apply.
  • The tool-completion hook only records candidates. Filesystem metadata checks and retrieval happen at the next model-context boundary. The feature does not read source contents itself or block a native source read on retrieval.
  • One file-selected lesson can supplement ordinary recall within the existing memory count. A bounded shortlist prefers a memory ID absent from ordinary selection, so a redundant semantic match does not consume the supplemental slot. Context is replaced as one block; observations reset at turn/compaction/session boundaries.

Default in this draft: enabled for the supported local V2/database path. chatMessage.fileAwareRecall: false opts out through configuration; no dashboard is required. A valid inherited global setting survives unrelated project settings, and an explicit project boolean overrides it. This default is a design choice for review, not a recommendation to release the experiment yet. V1, Markdown storage and managed/remote workspaces retain their existing behavior.

What the investigation found

An early version attached every observed file to the whole task memory. That recalled a TTL lesson for an incidentally inspected static vendor helper. The exact-path summary filter fixes that specific failure, including query-time filtering of existing broad metadata. A fresh natural capture also inspected package.json without naming it in the lesson; a later read correctly selected no associated memory.

A second issue was replacement: the newest matching episode could already be present semantically, yet displace the additional file lesson. The shortlist/novelty selection change addresses that case without adding sticky session memory state. Regression tests cover both the recall service and the shared plugin bridge, including an overlap-only result preserving cached bytes/profile state.

An earlier four-memory experiment at the default threshold/count delivered a TTL lesson absent from the initial three semantic memories. It stayed for one model request, then was displaced. That was additional delivery, not an answer-quality win: the enabled answer omitted a key TTL distinction that the disabled answer explained from source.

With the final implementation, repeating the broad inspection prompt produced no extra TTL delivery: the agent reached that file after exhausting the four-refresh budget. Early unmatched reads can consume that budget. The cost bound remains in this draft rather than increasing retrieval until an example succeeds.

Validation

Local validation against current main at 5b05e1f (the subsequent CI fixes only wire the documentation and canonicalize test fixture roots):

  • bun run build passed.
  • Documentation-site bun run build passed (148 pages). The initial Site CI failure exposed the missing README heading map; the follow-up commit maps the new section and explicitly marks it untranslated.
  • bun run check passed (Prettier, ESLint and TypeScript).
  • Full suite: 1,247 passed, 4 platform skips, 0 failures; 8,981 assertions across 165 files, using bun test --timeout 15000 with networking disabled and live config/store/cache masked. The longer timeout accommodates an existing large-tool-log stress test near its default five-second limit; the ordinary default-timeout invocation is not being claimed here.
  • After the fixture-only macOS correction, the affected capture/bridge tests passed locally: 14 tests, 145 assertions. A simulated symlinked temporary root passed both fixtures. Production path checks were unchanged; the platform matrix is being rerun.
  • Independent review covered the selection change, lifecycle boundaries, source/bundle identity and actual experiment evidence.
  • Installed OpenCode 2.2.5 ran the production plugin with real mistral-small-latest agent/summary calls and mistral-embed embeddings. Four sessions naturally generated memories, including a read-only investigation; no memory text, associations or vectors were fabricated.
  • Eight controlled recall cases passed: same file, summary-supported vendor comparison, naturally incidental file exclusion, unseen same basename, unrelated file, different project, opt-out and read-only lesson. These used threshold 1.0 to isolate file delivery; they are not evidence of quality at default semantic settings.

Outcome comparisons: benefit remains unproven

Three constructed historical-regression scenarios were declared before running: reintroduce the cache truthiness bug, label lowercasing and pagination slice-end bug; remove the matching visible regression test; ask the agent to repair the module and add tests. Both arms used identical frozen source/memory copies, normal threshold 0.6, count 3, and injectOn: first. Each pair ran once, enabled then disabled, with equal 16-step/180-second limits. The independent checks below cover ten behaviors across the generated project.

Scenario Enabled behavioral checks Disabled behavioral checks
Cache 10/10; did not add/run the requested tests 9/10; zero-TTL remained broken and added tests failed
Labels 10/10; focused tests added and passed 10/10; focused tests added and passed
Pagination 9/10; no repair or focused pagination test 10/10; repair and focused tests passed; protocol deviation below

All six comparison sessions hit the agent step limit, despite client exit zero. Harness validity, behavioral correctness, and completing the requested task are separate outcomes. Every pair already received identical semantic memory IDs/scores, including its target lesson, and no file-selected memory. These mixed results therefore do not demonstrate a benefit from file recall.

The disabled pagination run attempted one manual memory search. It failed with empty output; no successful extra retrieval was observed. The native-hook observer missed the attempt. Under the strict no-manual-memory protocol, that comparison is invalid. The audit was corrected to include client-visible attempts, including failed ones; the original report is preserved rather than retrospectively calling it a clean comparison.

Per-arm snapshots substantiate source-edit scope for labels and pagination. Cache was run before that evidence gap was noticed; its saved tool operations give partial evidence, but comprehensive unrelated-edit exclusion is unverified. Independent inspection of all 90 saved repair-tool inputs found no access to evaluation infrastructure. Oracles were outside the task/project but accessible in the isolated filesystem, so they are not claimed to have been inaccessible to the agent.

The latest four phases made 146 provider requests, all HTTP 200; every broker stopped with zero in flight. Evaluation used fresh copies in isolated filesystem/network/process namespaces. Live memory, configuration and installation were untouched. Deliberate host shutdown still exited 1 with repeated-SIGTERM diagnostics; hosts were reaped without forced kills and database integrity passed. That is not a claim of clean-zero server shutdown. The local evaluation harness/stores are not included in this patch; the automated regression tests are.

Known limitations / reasons to keep this draft unmerged

  • Old memories have no new file evidence. There is no historical backfill or reconstruction from prose/saved-evidence recovery. Useful associations accumulate from new captures only.
  • Naming a file is a relevance hint, not proof that a lesson applies or is true. Explicitly named incidental files and whole multi-file episodes can still add noise. Conservative parsing can miss basenames, absolute paths and unusual formatting.
  • A real captured retry lesson inferred total attempts and delay/backoff behavior that its source predicate did not establish; later answers repeated those claims. Exact association filtering does not validate summary faithfulness.
  • At most 16 paths and four refreshes per authored turn, with a single supplemental memory. The refresh cap can miss late files; another genuinely additional lesson can replace the earlier one. Candidate reads are capped; large/association-heavy stores and stale candidate windows can conservatively miss matches.
  • No rename tracking or source-version freshness check. Current code remains authoritative. Positive file matches can refresh ordinary recall using existing providers, with possible embedding latency/cost; this is not a zero-cost feature.
  • Real-model evidence is one small generated corpus, one provider and one stochastic pair per scenario. Named-module prompts often already retrieve the lesson semantically. None of this establishes general accuracy or performance gains.
  • Linux/local OpenCode 2.2.5 was exercised; Windows/macOS acceptance and other host versions remain unverified.

My current recommendation is to hold merging. The narrow association and novelty fixes are justified, but more retrieval machinery is not justified by these results. A maintainer go/no-go would help decide whether to pursue stronger outcome evidence, narrow the proposal, or stop here.

Inspiration

The idea was informed by AgentMemory's OpenCode integration and Codemem's file-context ranking. This draft uses opencode-mem's own capture, storage and native V2 lifecycle rather than adopting their broader cross-agent architecture.

Checklist

  • Branch is based on current main.
  • Exercised locally; tests and their limits are recorded above.
  • Typecheck and formatting/lint checks pass.
  • README and example configuration updated.
  • Follow-up work and known limitations recorded.
  • Ready to merge — deliberately unchecked; maintainer go/no-go requested.

@tickernelz

tickernelz commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

thanks for the draft, and for writing the failed runs up honestly instead of hiding them.

my call is no-go for now. not about the code, about the evidence. in your own numbers both arms got the same semantic memories incl. the target lesson, no file memory was delivered in those comparison runs, and all six sessions hit the step limit. so nothing shows it beats normal recall.

and it's not free: ~800 lines in src (v2/adapter.ts, index.ts), on by default, so every V2 database user pays for extra lookups and maybe embedding calls for a gain we haven't measured.

the idea is fine, i'm not against it. open it again with:

  • a case where semantic recall really misses the lesson and only the file path finds it
  • more than one run per case, finishing inside the step limit
  • the file version beating the disabled one
  • default off

then i'll look again.

closing this for now, thanks again for the careful work.

@tickernelz tickernelz closed this Nov 5, 2026
@Phototonic

Copy link
Copy Markdown
Contributor Author

Thanks for the clear feedback, and for taking the time to look through the draft. I tried another local testing round along the lines you suggested before deciding whether to take this any further.

I changed the experiment to default-off and ran eight sessions across two scenarios, with each scenario repeated twice in both modes. All eight finished within the step limit and ran focused tests.

There was a small success with the labels scenario. In both repetitions, the lesson was missing from the usual top-three semantic memories, file recall brought it in after the file was read, and the enabled version preserved capitalization. It passed all ten independent behavioral checks, while the disabled version passed nine and missed that constraint. The cache scenarios were ties because semantic recall already supplied the relevant lesson.

The more telling check was increasing ordinary semantic recall from three results to four: that also included the labels lesson. I didn't run another answer-quality trial at four slots, so I can't claim equivalent outcomes, but it shows the lesson was already available through the existing system. The improvement we saw was selecting it within a smaller context budget, rather than finding something semantic retrieval couldn't find.

Given how narrow that result is, I don't think it justifies the extra code and lookup complexity. I'm going to set this implementation aside and leave the PR closed. Thanks again for the thoughtful review; it was useful to test the idea properly before adding something the project would have to maintain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants