diff --git a/.docs/design-3084-session-search-over-grok-cli.md b/.docs/design-3084-session-search-over-grok-cli.md new file mode 100644 index 000000000..b5a2da61c --- /dev/null +++ b/.docs/design-3084-session-search-over-grok-cli.md @@ -0,0 +1,381 @@ +# Implementation Plan: Terraphim Session Search Over Grok CLI Search (#3084) + +**Status**: Draft +**Research Doc**: `.docs/research-3084-session-search-over-grok-cli.md` +**Author**: OpenCode +**Date**: 2026-07-12 +**Estimated Effort**: 1.5 to 2 days + +## Overview + +### Summary + +Add Grok CLI sessions as a first-class Terraphim search source so `terraphim-agent sessions search` can become the preferred local search command over `grok sessions search`. The revised design is a minimal feature-gated direct-query adapter in `terraphim_sessions` that opens Grok's existing `session_search.sqlite` read-only, executes FTS queries, and merges transient Grok hits with Terraphim's existing session search results. + +### Approach + +Do not wrap the Grok CLI and do not import/copy Grok's search store. Query Grok's SQLite FTS table directly and return a unified result projection that can be displayed beside existing Terraphim session hits. + +### Scope + +**In Scope:** + +- Add `grok-search` feature to `crates/terraphim_sessions`. +- Add a direct-query Grok search provider, e.g. `crates/terraphim_sessions/src/search_provider/grok.rs` or `src/grok_search.rs`. +- Detect Grok search DB from `GROK_HOME/sessions/session_search.sqlite` or `~/.grok/sessions/session_search.sqlite`. +- Query `session_docs_fts` and `session_docs` read-only. +- Merge transient Grok hits with existing Terraphim session search hits. +- Add tests with real temporary SQLite files. +- Update user-facing documentation for preferred Terraphim search workflow. + +**Out of Scope:** + +- Modifying Grok CLI itself. +- Shelling out to `grok sessions search` at runtime. +- Importing Grok `session_docs` rows into Terraphim's session cache. +- Creating a duplicate Terraphim index over Grok data. +- Importing rewind file snapshots. +- Real-time watching for Grok sessions in the first implementation. +- Replacing Terraphim BM25/KG scoring. +- Adding new external services or network calls. + +**Avoid At All Cost:** + +- A second search engine just for Grok. +- A hard dependency on Grok being installed in `$PATH`. +- Copying Grok's search index into Terraphim. +- Broad public model changes before connector needs prove them necessary. +- Test mocks for Grok storage. + +## Architecture + +### Component Diagram + +```text +~/.grok or $GROK_HOME + -> sessions/session_search.sqlite + -> GrokSearchProvider (read-only SQLite FTS) + -> Vec + -> SessionService search merge + -> terraphim-agent sessions search +``` + +### Data Flow + +```text +sessions search + -> maybe_auto_import + -> existing cached/imported session search + -> GrokSearchProvider::search(query, limit) reads session_search.sqlite + -> normalise and merge scores + -> render unified results +``` + +### Key Design Decisions + +| Decision | Rationale | Alternatives Rejected | +|----------|-----------|----------------------| +| Add direct `GrokSearchProvider` under `terraphim_sessions` | Avoids forcing an external FTS store into an import/cache trait | `SessionConnector` import implementation. | +| Feature-gate as `grok-search` | Keeps optional SQLite/Grok-specific code isolated | Always-on dependency. | +| Use `GROK_HOME` before `~/.grok` | Supports documented override and default install | Hard-code only `~/.grok`. | +| Query `session_docs_fts` directly | Grok already maintains FTS and content hashes | Parse raw session directories or duplicate index. | +| Return a unified hit projection | Cached sessions and direct SQLite hits need a shared display shape | Force every result into full `Session`. | + +### Eliminated Options + +| Option Rejected | Why Rejected | Risk of Including | +|-----------------|--------------|-------------------| +| Runtime `grok sessions search` wrapper | Search results would remain dependent on another CLI process | External process failures and poor result control. | +| Import Grok `session_docs` rows | Duplicates Grok's search store and creates stale cache risk | Sync complexity and ownership ambiguity. | +| Parse raw Grok session directories first | SQLite FTS already provides indexed searchable content | More format drift and larger scope. | +| Watcher support in first pass | Direct SQLite query sees current Grok indexed state | More concurrency and debounce complexity. | + +### Simplicity Check + +The easiest correct design is one small read-only SQLite search provider plus a unified search-result projection. The existing import connectors remain unchanged, and Grok remains the owner of its own search database. This avoids speculative indexing and keeps the first change reviewable. + +**Nothing Speculative Checklist:** + +- [x] No features the user did not request. +- [x] Only one small abstraction where the current import trait is the wrong fit. +- [x] No runtime dependency on the Grok CLI. +- [x] No new search scorer. +- [x] No watcher or import step for Grok. + +## File Changes + +### New Files + +| File | Purpose | +|------|---------| +| `crates/terraphim_sessions/src/search_provider/mod.rs` | Defines direct search-provider abstraction for external indexed stores. | +| `crates/terraphim_sessions/src/search_provider/grok.rs` | Implements read-only Grok SQLite FTS queries. | +| `crates/terraphim_sessions/tests/grok_search_provider.rs` | Integration tests using real temporary Grok-shaped SQLite storage. | + +### Modified Files + +| File | Changes | +|------|---------| +| `crates/terraphim_sessions/Cargo.toml` | Add `grok-search` feature and reuse optional `rusqlite`. | +| `crates/terraphim_sessions/src/lib.rs` | Export `search_provider` behind the feature if downstream callers need it. | +| `crates/terraphim_sessions/src/service.rs` | Merge direct-provider hits into session search output without importing Grok rows. | +| `docs/specifications/terraphim-agent-session-search-spec.md` | Add Grok to supported sources and document preferred Terraphim-over-Grok workflow. | +| `README.md` | Add concise example: query/search Grok sessions with Terraphim. | + +### Optional Follow-Up Files + +| File | Changes | +|------|---------| +| `crates/terraphim_agent/src/repl/commands.rs` | Add `sessions search --source --limit ` parsing. | +| `crates/terraphim_agent/src/repl/handler.rs` | Apply search source/limit and show snippets/scores if available. | + +## API Design + +### Unified Result Type + +```rust +#[derive(Debug, Clone)] +pub struct SessionSearchHit { + pub source: String, + pub session_id: String, + pub external_id: String, + pub title: Option, + pub project_path: Option, + pub source_path: Option, + pub score: f64, + pub snippet: Option, + pub updated_at: Option, +} +``` + +### Direct Search Provider Trait + +```rust +#[async_trait] +pub trait SessionSearchProvider: Send + Sync { + fn source_id(&self) -> &str; + fn display_name(&self) -> &str; + fn detect(&self) -> ConnectorStatus; + async fn search(&self, query: &str, limit: usize) -> Result>; + async fn get(&self, session_id: &str) -> Result>; +} +``` + +### Internal Helper Signatures + +```rust +#[derive(Debug, Default)] +pub struct GrokSearchProvider; + +impl GrokSearchProvider { + fn grok_home(&self) -> Option; + fn database_path(&self) -> Option; + fn open_readonly(&self, db_path: &Path) -> Result; + fn escape_fts_query(query: &str) -> String; + fn normalise_grok_score(raw_bm25: f64) -> f64; +} +``` + +### Suggested Internal Data Types + +```rust +#[derive(Debug, Deserialize)] +struct GrokSqliteRow { + session_id: String, + cwd: String, + updated_at: i64, + title: String, + content: String, + content_hash: String, + last_indexed_offset: i64, +} +``` + +Keep these private to the Grok search provider module. + +### SQLite Mapping Rules + +| Grok Field | Terraphim Field | +|------------|-----------------| +| `session_id` | `SessionSearchHit.session_id = format!("grok:{session_id}")`, `external_id = session_id` | +| `cwd` | `project_path` | +| `title` | `title` | +| `updated_at` | `updated_at` after timestamp conversion | +| `bm25(session_docs_fts)` | Normalised `score` | +| `snippet(...)` | `snippet` | +| `content_hash`, `last_indexed_offset` | Optional internal metadata if a metadata map is added later | + +## Test Strategy + +### Unit Tests + +| Test | Location | Purpose | +|------|----------|---------| +| `detects_grok_database_from_env` | `grok_search_provider.rs` | Verifies `GROK_HOME` override is honoured. | +| `detects_missing_database` | `grok_search_provider.rs` | Returns `NotFound` when no SQLite store exists. | +| `queries_sqlite_fts` | `grok_search_provider.rs` | Builds a real SQLite DB and returns FTS hits. | +| `returns_snippets_and_project_path` | `grok_search_provider.rs` | Verifies snippet/title/cwd projection. | +| `handles_title_only_rows` | `grok_search_provider.rs` | Rows with empty content can still be returned for title matches. | +| `opens_database_readonly` | `grok_search_provider.rs` | Verifies no writes are required. | + +### Integration Tests + +| Test | Location | Purpose | +|------|----------|---------| +| `grok_hits_are_merged_with_session_hits` | `crates/terraphim_sessions/tests/grok_search_provider.rs` | Query Grok fixture and existing session fixture, assert unified output includes both sources. | +| `source_filter_limits_to_grok` | `crates/terraphim_sessions/tests/grok_search_provider.rs` | Search with source filter and assert only Grok hits are returned. | + +### Manual Verification + +```bash +`cargo test -p terraphim_sessions --features grok-search,search-index` +cargo test -p terraphim_sessions --features full +~/.cargo/bin/terraphim-agent sessions sources +~/.cargo/bin/terraphim-agent sessions search "session search" +/home/alex/.grok/bin/grok sessions search "session search" --limit 5 +``` + +Expected manual outcome: + +- Terraphim detects `grok` as a direct search source. +- Terraphim returns Grok hits plus any matching non-Grok sessions. +- Terraphim does not create a copy of Grok's search store or require `sessions import --source grok`. + +### Coverage Check + +After implementation, run a scoped coverage check for changed Rust crates: + +```bash +cargo llvm-cov -p terraphim_sessions --features grok-search,search-index --summary-only +``` + +If `cargo llvm-cov` is not installed, document that gap and run the full test suite above plus `ubs` on changed files. + +## Implementation Steps + +### Step 1: Add Feature Flag and Search Provider Skeleton + +**Files:** `crates/terraphim_sessions/Cargo.toml`, `crates/terraphim_sessions/src/search_provider/mod.rs`, `crates/terraphim_sessions/src/search_provider/grok.rs` + +**Description:** Add `grok-search = ["dep:rusqlite"]`; define `SessionSearchProvider` and `SessionSearchHit`; add a skeletal `GrokSearchProvider` under the feature gate. + +**Tests:** `cargo test -p terraphim_sessions --no-default-features` + +**Estimated:** 1 hour + +### Step 2: Implement Grok Database Detection + +**Files:** `crates/terraphim_sessions/src/search_provider/grok.rs` + +**Description:** Implement `source_id`, `display_name`, `grok_home`, `database_path`, and `detect`. Detection should return `Available` when `session_search.sqlite` exists and estimate count from `session_docs`. + +**Tests:** detection tests with temporary real directories. + +**Estimated:** 2 hours + +### Step 3: Implement Read-Only SQLite FTS Search + +**Files:** `crates/terraphim_sessions/src/search_provider/grok.rs` + +**Description:** Open `session_search.sqlite` read-only. Query `session_docs_fts` joined to `session_docs`, returning `SessionSearchHit` values with session ID, cwd, title, snippet, updated timestamp, and normalised score. + +**Tests:** create a real SQLite database in a temp dir and import it. + +**Estimated:** 3 hours + +### Step 4: Merge Direct Hits With Existing Session Search + +**Files:** `crates/terraphim_sessions/src/service.rs`, `crates/terraphim_sessions/src/search.rs` + +**Description:** Add a search path that projects existing `Scored` results into `SessionSearchHit`, queries `GrokSearchProvider`, normalises scores, merges, sorts, and limits. + +**Tests:** temp directory fixture with two sessions and one malformed record. + +**Estimated:** 4 hours + +### Step 5: CLI Surface + +**Files:** `crates/terraphim_agent/src/repl/commands.rs`, `crates/terraphim_agent/src/repl/handler.rs` + +**Description:** Add `sessions search --source --limit ` if feasible. Ensure `--source grok` uses direct-provider results and does not call import. + +**Tests:** fixture where the same ID appears in SQLite and directory data. + +**Estimated:** 2 hours + +### Step 6: Search Pipeline Regression Test + +**Files:** `crates/terraphim_sessions/tests/grok_search_provider.rs` + +**Description:** Build a real SQLite FTS fixture, search for a term in indexed content, and assert source is `grok` without any import step. + +**Tests:** run `cargo test -p terraphim_sessions --features grok-search,search-index`. + +**Estimated:** 2 hours + +### Step 7: Documentation Update + +**Files:** `README.md`, `docs/specifications/terraphim-agent-session-search-spec.md` + +**Description:** Document Grok as a supported source and position Terraphim session search as the canonical cross-tool search path. Include comparison to `grok sessions search` without disparaging Grok. + +**Tests:** documentation review; no code tests. + +**Estimated:** 1 hour + +### Step 8: Quality Gates + +**Files:** changed files only + +**Commands:** + +```bash +cargo fmt --check +cargo clippy -p terraphim_sessions --features grok-search,search-index --all-targets -- -D warnings +cargo test -p terraphim_sessions --features grok-search,search-index +cargo test -p terraphim_sessions --features full +ubs crates/terraphim_sessions/src/search_provider/grok.rs crates/terraphim_sessions/src/search_provider/mod.rs crates/terraphim_sessions/src/service.rs crates/terraphim_sessions/Cargo.toml +``` + +**Estimated:** 2 hours + +## Rollback Plan + +If the Grok direct-query provider causes regressions: + +1. Remove `grok-search` from `full` while leaving the module behind the explicit feature. +2. Disable direct-provider merging while keeping existing imported-session search unchanged. +3. Keep documentation marked experimental until the provider passes local verification. + +## Dependencies + +### New Dependencies + +No new crate is required if `rusqlite` is reused behind `grok-search`. + +### Dependency Updates + +None planned. + +## Performance Considerations + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Grok query latency | Use SQLite FTS directly with `LIMIT` | Integration fixture plus manual local query timing. | +| Search merge latency | Preserve existing session-search target where practical | Search against local corpus with Grok provider enabled. | +| Memory | Avoid loading all Grok rows | Fetch only limited FTS hits. | + +## Open Items + +| Item | Status | Owner | +|------|--------|-------| +| Decide if `grok-search` joins `full` immediately | Pending | Human reviewer | +| Define exact score normalisation for Grok FTS versus Terraphim BM25 | Pending | Implementer | +| Decide first-pass `--source` and `--limit` CLI support | Pending | Human reviewer | + +## Approval + +- [ ] Technical review complete +- [ ] Test strategy approved +- [ ] Grok direct-query feature inclusion decided +- [ ] Human approval received before implementation diff --git a/.docs/handover-2026-07-06.md b/.docs/handover-2026-07-06.md new file mode 100644 index 000000000..6d0b4e67c --- /dev/null +++ b/.docs/handover-2026-07-06.md @@ -0,0 +1,117 @@ +# Session Handover: 2026-07-06 + +## Progress Summary + +### Tasks Completed + +1. **Installed terraphim-grep and terraphim-agent at v1.21.8** (GNU, from terraphim-clients GitHub releases) + - Previous: grep 1.20.5, agent 1.20.3 + - Current: both at 1.21.8 (x86_64-unknown-linux-gnu) + - Backups at `~/.cargo/bin/*.bak-1.20.5` + - Verified code search compiled in (`--haystack code` with `all, code, docs` options) + +2. **Fixed terraphim-agent config** (`~/.config/terraphim/embedded_config.json`) + - Changed invalid `"id": "local-terraphim-config"` to `"id": "Embedded"` + - Agent now starts without config errors + +3. **Validated both tools via tmux** + - terraphim-grep: code search returns results (56ms, 3009 files scanned) + - terraphim-agent: roles list (5 roles), search, learn query, sessions search all functional + - Skills confirmed present in both opencode and claude code + +4. **Filed parse error bug** -- Gitea [#3081](https://git.terraphim.cloud/terraphim/terraphim-ai/issues/3081) + - `terraphim_session_analyzer::parser` fails on 13 Claude Code JSONL entry types (~2430 WARN lines/scan) + - Root cause: `SessionEntry` struct requires `uuid` + `message` fields that metadata entries lack + +5. **Disciplined research + design** for #3081 + - Research doc: `.docs/adf/3081/research.md` + - Design doc: `.docs/adf/3081/design.md` + - Approach: pre-filter by entry type before deserialization (15 lines, one location) + +6. **Implemented the fix** in terraphim-clients + - Vendored `terraphim-session-analyzer` crate into workspace as path dependency + - Added `SKIP_ENTRY_TYPES` constant (12 metadata types) + `EntryTypePeek` struct + - Added pre-filter in `parser.rs::from_file()` + - 5 new tests, all 114 pass, clippy clean, fmt clean + - **Live validation: WARN count dropped from 2430 to 435 (82% reduction)** + +7. **Structural PR review** -- posted to [#3081](https://git.terraphim.cloud/terraphim/terraphim-ai/issues/3081#issuecomment-60675) + - Confidence score: 4/5 + - 0 P0, 0 P1, 3 P2 findings (double JSON parse overhead, version pin too restrictive, path dep version drift) + - 1 UBS false positive (pre-existing `entry.uuid == message_id` flagged as timing attack) + - Full review at `.docs/adf/3081/pr-review.md` + +8. **Pushed both repos to GitHub and Gitea** + - terraphim-clients: commit `eac88d0` (fix) + `bd4ebe6` (merge) on both remotes + - terraphim-ai: commit `ca740ffad` (docs) + `0db4f0f9e` (merge) on both remotes + - Branch protection on Gitea terraphim-clients temporarily disabled, then restored + +### Current Implementation State + +**Working:** +- terraphim-grep 1.21.8 -- code search, thesaurus matching, KG scoring +- terraphim-agent 1.21.8 -- roles, search, learn, sessions (with 82% fewer WARN lines) +- Session analyzer pre-filter eliminates all 12 metadata entry types +- Both binaries configured for opencode and claude code (skills + PATH) + +**Blocked / Deferred:** +- 427 `assistant` entry parse failures remain (genuine format issues, separate investigation needed) +- `terraphim-clients/Cargo.toml` has `terraphim_service` pinned to `=1.20.6` (overly restrictive, P2 from review) +- Pre-existing clippy lints in vendored session-analyzer crate (12 warnings, not from our changes) +- terraphim-grep has no `check-update` subcommand (the quickstart doc references one) + +## Technical Context + +### terraphim-ai +```bash +# Branch: main +git log --oneline -3 +0db4f0f9e Merge remote-tracking branch 'origin/main' +ca740ffad docs(adf): add #3081 research, design, and PR review documents +5d5c675ca chore(deps)(deps): bump sysinfo from 0.30.13 to 0.39.5 (#929) + +# Status: clean (all pushed to origin + gitea) +``` + +### terraphim-clients +```bash +# Branch: main +git log --oneline -3 +bd4ebe6 Merge remote-tracking branch 'gitea/main' +eac88d0 fix(session-analyzer): skip Claude Code metadata entry types in JSONL parser +90989f9 Merge pull request 'Fix #1384: add /health readiness endpoint to MCP SSE server' (#31) + +# Status: clean (all pushed to origin + gitea) +# Gitea remote added: https://git.terraphim.cloud/terraphim/terraphim-clients.git +``` + +### Key Files Changed + +| File | Repo | Change | +|------|------|--------| +| `crates/terraphim-session-analyzer/src/parser.rs` | clients | Pre-filter: SKIP_ENTRY_TYPES + EntryTypePeek + from_file() skip logic + 5 tests | +| `Cargo.toml` | clients | Added workspace member, pinned terraphim_service to =1.20.6 | +| `crates/terraphim_sessions/Cargo.toml` | clients | Changed dep to path: ../../crates/terraphim-session-analyzer | +| `~/.config/terraphim/embedded_config.json` | local | Fixed id field: "local-terraphim-config" -> "Embedded" | +| `.docs/adf/3081/research.md` | ai | Phase 1 research document | +| `.docs/adf/3081/design.md` | ai | Phase 2 implementation plan | +| `.docs/adf/3081/pr-review.md` | ai | Structural PR review | + +### Installed Binaries + +| Binary | Version | Path | Notes | +|--------|---------|------|-------| +| terraphim-grep | 1.21.8 | ~/.cargo/bin/terraphim-grep | GNU, code search compiled in | +| terraphim-agent | 1.21.0* | ~/.cargo/bin/terraphim-agent | Built from clients repo with fix | + +*Note: installed agent reports 1.21.0 (workspace version) not 1.21.8 (release tag) because it was built from source, not downloaded from release. + +### Open Items + +| Item | Priority | Issue | +|------|----------|-------| +| 427 `assistant` parse failures | Medium | File follow-up issue for investigation | +| Release terraphim-agent 1.21.8 with fix | High | Need to publish session-analyzer patch and rebuild release | +| `=1.20.6` version pin | Low (P2) | Remove `=` prefix to allow patch updates | +| Pre-existing clippy lints in session-analyzer | Low | 12 lints in original code, not from our changes | +| terraphim-grep check-update doc | Low | Quickstart references non-existent subcommand | diff --git a/.docs/plan-terraphim-agent-configuration-manifest.md b/.docs/plan-terraphim-agent-configuration-manifest.md new file mode 100644 index 000000000..fcaee749b --- /dev/null +++ b/.docs/plan-terraphim-agent-configuration-manifest.md @@ -0,0 +1,214 @@ +# Terraphim AI Agent Configuration Planning Manifest + +Status: planning only. This document proposes a Terraphim AI agent configuration and crate/dependency manifest. It does not implement or activate any agent configuration. + +## Source Comparison + +The requested Odilo path `~/projects/zestic-ai/odilo` was not present. The active Odilo repository used for comparison was `/home/alex/projects/zestic-at/odilo`. + +Odilo patterns worth adopting: + +- `manifest.yaml` is the canonical component registry for agent routing, ownership, dependencies, deployment targets, secrets, policies, and compatible skills. +- Root `AGENTS.md` defines an on-ramp: read manifest, component `.agent` contract, policies, global boundaries, build docs, and file summaries before changing code. +- `.agent/BOUNDARIES.md` defines repo-wide no-touch paths and command constraints. +- `.agent/POLICIES.md` maps component policies to merge/push gates such as `production-code-pairing`, `auto-mode-ok`, and `human-review-only`. +- `.agents/skills/` stores project-local skills so dispatched agents get reproducible behaviour independent of global skill drift. + +Terraphim-specific patterns already present: + +- `.terraphim/orchestrator.toml.bigbox` is the deployed ADF-style root config and includes `conf.d/*.toml`. +- `scripts/adf-setup/agents/*.toml` contains version-controlled agent templates. +- `crates/terraphim_orchestrator/orchestrator.example.toml` documents the larger fleet shape, routing, pre-checks, and Gitea posting conventions. +- Gitea is the task authority for `terraphim/terraphim-ai` at `https://git.terraphim.cloud`. + +## Proposed Configuration Shape + +Create a Terraphim equivalent of the Odilo manifest rather than adding more ad hoc agent TOML: + +- `manifest.yaml`: canonical component and dependency registry for this repository. +- `.agent/BOUNDARIES.md`: Terraphim-specific global boundaries, including no `.env` edits, no machine-local Cargo overrides committed, and no changes to extracted polyrepo ownership without an issue/ADR. +- `.agent/POLICIES.md`: policy definitions aligned with Gitea PageRank workflow and ADF merge gates. +- `.agents/README.md`: project-local skill inventory and provenance. +- `.agents/skills/`: only if reproducible project-local skill pinning is required; otherwise reference existing global skills. +- `.terraphim/adf.toml`: project source config already referenced by `.terraphim/orchestrator.toml.bigbox`; should become the project-specific manifest-to-ADF bridge if not already present. +- `conf.d/*.toml`: generated or hand-curated concrete agent definitions after the manifest is approved. + +No implementation should happen until the manifest schema, component list, and policy mapping are approved. + +## Proposed Agent Policies + +- `production-code-pairing`: human approval required; use for server runtime, Gitea runner, orchestrator, merge coordination, registry publishing, and deployment-sensitive crates. +- `auto-mode-ok`: CI-gated autonomous implementation allowed; use for tooling crates with low blast radius and strong test coverage. +- `human-review-only`: agents may analyse and propose changes but must not merge; use for governance docs, manifest/policy files, release/versioning decisions, private registry configuration, and security boundaries. +- `event-only`: agents can only be dispatched by orchestrator events; use for build runner, PR reviewer, merge coordinator, and webhook/status agents. + +## Proposed Agent Fleet + +The fleet should be narrower than the generic example and biased towards Terraphim's actual risks: polyrepo registry boundaries, Gitea automation, Rust workspace quality, and ADF reliability. + +| Agent | Dispatch | Policy | Purpose | Template Basis | +| --- | --- | --- | --- | --- | +| `build-runner` | push/PR event | `event-only` | deterministic `fmt`, `clippy`, and tests via `rch` | `scripts/adf-setup/agents/build-runner.toml` | +| `pr-reviewer` | PR event | `event-only` | bounded structural PR review and `adf:gate-result` output | `scripts/adf-setup/agents/pr-reviewer.toml` | +| `merge-coordinator` | PR event/comment | `production-code-pairing` | coordinate required checks and issue closure | `scripts/adf-setup/agents/merge-coordinator.toml` | +| `repo-steward` | scheduled | `human-review-only` | detect stale branches, duplicate issues, dependency drift, and documentation drift | `scripts/adf-setup/agents/repo-steward.toml` | +| `runtime-guardian` | scheduled/event | `production-code-pairing` | monitor orchestrator/systemd/Quickwit/webhook health | `scripts/adf-setup/agents/runtime-guardian.toml` | +| `registry-steward` | scheduled/manual | `human-review-only` | audit private `terraphim` registry versions, local-vs-registry duplication, and publish readiness | new proposed template | +| `gitea-runner-steward` | scheduled/manual | `production-code-pairing` | maintain native Gitea runner and Gitea API integration | new proposed template | +| `product-development` | scheduled | `human-review-only` | PageRank prioritisation only; no code, no branches, no PRs | `scripts/adf-setup/agents/product-development.toml` | + +Recommended first implementation slice after approval: + +1. Add `manifest.yaml`, `.agent/BOUNDARIES.md`, and `.agent/POLICIES.md` only. +2. Add `registry-steward.toml` and `gitea-runner-steward.toml` as disabled templates. +3. Wire enabled agents only after validating the manifest with a small schema/test. + +## Cargo Registry And Dependency Sources + +Cargo registry configuration is in `.cargo/config.toml`: + +- crates.io uses sparse protocol. +- private registry `terraphim` points at `sparse+https://git.terraphim.cloud/api/packages/terraphim/cargo/`. +- reads and publishes require `CARGO_REGISTRIES_TERRAPHIM_TOKEN="Bearer $GITEA_TOKEN"`. + +Resolved dependency source counts from `cargo metadata --format-version 1`: + +| Source | Count | Notes | +| --- | ---: | --- | +| crates.io | 930 | normal third-party transitive dependency set | +| local/path | 17 | active workspace members in this checkout | +| `terraphim` private registry | 14 | extracted Terraphim crates consumed from Gitea Cargo registry | +| git | 2 | patched dependencies from GitHub | + +Private `terraphim` registry packages currently resolved: + +- `terraphim-markdown-parser` 1.20.2 +- `terraphim_agent_evolution` 1.20.2 +- `terraphim_automata` 1.20.2 +- `terraphim_config` 1.20.2 +- `terraphim_mcp_search` 0.1.0 +- `terraphim_orchestrator` 1.20.2 +- `terraphim_persistence` 1.20.2 +- `terraphim_rolegraph` 1.20.2 +- `terraphim_router` 1.20.2 +- `terraphim_service` 1.20.5 +- `terraphim_settings` 1.20.2 +- `terraphim_spawner` 1.21.0 +- `terraphim_tracker` 1.20.2 +- `terraphim_types` 1.20.2 + +Git dependencies currently resolved: + +- `rustls-webpki` 0.103.12 from `https://github.com/rustls/webpki.git`, tag `v/0.103.12`. +- `self_update` 0.42.0 from `https://github.com/AlexMikhalev/self_update.git`, branch `update-zipsign-api-v0.2`. + +## Local Cargo Manifest Inventory + +Active workspace members: + +| Manifest | Package | Category | +| --- | --- | --- | +| `terraphim_server/Cargo.toml` | `terraphim_server` | server/runtime | +| `terraphim_firecracker/Cargo.toml` | `terraphim-firecracker` | sandbox/runtime | +| `terraphim_ai_nodejs/Cargo.toml` | `terraphim_ai_nodejs` | Node integration | +| `crates/terraphim_dsm/Cargo.toml` | `terraphim_dsm` | architecture analysis | +| `crates/terraphim_eval_check/Cargo.toml` | `terraphim_eval_check` | evaluation tooling | +| `crates/terraphim_github_runner/Cargo.toml` | `terraphim_github_runner` | CI runner | +| `crates/terraphim_lsp/Cargo.toml` | `terraphim_lsp` | LSP tooling | +| `crates/terraphim_mcp_search/Cargo.toml` | `terraphim_mcp_search` | MCP/search | +| `crates/terraphim_merge_coordinator/Cargo.toml` | `terraphim_merge_coordinator` | merge automation | +| `crates/terraphim_rlm/Cargo.toml` | `terraphim_rlm` | recursive language model/sandbox | +| `crates/terraphim_sessions/Cargo.toml` | `terraphim_sessions` | session search/history | +| `crates/terraphim_spawner/Cargo.toml` | `terraphim_spawner` | process spawning | +| `crates/terraphim_tinyclaw/Cargo.toml` | `terraphim_tinyclaw` | agent/proxy runtime | +| `crates/terraphim_update/Cargo.toml` | `terraphim_update` | updater | +| `crates/terraphim_validation/Cargo.toml` | `terraphim_validation` | validation tooling | +| `crates/terraphim_weather_report/Cargo.toml` | `terraphim_weather_report` | weather/reporting | +| `crates/terraphim_workspace/Cargo.toml` | `terraphim_workspace` | workspace tooling | + +Local manifests present but not active workspace members: + +- `crates/terraphim_orchestrator/Cargo.toml`: excluded from this workspace; Bigbox builds orchestrator from `/home/alex/projects/terraphim/terraphim-agents` per `AGENTS.md`. +- `crates/terraphim_gitea_runner/Cargo.toml`: native Gitea runner source exists but is excluded from workspace builds. +- `crates/terraphim_github_runner_server/Cargo.toml`: host runner server, excluded. +- `crates/terraphim_agent_application/Cargo.toml`: experimental, excluded. +- `crates/terraphim_automata_py/Cargo.toml` and `crates/terraphim_rolegraph_py/Cargo.toml`: Python bindings, excluded. +- `crates/haystack_atlassian/Cargo.toml` and `crates/haystack_discourse/Cargo.toml`: future haystack providers, excluded. +- `crates/terraphim_symphony/Cargo.toml`: separate build path. +- `browser_extensions/TerraphimAIParseExtension/wasm/Cargo.toml`: browser extension WASM build. +- `vs-code-terraphim-it/rust-lib/Cargo.toml`: VS Code integration Rust library. +- `lab/parking-lot/terraphim-grep/Cargo.toml`: parked lab source. + +## Direct Private Registry References By Manifest + +These manifests directly reference the `terraphim` private registry and need registry-aware agent handling: + +- Root `Cargo.toml`: `[patch.crates-io] terraphim_service = { version = "1.20.5", registry = "terraphim" }`. +- `crates/haystack_atlassian/Cargo.toml`: `haystack_core`, `terraphim_types`. +- `crates/haystack_discourse/Cargo.toml`: `haystack_core`, `terraphim_types`. +- `crates/terraphim_agent_application/Cargo.toml`: agent supervisor, messaging, registry, KG orchestration, task decomposition, goal alignment, and `terraphim_types`. +- `crates/terraphim_automata_py/Cargo.toml`: `terraphim_automata`, `terraphim_types`. +- `crates/terraphim_github_runner/Cargo.toml`: optional `terraphim_agent_evolution`. +- `crates/terraphim_github_runner_server/Cargo.toml`: `terraphim_service`, `terraphim_config`, `terraphim_test_utils`. +- `crates/terraphim_lsp/Cargo.toml`: `terraphim_automata`, `terraphim_types`, `terraphim_rolegraph`. +- `crates/terraphim_rlm/Cargo.toml`: optional `terraphim_mcp_search`. +- `crates/terraphim_rolegraph_py/Cargo.toml`: `terraphim_rolegraph`, `terraphim_types`, `terraphim_automata`. +- `crates/terraphim_sessions/Cargo.toml`: session analyser, markdown parser, automata, rolegraph, and types. +- `crates/terraphim_spawner/Cargo.toml`: `terraphim_types`. +- `crates/terraphim_symphony/Cargo.toml`: `terraphim_tracker`. +- `crates/terraphim_tinyclaw/Cargo.toml`: `terraphim_mcp_search`. +- `crates/terraphim_validation/Cargo.toml`: `terraphim_config`, `terraphim_types`. +- `crates/terraphim_weather_report/Cargo.toml`: `terraphim_automata`, `terraphim_orchestrator`, `terraphim_types`. + +## Gitea-Specific Sources And Risks + +Gitea-relevant local sources: + +- `crates/terraphim_gitea_runner`: native Gitea Actions runner; present but excluded from workspace builds. +- `crates/terraphim_github_runner`: shared runner stack with Gitea payload support comments. +- `crates/terraphim_merge_coordinator`: merge automation and PR coordination. +- `crates/terraphim_orchestrator`: ADF dispatch, Gitea skill loader, PR gate, webhook, commit-status, and mention handling source; excluded in this workspace. +- `.cargo/config.toml`: private Gitea Cargo registry configuration. +- `.terraphim/orchestrator.toml.bigbox`: Gitea tracker configuration for `terraphim/terraphim-ai`. + +Agent planning implications: + +- Any agent touching registry configuration, `Cargo.lock`, or `Cargo.toml` entries with `registry = "terraphim"` should use `production-code-pairing` or `human-review-only` until registry drift checks are automated. +- The duplicated names `terraphim_mcp_search` and `terraphim_spawner` appear as both local workspace packages and private registry packages in the resolved graph. A `registry-steward` should detect and explain these cases before any version bump. +- `terraphim_gitea_runner` is present locally but excluded; a `gitea-runner-steward` should not assume it is covered by `cargo test --workspace`. + +## Proposed Manifest Components + +Initial `manifest.yaml` components should be coarse, not one component per crate: + +| Component | Source Paths | Policy | Notes | +| --- | --- | --- | --- | +| `server-runtime` | `terraphim_server`, `terraphim_ai_nodejs` | `production-code-pairing` | user-facing runtime and integration surface | +| `sandbox-runtime` | `terraphim_firecracker`, `crates/terraphim_rlm` | `production-code-pairing` | sandbox/security sensitive | +| `adf-orchestration` | `crates/terraphim_orchestrator`, `.terraphim`, `scripts/adf-setup` | `production-code-pairing` | runtime deployed from agents repo; do not commit Bigbox-only overrides | +| `gitea-automation` | `crates/terraphim_gitea_runner`, `crates/terraphim_github_runner`, `crates/terraphim_merge_coordinator` | `production-code-pairing` | CI and merge status authority | +| `registry-and-polyrepo` | `Cargo.toml`, `Cargo.lock`, `.cargo/config.toml`, registry-consuming manifests | `human-review-only` | private registry and extracted crate boundary | +| `search-and-kg` | `crates/terraphim_mcp_search`, `crates/terraphim_lsp`, `crates/terraphim_sessions`, `.terraphim/kg` | `auto-mode-ok` with registry guard | core search and KG tooling | +| `validation-and-quality` | `crates/terraphim_validation`, `crates/terraphim_eval_check`, `crates/terraphim_dsm` | `auto-mode-ok` | quality tooling, with coverage evidence required | +| `desktop-and-extensions` | `desktop`, `browser_extensions`, `vs-code-terraphim-it` | `production-code-pairing` | separate build/test commands | +| `experimental-and-parked` | `lab/parking-lot`, `crates/terraphim_agent_application`, `crates/terraphim_symphony` | `human-review-only` | avoid accidental activation | +| `documentation-and-governance` | `.docs`, `docs`, `AGENTS.md`, manifest/policy files | `human-review-only` | governance artefacts | + +## Validation Plan Before Implementation + +Before implementing any config changes: + +1. Confirm whether Terraphim should copy Odilo's exact `manifest.yaml` schema or use a Terraphim-specific schema. +2. Confirm whether project-local `.agents/skills` should be vendored, or whether global skills remain authoritative. +3. Add a manifest validator test that checks source paths exist, policies are defined, and registry-sensitive components are not `auto-mode-ok`. +4. Run `cargo metadata --format-version 1` in CI to generate the dependency source inventory and fail on unexpected source drift. +5. Add a specific check for local-vs-registry duplicate crate names. +6. Only then generate or update `conf.d/*.toml` agent configs. + +## Non-Goals For This Planning Pass + +- Do not edit `.terraphim/orchestrator.toml.bigbox`. +- Do not add or enable any `conf.d/*.toml` agent. +- Do not change `Cargo.toml`, `Cargo.lock`, `.cargo/config.toml`, or registry dependencies. +- Do not create Gitea issues or PRs until the manifest plan is reviewed. +- Do not commit this planning artefact unless explicitly requested. diff --git a/.docs/research-3084-session-search-over-grok-cli.md b/.docs/research-3084-session-search-over-grok-cli.md new file mode 100644 index 000000000..591d05760 --- /dev/null +++ b/.docs/research-3084-session-search-over-grok-cli.md @@ -0,0 +1,288 @@ +# Research Document: Terraphim Session Search Over Grok CLI Search (#3084) + +**Status**: Draft +**Author**: OpenCode +**Date**: 2026-07-12 +**Issue**: #3084 + +## Executive Summary + +The local repository was fast-forwarded to `origin/main` and `gitea/main` at `101532826` before research. Follow-up validation confirmed that Grok already maintains a local SQLite FTS search store at `~/.grok/sessions/session_search.sqlite`; Terraphim should query that store read-only rather than importing or copying Grok search data. Grok CLI search is useful but isolated; Terraphim can provide a unified local search surface by adding a direct-query Grok adapter and merging those transient results with existing Terraphim session results. + +## Essential Questions Check + +| Question | Answer | Evidence | +|----------|--------|----------| +| Energising? | Yes | Terraphim's core value is local knowledge search across AI coding history. | +| Leverages strengths? | Yes | Existing `terraphim_sessions` connectors, BM25 ranking, KG enrichment, and Terraphim CLI make this a natural fit. | +| Meets real need? | Yes | The user explicitly asked to plan updating session search over Grok CLI search; Grok sessions are currently outside Terraphim's connector registry. | + +**Proceed**: Yes - 3/3 yes. + +## Problem Statement + +### Description + +Terraphim session search should become the preferred local search path for AI coding sessions, including sessions created by Grok CLI. Today, Grok can search its own sessions via `grok sessions search`, but Terraphim cannot query Grok's local search index and therefore cannot return Grok matches alongside Claude Code, OpenCode, Codex, Aider, and Cline sessions. + +### Impact + +- Developers must remember which assistant produced the relevant context. +- Grok CLI search is isolated from Terraphim's cross-source session search. +- Terraphim cannot enrich Grok conversations with role concepts or return them alongside Claude Code, OpenCode, Codex, Aider, and Cline results. +- AI agents cannot rely on one canonical local session-search command. + +### Success Criteria + +1. `terraphim-agent sessions sources` detects Grok when `~/.grok/sessions/session_search.sqlite` exists or when `GROK_HOME` points to an equivalent directory. +2. `terraphim-agent sessions search ` queries Grok's SQLite FTS store read-only without copying rows into Terraphim's session cache or writing a second Terraphim index. +3. Terraphim returns Grok matches alongside other session sources with source metadata, title, project path, score, and snippet. +4. Terraphim preserves Grok as the owner of its own search store; no import, migration, or mutation of Grok data is required. +5. Tests use real temporary SQLite databases, not mocks. + +## Current State Analysis + +### Repository Update State + +| Item | Result | +|------|--------| +| Current branch | `main` | +| Pull action | `git fetch --all --prune`, then `git merge --ff-only origin/main` | +| New local HEAD | `101532826` | +| Relevant remotes updated | `origin/main`, `gitea/main` | +| Remote fetch caveat | Remote `terraphim` failed with `Repository not found`; it is not a relevant source for this plan. | +| Existing untracked file | `.docs/handover-2026-07-06.md`, left untouched. | +| Merge hook caveat | Post-merge hook reported `Failed to import bd changes after merge`; tracked code was fast-forwarded successfully. | + +### Existing Implementation + +| Component | Location | Purpose | +|-----------|----------|---------| +| Session model | `crates/terraphim_sessions/src/model.rs` | Unified `Session`, `Message`, `SessionMetadata`, and message roles. | +| Connector trait and registry | `crates/terraphim_sessions/src/connector/mod.rs` | Source detection, import options, feature-gated connector registration. | +| Native Claude connector | `crates/terraphim_sessions/src/connector/native.rs` | Always-available Claude Code JSONL importer and file watcher. | +| OpenCode connector | `crates/terraphim_sessions/src/connector/opencode.rs` | SQLite and legacy JSONL import pattern. | +| Codex connector | `crates/terraphim_sessions/src/connector/codex.rs` | JSONL session importer closest to Grok's per-session files. | +| Search adapter | `crates/terraphim_sessions/src/search.rs` | Converts sessions to `Document`, scores with BM25, and optionally boosts KG matches. | +| Session service | `crates/terraphim_sessions/src/service.rs` | Auto-import, cache, search, statistics, source filtering. | +| REPL command parser | `crates/terraphim_agent/src/repl/commands.rs` | Parses `/sessions search`, `/sessions import`, `/sessions list`, etc. | +| REPL session handler | `crates/terraphim_agent/src/repl/handler.rs` | Formats session source, import, list, search, stats, show, and expand output. | +| Session-search specification | `docs/specifications/terraphim-agent-session-search-spec.md` | Existing goals for cross-agent search, robot mode, aliases, token-aware output, and KG enrichment. | + +### Current Terraphim Search Flow + +```text +terraphim-agent sessions search + -> ReplCommand::Sessions { Search { query } } + -> ReplHandler::handle_sessions + -> static SessionService + -> maybe_auto_import + -> ConnectorRegistry::import_all + -> SessionService::search or search_with_thesaurus + -> search_sessions or search_sessions_hybrid + -> human-readable table output +``` + +### Current Search Behaviour + +- `search_sessions` builds a `Document` per session from project path, model, and every non-empty message, capped at `MAX_BODY_LENGTH = 50_000` bytes. +- Results are capped at `MAX_SEARCH_RESULTS = 50` and filtered to scores at least `10%` of the top score. +- With `enrichment`, `search_sessions_hybrid` applies KG concept boosting when a role thesaurus is available. +- Without `search-index`, `SessionService` falls back to substring matching over title, project path, and message content. + +### Grok CLI Search Baseline + +Observed CLI surface: + +```text +grok sessions search [OPTIONS] + -n, --limit default 20 +``` + +Help text states that Grok search searches "summaries and first prompts". A sample local command returned one matching session for `session search`. Grok also exposes `grok sessions list --limit `. + +### Grok Local Storage Findings + +Grok stores session data under `~/.grok/sessions` by default and supports `GROK_HOME` according to its local documentation. + +Observed files: + +```text +~/.grok/sessions/session_search.sqlite +~/.grok/sessions//prompt_history.jsonl +~/.grok/sessions///summary.json +~/.grok/sessions///updates.jsonl +~/.grok/sessions///chat_history.jsonl +~/.grok/sessions///signals.json +``` + +Observed SQLite schema, validated with `sqlite3 ~/.grok/sessions/session_search.sqlite .schema`: + +```sql +CREATE TABLE session_docs ( + session_id TEXT PRIMARY KEY, + cwd TEXT NOT NULL, + updated_at INTEGER NOT NULL, + title TEXT NOT NULL, + content TEXT NOT NULL, + content_hash TEXT NOT NULL, + last_indexed_offset INTEGER NOT NULL DEFAULT 0 +); +CREATE VIRTUAL TABLE session_docs_fts USING fts5(title, content, content='session_docs', content_rowid='rowid'); +``` + +Observed `prompt_history.jsonl` entries contain `timestamp`, `session_id`, `prompt`, and `is_bash`. + +Follow-up validation confirmed direct FTS querying works: + +```sql +SELECT d.session_id, + bm25(session_docs_fts) AS score, + snippet(session_docs_fts, 1, '[', ']', '...', 16) AS snippet +FROM session_docs_fts +JOIN session_docs d ON d.rowid = session_docs_fts.rowid +WHERE session_docs_fts MATCH 'session search' +ORDER BY score +LIMIT 5; +``` + +The local store contained 29 `session_docs` rows at validation time. Some rows have empty `content`, so the adapter must tolerate title-only entries. + +## Constraints + +### Technical Constraints + +- `terraphim_sessions` is in the workspace; `crates/terraphim_agent` is currently excluded from workspace builds but still present and used by the installed CLI path. +- The current `SessionConnector` trait is import-oriented, so direct Grok querying should be introduced as a separate search-provider extension rather than forcing Grok into the import/cache model. +- SQLite support already exists through optional `rusqlite` for the OpenCode connector, so a Grok direct-query adapter can reuse that dependency behind a `grok-search` feature. +- Tests must not use mocks. Use temporary directories, real JSONL files, and real SQLite databases. +- Search should stay local and privacy-first; do not shell out to `grok`, mutate Grok's SQLite database, or duplicate Grok's search store. + +### Business Constraints + +- The design should minimise maintenance by reusing `SessionConnector`, `SessionService`, and `search_sessions` rather than adding a parallel search engine. +- The implementation should be incremental: direct-query adapter first, result merging second, CLI ergonomics third. + +### Non-Functional Requirements + +| Requirement | Target | Current | +|-------------|--------|---------| +| Local-only operation | No network or Grok API calls | Terraphim and Grok both store local files. | +| No duplicate index | Query Grok's FTS store in place | `session_search.sqlite` already contains `session_docs_fts`. | +| Read-only safety | Open SQLite with read-only flags | Avoids mutating user-owned Grok data. | +| Performance | Use Grok FTS for Grok results | Avoids loading every Grok row into Terraphim memory. | + +## Vital Few + +| Constraint | Why It's Vital | Evidence | +|------------|----------------|----------| +| Add a direct-query Grok search adapter instead of shelling out or importing | Keeps Terraphim the canonical search path while preserving Grok's search store ownership | Grok SQLite FTS schema is accessible and queryable. | +| Preserve a unified result model | Allows Grok hits to appear beside existing Terraphim session hits | Existing CLI needs source/title/snippet/session ID fields. | +| Use real SQLite artefacts in tests | Prevents format drift and complies with project test rules | No mocks are allowed. | + +## Eliminated From Scope + +| Eliminated Item | Why Eliminated | +|-----------------|----------------| +| Replacing BM25 scoring | Existing scoring already works and supports KG boost. | +| Calling `grok sessions search` at runtime | Would make Terraphim depend on an external CLI and lose result-shaping control. | +| Importing Grok `session_docs` rows into Terraphim cache | Duplicates an existing FTS store and creates synchronisation/staleness risk. | +| Importing file snapshots or rewind points | High privacy and size cost; not needed for session search parity. | +| Real-time Grok watching in the first change | Direct SQLite queries always see Grok's current indexed state. | +| Changing Grok's own index | Out of Terraphim scope and creates upgrade risk. | + +## Dependencies + +### Internal Dependencies + +| Dependency | Impact | Risk | +|------------|--------|------| +| New direct search provider trait | Required extension point for Grok FTS queries without import | Medium | +| `SessionService::search` | Needs to merge cached session results with direct-provider results | Medium | +| Result projection type | Needed to represent source/title/snippet/score without requiring full `Session` import | Medium | +| REPL parser and handler | Needed for source filtering and display improvements | Medium, because command parser currently has limited flags for search. | + +### External Dependencies + +| Dependency | Version | Risk | Alternative | +|------------|---------|------|-------------| +| `rusqlite` | Existing optional `0.32` bundled | Low, already used by OpenCode connector | Shell out to `sqlite3`, rejected. | +| Grok storage format | CLI-managed, not versioned here | Medium, may change | Prefer documented per-session JSON files and tolerate missing fields. | + +## Risks and Unknowns + +| Risk | Likelihood | Impact | Mitigation | +|------|------------|--------|------------| +| Grok file format changes | Medium | Medium | Prefer tolerant JSON parsing and use `session_search.sqlite` only as an optional fast path. | +| SQLite `content` duplicates or summarises rather than preserving full history | Medium | Medium | Treat Grok parity target as querying Grok's own search store; richer raw-log indexing is explicitly out of scope unless Grok exposes it in SQLite. | +| Importing tool outputs leaks too much sensitive data into search body | Medium | High | Limit first pass to prompts and assistant text; include tool names and compact summaries, not raw large payloads. | +| Current `terraphim-agent sessions search` emits noisy Cursor warnings | High | Low for Grok, medium for UX | Track separately or suppress unavailable Cursor connector warnings; not required for Grok connector. | +| `crates/terraphim_agent` excluded from workspace complicates CLI verification | Medium | Medium | Test `terraphim_sessions` crate directly; verify installed CLI separately if agent crate cannot be built in workspace. | + +### Open Questions + +1. Should the direct Grok search adapter be enabled by default under `full`, or should it require an explicit `grok-search` feature only? +2. What unified result type should carry both cached `Session` hits and direct SQLite hits without requiring Grok import? +3. Should `terraphim-agent sessions search` grow a `--source grok` and `--limit` parser in the first implementation? + +### Assumptions Explicitly Stated + +| Assumption | Basis | Risk if Wrong | Verified? | +|------------|-------|---------------|-----------| +| Grok stores its search DB under `GROK_HOME/sessions` or `~/.grok/sessions` | Grok docs and observed filesystem | Adapter misses sessions if installation differs | Partially | +| `session_search.sqlite` is sufficient for the first Terraphim/Grok integration | Validated schema and FTS query | Some rich raw conversation fields may not be available | Yes | +| Terraphim should not shell out to `grok` for normal search | Terraphim privacy-first architecture and unified CLI goal | CLI fallback might be simpler but weaker | Yes | +| Terraphim should not import/copy Grok's search store | Existing SQLite FTS already solves storage/indexing | Direct result merging requires a small service change | Yes | + +### Multiple Interpretations Considered + +| Interpretation | Implications | Why Chosen/Rejected | +|----------------|--------------|---------------------| +| "Update session search over Grok CLI search" means replace Grok's search implementation | Requires modifying Grok itself, outside this repo | Rejected. | +| It means make Terraphim use `grok sessions search` as a backend | Fast but external CLI-dependent and loses result-shaping control | Rejected. | +| It means import Grok sessions into Terraphim's cache/index | Fits existing import connectors but duplicates Grok's search store | Rejected after validation. | +| It means make Terraphim query Grok's local SQLite search store directly and merge results | Avoids duplicate storage and preserves Grok ownership | Chosen. | + +## Research Findings + +1. `terraphim_sessions` is already the correct extension point. It exposes `SessionConnector`, `ConnectorRegistry`, `SessionService`, and BM25/KG search. +2. Grok has enough local data for direct search integration: `session_search.sqlite` contains `session_docs`, `session_docs_fts`, content hashes, and triggers. +3. A direct SQLite adapter is preferable to import because it avoids duplicate storage, stale cached copies, and ownership ambiguity. +4. The smallest viable implementation is a feature-gated `GrokSearchProvider` that opens `session_search.sqlite` read-only, runs FTS queries, and returns transient session-search hits. +5. CLI search ergonomics lag behind the original session-search spec: parser support for `--source`, `--limit`, robot output, and richer result snippets should be planned around a unified search-result type. + +## Recommendations + +### Proceed + +Proceed with a direct-query adapter implementation. Do not import Grok rows into Terraphim's session cache, do not create a second Terraphim index for Grok data, and do not wrap `grok sessions search`. + +### Scope Recommendations + +Phase 1 implementation should include: + +- A feature-gated `grok-search` adapter in `terraphim_sessions`. +- Source detection using `GROK_HOME` then `~/.grok`. +- Read-only FTS queries against `session_search.sqlite`. +- Result merging between Terraphim cached session hits and Grok direct-query hits. +- Direct unit/integration tests with temporary real SQLite databases. +- Documentation updates showing `terraphim-agent sessions search` as the preferred path; no `sessions import --source grok` workflow. + +Phase 2 follow-up should include: + +- CLI flags for `sessions search --source grok --limit n`. +- Robot output for session search if not already available in the active CLI path. +- Optional watcher support after format stability is proven. + +### Risk Mitigation Recommendations + +- Open Grok SQLite with read-only flags. +- Normalise FTS `bm25()` scores before merging with Terraphim BM25 scores. +- Keep Grok-specific result metadata in a result projection rather than expanding `Session` unless a full-session view is needed. +- Add regression fixtures for SQLite-only Grok storage, including empty `content` rows and FTS snippets. + +## Next Steps + +1. Review and approve `.docs/design-3084-session-search-over-grok-cli.md`. +2. Decide whether `grok-search` belongs in `full` immediately or behind explicit opt-in for one release. +3. Implement with tests in a dedicated task branch, referencing #3084 or a follow-up implementation issue.