diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 3c9fd1ad..f0d6755d 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,17 @@ +## 2026-07-18 — EpisodicMemory::basins() — usable episodic-witness basin partition (operator directive: "usable code, not paper churn") — main thread + +- **Task:** the operator asked for the AriGraph episodic-witness basins as USABLE CODE (not a probe, not jc-coupled). AriGraph had `communities()` (structural, #714) but no experiential partition (`grep basin` → nothing). Built the complement. +- **Change:** `arigraph/episodic.rs` — `EpisodicBasins { entities, labels, num_basins }` + `basin_of`/`members` (same accessor shape as `Communities`, so caller can compare the two partitions directly) and `EpisodicMemory::basins()`: deterministic union-find over per-episode entity **co-occurrence** (parse each `"subject - relation - object"` triplet, union all entities seen in one episode), sorted-entity + union-by-lower-root → stable partition. Re-exported in `arigraph/mod.rs`. +- **Honest scope:** this is the experiential complement to `communities()` — entities grouped by *what was observed together*, NOT the le-contract 6-slot horizontal `basin:role` register frame (that stays EVENTUAL per `context-role-traversal-tissue.md`). No similarity pooling; structure only. +- **Verified:** `cargo test -p lance-graph --lib -- graph::arigraph::episodic` 16/16 green (5 new basin tests: co-occurrence grouping, shared-entity merge, single-episode union, empty-safe, deterministic); clippy clean on the additions. Branch `claude/happy-hamilton-0azlw4`. + +## 2026-07-18 — REMOVED the S1 community-basin probe + jc dev-coupling (operator directive: "I don't want it in the JC crate") — main thread + +- **Task:** the S1 real-corpus probe pooled typed relation planes into a similarity scalar and produced a retracted "mint" verdict; operator ruled it an embarrassment coupled to the scientific jc crate. Removed the entire probe line. +- **Change (branch reset to clean main, single commit):** deleted `examples/p_community_basin_agree.rs`; removed `jc = { path = "../jc" }` from `crates/lance-graph/Cargo.toml` + Cargo.lock (jc was reachable only via this dev-dep); the #722-only real-corpus example + schema.org data blob + python extractor never merged (dropped on reset). G0 harness (#716) untouched (no jc). +- **Verified:** `cargo build -p lance-graph --examples` green without jc; only remaining example is `g0_graph_loadbearing.rs`. +- **Board:** EPIPHANIES `E-S1-PROBE-REMOVED-1`. PR #722 repurposed to the removal; force-pushed onto clean main. + ## 2026-07-17 — E-CONTEXT-ROLE-TISSUE-1 capture — connecting-tissue traversal doctrine (main thread, no subagents) - **Task:** capture the operator's context:role generalization (vertical HHTL / horizontal 6-context) and find the clean cross-domain reuse (screens, documents, time series, AriGraph) with classid+appid+ClassView/WideFieldMask. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 64232c20..0be06475 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,11 @@ +## 2026-07-18 — E-S1-PROBE-REMOVED-1 — the S1 community-basin probe and the lance-graph `jc` dev-coupling are REMOVED. The probe pooled ~1450 typed relation planes into one undirected similarity graph and read a scalar φ as a "community-id earns the mint" verdict — a flattening artifact. jc is the scientific crate; this empirical probe does not belong coupled to it. + +**Status:** SHIPPED (deletes `crates/lance-graph/examples/p_community_basin_agree.rs`; removes the `jc` dev-dependency from lance-graph `Cargo.toml` + `Cargo.lock`. The #722-only real-corpus variant + its schema.org data blob + python extractor never merged and are dropped by resetting the branch to clean main.) + +- **What was wrong.** The probe collapsed typed structure — every relation is its own context:role plane — into one weight-summed adjacency, partitioned the blob, and reported a scalar agreement as a decision. Structure ≠ similarity; the `(8:8)`/ClassView substrate exists to enforce exactly that, and the probe violated it *after* the rule had been stated. The "DISTINCT → community-id earns the mint" verdict was retracted; the whole probe line is now removed rather than iterated. +- **Stands on its own (unaffected):** the context:role traversal-tissue doctrine (`E-CONTEXT-ROLE-TISSUE-1`), the jc-scientific / empirical boundary (`E-EMPIRICAL-VS-SCIENTIFIC-JC-1`), the G0 harness (#716, no jc). The community≡basin identity (`E-GRAPHRAG-DGR3B-1`) is an UNRUN hypothesis again — there is no valid probe for it in-tree. +- **Rule (operator, reinforced):** no empirical probe of mine sits next to or dev-depends on jc. A community/basin agreement, if ever measured, is decomposed per typed plane and does not borrow jc's adjacency. + ## 2026-07-17 — E-CONTEXT-ROLE-TISSUE-1 — operator ruling: `part_of:is_a` ≡ **context:role**, one polymorphic `6×(8:8)` register with TWO orientations — VERTICAL (stacked exactness = HHTL family identity, q2-FMA-helix-attested) and HORIZONTAL (6-context / episodic-witness `basin:role`, eventual) — reused with classid+appid+ClassView/WideFieldMask as the CONNECTING-TISSUE TRAVERSAL for screens, documents, time series, and AriGraph **Status:** CAPTURED (knowledge doc `.claude/knowledge/context-role-traversal-tissue.md`, this commit). Doctrine + reuse checklist + probe pair; no code this arc (the anchors already exist — that is the finding). diff --git a/Cargo.lock b/Cargo.lock index 8e17e3e3..29a475e3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4337,10 +4337,6 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" -[[package]] -name = "jc" -version = "0.1.0" - [[package]] name = "jieba-macros" version = "0.9.0" @@ -4825,7 +4821,6 @@ dependencies = [ "datafusion-sql", "deltalake", "futures", - "jc", "lance", "lance-arrow", "lance-graph-catalog", diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index f8fecc2d..89280fee 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -103,5 +103,4 @@ lance-index = "=7.0.0" tempfile = "3" tokio = { version = "1.37", features = ["macros", "rt-multi-thread"] } causal-edge = { path = "../causal-edge" } -jc = { path = "../jc" } lance-graph-planner = { path = "../lance-graph-planner" } diff --git a/crates/lance-graph/examples/p_community_basin_agree.rs b/crates/lance-graph/examples/p_community_basin_agree.rs deleted file mode 100644 index a50f7a07..00000000 --- a/crates/lance-graph/examples/p_community_basin_agree.rs +++ /dev/null @@ -1,262 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright The Lance Authors - -//! P-COMMUNITY-BASIN-AGREE — the **empirical** probe for the operator's identity -//! ruling: *`part_of:is_a` category ≡ Leiden community ≡ episodic-witness basin — -//! one concept* (graphrag plan §3b.1). -//! -//! # Empirical, not scientific (the layer boundary) -//! -//! This harness is **empirical**: it sets up a measurement over real graph -//! structure and reports an observation. The **statistic** it reports is -//! `jc`'s — the certified scientific crate ([`jc::reliability::pearson`] / -//! [`spearman`]). The empirical construction (the two partitions, their pairwise -//! co-membership vectors) is this probe's; the science is consumed, never -//! re-implemented or extended here. jc has no partition-agreement metric -//! (Rand/ARI/NMI); if the empirical reading turns out to need one, that is an -//! **observation reported to the scientific track**, not a metric bolted on from -//! the empirical side. -//! -//! # What it measures -//! -//! Two partitions of the same entity set: -//! - **Community** — [`TripletGraph::communities`] (multi-level Louvain + -//! Leiden refinement) over the *whole* relational field (all edges). -//! - **Basin** — the connected components under **only** the `is_a`/`part_of` -//! edges (the inherited/taxonomic grouping the plan names as the basin rail). -//! -//! Agreement = Pearson (φ) of the two partitions' pairwise co-membership vectors -//! (`same-community?` vs `same-basin?` over every entity pair). φ = 1.0 iff the -//! partitions are identical up to relabeling — the **identity** the ruling -//! predicts. φ < 1.0 localises the disagreements: those entities are exactly the -//! *discovered bridges / revision candidates* (a community that crosses a basin). -//! -//! # The gate it feeds -//! -//! On a real `is_a`-annotated corpus this number is the S1 gate the other -//! session's D-TRI-1 classid-half mint waits on: **identity (φ→1) ⇒ community-id -//! collapses into the `is_a` rail, no new tenant minted; distinct (φ<1) ⇒ -//! community-id earns its place in the batched mint.** This synthetic fixture -//! exercises the *mechanism*; the verdict needs the real corpus. -//! -//! Run: `cargo run -p lance-graph --example p_community_basin_agree` - -use std::collections::HashMap; - -use jc::reliability::{pearson, spearman}; -use lance_graph::graph::arigraph::triplet_graph::{Triplet, TripletGraph}; - -/// A `(subject, relation, object)` fact. -type Fact = (&'static str, &'static str, &'static str); - -/// Relations that define the **basin** rail (the inherited/taxonomic grouping). -fn is_taxonomic(relation: &str) -> bool { - let r = relation.to_lowercase(); - r.contains("is_a") || r.contains("is a") || r.contains("part_of") || r.contains("part of") -} - -fn build_graph(facts: &[Fact]) -> TripletGraph { - let mut g = TripletGraph::new(); - let ts: Vec = facts - .iter() - .enumerate() - .map(|(i, (s, r, o))| Triplet::new(s, o, r, i as u64)) - .collect(); - g.add_triplets(&ts); - g -} - -/// Basin partition: dense connected-component ids over ONLY the `is_a`/`part_of` -/// edges, aligned to `entities` order. Entities in no taxonomic edge are their -/// own basin. A deterministic union-find (find with path compression, union by -/// lower root so ids are stable given the sorted `entities`). -fn basin_labels(graph: &TripletGraph, entities: &[String]) -> Vec { - let index: HashMap<&str, usize> = entities - .iter() - .enumerate() - .map(|(i, e)| (e.as_str(), i)) - .collect(); - let mut parent: Vec = (0..entities.len()).collect(); - - fn find(parent: &mut [usize], mut x: usize) -> usize { - while parent[x] != x { - parent[x] = parent[parent[x]]; - x = parent[x]; - } - x - } - - for t in &graph.triplets { - if t.is_deleted() || !is_taxonomic(&t.relation) { - continue; - } - let (Some(&a), Some(&b)) = (index.get(t.subject.as_str()), index.get(t.object.as_str())) - else { - continue; - }; - let (ra, rb) = (find(&mut parent, a), find(&mut parent, b)); - if ra != rb { - // union by lower root → deterministic given sorted entities. - let (lo, hi) = (ra.min(rb), ra.max(rb)); - parent[hi] = lo; - } - } - - // Densify roots to 0..k by first appearance (deterministic). - let mut root_to_dense: HashMap = HashMap::new(); - let mut next = 0u32; - (0..entities.len()) - .map(|i| { - let r = find(&mut parent, i); - *root_to_dense.entry(r).or_insert_with(|| { - let d = next; - next += 1; - d - }) - }) - .collect() -} - -/// Pairwise co-membership over all `i Vec { - let n = labels.len(); - let mut v = Vec::with_capacity(n * (n - 1) / 2); - for i in 0..n { - for j in (i + 1)..n { - v.push(if labels[i] == labels[j] { 1.0 } else { 0.0 }); - } - } - v -} - -fn distinct(labels: &[u32]) -> usize { - let mut s = labels.to_vec(); - s.sort_unstable(); - s.dedup(); - s.len() -} - -fn measure(title: &str, facts: &[Fact]) { - let graph = build_graph(facts); - let comms = graph.communities(); - let community = comms.labels.clone(); - let basin = basin_labels(&graph, &comms.entities); - - let comm_co = comembership(&community); - let basin_co = comembership(&basin); - - // Scientific statistic — jc, consumed as-is. - let phi = pearson(&comm_co, &basin_co); - let rho = spearman(&comm_co, &basin_co); - // Empirical descriptive — raw fraction of pairs that agree (Rand-like). - let agree_frac = if comm_co.is_empty() { - f64::NAN - } else { - comm_co - .iter() - .zip(&basin_co) - .filter(|(a, b)| a == b) - .count() as f64 - / comm_co.len() as f64 - }; - - // Bridges = entities whose basin differs from the MAJORITY basin of their - // community — the minority member a community pulled across a basin line - // (the discovered bridge / revision candidate, plan §3b.1). - let mut comm_basin_counts: HashMap> = HashMap::new(); - for i in 0..community.len() { - *comm_basin_counts - .entry(community[i]) - .or_default() - .entry(basin[i]) - .or_insert(0) += 1; - } - let majority_basin: HashMap = comm_basin_counts - .iter() - .map(|(&c, counts)| { - let maj = counts - .iter() - .max_by_key(|(_, &n)| n) - .map(|(&b, _)| b) - .unwrap_or(0); - (c, maj) - }) - .collect(); - let bridges: Vec<&str> = comms - .entities - .iter() - .enumerate() - .filter(|&(i, _)| basin[i] != majority_basin[&community[i]]) - .map(|(_, e)| e.as_str()) - .collect(); - - println!("── {title} ──"); - println!( - " entities={} communities={} basins={}", - comms.entities.len(), - distinct(&community), - distinct(&basin) - ); - println!( - " φ (jc::pearson) = {} ρ (jc::spearman) = {} pair-agreement = {:.3}", - phi.map(|x| format!("{x:.4}")) - .unwrap_or_else(|| "n/a (constant)".into()), - rho.map(|x| format!("{x:.4}")) - .unwrap_or_else(|| "n/a (constant)".into()), - agree_frac, - ); - match phi { - Some(p) if p > 0.999 => { - println!(" → IDENTITY on this fixture (φ≈1): community ≡ basin, no bridges.") - } - Some(_) => println!( - " → DISTINCT-with-bridges: {:?} cross a basin (the revision candidates).", - bridges - ), - None => println!(" → degenerate (a partition is trivial); not informative."), - } - println!(); -} - -fn main() { - println!("== P-COMMUNITY-BASIN-AGREE (empirical probe; statistic = jc science) ==\n"); - - // Scenario A — clean: two taxonomic basins, structure matches exactly. - // Expect φ≈1.0 (the identity the ruling predicts, cleanly demonstrated). - let clean: &[Fact] = &[ - ("cat", "is_a", "pet"), - ("dog", "is_a", "pet"), - ("hamster", "is_a", "pet"), - ("cat", "relates_to", "dog"), - ("dog", "relates_to", "hamster"), - ("hamster", "relates_to", "cat"), - ("oak", "is_a", "tree"), - ("pine", "is_a", "tree"), - ("birch", "is_a", "tree"), - ("oak", "relates_to", "pine"), - ("pine", "relates_to", "birch"), - ("birch", "relates_to", "oak"), - ]; - measure("A · aligned (identity expected)", clean); - - // Scenario B — one deliberate cross: `robot` is_a pet (basin=pet) but is - // structurally pulled entirely into the tree cluster (community=tree). - // Expect φ<1.0 with `robot` named as the single bridge / revision candidate. - let mut bridged: Vec = clean.to_vec(); - bridged.extend_from_slice(&[ - ("robot", "is_a", "pet"), - ("robot", "relates_to", "oak"), - ("robot", "relates_to", "pine"), - ("robot", "relates_to", "birch"), - ]); - measure("B · one cross-basin bridge (distinct expected)", &bridged); - - println!( - "NOTE: synthetic mechanism, NOT the verdict. The S1 gate reads φ on a REAL\n\ - is_a-annotated corpus: identity (φ→1) ⇒ community-id collapses into the\n\ - is_a rail, NO new tenant minted; distinct (φ<1) ⇒ community-id earns the\n\ - batched D-TRI-1 mint. jc supplies the statistic; if a Rand/ARI/NMI metric\n\ - is wanted, that is an observation for the scientific (jc) track." - ); -} diff --git a/crates/lance-graph/src/graph/arigraph/episodic.rs b/crates/lance-graph/src/graph/arigraph/episodic.rs index 8afeafbc..9a5bb318 100644 --- a/crates/lance-graph/src/graph/arigraph/episodic.rs +++ b/crates/lance-graph/src/graph/arigraph/episodic.rs @@ -6,6 +6,8 @@ //! Stores observation episodes with fingerprint-based similarity retrieval, //! enabling agents to recall relevant past experiences. +use std::collections::{BTreeSet, HashMap}; + use crate::graph::fingerprint::{hamming_distance, label_fp, Fingerprint}; use crate::graph::spo::truth::TruthValue; @@ -65,6 +67,43 @@ pub struct EpisodicMemory { capacity: usize, } +/// An **episodic-basin** partition over the entities observed in +/// [`EpisodicMemory`]. +/// +/// The *experiential* complement to +/// [`Communities`](super::community::Communities): entities are grouped by +/// **co-occurrence in episodes** (what was observed together), not by structural +/// graph connectivity (what is linked). Same accessor shape as `Communities`, so +/// the two partitions can be read the same way and compared by a caller. +#[derive(Debug, Clone)] +pub struct EpisodicBasins { + /// Sorted, deduped entity names observed across episodes. + pub entities: Vec, + /// Basin id per entity, parallel to [`Self::entities`]. + pub labels: Vec, + /// Number of distinct basins. + pub num_basins: usize, +} + +impl EpisodicBasins { + /// The basin id of `entity`, if it was observed in any episode. + pub fn basin_of(&self, entity: &str) -> Option { + self.entities + .iter() + .position(|e| e == entity) + .map(|i| self.labels[i]) + } + + /// The entity names in `basin`. + pub fn members(&self, basin: u32) -> Vec<&str> { + self.entities + .iter() + .zip(&self.labels) + .filter_map(|(e, &b)| if b == basin { Some(e.as_str()) } else { None }) + .collect() + } +} + impl EpisodicMemory { /// Create a new episodic memory with the given capacity. pub fn new(capacity: usize) -> Self { @@ -138,6 +177,90 @@ impl EpisodicMemory { self.episodes.len() } + /// Partition observed entities into **episodic basins**: two entities share + /// a basin iff they co-occur in at least one stored episode (both names + /// appear in that episode's `"subject - relation - object"` triplets). This + /// is the *experiential* grouping — the complement to the *structural* + /// [`TripletGraph::communities`](super::triplet_graph::TripletGraph::communities) + /// partition. An entity never seen alongside another is its own basin. + /// + /// Deterministic: entities are sorted, and the union-find merges by lower + /// root, so the same episode set always yields the same partition. + pub fn basins(&self) -> EpisodicBasins { + // 1. Per-episode entity sets + the global sorted entity list. + let mut per_episode: Vec> = Vec::with_capacity(self.episodes.len()); + let mut all: BTreeSet = BTreeSet::new(); + for ep in &self.episodes { + let mut ents: Vec = Vec::new(); + for triplet in &ep.triplets { + let parts: Vec<&str> = triplet.split(" - ").collect(); + if parts.len() == 3 { + for raw in [parts[0].trim(), parts[2].trim()] { + if !raw.is_empty() { + ents.push(raw.to_string()); + all.insert(raw.to_string()); + } + } + } + } + ents.sort_unstable(); + ents.dedup(); + per_episode.push(ents); + } + let entities: Vec = all.into_iter().collect(); + let index: HashMap<&str, usize> = entities + .iter() + .enumerate() + .map(|(i, e)| (e.as_str(), i)) + .collect(); + + // 2. Union-find over co-occurrence (all entities in one episode merge). + let mut parent: Vec = (0..entities.len()).collect(); + fn find(parent: &mut [usize], mut x: usize) -> usize { + while parent[x] != x { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + x + } + for ents in &per_episode { + let mut anchor: Option = None; + for e in ents { + let i = index[e.as_str()]; + match anchor { + None => anchor = Some(i), + Some(a) => { + let (ra, rb) = (find(&mut parent, a), find(&mut parent, i)); + if ra != rb { + let (lo, hi) = (ra.min(rb), ra.max(rb)); + parent[hi] = lo; + } + } + } + } + } + + // 3. Densify roots to 0..k by first appearance (deterministic). + let mut dense: HashMap = HashMap::new(); + let mut next = 0u32; + let labels: Vec = (0..entities.len()) + .map(|i| { + let r = find(&mut parent, i); + *dense.entry(r).or_insert_with(|| { + let d = next; + next += 1; + d + }) + }) + .collect(); + let num_basins = dense.len(); + EpisodicBasins { + entities, + labels, + num_basins, + } + } + /// True if no episodes are stored. pub fn is_empty(&self) -> bool { self.episodes.is_empty() @@ -423,4 +546,57 @@ mod tests { fn threshold_matches_contract_constant() { assert!((UNBUNDLE_HARDNESS_THRESHOLD - 0.8).abs() < 1e-6); } + + // ── episodic basins ────────────────────────────────────────────────── + + fn basin_mem(episodes: &[&[&str]]) -> EpisodicMemory { + let mut m = EpisodicMemory::new(64); + for (i, trips) in episodes.iter().enumerate() { + let ts: Vec = trips.iter().map(|s| s.to_string()).collect(); + m.add(&format!("obs{i}"), &ts, i as u64); + } + m + } + + #[test] + fn basins_group_co_occurring_entities() { + let m = basin_mem(&[&["alice - knows - bob"], &["carol - knows - dave"]]); + let b = m.basins(); + assert_eq!(b.num_basins, 2); + assert_eq!(b.basin_of("alice"), b.basin_of("bob")); + assert_eq!(b.basin_of("carol"), b.basin_of("dave")); + assert_ne!(b.basin_of("alice"), b.basin_of("carol")); + } + + #[test] + fn basins_merge_on_a_shared_entity() { + // `bob` bridges the two episodes → one basin of {alice, bob, carol}. + let m = basin_mem(&[&["alice - knows - bob"], &["bob - knows - carol"]]); + let b = m.basins(); + assert_eq!(b.num_basins, 1); + assert_eq!(b.members(b.basin_of("alice").unwrap()).len(), 3); + } + + #[test] + fn basins_union_all_entities_in_one_episode() { + let m = basin_mem(&[&["a - r - b", "b - r - c", "c - r - d"]]); + let b = m.basins(); + assert_eq!(b.num_basins, 1); + assert_eq!(b.entities.len(), 4); + } + + #[test] + fn basins_empty_memory_is_safe() { + let b = EpisodicMemory::new(8).basins(); + assert_eq!(b.num_basins, 0); + assert!(b.entities.is_empty()); + } + + #[test] + fn basins_deterministic() { + let m = basin_mem(&[&["alice - r - bob"], &["bob - r - carol"]]); + let (x, y) = (m.basins(), m.basins()); + assert_eq!(x.entities, y.entities); + assert_eq!(x.labels, y.labels); + } } diff --git a/crates/lance-graph/src/graph/arigraph/mod.rs b/crates/lance-graph/src/graph/arigraph/mod.rs index 8fd9ab2c..6af26b86 100644 --- a/crates/lance-graph/src/graph/arigraph/mod.rs +++ b/crates/lance-graph/src/graph/arigraph/mod.rs @@ -21,6 +21,7 @@ pub mod xai_client; pub use bm25::Bm25Index; pub use community::Communities; +pub use episodic::EpisodicBasins; pub use ppr::PersonalizedPageRank; pub use witness_corpus::{WitnessCorpus, WitnessEntry, WitnessId, WitnessIndexHashMap};