Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 13 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -926,7 +926,7 @@ jobs:
extractor.log

integration-postgres:
name: pytest integration (Postgres ${{ matrix.postgres }} memory authority)
name: pytest integration (Postgres ${{ matrix.postgres }}, ${{ matrix.profile }})
needs: [validate-ref, clickhouse]
runs-on: ubuntu-latest
timeout-minutes: 60
Expand All @@ -935,6 +935,7 @@ jobs:
fail-fast: false
matrix:
postgres: ["16", "18"]
profile: [storage, isolation]
env:
LOOM_CLICKHOUSE_IMAGE: ${{ needs.clickhouse.outputs.image }}
LOOM_CH_HOST: localhost
Expand Down Expand Up @@ -1016,6 +1017,7 @@ jobs:
tail -100 server.log
exit 1
- name: pytest (Postgres authority, memory, knowledge, isolation and the SQL surface)
if: ${{ matrix.profile == 'storage' }}
run: >-
pytest -q
tests/integration/test_pg_authority.py
Expand Down Expand Up @@ -1096,6 +1098,7 @@ jobs:
tests/integration/test_marked_durable_folds_native.py
tests/integration/test_marked_sql_surface_native.py
- name: pytest online-index durability on a private physical standby
if: ${{ matrix.profile == 'storage' }}
timeout-minutes: 8
env:
LOOM_PG_IMAGE: postgres:${{ matrix.postgres }}
Expand Down Expand Up @@ -1124,6 +1127,7 @@ jobs:
docker update --memory 1g --memory-swap 1g loom-scale-index-drill-pg loom-scale-index-drill-pg-standby
pytest -q tests/integration/test_pg_online_index_durability.py
- name: pytest placement durability on a private physical standby
if: ${{ matrix.profile == 'storage' }}
timeout-minutes: 8
env:
LOOM_PG_IMAGE: postgres:${{ matrix.postgres }}
Expand Down Expand Up @@ -1162,12 +1166,12 @@ jobs:
export LOOM_CELL_PLACE_DRILL_DSN="postgres://loom:loomdev@127.0.0.1:$port/loom"
docker update --memory 1g --memory-swap 1g loom-scale-place-drill-pg loom-scale-place-drill-pg-standby
pytest -q tests/integration/test_cell_place_durability.py
# The tenant-isolation gates again, against a second server that enforces auth:
# The tenant-isolation profile uses a second server that enforces auth:
# the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens.
# They run whether or not the memory suites passed (`!cancelled()`).
# Its matrix job runs independently of the storage profile's outcome.
- name: Boot control plane with auth on
id: auth_on
if: ${{ !cancelled() && steps.issuer.outcome == 'success' }}
if: ${{ !cancelled() && matrix.profile == 'isolation' && steps.issuer.outcome == 'success' }}
env:
LOOM_AUTH_REQUIRED: "true"
LOOM_INSTANCE_LABEL: "ci:integration-postgres-auth"
Expand All @@ -1187,7 +1191,7 @@ jobs:
# extraction queue, projection failures) hold no Postgres row: they run on the
# ClickHouse job.
- name: pytest isolation gates (Postgres authority, auth on)
if: ${{ !cancelled() && steps.auth_on.outcome == 'success' }}
if: ${{ !cancelled() && matrix.profile == 'isolation' && steps.auth_on.outcome == 'success' }}
env:
LOOM_AUTH_REQUIRED: "true"
LOOM_BASE_URL: http://localhost:7778
Expand All @@ -1209,16 +1213,16 @@ jobs:
tests/integration/test_pg_authority.py
-k "not test_a_duplicate_never_resolves_to_a_canonical_outside_the_scope
and not test_a_superseded_hit_never_names_a_successor_outside_the_scope"
# Last, on its own: `loom reconcile` acts on the whole database, every other test's
# rows included, so it runs after them all and in its file's order.
# Last in the storage profile: `loom reconcile` acts on the whole database,
# the storage tests' rows included, so it runs after them in its file's order.
- name: pytest (loom reconcile drills)
if: ${{ !cancelled() }}
if: ${{ !cancelled() && matrix.profile == 'storage' }}
run: pytest -q -p no:randomly tests/integration/test_restore_reconcile.py
- name: Upload server log on failure
if: failure()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: server-log-postgres-${{ matrix.postgres }}
name: server-log-postgres-${{ matrix.postgres }}-${{ matrix.profile }}
path: |
server.log
server-auth.log
Expand Down
21 changes: 13 additions & 8 deletions bench/conflict_resolver_long_session.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
namespace: six attributes (home city, employer, manager, standup time, CI
provider, diet) change once or twice, each change under a new key and in new
words, among restated, elaborated and neighbouring facts, a pinned fact that is
contradicted later, a fact deleted and brought back with Undo, and eighteen
contradicted later, a permanently deleted fact, and eighteen
unrelated ones (`bench/conflict_resolver_scenario.py`, the scenario and its
ground truth; `tests/integration/test_conflict_resolver_long_session_ch.py`
runs it with a scripted judge). The harness then waits for the server's
Expand Down Expand Up @@ -91,12 +91,13 @@
from bench.conflict_resolver_scenario import (
ATTRIBUTES,
BY_KEY,
CHECK_DELETED_IN,
CURRENT,
DELETE_IN,
DELETED,
FACTS,
KEPT,
OLDER,
RESTORE_IN,
SESSIONS,
dialogue,
relation_of,
Expand Down Expand Up @@ -150,7 +151,7 @@ def _entry(start: datetime, session: int, i: int, key: str) -> dict[str, Any]:


def write_set_session(post: Post, namespace: str, session: int, start: datetime) -> None:
"""The session's facts under the scenario's keys, then its delete or Undo."""
"""The session's facts under the scenario's keys, then its deletion or absence check."""
keys = [f.key for f in FACTS if f.session == session]
entries = [_entry(start, session, i, k) for i, k in enumerate(keys)]
ident = _identity(namespace, session)
Expand All @@ -162,9 +163,12 @@ def write_set_session(post: Post, namespace: str, session: int, start: datetime)
if session in DELETE_IN:
key = DELETE_IN[session]
post("memory.delete", {**_identity(namespace, BY_KEY[key].session), "memory_key": key})
if session in RESTORE_IN:
key = RESTORE_IN[session]
post("memory.restore", {**_identity(namespace, BY_KEY[key].session), "memory_key": key})
if session in CHECK_DELETED_IN:
key = CHECK_DELETED_IN[session]
absent = post(
"memory.get", {**_identity(namespace, BY_KEY[key].session), "memory_key": key}
)
assert absent["found"] is False, absent


def write_messages_session(post: Post, namespace: str, session: int, start: datetime) -> None:
Expand Down Expand Up @@ -320,7 +324,8 @@ def score_by_key(rows: dict[str, dict[str, Any]], top: dict[str, str]) -> dict[s
"flagged_current": sum(
1 for k in CURRENT if k in rows and rows[k]["status"] == "conflicted"
),
"missing_rows": sorted({f.key for f in FACTS} - set(rows)),
"missing_rows": sorted({f.key for f in FACTS} - DELETED - set(rows)),
"deleted_returned": sorted(DELETED & set(rows)),
"older_live": sorted(k for k in OLDER if k in rows and _live(rows[k])),
}

Expand Down Expand Up @@ -480,7 +485,7 @@ def print_plan(servers: list[tuple[str, str]], variants: list[str]) -> None:
for name, a in ATTRIBUTES.items():
values = " -> ".join(f"{BY_KEY[k].value} (s{BY_KEY[k].session})" for k in a.keys)
print(f" {name}: {values}")
print(f"kept live: {len(KEPT)} facts; deleted+restored: {sorted(set(DELETE_IN.values()))}")
print(f"kept live: {len(KEPT)} facts; permanently deleted: {sorted(set(DELETE_IN.values()))}")
for label, url in servers or [("on", "<url>"), ("off", "<url>")]:
for v in variants:
print(f"plan: {label} {url} variant={v}")
Expand Down
15 changes: 7 additions & 8 deletions bench/conflict_resolver_scenario.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
key and in new words. The last value is current (`CURRENT`); every earlier
one should end superseded behind the next (`OLDER`).
- Four facts are restated, four elaborated, four sit near a different fact, one
pinned fact is contradicted later, one fact is deleted and brought back with
Undo, and eighteen stand alone. All of them should stay live and unflagged
pinned fact is contradicted later, one fact is permanently deleted, and eighteen
stand alone. The other facts stay live and unflagged
(`KEPT`).
- `JUDGE` holds the right verdict for every pair a cluster should hold, except
for the scripted judge's deliberate mistakes: two wrong answers at low
Expand Down Expand Up @@ -170,7 +170,7 @@ def near(topic: str, cos: float, axis: int = 1) -> tuple[Part, ...]:
# Pinned, and contradicted later: a pinned row is never retired.
F(3, "code_freeze", "Code freeze starts every Friday at noon", alone("freeze"), pinned=True),
F(9, "freeze_change", "Code freeze now starts on Thursday evening", near("freeze", 0.75)),
# Deleted in session 4, brought back with Undo in session 6, elaborated in 9.
# Deleted in session 4; session 6 checks its key stays absent.
F(3, "badge", "The user's badge number is 4471", alone("badge")),
F(9, "badge_lab", "Badge 4471 also opens the third-floor lab", near("badge", 0.75)),
# Unrelated, each in its own direction: no peer, never judged.
Expand All @@ -196,9 +196,9 @@ def near(topic: str, cos: float, axis: int = 1) -> tuple[Part, ...]:
SESSIONS = 12
SOLO = {f.key for f in FACTS if f.at[0][0].startswith("solo")}

# The key deleted in a session, and the session whose Undo brings it back (after
# each session's writes).
DELETE_IN, RESTORE_IN = {4: "badge"}, {6: "badge"}
# The key deleted in a session, and the later session that checks its continued absence.
DELETE_IN, CHECK_DELETED_IN = {4: "badge"}, {6: "badge"}
DELETED = set(DELETE_IN.values())

# The relations are not single-valued, so no write closes a fact on its own.
ATTRIBUTES: dict[str, Attribute] = {
Expand Down Expand Up @@ -240,7 +240,6 @@ def near(topic: str, cos: float, axis: int = 1) -> tuple[Part, ...]:
frozenset({"marathon", "marathon_plan"}): ("elaboration", 0.96, True, True),
frozenset({"rust", "rust_plan"}): ("elaboration", 0.95, True, False),
frozenset({"guitar", "guitar_band"}): ("elaboration", 0.96, True, True),
frozenset({"badge", "badge_lab"}): ("elaboration", 0.96, True, False),
frozenset({"sister_city", "home_city"}): ("unrelated", 0.97, False, False),
frozenset({"skip_level", "manager"}): ("related", 0.95, False, False),
frozenset({"car", "partner_car"}): ("unrelated", 0.97, False, True),
Expand All @@ -262,7 +261,7 @@ def near(topic: str, cos: float, axis: int = 1) -> tuple[Part, ...]:

CURRENT = {a.keys[-1] for a in ATTRIBUTES.values()}
OLDER = {k for a in ATTRIBUTES.values() for k in a.keys[:-1]}
KEPT = {f.key for f in FACTS} - CURRENT - OLDER
KEPT = {f.key for f in FACTS} - CURRENT - OLDER - DELETED
BY_KEY = {f.key: f for f in FACTS}


Expand Down
17 changes: 15 additions & 2 deletions docs/architecture/clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -652,8 +652,9 @@ on `valid_to`, `invalid_at`, `version`; `set` on `status` and `memory_type`.

Loom deletes memory by writing a tombstone version (`tombstoned_at` set, every other column
carried). It is visible to the next `FINAL` read, keeps prior versions in
`memory_versions` and leaves `memory.restore` a window. Physical removal is for erasure
with `purge=true`, runner state, and the supersession rows `knowledge.restore` removes.
`memory_versions`. Only automatic retirement without permanent refusal has a restore window. Physical removal is for erasure
with `purge=true`, stale vector caches, runner state, and the supersession rows
`knowledge.restore` removes.

- **`knowledge.restore`** runs a lightweight `DELETE FROM loom.chunk_supersessions` for the
rows one delete wrote: the document's `source_uri`, the ingestions of its revision and
Expand Down Expand Up @@ -705,6 +706,18 @@ with `purge=true`, runner state, and the supersession rows `knowledge.restore` r
candidate review lease. An invalidated model attempt instead removes only its own
candidate versions through a lightweight `DELETE FROM`, bounded by the claim version,
so it cannot remove a later review decision.
- **Orphaned vector retirement** deletes cache versions below the caller's cutoff by
the complete vector sort key, across all creation partitions. A missing authoritative
source or matching erasure policy cannot be represented by a newly inserted marker:
even an empty marker would recreate residue after a purge. Admission fences stay held
until the synchronous lightweight delete is acknowledged; equal and newer versions stay.
- **Shared canonical retirement** uses the same bounded lightweight deletion when its
shared head is absent, source identities are unproven or erasure policy matches. The
cutoff preserves a later promotion and avoids writing a tombstone after a purge.
- **Refused extraction jobs** end through a control-only `ALTER TABLE … UPDATE` on the
exact identity, queue id, source version and payload hash. It writes a fixed reason and
terminal status without copying transcript bytes or creating a row. Unrelated jobs in
the same claim batch can still run.
- **Extraction retries** (`memory.extraction_retry_mine`) update failed queue rows in
place, asynchronously. The update targets the exact identity and version observed, with
the caller's namespace, principal and read scope. Copying the transcript into a new
Expand Down
21 changes: 11 additions & 10 deletions docs/architecture/memory-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -724,10 +724,10 @@ is left. Its triples close as for a row that holds none.

A revision (`memory.set`, `memory.bulk_set`, `memory.update_by_id`) of a row that
existed before the write closes the facts its row held that no row holds any longer, and
an Undo of a retirement or of a delete opens again those the restored row's current value
an Undo of reversible retirement opens again those the restored row's current value
holds by the same rule (another row's claim keeps a fact from closing, but never reopens
one); either, when it fails, is recorded for `projection_repair` as `triples_reconcile`.
A delete closes none of its row's facts, so the only facts a delete's Undo opens again
A tombstone closes none of its row's facts, so the only facts a reversible restore opens again
are ones another row's revision or retirement closed while the row was gone.

A dropped fact still valid later ("until Dec 1") ends now, and so does one a retired row
Expand Down Expand Up @@ -1004,14 +1004,15 @@ session, agent or run identity. A read bounded on one of those axes omits the ag
graph signals, including hub pruning and graph paths. Its fact legs retain their scoped
cosine and query-entity ranking. Namespace-wide reads use the graph aggregates.

Session digests carry the keys they compressed in `derived_from`. Before deduplication,
the summary leg checks those known sources under the digest's full identity. A source
that is missing, unserved, deleted, expired, outside its validity, high-sensitivity or at a newer
version withholds the digest. An unreadable source check withholds all unchecked digests.
Without source revision hashes, a later source version conservatively withholds a digest
during replay too. Lineage covers extracted facts, not every sentence of the transcript:
a digest without a known source remains session evidence, and does not prove that its
statements are current facts.
Session digests contain whole committed source values joined by newlines. Their ordered
`derived_from` and `key_facts` reconstruct every byte of `summary_text`; raw transcript
text and model-generated additions cannot contribute. Every digest reader verifies that
proof against the authoritative heads under the digest's exact agent, user and session.
Missing, changed, unserved, expired, truncated, sensitive or unreviewed sources withhold
the digest. A pin or touch that leaves the same eligible content does not invalidate it.
Legacy digests without complete proof are withheld. Source-read failures fail closed.
A digest hit cannot open unrelated raw turns from its session. Model-generated period
lessons and contrast reflections enter proposed state for independent review.

`memory.list` excludes rejected rows from both sort orders and counts. A key lookup can
inspect a rejected row. `memory.for_file` also checks `valid_to` before adding a fact to
Expand Down
Loading
Loading