Skip to content

did2: reopen closed connections; faster reads and writes (link and value side tables, schema-cache memo, read shortcut) - #218

Draft
jesshaley wants to merge 32 commits into
V2from
claude/ecstatic-pascal-md91fw
Draft

did2: reopen closed connections; faster reads and writes (link and value side tables, schema-cache memo, read shortcut)#218
jesshaley wants to merge 32 commits into
V2from
claude/ecstatic-pascal-md91fw

Conversation

@jesshaley

@jesshaley jesshaley commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

did2.database.sqlitedb now survives having its mksqlite connection closed by someone else, and never uses or closes a connection number that has since been given to another database.

Why

mksqlite numbers its connections, and code outside a database object closes them by number:

  • ndi.dataset runs mksqlite('close') (dbid 1) after opening a session, at eight call sites;
  • ndi.session.dir runs mksqlite(0, 'close') (all of them) before deleting a session, and says why: "DID reopens lazily on the next operation (issue #870)".

The legacy backend (did.implementations.sqlitedb) follows that contract. did2.database.sqlitedb opened once in its constructor and kept the number, so a V2 session opened from an ndi.dataset failed every search with MKSQLITE:ANY: database not open. This was found by running the NDI V2 object layer on the real Haley dataset (VH-Lab/NDI-matlab#1021).

A closed number is also reused by the next mksqlite(0, 'open'). A database still holding it would then query another database's file without an error. That is the more dangerous half.

What changed

  • ensureConnection() runs at the start of every public operation (add, remove, has, get, allIds, count, search, hasFile, filePath, fileNames, testHookDbId).
    • It checks with one PRAGMA database_list that the number is open and still names this database's file.
    • Otherwise it reopens the file on the next free number and re-applies foreign_keys = ON.
    • It never closes the old number, which may now be someone else's connection.
  • close() closes the number only while it is still this database's. A database closed by its owner stays closed: an operation on it raises did2:database:closed instead of silently reopening.
  • isOpen() is now false only after the owner's close(). A connection closed by someone else still counts as open, because it reopens on next use.
  • testHookReconnects() is added for tests.

Tests

tests/+did2/+unittest/testSqliteDbReconnect.m (run by test-builder.yml, which picks up testSqliteDb*):

  • reopens after mksqlite(0, 'close');
  • reopens after its own number is closed;
  • a reused number is not queried: A reads its own file, and B's connection is untouched;
  • close() leaves a reused number alone;
  • an owner-closed database stays closed (did2:database:closed), and close() is idempotent.

Also in this PR: statements by composition (did-schema#88)

did-schema#88 (signed 2026-10-08) makes a statement's class its direction (observation, ...) and lists its value kind (temperature, ...) in superclasses after the direction's own chain.

  • Validator (+schema/cache.m): valueKindRule, documentValueKind (did2:validation:missingValueKind / badValueKind), documentAncestors, documentChain, resolvePlacementFor. validateDocument, rehydrate and buildBlankDocument follow the kind.
  • Builder (did2.build.document): a ValueKind option; a leaf name such as 'temperature_observation' resolves to observation + temperature on a schema without that leaf (private/resolveLeaf.m), so callers work on both schemas. statement.m writes value.data_type or value.datum_type, whichever the schema has.
  • ensureClassBlocks keeps the kind. NDI's read path re-stamps every document's superclasses with it, and it rebuilt them from the class alone, so a composed statement lost its value kind on every read and its value could not be found. It now re-stamps with documentAncestors(class, kind); testBuild/testTheValidatorChecksTheValueKind checks a round trip.

Also in this PR: searches that name a link start from its index

Found with the new testHookExplain (a hidden helper that returns a query's SQL and SQLite's EXPLAIN QUERY PLAN), on the real Haley V2 dataset: 187,673 documents, one worm's 22 statements.

compileQuery wrote a named depends_on condition as a correlated EXISTS. SQLite evaluates that once per document, so a query with nothing else to narrow it read every document. That is what ndi.dataset sends: isa plus one link.

form plan time
before (EXISTS) SCAN documents 0.708 s
same query + session id SEARCH documents USING INDEX documents_q_base_session_id 0.029 s
starting from the link SEARCH d USING INDEX depends_on_name_document_id 0.0007 s

A named link is now documents.id IN (SELECT d.doc_id FROM depends_on d WHERE d.name = ? AND d.document_id = ?), and its negation is NOT IN (doc_id is NOT NULL). The wildcard name stays a correlated EXISTS, because no index has document_id first. No index is added, and existing databases are unchanged.

Tests:

  • testCompileQuery pins the SQL shape, including the new NOT IN case.
  • testSqliteDb/testSearchDependsOnCombinations pins the results: the link alone, negated, with isa, inside an or, with a scalar condition, and with the wildcard. It also checks that the plan does not scan documents.
  • testSqliteDbReconnect/testExplainReportsAPlanAndWritesNothing covers the helper.

Also in this PR: faster reads and writes, measured on the Haley V2 dataset

Each change was measured on a real dataset before and after. The read timings are ndi.subject.find over 10,536 subjects, warm. The write timings are importing one session (≈5,000 documents written) under MATLAB's profiler.

change where effect
v1_to_v2 option Audits (default on) +convert/v1_to_v2.m lets NDI's read path skip the report-only census (silentLoss etc.) on every read: 29.9 s → 7.8 s, together with NDI reading each document once
schema cache remembers superclasses / resolvePlacement per class +schema/cache.m 900,580 getClass calls → a few thousand: 7.8 s → 2.4 s
isAlreadyTarget, ensureClassBlocks made public +convert/ NDI skips the full read pass for documents already in V2 form: 2.4 s → 1.9 s
array-index paths split once and grouped by block; rows inserted several to a statement +database/sqlitedb.m 910,420 path lookups and ~80% of the SQL calls removed: 188 s → 176 s
queryable values in queryable_scalar_elem instead of 675 indexed generated columns +database/sqlitedb.m, compileQuery.m per-insert index upkeep removed: SQL write time 35.0 s → 1.0 s; writing a session 24 s → 5 s (dataset) and 21 s → 4 s (session folder); whole run 176 s → 141 s

queryable_scalar_elem changes the on-disk format. A V2 database written before it (with q_* generated columns) is converted once when opened: the documents table is rebuilt without the generated columns and their indexes, and the side table is filled from the stored bodies. Older DID-matlab opening a converted file would rebuild the generated columns again, so the change is safe both ways but slow on each switch. testSqliteDb/testAnOldLayoutDatabaseIsConvertedOnOpen covers the conversion. That test caught a real bug: pragma_table_info does not list generated columns, so detection uses pragma_table_xinfo.

The quick suite's failures are the same 160 throughout, identical to #217's. The did2.build tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf

Generated by Claude Code

Generated by Claude Code

@claude
did2.database.sqlitedb: reopen a connection closed under it; never us…
c50b595 
…e a reused number

mksqlite numbers its connections, and code outside a database object
closes them by number: ndi.dataset runs mksqlite('close') after opening a
session, ndi.session.dir runs mksqlite(0, 'close') before deleting one.
The legacy backend reopens on next use (issue #870); sqlitedb opened once
in its constructor and kept the number, so a V2 session opened from an
ndi.dataset failed every search with "database not open". A closed number
is also reused by the next open, so a database still holding it could
have read another database's file without an error.

Every public operation now checks, with one PRAGMA database_list, that its
number is open AND still names its own file, and otherwise reopens on the
next free number. close() closes the number only while it is still its
own, and a database closed by its owner stays closed
(did2:database:closed). testSqliteDbReconnect pins all four cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@jesshaley Claude

jesshaley commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Quick migrator tests is red on this PR, and the failures are not this PR's: the failing list is identical, test for test, to the one on 4f0558c (#217, merged into V2). Both runs list the same 160 entries in their failure summary (epoch minting, family uniqueness, batch-pass wiring and others). None of them touches did2.database.sqlitedb. No fix for them exists yet, so there is nothing to port into this PR.

The checks that run this change are green: did2.build tests (V_eta), which runs every testSqliteDb* file including the new testSqliteDbReconnect, and Fixture corpus (fast validate). Test code is still running.

Generated by Claude Code

claude added 2 commits October 6, 2026 20:34
@claude
did2.database.sqlitedb: testHookExplain, the SQL and SQLite plan of a…
e172446 
… query

For diagnosing slow searches: returns the statement search() runs, its
bound values and EXPLAIN QUERY PLAN. Runs nothing else, writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
compileQuery: a named depends_on is documents.id IN (...), so SQLite …
d95c5c7 
…starts from the edge index

A correlated EXISTS is evaluated once per document, so the planner had to
walk the documents table first; a query with nothing else to narrow it
(an isa plus an edge, as ndi.dataset sends) scanned every document.
Measured on the Haley V2 dataset (187,673 documents, a worm's 22
statements, EXPLAIN QUERY PLAN through testHookExplain): SCAN documents,
0.708 s; starting from depends_on(name, document_id), 0.0007 s.

The negated form is NOT IN (doc_id is NOT NULL). The wildcard name stays a
correlated EXISTS: no index has document_id first. testSqliteDb pins the
answers (alone, negated, with isa, in an or, with a scalar, wildcard) and
that the plan does not scan documents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@jesshaley jesshaley changed the title did2.database.sqlitedb: reopen a connection closed under it; never use a reused number did2.database.sqlitedb: survive connections closed under it; searches that name a link start from its index Oct 6, 2026
claude added 7 commits October 6, 2026 20:59
@claude
v1_to_v2: 'Audits' option, so a read path can skip the report-only ce…
3b99cad 
…nsus

silentLoss, fileList and timeReferenceFamilies ran on every call, including
NDI's one-document read normalisation, which discards them. Profiled on
NDI reading the 10,536 Haley V2 subjects: 23.5 s of 51 s (silentLoss
17.5 s alone). Default true, so migration runs and their census are
unchanged; when false each field is struct('audit_skipped', true), never
an empty census that would read as clean. testV1ToV2Audits pins all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
schema cache: remember superclasses() and resolvePlacement() per class
5f4b83e 
Both are pure functions of the loaded schemas, which only grow for the
life of a cache instance, and every document read recomputed them:
listing the 10,536 Haley V2 subjects through NDI called getClass 900,580
times (profiled; rehydrate on decode and v1_to_v2's ensureClassBlocks are
the two per-document callers). rehydrate's own header already named this
("Memoise before quoting a corpus timing"). An error is not remembered.
The remembered placement's fieldsByBlock is a shared containers.Map;
callers read it and none modifies it, which the docstring now says.
testSchemaCacheMemo: remembered answers equal a fresh cache's for every
fixture class, errors raise every time, and decoding is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
testSchemaCacheMemo: only the concrete collision fixture raises
753e2db 
demoCollideAbstract is the valid abstract parent; demoCollideConcrete is
the class whose placement collides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
did2.convert: isAlreadyTarget and ensureClassBlocks are public
156b5d3 
Both were local to v1_to_v2. For a body already at its target, with
validation off, v1_to_v2 reduces to exactly these two (the base-field
rename is a no-op below V_eta and did2.document(struct).toStruct() is the
identity), so a READ path can apply them directly instead of running the
batch machinery per document (NDI's applyReadNormalization: 1.2 s of 2.4 s
listing the 10,536 Haley V2 subjects). Moved unchanged; v1_to_v2 calls
the package functions, so there is still one definition of each.
testV1ToV2Audits: the shortcut equals the full pass, including a body
missing a block and one with a stale superclass list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
sqlitedb writes: index the array defs by block, and insert rows sever…
3e1189e 
…al to a statement

Profiled on the Haley V2 import of one session (4,975 documents written):
- insertSidecarRowsFromStruct tried every indexed array def of the whole
  schema on every document and split its path each time (910,420
  resolveParentArray calls, 953,000 strsplit/join); paths are now split
  once and a document only visits the defs of blocks it has, in the same
  sorted order
- one mksqlite call per row (99,800 calls, 35 s): superclasses, depends_on
  and sidecar rows now go in as multi-row INSERTs, chunked under SQLite's
  999 bound-variable limit
The tables hold the same rows in the same order. testSqliteDb: rows past
the chunk boundaries (450 array elements, 700 links) are all stored in
order, and a document with no array block writes none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
sqlitedb: queryable scalar values in a side table, not 675 indexed ge…
8945380 
…nerated columns

Each queryable scalar path of the schema was a q_<flat> STORED generated
column on `documents`, each with its own index: 675 for V_eta, all
maintained on every insert, almost all for empty values. Measured
2026-10-06 (SQLite 3.45, 2,158 documents shaped like the Haley V2 import):
3.9 ms per document as generated columns + indexes, 0.26 ms with one
side table indexed by (path, value).

- queryable_scalar_elem(doc_id, path, value_text, value_num, value_raw):
  one row per value a document has, in the column of the path's affinity,
  holding what json_extract gave the old column (so SQLite converts it the
  same way); deleted with the document (ON DELETE CASCADE)
- compileQuery: a queryable path is `documents.id IN (SELECT ... FROM
  queryable_scalar_elem ...)`, its negation NOT IN (a document without
  the value matches, as `column IS NULL OR NOT ...` did)
- a database in the old layout is converted once on open: rebuilt
  without the q_ columns and their documents_q_ indexes, then the side
  table is filled from the stored bodies
- tests replaced, not loosened: the side table exists and documents has no
  generated column; values are stored at insert and removed with the
  document; an old-layout database converts on open and searches still
  work; the SQL routes through the (path, value) index; negation is NOT IN

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
sqlitedb: detect old-layout q_ columns with pragma_table_xinfo
728d1cf 
Generated columns are hidden columns: pragma_table_info leaves them out,
so it counted 0 on an old-layout database and the conversion never ran
(caught by testAnOldLayoutDatabaseIsConvertedOnOpen; checked on SQLite
3.45: table_info 0, table_xinfo 2 for one STORED and one VIRTUAL column).
The tests' own checks used the same blind query and now use table_xinfo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@jesshaley jesshaley changed the title did2.database.sqlitedb: survive connections closed under it; searches that name a link start from its index did2: reopen closed connections; faster reads and writes (link and value side tables, schema-cache memo, read shortcut) Oct 6, 2026
claude added 6 commits October 7, 2026 16:30
@claude
did2.query: a depends_on target may be a list of ids or a query
2ba6340 
depends_on's param2 may now be a cellstr (an edge to any of these ids) or
a did2.query (an edge to a document that query matches): 'formulations
with peptone as an ingredient' in one search, without looking peptone's
id up first; nesting goes as deep as wanted, inside or, negated, and on
the wildcard edge too.

compileQuery: a list binds as ONE json_each(?) parameter (no 999-variable
limit); a query target compiles to a subquery, and to 1=1 when negated
(NOT IN an over-approximation would drop true matches). sqlitedb.search
resolves each query target to its ids first, because the in-memory
recheck needs them, and sends the list. matches() on one document
refuses a query target with did2:query:nestedNeedsDatabase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
testSqliteDb: a query target matching one document resolves to = ?, n…
01c6b77 
…ot json_each

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
did2.query: a 'wildcard' operator ('*' any run, '\*' a literal star, …
f6af1b1 
…case ignored)

Compiles to SQL LIKE ? ESCAPE '\' (SQL's own % and _ escaped, so they are
literal), so a pattern runs in the database on an indexed field; the
in-memory recheck matches the same pattern as an anchored, case-blind
regexp. SQLite's LIKE ignores case for ASCII only, so a pattern with
non-ASCII characters compiles to 1=1 and is matched in memory. did.query
accepts 'wildcard' and '~wildcard' so ndi.query can send it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Statement family renamed: statement/assertion/.../calculation, entity…
8a3368f 
…_id, value

Follows DID-schema (V_eta_tenets.md T2 amendment, 2026-10-08). The V_eta
migrators (+migrators_j, the batch passes, did2.build) write the statement
edge as entity_id and name the classes without subject_; reads of emitted
statements take entity_id and fall back to subject_id. did_v1 edges read
from source documents, the V_zeta bath path, method_parameters' scope edge
and the +migrators_e/+migrators_i targets keep subject_id. Tests follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Contents.m roster regenerated for the rename; testFixtureCorpus names…
5718fc3 
… each quarantined fixture

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Undo the migrator rename; did2.build asks the schema for the statemen…
1dae6e1 
…t names

The migrators, batch passes and validators follow the pinned pre-#73 schema
(test-fixtures.yml / test-code.yml: v_eta-pre73), which keeps the old names;
renaming them quarantined 28 fixtures against that pin. They move with the
PR #76 checklist, like the rest of #73. Only did2.build, which targets the
current schema, follows the 2026-10-08 rename -- and it asks the schema in use
(private/vetaNames: 'statement'/'entity_id' when the schema has ,
else 'subject_statement'/'subject_id'), so it builds valid documents against
did-schema main today and after the rename merges. testBuild asks the same
way. testFixtureCorpus keeps naming each quarantined fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@jesshaley jesshaley mentioned this pull request Oct 8, 2026
Draft
claude added 3 commits October 8, 2026 18:46
@claude
Statements by composition: the validator and did2.build take a value …
fd9e93e 
…kind

did-schema V_eta_entity_composition_plan.md sec. 1 (a proposal built ahead of
signature): a statement direction declares `value_kind {root, count}`, and a
document of it lists one concrete value kind's chain after its own.

did2.schema.cache
- valueKindRule, documentValueKind, documentAncestors, documentChain and
  resolvePlacementFor: a document's expected superclasses, blocks and fields
  are its class's plus its kind's.
- validateDocument checks the kind (missingValueKind, badValueKind), validates
  the kind's blocks and its required edges; rehydrate and buildBlankDocument
  follow the kind.
- A class without `value_kind` (every pinned pre-#73 schema) is unchanged.

did2.build
- document takes 'ValueKind'; a leaf name the schema lacks of the form
  <kind>_<direction> ('temperature_observation') builds that direction with
  that kind, so callers written for either schema build valid documents on
  both. A direction given no kind is did2:build:missingValueKind.
- statement writes `data_type` when the schema renamed `datum_type`;
  checkRules knows `data_type_when_bytes`.
- testBuild: schema-aware subject and leaf expectations, plus tests of the
  kind in the builder and in the validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
testBuild: compare the bound part_of node case-insensitively (prefixe…
a4f2b69 
…s are lowercase since 2026-10-08)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
ensureClassBlocks keeps the value kind a document names
d846092 
The read path's re-stamp rebuilt document_class.superclasses from the class's
own chain, so a composed statement (an `observation` listing `temperature`)
lost its kind on every NDI read, and its value could not be found. It now
re-stamps with documentAncestors(class, kind) and pads the kind's blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@jesshaley jesshaley mentioned this pull request Oct 8, 2026
Draft
@claude
Lightsheet migrators: lightsheet_zarr_pyramid and lightsheet_zarr_lev…
1aece9a 
…el (#73 L1-L4, unsigned)

lightsheetZarrPyramid -> an `observation` of intensity (built with did2.build
as intensity_observation, so a composed observation listing `intensity`), id
preserved; entity_id <- subject_id, instrument_id <- element_id, software_id
<- an `entity` of type software named by pipeline_version; value.data_type
from dtype (NGFF typestrings now read by jDatumType, which also returns the
byte order); and, for an OME-Zarr source (source_file_id + an NGFF dtype), an
unheld opaque_body with keys for the full-resolution array. No time
reference: the pyramid carries none.

lightsheetZarrLevel -> a sampled_body owned by the pyramid's observation when
it holds bytes (chunk.bin manifest -> body_data, chunk.bin_k -> body_data_{k-1};
chunks on each key; datum_order, byte_order, fill_value, complete; blosc ->
compression 'blosc', raw -> none, codec_params dropped; level > 0 redundant
with the body condition `summary statistic`), or nothing when it holds none.
v1_to_v2 now counts that as summary.emitted_nothing_count / _by_class.

did2.validate.fileList recognises file-series members (`<stem>_<digits>`),
as did2.build already did. Tests: testMigratorsJLightsheet (the L4 real-chunk
check skips: no real NDI blosc chunk exists in any repository searched) and
two series tests in testFileList. Roster regenerated (95 migrators).

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
github-advanced-security[bot]
github-advanced-security Bot found potential problems Oct 14, 2026
View reviewed changes
Comment thread src/did/+did2/+convert/+migrators_j/private/jLightsheetKeys.m
entries = cell(1, nAx);
for k = 1:nAx
n = shape(k);
opts = {};
claude added 4 commits October 14, 2026 19:16
@claude
Lightsheet point 3: the store's fileReference becomes the observation…
3effc28 
…'s unheld body

Per signed #73 item 25 and Jess's 2026-10-14 decision (lightsheet point 3):
lightsheet_zarr_pyramid no longer makes its own store opaque_body. For an
OME-Zarr pyramid (source_file_id + NGFF dtype) it carries, TRANSIENTLY on the
observation, the v1 `source_file_id` edge verbatim, the full-resolution keys
(data.keys) and the store description (statement.notes).

foldGenericFiles gains its store-record half (makeSourceFile makes a
generic_file when it attaches a file and a fileReference when it only
describes one, so both records of an external file fold in one pass, which
ndi.migrate.local already runs): each file_reference named by exactly one
carrying observation is REPLACED by an opaque_body with the record's id,
owner_id = the observation, format application/x-ome-zarr, filename, member
body_data_0 recorded by location (original_path, location_type file, ingest 0),
size_bytes only when file_size > 0 (0 = unrecorded; a directory store is
always 0), file_created/file_modified from the datenums (no zone), the keys,
and the .zattrs checksum in `description`, never content_hash (L4). The
carrier is stripped on every path, folded or refused. A file_reference no
carrier names is left a passthrough and counted. New store_* counters are
rendered in census_digest.py. The harness chain size is unchanged (no new
pass); foldGenericFiles now declares `file_reference -> document: opaque_body`.

Tests: testMigratorsJLightsheet replaces the migrator-store-body test with
the carry test, the full fold (pyramid + fileReference + level), the
passthrough case and the absent-record case.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
File series: the member base follows the document's vintage (#73 item…
1a6bd9f 
… 17)

Point 6 (Jess, 2026-10-14: keep DID-schema #73 item 17). A V_eta document
numbers its file-series members from 0 (body_data_0 is manifest slot 0); a
did_v1 document keeps NAME_1 as slot 0. The base is read off the DOCUMENT,
never the file name: did.document/seriesMemberBase returns 0 when
document_class.schema_version ranks at or after V_eta
(did2.convert.schemaVersionRank) and 1 otherwise. The manifest format is
unchanged.

Applied in did.document (seriesMemberOf now returns the manifest INDEX plus
the NUMBER as written, and a number below the base is not a member;
addFileSeries defaults to base:base+N-1 and refuses a member below the base;
is_in_file_list follows through seriesMemberOf), did.database (seriesHas takes
a member number; seriesMembers returns member numbers), the sqlitedb member
lookup (unchanged logic, it takes seriesMemberOf's manifest index), and
did2.validate.fileList (`<stem>_0` is a member only on a V_eta document).

lightsheet_zarr_level: the chunk.bin_k -> body_data_{k-1} rename already kept
the slots; its MEMBER NUMBERING open point is recorded as decided, and a
not-yet-stripped ingest record's member numbers shift with the names.

Tests: TestDocumentFileSeries (V_eta member 0 -> slot 0, default zero-based,
negative refused; v1 NAME_1 -> slot 0, v1 member 0 refused),
TestFileSeriesRoundTrip (V_eta member 0 round-trips through sqlitedb; v1
NAME_1 still slot 0), testFileList (`_0` by vintage), testMigratorsJLightsheet
(body_data_0 on the migrated level is the slot chunk.bin_1 named).
tools/scratch.m runs the four did.unittest series classes, which no workflow
on a claude/** push otherwise runs.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
testMigratorsJLightsheet: seriesMemberOf parses names, it does not bo…
94db07c 
…und them

The assertion that body_data_4 is not a member was wrong: seriesMemberOf
answers whether a NAME is a member of a declared series, not whether the
manifest has that slot (that is seriesHas). Replaced with the check the
vintage rule does make: chunk.bin_0 is no member of the v1 level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 4: data_body.format is an ontology term; SmartSPIM f…
1dc4c2a 
…olders fold too

Jess, 2026-10-14 (did-schema 877bbde holds the schema half):

(a) `data_body.format` is a bound ontology_term, not an IANA media-type
string. did2.build.opaqueBody / sampledBody take Format as a term: a
{node, name} struct, or a bare CURIE that becomes the node with its name
filled from a small table of checked terms (+did2/+build/private/formatTerm.m:
EDAM TIFF / MPEG-4 / binary / Zarr, EMPTY:0000259 OME-Zarr); other text is the
name, the rule did2.build.document already applies to any ontology_term.
foldGenericFiles carries generic_file's format_ontology CURIE as the node,
and writes the OME-Zarr store body with EMPTY:0000259 from ONE named
constant (OME_ZARR_FORMAT). jSorterOutput's 'directory' becomes a name-only
term (a directory is not a file format; a term to find or mint).

(b) A SmartSPIM pyramid's fileReference (the folder of raw TIFF slices the
levels were stitched from) becomes an unheld opaque_body of the observation
in the same fold: the record's id, format TIFF (EDAM:format_3591), no keys,
`redundant` false. The pyramid migrator now carries source_file_id (and the
label notes) for every pyramid that names a source, keys only for OME-Zarr;
the pass tells the two apart by the same NGFF-dtype rule, read off the
observation's value.source_data_type. New counters stores_folded_ome_zarr /
stores_folded_tiff_folder, rendered in census_digest.py.

Tests: testBuild (CURIE -> node + name, struct, unknown CURIE),
testFixtureCorpus (format.node), testMigratorsJLightsheet (OME-Zarr term and
counters; SmartSPIM pyramid + fileReference + level -> one TIFF body with the
record's id; the SmartSPIM observation carries the join and no keys).

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
claude added 8 commits October 14, 2026 20:00
@claude
testBuild: an unknown format CURIE's empty name is dropped by the bui…
86f5226 
…lder

did2.build drops an absent ('') sub-field, so the term carries only its node;
assert the node and that no name is present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 8: the pyramid's t unit and channel names complete i…
20ff9b4 
…ts levels

Jess, 2026-10-14. A v1 level carries only voxel_size_units; axes_units (a t
unit) and channel_names exist only on the pyramid. lightsheet_zarr_pyramid now
carries its full-resolution keys on the observation for EVERY pyramid
(OME-Zarr or SmartSPIM, with or without a source), transient; they reach the
store body only for OME-Zarr (point 4).

foldGenericFiles' store half completes every sampled_body whose owner_id is
such an observation: `t` gets the pyramid's unit (the level's raw
origin/spacing scaled by the factor the pyramid's were), `c`'s positions are
named by channel when the counts agree, z/y/x units are CHECKED (canonical
unit and source spelling) and a disagreement is counted, never overwritten.
The carried keys are removed from every observation, folded or not. New
counters level_bodies_seen / _completed / _without_pyramid,
level_unit_disagreements, level_completion_quarantined, rendered in
census_digest.py. The level migrator's OPEN POINTS bullet records the
decision.

Tests: a level gets the t unit and channel names; a z unit mismatch is
counted and the level's own unit kept; a level whose pyramid is absent keeps
its keys and is counted; the SmartSPIM observation carries the keys.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
foldGenericFiles: pack completed level keys as one struct array
e81b9dd 
The validator requires data.keys to be a struct; a cell of differently-shaped
keys quarantined every completed level. Keys are now packed as did2.build's
mergeRagged packs them (union of fields, missing ones []), and the channel /
time checks read an empty field as absent, as a packed struct array has every
field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 10: unit, statistic and variable nodes (option C)
0e0cec0 
Jess, 2026-10-14 (nodes checked on OLS):
- units: the canonical unit on every lightsheet key carries its UO node
  (second UO:0000010; meter UO:0000008, the canonical length every space axis
  is converted to; micrometer UO:0000017 is in the table). An NGFF unit with
  no unambiguous mapping stays NAME-ONLY as the source spelled it (it used to
  be dropped) and foldGenericFiles counts it (pyramid_key_units_name_only,
  rendered in census_digest.py).
- the levels' `summary statistic` values: max -> NCIT:C25564 'Maximum', mean
  -> NCIT:C53319 'Arithmetic Mean'; any other reduction name-only.
- the observation's variable: NCIT:C184706 'Fluorescence Intensity'
  (replacing the bare name 'fluorescence').
The axis names and the variable 'summary statistic' stay names-only (term
worksheet). OPEN POINTS updated; tests assert the nodes, the name-only
statistic variable, and the name-only unit count.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Body formats follow the schema in use (fixture corpus on v_eta-pre73)
19256aa 
test-fixtures.yml pins did-schema to v_eta-pre73 on purpose (most migrators
still emit pre-#73 classes), where data_body.format is a char media type;
since point 4 the writers produced the {node, name} term and kilosort_clusters,
kiasort_clusters and opaque_body quarantined there. The decision now lives in
ONE place, did2.build.formatTerm (moved out of +build/private so +convert can
call it), which reads data_body.format's declared type through the schema
cache, as NDI's Haley import does (ndi.setup.conv.haley private/fileFormat.m):
an ontology_term schema gets the term; a char schema gets the media type
(TIFF image/tiff, MPEG-4 video/mp4, binary application/octet-stream, Zarr
application/x-zarr, OME-Zarr application/x-ome-zarr), an unlisted CURIE
verbatim (generic_file's format_ontology, as before) and a name-only term as
its name (jSorterOutput's 'directory', as before). Used by
did2.build.opaqueBody / sampledBody, foldGenericFiles' generic_file half and
jSorterOutput. testFixtureCorpus accepts the format in either form.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 1: the batch pass mints the pyramid's session time r…
7cfa864 
…eference

Jess, 2026-10-14, option A. jSessionAnchor and resolveSessionAnchors are not
used: their classes (session_relative_reference, relative_reference) are gone
from V_eta.

New reusable helper did2.convert.mintSessionDuring: for each document carrying
its TRANSIENT marker (a depends_on entry named mintSessionDuring('marker'),
valued with the document's own id so it can never dangle), it finds the
batch's session document for the document's base.session_id -- the
resolveSessionAnchors / jSessionDocId lookup: index the session documents (a
`session`, or an `entity` of type session) on their own base.session_id,
refuse a session id two documents claim -- mints
did2.build.relativeTimeReference(<session doc id>, 'Relation',
'intervalDuring'), points the document's time_reference_id at it and removes
the marker on every path. No session document: no reference, counted
(observations_without_session); an id is never invented. Written so the later
port of the 29 jSessionAnchor callers can call it; they are not ported.

lightsheet_zarr_pyramid marks its observation (built, as before, without a
time reference). foldGenericFiles runs the helper first, validates the minted
references with everything else and appends them; a reference that does not
validate takes its document's change with it, so no edge is left dangling.
New counters session_references_marked / _minted / _ambiguous /
_quarantined and observations_without_session, rendered in census_digest.py.

Tests: the observation gets exactly one time reference, a
relative_time_reference to the session document with relation
time:intervalDuring, marker removed, 0 orphans; with no session document it is
counted and has none.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 5: the software entity is the builder; pipeline_vers…
c2da2ea 
…ion is its version

Jess, 2026-10-14. `pipeline_version` is the writer's "tag for the pyramid"
(makePyramid.m:28), not a program name. lightsheet_zarr_pyramid now always
emits one `software` entity named 'NDI lightsheet pyramid builder
(ndi.fun.doc.lightsheet.makePyramid)' and always sets the observation's
software_id; when pipeline_version is non-empty it adds a `text_assertion`
(variable 'version') about that entity, built as ndi.setup.V2.entityDocuments
records a software version.

NO TIME REFERENCE ON THE ASSERTION, and that departs from the instruction on
evidence: an assertion declares no time_reference_id (statement.json's
depends_on is entity_id and value_id only; only `interaction` declares the
edge), and ndi.setup.V2.entityDocuments passes none. Marking it would put an
undeclared edge on it. Software dedup across pyramids stays NDI's
resolveSoftwareDedup's job. jSoftware.m and its callers are untouched.

Tests: the entity is named for the builder; with no pipeline_version the
entity and software_id remain and no version assertion is made; with one, a
'version' assertion about the entity carries it.

Not run locally: no MATLAB in the authoring container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
@claude
Lightsheet point 11: a level's sampled_body keeps the v1 level's base.id
000aeb8 
Jess 2026-10-14, option A. did2.build.sampledBody / opaqueBody already took
'Id' and forwarded it to did2.build.document (since 2e95ac9); the help now
names it, with '' (the default) minting a new id. lightsheet_zarr_level
passes baseId(preBody), and its header records the decision in place of
the "after the migrator confirms no inbound edge" text.

Tests: testBuild/testBodiesKeepAGivenIdAndMintOneOtherwise;
testMigratorsJLightsheet/testALevelsBodyKeepsTheLevelsId (single level, and
two levels through the batch pass: ids equal the v1 level ids, 0 orphans).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jesshaley @github-advanced-security @claude