did2 sqlitedb: a file store (ingestion)

[jesshaley]

did2.database.sqlitedb stored documents only. The legacy did.implementations.sqlitedb copies files into <db folder>/files/<uid>, but that was never ported to V2. So a V2 writer that creates bytes, like the Haley V2 import's masks and tracks, had nowhere to put them. NDI's did2sqlite errors on any file not recorded by location.

What changes

Ingestion

• On add, every files.file_info location with ingest true is copied into <db folder>/files/<uid>. This is the same layout as the legacy backend, so NDI's file_directory is unchanged.
• Each copy is recorded in a new files table.
• delete_original removes the source only after the documents are committed.
• A location with ingest false is recorded and read where it is.

Lookup: hasFile, filePath and fileNames. filePath raises an error that names the missing piece (noSuchFile, fileNotHere, missingDocument) and never returns an empty path.

Refusals instead of warnings: these are errors, and nothing in the batch is written. Files already copied for it are deleted.

• an unsafe uid;
• a missing source file;
• a location type other than file;
• a uid already stored with different bytes.

The legacy backend warned and stored the document anyway, which left a document whose file was missing.

Removal: deletes the bytes a document ingested, unless another document still records the same uid.

Older databases gain the files table when opened.

CI: testSqliteDbFiles adds 14 tests. test-builder.yml now also runs testSqliteDb* and triggers on +did2/+database changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf

---

Generated by Claude Code

[jesshaley]

Quick migrator tests / Migrator transform tests (no corpora) is red, but not because of this PR. Its check_empty_ontology_nodes.py --enforce step reads the did-schema checkout and reports "93 schema-side entries, baseline 86". did-schema #84 added seven subject.type terms with empty nodes and didn't raise the baseline. This PR touches none of that.

The fix is Waltham-Data-Science/DID-schema#87, which raises the baseline to 93. It's on the same-named did-schema branch, so this job picks it up. I've re-run the failed job once against it.

---

Generated by Claude Code

[jesshaley]

The re-run got past the ontology-term guard, so DID-schema#87 fixes that part. It then failed 78 migrator tests, for a reason that isn't this PR's either.

• The quick gate checks out the same-named did-schema branch when one exists (test-migrators-quick.yml:53). Otherwise it falls back to v_eta-pre73.
• So it validates the migrators against the post-Bug: did.database.findfilematch fails if input "expectedName" ends with # symbol #73 schema. They haven't caught up with that schema yet.
• The 78 failures are all migrator or schema validation tests (testSilentLoss, testTimeReferenceFamilies, testValidIntervalDecompose, testResolveEpochProbemap, …). None touch did2.database.
• testBuild: counts take a tolerance (CHANGE 7 amendment 1) #214, from this same branch, merged into V2 with the identical Quick migrator tests and Test code failures.

The tests this PR adds (testSqliteDbFiles, plus the existing testSqliteDb) run in did2.build tests, which is green. That job fails on any filtered or incomplete test.

No fix exists here: it needs the migrators brought up to the #73 schema, which is separate work.

---

Generated by Claude Code

[jesshaley]

Test code / Analyse and test code is red, but not because of this PR, for the same reason as the quick gate.

• All six corpora migrated globally clean: 0 quarantined, 0 orphans, 0 fragments, 0 empty edges, over 221,827 v1 documents.
• The job fails in the later coverage step. That step reads the same-named did-schema branch's tools/coverage.py and reports "anomalies: 101 across 60 class(es) (99 over a FAILED rung)". That is the stale-migrator mismatch against the post-Bug: did.database.findfilematch fails if input "expectedName" ends with # symbol #73 schema.
• testBuild: counts take a tolerance (CHANGE 7 amendment 1) #214, from the same branch, merged with this check red as well.
• Nothing in that step reads did2.database.

No fix belongs in this PR. I haven't re-run this ~5 h job: the one re-run went to the quick gate above.

---

Generated by Claude Code