Ingest-side path guard rejects legitimate ingest sources outside the DB directory (regression from #58 fix) #60

Summary

The guard added in PR #59 (commit 5de7ce4, "Refuse traversal-crafted uid/location in ingest and open_doc (#58)") places a _validated_local_path check on the ingest source at src/did/implementations/sqlitedb.py:625-627. That check refuses any source file that doesn't resolve inside the DB directory. Legitimate ingest workflows — where a caller says "take this file from wherever it lives and add it to my DB" — are now refused.

Reproduction

Downstream: Waltham-Data-Science/NDI-python#194 — 110 failed + 115 errored tests across the 3.10 / 3.11 / 3.12 matrix on every branch (including main), all with:

ValueError: Refusing ingest source for "epoch_binary_data.vhsb":
'/tmp/ndi-vhsb-…' resolves outside the database directory
'/tmp/pytest-of-runner/pytest-0/…/.ndi'. See DID-python issue #58.

NDI-python stages raw ingest sources under /tmp/ndi-vhsb-*/ (or similar), then adds them to a DB whose .ndi lives elsewhere. That is the normal use case, not a traversal attack.

Threat model reconsideration

• Destination: already protected by _is_safe_uid(uid) immediately above line 625. The uid can't contain path separators, ., .., or an empty basename, so the ingest write always lands under <file_dir>/<uid> no matter what the caller supplied.
• Source: provided by the caller — they explicitly said "add this file to my DB." Where the source file lives has no bearing on what gets written into the DB, and refusing sources outside the DB dir does not close any attack surface that _is_safe_uid doesn't already close.
• The scenario the guard was meant to address — attacker-controlled document JSON with a crafted location — still matters on the read path (open_doc), where a malicious stored location like ../../etc/passwd could otherwise cause the DB to read arbitrary files. That's why the read-side filter at sqlitedb.py:1293-1301 (via _is_safe_local_location) is the actual defense.

Proposed fix

Drop the _validated_local_path call at sqlitedb.py:625-627 — replace with a plain source_path = self._resolve_local(location) (or equivalent that rebases a relative path against db_dir without the containment check). Keep:

• the _is_safe_uid(uid) check just above (destination guard),
• the read-side filter in _locations_for_file / _is_safe_local_location (defense against unsafe stored locations).

Companion issue

DID-matlab has the analogous over-reach in validateIngestFileEntry (src/did/+did/+implementations/sqlitedb.m:1323-1358) raising DID:SQLITEDB:PathTraversal. A matching issue is being filed there.

https://claude.ai/code/session_01XFqZrEMdcdcK776WNnFPe1