[NoopDog]

Closes #604

What changed

• A slot-map entry can now reach files by folder (slot_map.py). Until now, an entry reached a file only through a cell that holds the file's DRS URI. A folder entry instead names a column that holds a top folder of the dataset's storage, plus one member that is not a slot, files_under. The entry reaches every anvil_file whose normalised file_path starts with <cell>/<files_under>/:

  ANVIL_NIA_CARD_Coriell_Cell_Lines_Open:
    sequencing:
      seq_filename:          # `HG002_R10`
        files_under: reads   # reaches `HG002_R10/reads/…`
        assay_type:
          - {cell: sequencing_assay}

  The loader refuses four things:
  • a files_under that is empty, absolute, or holds . or ..;
  • a column_name span on a folder entry, because the folder column's name describes no file;
  • samples on a folder entry, until NIA_CARD fast5: check each file's own sample against its folder's sequencing row #740;
  • a cell elsewhere in the table that reads the folder column as a value.
• The importer (anvil_evidence.py). folder_files finds the files under a row's folder, and the check and the import both use it. Each file found gets the same evidence line a DRS link would have written, keyed by its file_ref. The import counts rows whose cell isn't a folder name (not_folder) and rows whose folder reaches nothing (empty_folder). check holds a folder column to four rules:
  • some row carries the column;
  • every value is a top folder's name;
  • each folder reaches at least one of the dataset's files;
  • no file is reached from two rows;
  • no reached file is an index or checksum by its own name (x.bam.bai, x.fast5.md5). A link entry excludes those by its column's name (No index file is classified data_type: index — 223,731 inherit the parent's instead #437, Checksums are not_applicable on every carried dimension: ChecksumActivity passes nothing (reverses #571's checksum inheritance) #596), but a folder column's name says nothing about the files under it.
• One reader of a dataset's anvil_file rows. FileIndex / file_index is now shared with the lineage importer, whose own DatasetFiles / dataset_files are gone. It reads storage paths only for a dataset that has a folder entry (SlotMap.has_folders).
• Column names stated once: azul_manifest.ANVIL_FILE_REF and ANVIL_FILE_PATH, which snapshot_input now uses too.
• The mapping: NIA_CARD sequencing.seq_filename, files_under: reads, carries assay_type from sequencing_assay only. The existing sequencing_id_fileref entry and the value map are unchanged: row assay_type.wgs already gives WGS / genomic.
• Contract 2.7: a file link may be a folder where the map declares it. There are matching lines in CLAUDE.md and docs/interpreting-submitter-tables.md.

Why

The 12,394 ONT .fast5 files in ANVIL_NIA_CARD_Coriell_Cell_Lines_Open had no data_modality or assay_type. The submitter's sequencing table says WGS / DNA for all 6 runs. Its rows reach their files only by storage folder: seq_filename is HG002_R10, and the run's files sit under HG002_R10/. #604 records the walkthrough and the measurements behind trusting that link.

Assumptions I made

• The entry reaches reads/, not the whole top folder (decided with Dave). Measured on run 20261022_161028:
  • The whole folder would also reach 28 methylation BEDs, which are already epigenomic.methylation, and 24 hapdup assemblies, whose assay is not_applicable. WGS → genomic would make every one of those 52 files a conflict.
  • reads/ holds the 12,394 fast5s plus the run's 2 fast5 tarballs, 6 BAMs, 6 FASTQs and 9 QC and summary files, all of them the run's reads or reports on them.
  • The trial import reached 12,417 files.
• anvil_file.file_path, not the submitter's file_inventory.path. The file's own record carries its path since Carry each file's storage path from the snapshot's file_inventory onto the input, so files with the same name can be told apart #717, and the two agree on all 12,534 NIA_CARD files.
• Paths drop only their empty segments before matching (a//b is a/b). . and .. stay as written, because a storage object name is literal. Measured on anvil15, the two ANVIL_HPRC // paths are the only paths this changes.
• The sample cross-check moved here from Read each NIA_CARD fast5's sample from its HDF5 attributes (cross-check with #604's folder join when it lands) #605 is left out (decided with Dave), as NIA_CARD fast5: check each file's own sample against its folder's sequencing row #740. Mapping samples through the folder entry would compare the 4,245 HG002 fast5s' GM24385 (coriell:NA24385) with the row's HG002 (giab:HG002). Those two are kept apart until a crosswalk exists (Sample to donor: donor_of edges from the Azul manifest and public registries, pedigree as child_of #361), so reconcile would give those files no samples. The loader refuses samples on a folder entry until NIA_CARD fast5: check each file's own sample against its folder's sequencing row #740 decides.
• The lineage importer's output is unchanged by the shared index: importing all 11 datasets on main and on this branch gives identical line bodies, 18 files and 619,506 lines.

How to verify

1. make check-slot-map should print Slot map checked against anvil15: 317 column entries across 11 dataset(s).
2. Import into a scratch root:
   uv run python scripts/import_anvil_evidence.py --dataset ANVIL_NIA_CARD_Coriell_Cell_Lines_Open --evidence-root /tmp/ev604.
   It should print sequencing: 6 rows, 12,441 evidence rows and Wrote 12,441 evidence rows for 12,423 files. That is 12,417 files under the six reads/ folders plus the existing entry's 6 files.
3. Break the map on purpose:
   • Change files_under: reads to files_under: nope and run step 1 again. It should report that seq_filename "reaches none of the dataset's files under 'HG002/nope/', …".
   • Rename the entry key from seq_filename to sample_id. It should report that "4,741 files are reached from more than one row".
4. make test-all lint-all type format-check validate-metadata should all pass.
5. The run (done at b3b3927, committed as 5bd91e3; re-importing NIA_CARD at 0092b34, after the review fixes, gives identical evidence lines). NIA_CARD was imported into data/source_evidence/ as generation 20261023T165533Z, followed by make classify (run 20261023_095539) and make all-reports. Against 20261022_161028, the last run on main, file by file over all 708,088:
   • Only NIA_CARD changed. No other dataset, conflict, samples or generated_by changed.
   • 12,394 fast5s: assay_type goes from not_classified to WGS, credited filled_by_submitter, and data_modality from not_classified to genomic, credited filled_by_submitter_harmonized.
     • The two credits differ by design. Raw WGS is already our term, so the schema credits it verbatim. genomic comes through the assay_type.wgs translation row, so it is credited harmonized. The issue expected harmonized for both.
   • The other 23 files under reads/ (2 tarballs, 6 BAMs, 6 FASTQs, 9 QC and summary files) gain the same values.
     • The 3 R10 BAMs that inference already called genomic are now credited to the submitter, with the same value.
   • Totals: 12,417 files gain assay_type and 12,414 gain data_modality. NIA_CARD goes from 0.9% to 99.9% filled for data_modality and from 0.1% to 99.2% for assay_type (docs/reconcile-report.md).
   • To check one yourself: in output/anvil/20261023_095539/reconciled/fast5_classifications.ndjson.gz, read any NIA_CARD record, such as PAI80410_89549a63_1195.fast5. Its assay_type evidence cites sequencing.sequencing_assay, raw WGS.

The issue's definition of done, mapped to these steps:

• Code: a new folder entry form in slot_map.py, resolved in anvil_evidence.py: steps 1–3.
• Tests, and the make check-slot-map counts: steps 1 and 4.
• Mapping: NIA_CARD sequencing through seq_filename carries assay_type from sequencing_assay only, with no value-map change: step 2 (only assay_type lines from sequencing_assay).
• Contract: 2.7 amended. Read docs/claims-contract.md 2.7.
• Run: re-import NIA_CARD, make classify, the 12,394 records read WGS / genomic (credited filled_by_submitter / filled_by_submitter_harmonized, see step 5), make all-reports, and the fill reported: step 5.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
🐦 Built with cc-claude-tools

[Copilot]

🔵 Needs a closer look

The planned full-corpus reconciliation is still pending, leaving the expected metadata fills and absence of new conflicts unverified.

No open issues 🎉

What changed in this PR

Adds folder-based evidence joins so NIA_CARD’s fast5 files can receive assay metadata from their sequencing rows, without extending the mapping to assemblies or methylation outputs.

Changes:

• Adds validated files_under slot-map entries and folder-resolution diagnostics.
• Maps NIA_CARD’s reads/ folders to sequencing assay evidence and shares the file index with the lineage importer.
• Adds focused tests and documents the folder-link contract; sample cross-checking remains deferred to #740.

File	Description
tests/​test_slot_map.py	Tests folder-entry validation and restrictions.
tests/​test_anvil_evidence.py	Tests folder joins, diagnostics, and shared indexing.
src/​meta_disco/​sources/​anvil_slot_map.yaml	Maps NIA_CARD read folders to assay evidence.
src/​meta_disco/​snapshot_input.py	Uses shared file-reference and storage-path constants.
src/​meta_disco/​slot_map.py	Defines and validates folder entries.
src/​meta_disco/​azul_manifest.py	Declares shared column-name constants.
src/​meta_disco/​anvil_lineage.py	Reuses the shared file index.
src/​meta_disco/​anvil_evidence.py	Resolves folder entries and reports unmatched or invalid folders.
docs/​interpreting-submitter-tables.md	Explains folder-based file links.
docs/​claims-contract.md	Extends contract 2.7 to declared folder links.
CLAUDE.md	Documents the folder-entry import route.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The scoped folder matching, validation safeguards, and shared-index refactor have no identified blocking defects and are supported by focused regression coverage.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Folder validation misses supported index and compressed checksum filenames, allowing payload metadata onto sidecar files.

1 open issue

• Recognize compressed index and checksum sidecars in folder guard · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Sidecar validation uses storage basenames rather than catalog filenames, allowing indexes with differing names to receive folder-level payload metadata.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Validate sidecars using catalog filenames, not storage basenames

src/​meta_disco/​anvil_evidence.py:393

Check sidecars using anvil_file.file_name, not the basename of file_path. snapshot_input.record_from_file_row preserves these as separate columns, while producers.route routes files by file_name. A file named a.bam.bai stored at HG002/reads/object-1 therefore passes this guard even though the pipeline treats it as an index, allowing folder-level WGS evidence to be imported onto it. Retain the catalog filename or sidecar status in FileIndex and validate the reached files against it. Add a regression test where the catalog filename and storage basename differ.

1 resolved since last review

• Recognize compressed index and checksum sidecars in folder guard

_{Review effort: Balanced}

[NoopDog]

Copilot's previously missed finding (validate sidecars using catalog filenames, anvil_evidence.py:393) is fixed in 765b65c. FileIndex.sidecars holds the file_ref of each indexed file whose catalog file_name is_sidecar_file calls an index or checksum, and check reads that rather than the storage basename. A regression test covers a catalog name that differs from the storage name in both directions.

[Copilot]

🟢 Approval recommended

Static review found no unresolved blocking issues in folder joins, validation, or shared-index compatibility; project tests were not executed during review.

No open issues 🎉

_{Review effort: Balanced}