Skip to content

Add AlignedDynamicTable non-generated base class with category table validation#837

Merged
ehennestad merged 14 commits into
mainfrom
codex/aligned-dynamic-table-add-category
Jul 2, 2026
Merged

Add AlignedDynamicTable non-generated base class with category table validation#837
ehennestad merged 14 commits into
mainfrom
codex/aligned-dynamic-table-add-category

Conversation

@ehennestad

@ehennestad ehennestad commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

⚠️ Depends on #836 — merge that first - merged

Motivation

The AlignedDynamicTable schema defines two requirements in docstrings that cannot be enforced by the generated class alone:

  1. All sub-tables added to an AlignedDynamicTable "MUST have the same number of rows" (source).
  2. The names in the categories property MUST match the names of category tables in the AlignedDynamicTable (source).

Without additional runtime validation, users can create invalid NWB files when using AlignedDynamicTable or its subtypes.

This PR adds validation hooks for AlignedDynamicTable through a custom base class. It also adds convenience APIs so valid tables are easier to construct: schema-defined category tables auto-register in categories when assigned, and custom category tables can be added with addCategory(name, table).

This PR stacks on #836 and uses its schema validation reporting context to maintain backwards-compatibility by making AlignedDynamicTable validation on construction warn instead of error to ensure pre-PR icephys tutorial examples can still run without breaking.

Example usage (updated snippets from icephys tutorial)

1. categories do not need to be listed when the table is initialized with rows — assigning a category table later registers it automatically:

% before
ic_rec_table = types.core.IntracellularRecordingsTable( ...
    'categories', {'electrodes', 'stimuli', 'responses'}, ...  % name is manually added to categories
    'colnames', {'recordings_tag'}, ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64([0; 1; 2])), ...
    'recordings_tag', ...);
ic_rec_table.electrodes = ...;   % table is assigned afterwards

% after
ic_rec_table = types.core.IntracellularRecordingsTable( ...
    'colnames', {'recordings_tag'}, ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64([0; 1; 2])), ...
    'recordings_tag', ...);
ic_rec_table.electrodes = ...;   % auto-registers in name `categories`

2. Use addCategory for custom categories instead of the manual two-step:

% before
ic_rec_table.categories = [ic_rec_table.categories, {'recording_lab_data'}];
ic_rec_table.dynamictable.set('recording_lab_data', customTable);

% after
ic_rec_table.addCategory("recording_lab_data", customTable);

How to test the behavior?

runtests('+tests/+system/AlignedDynamicTableTest.m')

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Codex

@ehennestad ehennestad changed the title Add AlignedDynamicTable category utilities Add AlignedDynamicTable non-generated base class with category table validation Jun 26, 2026
@bendichter

Copy link
Copy Markdown
Contributor

What would be the warning with the old syntax?

@ehennestad

Copy link
Copy Markdown
Collaborator Author

What would be the warning with the old syntax?

ic_rec_table = types.core.IntracellularRecordingsTable( ...
    'categories', {'electrodes', 'stimuli', 'responses'}, ...
    'colnames', {'recordings_tag'}, ...
    'description', [ ...
        'A table to group together a stimulus and response from a single ', ...
        'electrode and a single simultaneous recording and for storing ', ...
        'metadata about the intracellular recording.'], ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64([0; 1; 2])), ...
    'recordings_tag', types.hdmf_common.VectorData( ...
        'data', repmat({'Tag'}, 3, 1), ...
        'description', 'Column for storing a custom recordings tag' ...
        ) ...
);

Current warning:
Warning: The categories property lists category table(s) that have not been added to the AlignedDynamicTable: electrodes, stimuli, responses.
Add the missing table(s) with the addCategory method (or by setting the corresponding schema category property), or list categories only before the table has rows. The non-conforming value is kept. If you maintain this data, consider correcting it before export.

Maybe a better version:
Warning: The categories property lists category table(s) that have not been added to the IntracellularRecordingsTable: electrodes, stimuli, responses.
The table already has <num_rows> and all category tables must have the same height as the parent table. Add the missing table(s) with the addCategory method, or list categories only before the table has rows. The non-conforming value is kept. This needs to be corrected before export.

@ehennestad ehennestad changed the base branch from codex/report-schema-violation-context to main June 30, 2026 16:01
@ehennestad ehennestad marked this pull request as ready for review June 30, 2026 16:01
@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.53480% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.43%. Comparing base (0343320) to head (961a13b).

Files with missing lines Patch % Lines
...es/+util/+dynamictable/+internal/getColumnHeight.m 90.00% 2 Missing ⚠️
...nternal/collectConstantPropertiesAcrossHierarchy.m 91.66% 1 Missing ⚠️
+matnwb/+neurodata/AlignedDynamicTableBase.m 99.34% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #837      +/-   ##
==========================================
+ Coverage   95.33%   95.43%   +0.10%     
==========================================
  Files         226      228       +2     
  Lines        7821     8064     +243     
==========================================
+ Hits         7456     7696     +240     
- Misses        365      368       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ehennestad ehennestad requested a review from bendichter July 1, 2026 07:38
@ehennestad ehennestad force-pushed the codex/aligned-dynamic-table-add-category branch from b6fb4bc to 3b9944c Compare July 1, 2026 15:15
@ehennestad ehennestad changed the base branch from main to add-dynamictable-base-class July 1, 2026 15:16
ehennestad and others added 7 commits July 1, 2026 17:26
Introduce matnwb.neurodata.AlignedDynamicTableBase, the non-generated base class
that owns aligned-table behavior the generated schema class cannot express:
category table registration and lookup, category-name validation, and row-height
consistency between the parent table and its category tables. It extends
DynamicTableBase so aligned tables inherit the shared DynamicTable behavior.

Add matnwb.neurodata.internal.collectConstantPropertiesAcrossHierarchy, used to
gather schema-declared categories across the class hierarchy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
getTableHeight now also reports whether the table height is established,
getColumnHeight accounts for unbound DataPipe offsets, and initDynamicTableId
fills an existing but empty id column. These helpers back the row-height
consistency checks used by both DynamicTable and AlignedDynamicTable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Attach matnwb.neurodata.AlignedDynamicTableBase to AlignedDynamicTable via the
customBaseClasses map and emit its schema-declared category constant block.
Route aligned tables through obj.ensureAlignedTableConsistency() in the
constructor and custom-constraint hooks, and register schema-defined category
properties with post-set hooks. Replace file.isDynamicTableDescendant with the
more general file.internal.isDescendantOf.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AlignedDynamicTable inherits matnwb.neurodata.AlignedDynamicTableBase and gains
its DeclaredSchemaCategories constant and category post-set hooks;
IntracellularRecordingsTable, an AlignedDynamicTable descendant, is regenerated
accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add system tests for AlignedDynamicTable category construction and a DynamicTable
test factory double, plus unit coverage for the refined height and id helpers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use addCategory to add custom category table
@ehennestad ehennestad force-pushed the codex/aligned-dynamic-table-add-category branch from 3b9944c to e5b48fd Compare July 1, 2026 15:27
@ehennestad ehennestad changed the base branch from add-dynamictable-base-class to main July 1, 2026 15:27
@ehennestad ehennestad enabled auto-merge July 1, 2026 15:27
@ehennestad ehennestad marked this pull request as draft July 2, 2026 14:11
auto-merge was automatically disabled July 2, 2026 14:11

Pull request was converted to draft

@ehennestad ehennestad marked this pull request as ready for review July 2, 2026 14:11
@bendichter

bendichter commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Warning: The categories property lists category table(s) that have not been added to the IntracellularRecordingsTable: electrodes, stimuli, responses. Add the missing table(s) with the IntracellularRecordingsTable.addCategory method before exporting.

@ehennestad ehennestad enabled auto-merge July 2, 2026 14:55
@ehennestad

Copy link
Copy Markdown
Collaborator Author

Warning: The categories property lists category table(s) that have not been added to the IntracellularRecordingsTable: electrodes, stimuli, responses. Add the missing table(s) with the IntracellularRecordingsTable.addCategory method before exporting.

Warning is updated to that text here: 7f3bb6a

@ehennestad ehennestad added this pull request to the merge queue Jul 2, 2026
Merged via the queue into main with commit 13b0f9a Jul 2, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants