feat(dynamictable): add ragged and doubly-ragged array column helpers

[ehennestad]

Todo

• Support N-D arrays

Motivation

Background — NWB DynamicTables store a variable number of items per row as ragged arrays (§2.5), and some columns — notably the Units table's waveforms — use a doubly ragged array (§2.6) with two levels of indexing.

Problem — MatNWB had no ergonomic way to build these. For a doubly-ragged column, addRow silently produces a non-conformant structure (a single index level, which schema-aware readers such as pynwb misinterpret), and doing it correctly meant hand-constructing a VectorData plus two VectorIndex objects with the right cumulative offsets and ObjectView targets. Even a single-level ragged column required the two-step util.create_indexed_column + addColumn dance.

Solution — Add two DynamicTable methods, addRaggedArray and addDoublyRaggedArray (plus the underlying util.create_doubly_indexed_column primitive), that build and wire all the objects in a single call. The names mirror the NWB spec section titles.

What changed

• DynamicTable.addRaggedArray(name, data) — builds a ragged column (a VectorData + its VectorIndex) from a cell array and adds it, updating colnames and id. An optional table argument produces a DynamicTableRegion column instead.
• DynamicTable.addDoublyRaggedArray(name, data) — builds a doubly-ragged column (a VectorData + two VectorIndex levels) and wires name, name_index, and name_index_index.
• util.create_doubly_indexed_column — the low-level builder, mirroring the existing util.create_indexed_column.
• Both methods are inherited by every table type (Units, TimeIntervals, ElectrodesTable, …) and survive generateCore, since they live in the hand-written matnwb.neurodata.DynamicTableBase.
• Documentation — a new how-to guide, Storing Ragged and Doubly-Ragged Array Columns, covering ragged columns, single-electrode and multi-channel doubly-ragged waveforms, and the underlying helpers. Its code snippets live in a runnable example file, embedded via literalinclude and executed by a unit test, so the documented code cannot drift from the API.

Implementation notes

• addDoublyRaggedArray wires the three objects directly rather than delegating to addColumn, because addColumn's height check only follows a single index level and would reject a doubly-ragged column.
• create_doubly_indexed_column accepts either a per-row [nSubGroups x nSamples] matrix (single-element shortcut, e.g. one electrode per unit) or a nested cell (data{i}{j} = [nElements x nSamples]), and validates that all elements share the same sample length.

Examples

Doubly-ragged Units.waveforms

Shared setup:

nSamples = 4;
unit1 = reshape(1:(3*nSamples), 3, nSamples);        % 3 spikes x 4 samples
unit2 = 100 + reshape(1:(4*nSamples), 4, nSamples);  % 4 spikes x 4 samples

Before — manual construction of three objects:

allWaveforms  = [unit1.', unit2.'];                  % [nSamples x totalSpikes]
perUnitCounts = [size(unit1,1), size(unit2,1)];

wf = types.hdmf_common.VectorData('description','spike waveforms','data', allWaveforms);
wfIndex = types.hdmf_common.VectorIndex('description','', ...
    'target', types.untyped.ObjectView(wf), 'data', (1:size(allWaveforms,2))');
wfIndexIndex = types.hdmf_common.VectorIndex('description','', ...
    'target', types.untyped.ObjectView(wfIndex), 'data', cumsum(perUnitCounts)');

units = types.core.Units('colnames', {'waveforms'}, 'description', 'units', ...
    'waveforms', wf, 'waveforms_index', wfIndex, 'waveforms_index_index', wfIndexIndex, ...
    'id', types.hdmf_common.ElementIdentifiers('data', [0;1]));

waveforms_index (per spike event):    1   2   3   4   5   6   7
waveforms_index_index (per unit):     3   7

After — one call:

units = types.core.Units('colnames', {}, 'description', 'units');
units.addDoublyRaggedArray('waveforms', {unit1, unit2}, 'description', 'spike waveforms');

waveforms data size:                  4   7
waveforms_index (per spike event):    1   2   3   4   5   6   7
waveforms_index_index (per unit):     3   7
colnames:                             {'waveforms'}

Single-level ragged Units.spike_times

Before:

units = types.core.Units('colnames', {}, 'description', 'units');
[vec, idx] = util.create_indexed_column({[0.1 0.2 0.3], [0.5 0.6]}, 'spike times');
units.addColumn('spike_times', vec, 'spike_times_index', idx);
% spike_times_index:  3   5

After:

units = types.core.Units('colnames', {}, 'description', 'units');
units.addRaggedArray('spike_times', {[0.1 0.2 0.3], [0.5 0.6]}, 'description', 'spike times');
% spike_times_index:  3   5

How to test

nSamples = 4;
unit1 = reshape(1:(3*nSamples), 3, nSamples);
unit2 = 100 + reshape(1:(4*nSamples), 4, nSamples);

units = types.core.Units('colnames', {}, 'description', 'units');
units.addDoublyRaggedArray('waveforms', {unit1, unit2});

assert(isequal(units.waveforms_index.data(:).',       uint64(1:7)))   % per spike event
assert(isequal(units.waveforms_index_index.data(:).', uint64([3 7]))) % per unit

New unit tests cover the helper, the methods, and the documentation examples:

+tests/+unit/createDoublyIndexedColumnTest.m   (6 tests)
+tests/+unit/dynamicTableRaggedArrayTest.m     (5 tests)
+tests/+unit/howToRaggedArrayExamplesTest.m    (runs the how-to guide's example code)

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.50%. Comparing base (2ee6a6b) to head (bab50cc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
+ Coverage   95.45%   95.50%   +0.04%     
==========================================
  Files         228      229       +1     
  Lines        8080     8165      +85     
==========================================
+ Hits         7713     7798      +85     
  Misses        367      367              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.