Skip to content

docs(tutorials): rework convertTrials tutorial (ragged-array API, correctness fixes, reordered flow)#857

Draft
ehennestad wants to merge 6 commits into
add-ragged-array-methodsfrom
update-convert-trials-tutorial
Draft

docs(tutorials): rework convertTrials tutorial (ragged-array API, correctness fixes, reordered flow)#857
ehennestad wants to merge 6 commits into
add-ragged-array-methodsfrom
update-convert-trials-tutorial

Conversation

@ehennestad

@ehennestad ehennestad commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Depends on #855.

Motivation

Problem — For someone learning trial-based conversion from the convertTrials tutorial, the previous version taught outdated patterns and produced a subtly incorrect file. It stored spike waveforms through a workaround column, wrote null object references that break pynwb's to_dataframe, used a 1-based idx_start for its trial timeseries references (off by one for any schema-aware reader), stored the ElectricalSeries data in the wrong dimension order, and stepped through the sections in an order that made the trials table hard to follow.

Solution — Rework the tutorial to use current MatNWB idioms and produce a schema-faithful, pynwb-interoperable file, with a section order and prose that read cleanly top to bottom.

What changed

  • Spike data now uses the standard Units-table columns: a doubly-ragged waveforms column plus waveform_mean / waveform_sd, built with the new addDoublyRaggedArray helper — no SpikeEventSeries workaround.
  • Raw per-trial voltage traces are linked from the trials table with addRaggedArray, so trials without raw data simply have no reference instead of a null one (the previous null references broke pynwb's to_dataframe).
  • Devices are modeled with DeviceModel + Device.
  • All cross-references use object-based ObjectView / SoftLink rather than hand-written HDF5 paths.
  • Correctness fixes in the exported file: idx_start for trial timeseries references is now 0-based
  • Use starting_time and starting_time_rate instead of timestamps for ElectricalSeries;
  • ElectricalSeries data is written as channels × time; time-unit names are normalized to NWB's plural form (e.g. seconds).
  • Sections reordered for flow (trials table → trial-aligned timeseries references → units → raw acquisition), prose rewritten, and the trials table is displayed with toTable(); the file is checked with inspectNwbFile at the end.
  • basicUsage.mlx updated (it depends on convertTrials)
Implementation notes
  • The waveforms shortcut form (data{unit} = [numSpikes x numSamples]) assumes one electrode per unit, which holds for ALM-3; the tutorial documents the nested data{unit}{spike} = [numElectrodes x numSamples] form for the multi-electrode case.
  • Depends on the addRaggedArray / addDoublyRaggedArray methods and util.create_doubly_indexed_column from feat(dynamictable): add ragged and doubly-ragged array column helpers #855.

How to test

Requires the CRCNS ALM-3 dataset (see the links at the top of the tutorial). With CRCNS_ALM3_ROOT_PATH pointing at the downloaded data, run the tutorial end to end:

run('tutorials/convertTrials.mlx')   % or open and run the live script

It exports out/ANM255201_20141124.nwb and finishes with inspectNwbFile. As a cross-check, the trials and units tables should read cleanly in pynwb, including to_dataframe().

Current tutorial: https://matnwb--852.org.readthedocs.build/en/852/pages/tutorials/convertTrials.html
New Tutorial: https://matnwb--857.org.readthedocs.build/en/857/pages/tutorials/convertTrials.html

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant