Bring kilosort 4 into the pipeline

[oliche]

There is evidence from Allen https://www.biorxiv.org/content/10.1101/2025.11.12.687966v1 that the clustering of KS4 is superior to KS2.5.
Furthermore, we want to pave the way to comparing different clustering algorithms such as Lupin and Dartsort, using the same pre and post processing.

There is also an incentive to move to spike interface for the drift detection, the clustering and the post-processing, for ease of maintenance. For a first pass, we can keep the pre-processing from ibl-neuropixel as some features are not yet in spike interface.

Data Preparation

• pick a 2-3 recordings from the benchmark set, reproducible site, cortex, one more challenging and one NP2.4
• export 45 mins cbin files
• export a single 10 mins dataset (reproducible site or cortex)
• run the iblsorter baseline on AOTA
• run the Allen Ks4 baseline on AOTA
• [ ] run pre-processing from ibl-neuropixel. Do we need to export as kilosort 2.5 or flat binary is fine for spike interface ?
• prepare spikeinterface based preprocessing

Reveal websites

• drift maps: scatter both overlayed and on top
• raw data snippet with spike overlaid

Post-processing

• Look at the KS4 to alf conversion: Add amplitude calculation to kilosort4 importer SpikeInterface/spikeinterface#4503
• Extract waveforms: compare sorting analyzer speed and results to
• Extract true spikes from the sorting analyzer ? How long does it take ?
• Check the overall output files signature of this version against 2.5

Drift correction

• make sure dredge is used with IBL parameters (should be standard parameters)

Spike-sorter

• check that the whitening stabilization is enabled in KS4
• check input parameters against 2.5. Is there similar amplitude thresholds that need adjusting ?
• Output intermediate QCs as for iblsorter (whitening matrix etc..)

Results Integrity

• Output final QCs as for iblsorter: raster maps with drift curves
• Compare iblsorter baseline with new results