feat: use multiple reference sequences for minimizer index generation

[ivan-aksamentov]

Background:

Some pathogen datasets have significant genetic diversity that a single reference sequence cannot fully represent. This limits the accuracy of dataset auto-detection when query sequences are distant from the chosen reference. By allowing multiple reference sequences per dataset, the minimizer index can capture broader sequence diversity and improve detection rates.

Implementation:

• Add optional files.minimizerReferences field in pathogen.json (array of FASTA file paths)
• New get_minimizer_refs() function reads sequences from all listed files, falls back to main reference if field is absent
• make_ref_search_index() combines minimizers from all references using set union; uses average length for normalization
• Backward compatible: existing datasets work unchanged

Usage:

In pathogen.json, add array of FASTA paths containing representative sequences for the dataset:

{
  "files": {
    "reference": "reference.fasta",
    "minimizerReferences": [
      "clade_a.fasta",
      "clade_b.fasta"
    ]
  }
}

Each FASTA file can contain one or more sequences. All sequences across all files contribute minimizers to the dataset's index entry.

Checklist

• Check if changes affect downstream workflows which depend on this dataset. For instance, Nextstrain ingest workflows may break if clade nomenclature changes. Consider fixing those workflows or making an issue at least. Not applicable

[nneune]

I tested the multi-ref minimizer functionality in this branch with CVA16: https://github.com/nextstrain/nextclade_data/tree/cva16-testing-sample

[anna-parker]

@nneune here the length of the reference is being set to the average length of over all minimizers

[anna-parker]

list of unique kmer hashes -> this means if the same kmer hash comes up in multiple references it still is only added once to the list

[nneune]

One thing we noticed is that when multiple references are merged, using literal merged nMinimizers for normalization can hurt scoring.
Current scoring uses qry_hits / ref_minimizers (with length correction; see https://github.com/nextstrain/nextclade/blob/4cb5cf04d66b0d1d19882bddd5eb500ef912dd83/packages/nextclade/src/sort/minimizer_search.rs#L68 ), and historically this worked well for single-reference datasets because ref_minimizers approximated how many minimizers a true match is expected to hit.

In the multi-reference case, this assumption changes: a query close to reference A is not expected to hit all minimizers contributed by A+B+C after merging (especially with deduplication), which can make the denominator too large and lower otherwise good matches.

We suggest fixing this at minimizer generation time (no schema change): for multi-reference datasets, set nMinimizers in minimizer.json to expected minimizer hits (E_minimizer_hits) instead of literal merged unique minimizer count.

A practical approximation is:

E_minimizer_hits ≈ reference_length * 2^(cutoff - 32)

(or more exactly: (reference_length - k + 1) * (2^cutoff / 2^32), minus duplicate k-mers).

This keeps Nextclade code/schema unchanged, preserves backward compatibility, and naturally reduces to current behavior for single-reference datasets.

Notes on this matter here: https://docs.google.com/document/d/1EGIaSpCdSfgnPO0m4h9_OweGnw8wSGCio3ATK-xfof8/edit?tab=t.0#heading=h.tetsokd7p2ro