Fix ladder LevelEncoder ignoring value order and spacing (#69)

[MichielStock]

Closes #69.

Problem

The ladder LevelEncoder (every type except FHRR's fractional power path) had three problems:

1. Levels were matched to values by list position. ladder paired level i with values[i]. With unsorted input, such as raw PageRank scores, numerically close values could sit far apart on the ladder and get near-orthogonal hypervectors.
2. Duplicate values each got their own level. encode only ever returns the first one, so the extra levels did nothing but push the later levels further apart.
3. Steps were spaced by rank, not by value. Every step flipped the same fraction bandwidth, so even sorted input gave similarity that followed rank distance. That's why unique(sort(p)) still didn't match the evenly spaced range version.

Fix

• ladder sorts and deduplicates values. The n ≥ 2 check and the default bandwidth = 2/n use the deduplicated count.
• Each step flips bandwidth × gap / mean gap, capped at 1, so similarity follows value distance, as it does for FHRR.
• On an evenly spaced grid every step still flips exactly bandwidth, so range-based encoders, seeded results and the docstring example are unchanged.
• The docstring now covers unsorted and repeated input.

Check

60 skewed values in random order: rank correlation between the similarity matrix and -|xᵢ - xⱼ|

	before	after
BinaryHV / BipolarHV	0.03	0.999
TernaryHV	0.029	1.0
FHRR (unaffected)	1.0	1.0

The full test suite passes. The new regression testset covers shuffled input, duplicates, skewed data and equivalence with the range form.

Behaviour change: unevenly spaced value vectors now get levels spaced by value instead of by rank. To get rank-based levels on purpose, pass the ranks as values.

Follow-up: large gaps and non-linear scales

Spacing steps by gap introduced a regression. One large gap pushed a step's flip fraction to the 1.0 cap, and for BinaryHV/BipolarHV, flipping every bit gives the exact opposite vector: LevelEncoder(BipolarHV, [0, 0.001, 1]) produced similarity −1.

• Steps stop at orthogonal. Each step, and bandwidth itself, is capped at 0.5 for bit-flip types (BinaryHV, BipolarHV) and at 1.0 for types that redraw elements. A large gap now gives unrelated levels, at the type's chance similarity. The 2/n default is capped too: with two levels it used to be 1.0, which made the two bipolar levels exact opposites. An explicit bandwidth above the cap throws an ArgumentError.
• Warning for non-linear spacing. Level encoders assume an approximately linear scale, as in Torchhd's Level, where skewed data is log- or rank-transformed first. The constructor measures how far the sorted values are from an evenly spaced grid (the Kolmogorov–Smirnov distance to a uniform spread). It warns above max(0.25, 1.63/√n), the 1% critical value, and suggests a transform or LevelEncoder(HV, extrema(x), n). Calibration:
  • uniform data: at most 0.7% false alarms;
  • exponential data: always warns from n ≈ 50;
  • evenly spaced ranges: never warn.
• NaN/Inf values throw an ArgumentError instead of an internal assertion failure.

🤖 Generated with Claude Code

[MichielStock]

Hi @cvigilv, this should (correctly) close #70

[cvigilv]

I'll test it and report back, but it looks good from my skim through

[cvigilv]

My only comment is that the docstrings got way to big to show in an IDE, therefore I would either (i) summarize it or strip the information on implementation within it or (ii) move most of the extra information added in this PR to the "Additional information" section (which is accessible via ??LevelEncoder)

[cvigilv]

My only comment is that the docstrings got way to big to show in an IDE, therefore I would either (i) summarize it or strip the information on implementation within it or (ii) move most of the extra information added in this PR to the "Additional information" section (which is accessible via ??LevelEncoder)

(I think we need to do a pass over the docstrings to reduce complexity; the last addition to the package made them a bit overwhelming when accessed. We could merge this, and I can do a docstring pass tomorrow, since I was planning to add a few things to the docs).

[MichielStock]

Maybe indeed merge and make a new issue about docstrings?

…

On Tue, 6 Oct 2026, 18:02 Carlos Vigil-Vásquez, ***@***.***> wrote: *cvigilv* left a comment (Kermit-UGent/HyperdimensionalComputing.jl#70) <#70 (comment)> My only comment is that the docstrings got way to big to show in an IDE, therefore I would either (i) summarize it or strip the information on implementation within it or (ii) move most of the extra information added in this PR to the "Additional information" section (which is accessible via ??LevelEncoder) (I think we need to do a pass over the docstrings to reduce complexity; the last addition to the package made them a bit overwhelming when accessed. We could merge this, and I can do a docstring pass tomorrow, since I was planning to add a few things to the docs). — Reply to this email directly, view it on GitHub <#70?email_source=notifications&email_token=ABN5PI46RS4RMWCETXW3VUL5SUJSLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGAZDIMBYGQYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-6020240840>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABN5PIYTFYO6ERD4ELIT7U35SUJSLAVCNFSNUABFKJSXA33TNF2G64TZHM2DCNZRHE4TQNJSHNEXG43VMU5TKNZSHAYDMNZQGA22C5QC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cvigilv]

Maybe indeed merge and make a new issue about docstrings?
…
On Tue, 6 Oct 2026, 18:02 Carlos Vigil-Vásquez, @.> wrote: cvigilv left a comment (Kermit-UGent/HyperdimensionalComputing.jl#70) <#70 (comment)> My only comment is that the docstrings got way to big to show in an IDE, therefore I would either (i) summarize it or strip the information on implementation within it or (ii) move most of the extra information added in this PR to the "Additional information" section (which is accessible via ??LevelEncoder) (I think we need to do a pass over the docstrings to reduce complexity; the last addition to the package made them a bit overwhelming when accessed. We could merge this, and I can do a docstring pass tomorrow, since I was planning to add a few things to the docs). — Reply to this email directly, view it on GitHub <#70?email_source=notifications&email_token=ABN5PI46RS4RMWCETXW3VUL5SUJSLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGAZDIMBYGQYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-6020240840>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABN5PIYTFYO6ERD4ELIT7U35SUJSLAVCNFSNUABFKJSXA33TNF2G64TZHM2DCNZRHE4TQNJSHNEXG43VMU5TKNZSHAYDMNZQGA22C5QC . You are receiving this because you authored the thread.Message ID: @.>

yes, better