fix: compute Hypergeometric pmf in log space for large populations - #430
Conversation
|
The Coverage (nightly) failure here is essentially unrelated to this change. Recent nightly (1.99.0-nightly, 2026-08-06) started firing the It reproduces on a clean checkout of main (02fdab6) with nothing applied: 90 errors. This branch gives 91 — the one extra is the
Happy to send a separate PR swapping those paths over to the associated constants (or dropping the |
pmf multiplied and divided binomial coefficients in f64. Once a coefficient exceeded f64::MAX the result became 0.0 (denominator-only overflow) or NaN (inf/inf). cdf and ln_pmf already used ln_binomial. Delegate pmf to ln_pmf(x).exp(), matching the Multinomial fix, and guard ln_pmf for out-of-support x so it returns NEG_INFINITY instead of relying on ln_binomial edge cases alone. Adds regression tests from statrs-dev#426. Fixes statrs-dev#426
817a255 to
a7875e7
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #430 +/- ##
=======================================
Coverage 95.08% 95.08%
=======================================
Files 62 62
Lines 14087 14108 +21
=======================================
+ Hits 13394 13415 +21
Misses 693 693 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
should consider sharing this implementation detail if there are gotchas around it
|
good call, thanks for this fix and adjusting the tests for the exact integer versions. If someone finds a gotcha due to this, we may need to branch between log space and an actual rational implementation. |
Problem
Hypergeometric::pmfmultiplies and divides three binomial coefficients held inf64. Once any coefficient exceedsf64::MAX, the result is wrong:Hypergeometric::new(1030, 1, 515):pmf(0)andpmf(1)return0.0(exact value0.5) because only the denominator overflows.Hypergeometric::new(20000, 200, 300):pmf(0)andpmf(3)returnNaNbecause numerator and denominator both overflow (inf/inf).ln_pmf,cdf, andmeanon the same objects were already correct.Root cause
Direct evaluation of
(K choose x) * (N-K choose n-x) / (N choose n)viafactorial::binomial, which returns a roundedf64that can be+inffor large arguments.Fix
Compute the PMF in log space, matching the Multinomial fix style already in this repo:
pmf(x)delegates toself.ln_pmf(x).exp()ln_pmfreturnsf64::NEG_INFINITYfor out-of-supportx(x > draws,x > successes, orx < min()), sopmfreturns0.0consistentlycdf/sfalready sum exp of log-binomial terms and are left unchangedTest plan
pmf/ln_pmfcargo test --lib(774 passed, 2 ignored)Fixes #426