Skip to content

test: migrate stats/ttest2 to ULP-based assertions - #15481

Merged
kgryte merged 1 commit into
developfrom
kgryte/ulp-stats-ttest2
Sep 24, 2026
Merged

kgryte merged 1 commit into
developfrom
kgryte/ulp-stats-ttest2

Conversation

@kgryte

@kgryte kgryte commented Sep 24, 2026

Copy link
Copy Markdown
Member

Resolves a part of #11352.

Description

What is the purpose of this pull request?

This pull request:

  • migrates the tests for stats/ttest2 from relative tolerance testing to ULP difference testing.

Specifically, in test/test.js, each of the 16 relative tolerance assertions of the form

expected = unequal.pValue;
delta = abs( out.pValue - expected );
tol = 18.0 * EPS * abs( expected );
t.ok( delta <= tol, 'within tolerance. a: ['+a+']. b: ['+b+']. pValue: '+out.pValue+'. E: '+expected+'. Δ: '+delta+'. tol: '+tol );

is replaced by

expected = unequal.pValue;
t.strictEqual( isAlmostSameValue( out.pValue, expected, 17 ), true, 'returns expected value' );

along with the corresponding require changes (@stdlib/math/base/special/abs and @stdlib/constants/float64/eps dropped in favor of @stdlib/assert/is-almost-same-value) and the removal of the now-unused delta/tol locals.

ULP bounds (all measured minimums). Because each assertion compares a distinct output quantity against its own R-derived fixture value, the bounds are per-assertion rather than a single package-wide constant:

Test case Quantity ULP
Welch, unequal variances (default) pValue 17
statistic 0
ci[0] 0
ci[1] 1
variance: 'equal' pValue 1
statistic 0
ci[0] 0
ci[1] 0
difference: -2.0 ci[0] 0
ci[1] 0
alternative: 'less' pValue 17
statistic 0
ci[1] 0
alternative: 'greater' pValue 0
statistic 0
ci[0] 0

Each bound was determined empirically by computing, for every (actual, expected) pair, the smallest integer N for which isAlmostSameValue( actual, expected, N ) returns true. The three non-zero bounds were additionally confirmed to fail at N-1, so none can be tightened further:

Assertion N N-1
Welch pValue 17 → true 16 → false
Welch ci[1] 1 → true 0 → false
variance: 'equal' pValue 1 → true 0 → false
alternative: 'less' pValue 17 → true 16 → false

The remaining twelve comparisons are bit-exact against the R fixtures and therefore use 0. The suite was run twice at the final bounds to confirm determinism (identical results, no FMA/arch variation).

The two loosest bounds (17 for the Welch and one-sided p-values) correspond to the pre-existing loosest relative tolerance in the file (18.0 * EPS); these p-values are ~1e-5 tail probabilities computed via t/cdf, where the accumulated error is largest.

Notes:

  • The package has no test/test.native.js, so test/test.js is the only file changed.
  • Exact assertions elsewhere in the file were left untouched, including t.strictEqual( out.pValue, diff.pValue, ... ) and t.strictEqual( out.statistic, diff.statistic, ... ) in the difference test case (already bit-exact), t.strictEqual( out.ci[0], NINF, ... ), t.strictEqual( out.ci[1], PINF, ... ), and the .print() and argument-validation tests.
  • test/test.validate.js contains no relative tolerance assertions and is unchanged.

Verification:

  • make test TESTS_FILTER=".*/stats/ttest2/.*" — 46/46 and 89/89 passing (run twice, identical results).
  • make eslint-tests TESTS_FILTER=".*/stats/ttest2/.*" — clean.

Related Issues

Does this pull request have any related issues?

This pull request has the following related issues:

Questions

Any questions for reviewers of this pull request?

No.

Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

The change mirrors the idiom established in stats/ttest (#15147), the closest prior conversion in the same family, which has the same per-quantity fixture structure.

Two environment notes, neither affecting the diff:

  • make install-node-modules failed in this environment because the npm registry resolved for this session has no es-object-atoms@^1.1.2 (a transitive dependency); dev dependencies were installed with that dependency pinned to 1.1.1 so that the test and lint targets could run. package.json is unchanged in this PR.
  • The pre-commit hook could not run lint-editorconfig, as it downloads the editorconfig-checker binary from GitHub releases and network access to that repository is not available in this session. The changed file was verified manually to be tab-indented, LF-terminated, free of trailing whitespace, ASCII-only, and to end with a final newline.

Checklist

Please ensure the following tasks are completed before submitting this pull request.

AI Assistance

When authoring the changes proposed in this PR, did you use any kind of AI assistance?

  • Yes
  • No

If you answered "yes" above, how did you use AI assistance?

  • Code generation (e.g., when writing an implementation or fixing a bug)
  • Test/benchmark generation
  • Documentation (including examples)
  • Research and understanding

Disclosure

If you answered "yes" to using AI assistance, please provide a short disclosure indicating how you used AI assistance. This helps reviewers determine how much scrutiny to apply when reviewing your contribution. Example disclosures: "This PR was written primarily by Claude Code." or "I consulted ChatGPT to understand the codebase, but the proposed changes were fully authored manually by myself.".

This PR was authored by Claude Code, running unattended as a scheduled task. It reviewed prior converted packages to match the established idiom, applied the conversion, and empirically determined the minimum passing ULP bound for each assertion by measuring the ULP difference against the R fixtures, confirming each non-zero bound fails at N-1, and re-running the suite to confirm determinism.


@stdlib-js/reviewers

🤖 Generated with Claude Code

https://claude.ai/code/session_01RokNCyA4ZqnTxGzuyCAzqH


Generated by Claude Code

Migrate the tests for `stats/ttest2` from relative tolerance testing to
ULP difference testing, replacing the computed `delta`/`tol` comparisons
with `@stdlib/assert/is-almost-same-value` and using the minimum required
ULP value for each assertion.

Resolves a part of #11352.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RokNCyA4ZqnTxGzuyCAzqH
@stdlib-bot stdlib-bot added Statistics Issue or pull request related to statistical functionality. Good First PR A pull request resolving a Good First Issue. labels Sep 24, 2026
@stdlib-bot

Copy link
Copy Markdown
Contributor

Coverage Report

Package Statements Branches Functions Lines
stats/ttest2 $\\color{green}416/416$
$\\color{green}+100.00\\%$
$\\color{green}71/71$
$\\color{green}+100.00\\%$
$\\color{green}3/3$
$\\color{green}+100.00\\%$
$\\color{green}416/416$
$\\color{green}+100.00\\%$

The above coverage report was generated for the changes in this PR.

@kgryte kgryte added the Tests Pull requests specifically adding tests. label Sep 24, 2026
@kgryte
kgryte marked this pull request as ready for review September 24, 2026 06:11
@kgryte
kgryte requested a review from a team September 24, 2026 06:11
@stdlib-bot stdlib-bot added the Needs Review A pull request which needs code review. label Sep 24, 2026
@kgryte
kgryte merged commit bb22129 into develop Sep 24, 2026
82 checks passed
@kgryte
kgryte deleted the kgryte/ulp-stats-ttest2 branch September 24, 2026 06:12
@stdlib-bot stdlib-bot removed the Needs Review A pull request which needs code review. label Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Good First PR A pull request resolving a Good First Issue. Statistics Issue or pull request related to statistical functionality. Tests Pull requests specifically adding tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants