feat(report): paired-difference significance test + tool_call_collection#205
Merged
Conversation
DavidSouther
force-pushed
the
feature/f-report-stats
branch
from
808a962 to
12d7e5c
Compare
… (paired-difference test + tool_call_collection)
… tool_call_collection Adds PairedDifferenceTest, PAIRED_DIFFERENCE_ALPHA, ComparisonReport.paired_difference, Assertion::ToolCallCollection, and stub dispatch/class_tag wiring so the RED feature test (tests/report_statistics.rs) compiles and fails at todo!()/assertion runtime rather than at compile time. No behavior implemented yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…alue Implements log_gamma (Lanczos approximation), incomplete_beta_continued_fraction (Lentz's method), regularized_incomplete_beta, and paired_difference_p_value, pinned by three unit tests: the standard printed two-tailed-5%-critical-value table entry (t=2.262, df=9 -> p~=0.0500), t=0 -> p=1.0 for any df, and a monotonicity spot-check. Not yet wired into compute_comparison (Step 2). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…te_comparison Collects the per-pair +1.0/-1.0/0.0 diff alongside the existing bucket-count increments, then folds it through compute_paired_difference's general-case branch (mean, sample std dev, SEM, t-statistic, df, p-value via Step 1's paired_difference_p_value, significance at PAIRED_DIFFERENCE_ALPHA). InsufficientPairs and zero-variance remain todo!() until Step 3. Story 1's feature test (compute_comparison_reports_paired_difference_test_with_standard_error) now passes; full lib test suite (216 tests) and tests/report_cmd.rs stay green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…conventions
compute_paired_difference now returns InsufficientPairs { n } for n < 2, and
for n >= 2 with zero variance returns Computed with t_statistic: None and
the fixed limiting values design.md names: p_value=1.0/significant=false
when mean_difference is also zero (vacuous comparison), p_value=0.0/
significant=true otherwise (perfect unanimous shift). Pinned by three new
unit tests. Full lib suite (219 tests) green.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Order-insensitive multiset check over extract_tool_uses/tool_use_name: every name in tools must appear at least as many times as listed; extra/intervening calls tolerated. Empty tools: [] is vacuously satisfied (pinned by a new unit test; the feature test doesn't cover it). Fail reason names every shortfall tool plus required/observed counts. Both feature-test functions in tests/report_statistics.rs now pass (Story 1 and Story 2). Full lib suite (220 tests) green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…on grammar render_comparison_markdown gains a **Paired-difference test:** paragraph after the existing Summary line, naming n, mean_difference, standard_error, t_statistic (or "undefined" when None), degrees_of_freedom, p_value, and a significant/not-significant verdict at PAIRED_DIFFERENCE_ALPHA; InsufficientPairs renders a one-line "insufficient paired assertions" message naming n. render_single_markdown is untouched (design's settled scope boundary) - tests/report_cmd.rs's existing single-mode assertions stay green unchanged. DESIGN.md's Assertion grammar gains the tool_call_collection line in the Tool-call assertions group, immediately after tool_call_order. Pinned by two new unit tests (facts-present, not exact-string). Full test suite (222 lib tests + all integration test binaries) green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the one Follow-up item design.md's Summary names as deferred to TASKS.md for feature-f-report-stats: once tool_call_collection exists, update skills/ailly-skill-eval/SKILL.md and references/method.md to recommend it over tool_call_order by default for new suites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t tests Independent re-verification found `cargo clippy --all-features --all-targets -- -D warnings` actually failing (contradicting the build's own self-report) with 8 errors, all in code this feature-step introduced: missing backticks around identifiers in tests/report_statistics.rs doc comments (clippy::doc_markdown), and `other => panic!(...)` wildcard matches against PairedDifferenceTest's two variants (clippy::match_wildcard_for_single_variants) in both the feature test and report.rs's own new unit tests. Replaced each wildcard arm with an explicit, named binding on the specific unwanted variant. Co-Authored-By: "Ailly <developer@ailly.dev>"
… comment Caught in independent pre-merge review: ToolCallCollection made this 13, not 12. Co-Authored-By: "Ailly <developer@ailly.dev>"
…S.md Per-case paired-difference granularity and a round-trip serde test for Assertion::ToolCallCollection, both named as deferred-to-TASKS in the feature's design.md Summary. The tool_call_collection guidance item was already recorded during Build. Co-Authored-By: "Ailly <developer@ailly.dev>"
…eature G comparison_report_with (a test helper added by Step 5) was written before Feature G's ComparisonReport::falsification_gate field existed on main_two; rebasing this branch onto Feature G's merge surfaced it as a compile error under --all-targets, not caught by cargo test alone. Co-Authored-By: "Ailly <developer@ailly.dev>"
DavidSouther
force-pushed
the
feature/f-report-stats
branch
from
12d7e5c to
b5cf463
Compare
Owner
Author
|
Independent pre-merge review + rebase completed (2026-07-08):
Now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of the
2026-07-06-A-ailly-evalsproject, Feature-step F.compute_comparisonbuckets a two-run comparison into Improved/Regressed/UnchangedPass/UnchangedFail, but gives no way to tell whether a count change is a real regression or ordinary run-to-run LLM variance. Adds a paired-difference significance test (regularized incomplete beta / two-tailed t p-value) plus standard-error-of-the-mean reporting, wired into the existing JSON+Markdown comparison output.Opportunistically also adds
tool_call_collection(order-insensitive multiset check alongside the existing order-sensitivetool_call_order), reusingtool_call_order's own helpers — small, low-risk, same file, bundled per the parent design rather than split into its own feature-step.Status
9 commits, feature test green, clippy clean. Note: this branch has not yet been through the same independent adversarial-review pass that A/B/E/G received — worth extra scrutiny on the statistical-test math specifically.
Test plan
cargo testandcargo clippy