feat(foundation): real-data ablation, sparse-aware projection, FinaleDB pretrained checkpoint (with audit-2 fixes merged) - #6
Merged
Conversation
…tats, shuffled-label control
Five additive fixes from the 2026-09-21 biomedical review. None change
default behavior; all are opt-in flags so existing benchmark numbers
are preserved.
## A. SensAtSpecLoss wired into training loops (P0 from review)
The review flagged that `src/foundation/losses.py` defined a focal-BCE
loss purpose-built for ultra-low VAF cohorts ("values in [10, 50] are
good starting points for 0.1% VAF analytical cohorts"), but every
training loop used `F.cross_entropy`. Wired focal-BCE as opt-in
`loss='sens_at_spec'` in:
- CrossAttentionFusion (binary only; multi-class TOO stays CE)
- EarlyLateFusion (binary only)
- FoundationDownstream (binary only)
- `alpha_pos` (default 20.0) and `gamma` (default 2.0) exposed
- Invalid loss string raises ValueError at construction time
## B. Modality schema-fingerprint validator (catches silent fallbacks)
FoundationDownstream.fit() now accepts `validate_schema=True` which
runs `_validate_modality_schema` before training and raises
ValueError on out-of-range per-row medians or negative values in
non-negative modalities. Default off. `MODALITY_SCHEMAS` ships with
generous bounds derived from HEALTHY_RANGES. Catches the
silent-failure class where the model trains on noise.
## C. CAFFCalculator.from_fragments drop-stats counter
Opt-in `return_drop_stats=False` parameter. When True, returns
`(per_arm_coverage, drop_stats)` where drop_stats is
`{raw_chrom: n_dropped}`. Default False preserves backward compat.
Existing tests still pass; 3 new tests verify the contract.
## F. Shuffled-label negative control for paired-design validation
`real_tcga_validation.py` gets `--shuffled-label-control` and
`--n-shuffles` flags. Computes `signal_to_artifact_ratio` at
TF=0.1% per the cfdna-early-detection-validation skill's diagnostic.
Required for Nature Medicine / Cancer Discovery publication
readiness. Default off to preserve existing benchmark numbers.
## Tests
10 new tests added (test_biomedical_review_fixes.py). Test count
grows 66 -> 76. All 79 targeted tests pass; 334/335 in broader
suite pass (the foundation smoke test is a pre-existing flake on
n=40 cohorts, unrelated to this change).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
…DB pretrained checkpoint
Three additions completing the biomedical-review follow-up. All
defaults preserved; the pretrained checkpoint and its loader are
regenerable from the committed scripts (see docs/PRETRAINING.md).
## 1. Honest null result: focal-BCE ablation (docs/SENS_AT_SPEC_ABLATION.md)
5-seed × 5-fold CV on the real 20-patient TCGA-LUAD panel at TF=0.1%,
compared loss="ce" vs loss="sens_at_spec" (alpha_pos=20).
| Metric | CE | sens_at_spec | Paired Δ (95% CI) | p |
|---|---:|---:|---:|---:|
| Foundation AUC | 0.941 ± 0.025 | 0.948 ± 0.024 | +0.0075 [-0.006, +0.021] | 0.208 |
| Foundation Sens@99% | 0.480 ± 0.323 | 0.590 ± 0.225 | +0.110 [-0.09, +0.31] | 0.207 |
The positive point estimate is in the expected direction but
**not statistically significant at n=5**. The biomedical review's
"+5-15pp Sens@99" prediction over-claimed; the real-data lift at
this cohort size is +0.11pp with a wide CI that crosses zero.
**Recommendation: keep loss="ce" as default.** The new code path
is functional, non-regressing, and ready for users who want to opt
in. n=20 cohorts need ≥10 seeds (ideally 20) for statistical
significance on this effect size.
Files: scripts/sens_at_spec_ablation.py, scripts/foundation_real_smoke.py
(Loss/alpha_pos flags plumbed through to all 3 FoundationDownstream
instantiations), results/sens_at_spec_*.json (raw + paired t-test).
## 2. SparseAwareLinearProjection (opt-in per-modality)
Adds a class that emits a learned missing-token when input sparsity
exceeds a threshold. Motivation: at 0.1% VAF the panel-LLR modality is
~99.9% zeros; plain Linear+LayerNorm collapses the constant bias
vector through the transformer as if it were signal.
Wired via:
- LinearProjection (unchanged, default)
- SparseAwareLinearProjection (new, opt-in)
- make_projection(kind="linear"|"sparse_aware") factory
- MultiModalEncoder accepts projection_kinds={"mod": "sparse_aware"}
for per-modality routing; defaults preserve LinearProjection.
7 new tests in test/test_sparse_aware_projection.py:
sparse-row emits missing-token, dense row matches LinearProjection,
forward-shape parity, threshold respected, gradient flow through
both paths, end-to-end MultiModalEncoder test, factory error path.
## 3. Real-data FinaleDB pretrained checkpoint (docs/PRETRAINING.md)
Pipeline scripts/pretrain_real_finaledb.py produces a real-cohort
checkpoint at checkpoints/foundation_pretrained_finaledb.pt
(470 KB, gitignored — regenerable in <1 sec). 16 samples (8
healthy + 8 cancer, balanced across Cristiano 2019 + Jiang 2015).
Trained PROTOTYPE_CONFIG (embed_dim=64, n_layers=2, n_heads=2) for
5 epochs MMP + 2 epochs contrastive on the 16-sample subset.
scripts/finaledb_pretrained_loader.py splits the flat 2256-dim
feature matrix (83 mod summary + 2173 raw DELFI) back into the 6
per-modality dict that FoundationDownstream._validate_modalities
requires. End-to-end load verified: forward pass on the 16-sample
cohort produces finite (16, 6, 64) joint embeddings.
Honest limitations: small cohort (16 of 657), PROTOTYPE_CONFIG only,
no held-out validation (self-supervised, all samples seen), CPU-only
default. Live-fetch from FinaleDB S3 was attempted but the local
network truncates multi-part S3 objects (HEAD 54MB → downloaded
16-23MB). The pre-extracted cache path is the canonical artifact;
the inline extract_5channel_from_frag() function documents the
extract step for future runs on a non-truncating network.
3 regression tests in test/test_finaledb_pretrained_loader.py:
modality-shape split, end-to-end load, layout-constant guard.
## Test count
66 → 79 → **89 targeted** (+10: 7 sparse projection + 3 loader).
Broader suite: 334 → **344 passed** (excludes 2 known-flaky smoke
tests that fail on main too).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts: # scripts/foundation_real_smoke.py # test/test_biomedical_review_fixes.py
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the biomedical-review follow-up work onto current main (which
includes the audit-2 P0 fixes — GroupKFold, real frag channel,
torch-cpu install, no-torch-import test). All defaults preserved.
Conflicts resolved (3 regions, all additive)
scripts/foundation_real_smoke.pyline 360 — function signature:kept BOTH my
loss/alpha_posparams AND main'spatient_groups/n_folds/n_ensembleparams.scripts/foundation_real_smoke.pyline 806 — call site:pass all new args (loss, alpha_pos, n_folds, n_ensemble, patient_groups).
test/test_biomedical_review_fixes.pylines 452-720 — kept BOTHmy schema-validator test block AND main's no-torch-import regression
test.
What this PR adds
1. Real-data focal-BCE ablation (docs/SENS_AT_SPEC_ABLATION.md):
5-seed × 5-fold CV on the 20-patient TCGA-LUAD panel at TF=0.1%,
loss="ce" vs loss="sens_at_spec" (alpha_pos=20).
Recommend keeping loss="ce" as default.
2. SparseAwareLinearProjection (opt-in per-modality):
LinearProjectionis replaced with a learned missing-token when inputsparsity > threshold. Useful for 99.9%-zero panel-LLR inputs at 0.1%
VAF. Wired via
projection_kinds={"panel_llr": "sparse_aware"}—defaults unchanged. +7 tests.
3. Real-data FinaleDB pretrained checkpoint
(docs/PRETRAINING.md):
16-sample (8 healthy + 8 cancer, Cristiano 2019 + Jiang 2015)
checkpoint at
checkpoints/foundation_pretrained_finaledb.pt—regenerable in <1s from committed scripts. End-to-end
FoundationDownstream(pretrained=True, checkpoint_path=...)loadverified via
scripts/finaledb_pretrained_loader.py+ 3 regressiontests.
Test count
in my added tests
Honest limitations
CPU-only default
on local network. Pre-extracted cache is the canonical artifact.
on 20-patient cohort, lr_baseline_sens@99=0.37 vs 0.40 threshold)
— this is symmetric and does NOT affect the ablation's paired t-test
validity.
🤖 Generated with Claude Code