Repository navigation
Pair search over every candidate, up to a pair budget - #16
Merged
Merged
Conversation
The synergy-aware pair search took the remaining candidates in (lag, process) order and stopped at the first 20, so jointly informative (XOR) parents at longer lags or in larger networks were never searched. With 12 processes the old pool recovered 2 of 10 XOR pairs; with 20 processes it recovered none. - SkeletonSettings.max_pairs (default 5000 pairs, i.e. 100 candidates) replaces the 20-candidate cap. max_pair_candidates now defaults to None and still caps the pool if set. The order is unchanged and does not look at the target, so the maximum statistic stays valid: on null targets the pair test rejected 5 of 100 with 1540 pairs and 6 of 100 with 4560. - Candidates left out are listed in TargetSkeleton.pair_pool_dropped, with one warning. - The pair test is curtailed for every estimator unless curtail=False: it reports no p-value, and curtailment never changes a decision. - Joint codes are built pair by pair; materialising them all would mean up to 4950 columns as long as the data. - validation/pair_pool.py and a table in the validation docs: 10 of 10 XOR pairs recovered with 8, 12 and 20 processes, at 1.7 to 7 times the time per target.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The synergy-aware pair search finds parents that inform the target only jointly, as in an XOR. Its pool took the remaining candidates in (lag, process) order and stopped at the first 20. In larger networks, XOR parents at longer lags were therefore never searched. For example, with 8 processes and parents (5, lag 3) and (6, lag 2), the pair sits beyond the 20th candidate and is missed entirely.
Change
SkeletonSettings.max_pairs(default 5000 pairs, i.e. 100 candidates) replaces the 20-candidate cap.max_pair_candidatesnow defaults toNoneand still caps the pool when set. The order is unchanged and does not look at the target, so the maximum statistic stays a valid test.TargetSkeleton.pair_pool_droppedlists the candidates left out, and a single warning says so.curtail=False. It reports no p-value, and curtailment never changes a decision.Results
From
validation/pair_pool.py, with 4000 samples, the default permutation counts, and 10 XOR plus 10 null targets per size:The pair test keeps its level with the larger pool. On null targets it rejected 5 of 100 with 12 processes (1540 pairs) and 6 of 100 with 20 processes (4560 pairs). At 20 processes, the false parents were one greedy addition and two spurious pairs across 20 targets, which is consistent with the per-search level.
Tests
test_pair_pool_order_and_budget.test_pair_search_reaches_candidates_beyond_twenty, which also checks the warning andpair_pool_dropped.test_pair_search_curtailment_keeps_the_decision.-Ware clean.Motivated by Hourican & Quax, "Operationalising Weaker Faithfulness" (arXiv:2610.31724).