Skip to content

Synergy benchmarks: Hourican & Quax mechanisms and planted modules - #18

Merged
Autoplectic merged 1 commit into
mainfrom
synergy-benchmarks
Nov 14, 2026
Merged

Autoplectic merged 1 commit into
mainfrom
synergy-benchmarks

Conversation

@Autoplectic

Copy link
Copy Markdown
Member

Summary

This adds benchmarks for the mechanisms of Hourican & Quax, "Operationalising Weaker Faithfulness" (arXiv:2610.31724). It checks #16 (the pair-search budget) and #17 (lag-0 explanation families) against them.

  • datasets: xor_descendant, asymmetric_interaction, or_gate and xor_among, plus contemporaneous_motif(kind, law=) with CONTEMPORANEOUS_MOTIFS. The lag-0 motifs are sampled from their exact laws. law=True returns the dit distribution, so reference families come from explanation_family(decisions_from_distribution(...)).
  • benchmarks.synergy_validation:
    • Lagged motifs are scored on whether the parents are recovered.
    • Lag-0 motifs are scored on whether contemporaneous_families returns exactly the population family.
    • Rates come with Clopper–Pearson intervals.
  • benchmarks.planted_module_validation: synthetic modules (XOR, XOR with a descendant, OR, independent bits) are planted beside a fixed background, following the paper's neural application. It reports recovered parents, exact parents and false parents.
  • validation/synergy.py, a new section in docs/validation.rst, and a literature-table entry.

Results

From validation/synergy.py --planted, with 4000 samples, default settings and 10 replicates:

Check Rate
XOR, flip 0.1 / 0.25 / 0.45: both parents 10/10 each
XOR with a descendant: y from x1 and x2 10/10
XOR with a descendant: w from y only 9/10
Asymmetric interaction; OR; XOR among 12 processes 10/10 each
Lag-0 XOR: population family 9/10
Lag-0 XOR with a descendant, OR, chain, dependent baseline (empty family) 10/10 each
Lag-0 independent baseline 9/10

For modules planted beside a noisy rule-90 automaton of six cells:

  • Every true parent was found in 10/10 replicates of every module.
  • Every module variable got exactly its parents in 8, 7, 8 and 9 of 10 replicates.
  • The remaining replicates had 0.2–0.4 wrong extra parents each, which is consistent with the per-stage levels.

The XOR at flip 0.45 (0.007 bits of whole-minus-sum information) is found in every run. That signal is below the paper's 0.3-bit protection threshold.

Tests

  • tests/test_synergy.py: generators, motif laws, and small runs of both validations.
  • Full suite: 423 passed, 13 skipped. ruff, ty and sphinx -W are clean.

- datasets: xor_descendant, asymmetric_interaction, or_gate and xor_among, plus
  contemporaneous_motif(kind, law=) with CONTEMPORANEOUS_MOTIFS. The lag-0
  motifs are sampled from their exact laws, and law=True returns the dit
  distribution for reference families.
- benchmarks.synergy_validation scores parent recovery on the lagged motifs
  (XOR at three noise levels, XOR with a descendant, an asymmetric interaction,
  OR, and XOR among 12 processes). On the lag-0 motifs it scores whether
  contemporaneous_families returns exactly the population family. Rates come
  with Clopper-Pearson intervals.
- benchmarks.planted_module_validation plants synthetic modules (XOR, XOR with a
  descendant, OR, independent bits) beside a fixed background. It reports
  recovered parents, exact parents and false parents.
- validation/synergy.py and a validation docs section. Every check scores 9 or
  10 of 10. Planted modules recovered every true parent in 10 of 10 replicates
  and were exact in 7 to 9.
@Autoplectic
Autoplectic marked this pull request as ready for review November 14, 2026 21:54
@Autoplectic
Autoplectic merged commit 681408c into main Nov 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant