refactor: name things after the method, not the falsified conclusion (#46)

README.md

@@ -12,7 +12,8 @@ frozen forward → capture per-layer residual stream → [optional intervention]
 
 - **Exp4** — train per-layer decoders (the logit-lens tool itself; the one asset that costs compute).
 - **Exp5** — looping / repeating layers, quantified with the logit lens.
-- **Exp6** — self-repair: skip a layer, quantify how downstream layers compensate.
+- **Exp6** — ablate a layer, decode every depth (the paper's self-repair figure).
+  Whether downstream actually compensates is a separate question — see `evaluation/path_patching.py`.
 
 Model-specific code lives only in `adapters/`; capture and interventions use
 plain PyTorch forward hooks (no model-source surgery, no third-party lens libs).

analysis/de-te-figures/README.md

@@ -38,13 +38,13 @@ uv run --group viz python scripts/plot_de_te_perrow.py \
     --in-dir out/perrow --coord margin --hist-only --out out/fig2_perrow_cehist_margin.png
 
 # E2 — per-layer CE~DE law across depth (Hydra Fig-4d)
-uv run --group viz python scripts/plot_ce_de_law.py \
+uv run --group viz python scripts/plot_compensation_fit.py \
     --in-dir out/perrow --coord margin --agg --out out/layer_law_agg_margin.png
 
 # E2 — DE vs CE at the apex layer (Hydra Fig-4b)
-uv run --group viz python scripts/plot_ce_de_law.py \
+uv run --group viz python scripts/plot_compensation_fit.py \
     --in-dir out/perrow --coord margin --agg --apex-scatter --out out/apex_scatter_agg_margin.png
 ```
 
-- `dip_recover_fig8.png` = a copy of `out/self_repair_combined_fig8.png` (exp6 tuned-decoder trajectory), from an uncommitted local plot script.
+- `dip_recover_fig8.png` = a copy of `out/balef_exp6_combined_fig8.png` (exp6 tuned-decoder trajectory), from an uncommitted local plot script.
 - `hydra_ref_*.png` are screenshots from McGrath et al. 2023 — not reproducible from this repo.

docs/self-repair.md

@@ -16,7 +16,7 @@ Context for anyone picking up a self-repair issue (#34–#37). Defines the terms
 
 ## 3. Reproduction & the open question
 
-We reproduced Figure 8 across **LimiX-2M, TabICLv2, Mitra, TabFM** (`evaluation/self_repair.py`, `scripts/plot_self_repair*.py`).
+We reproduced Figure 8 across **LimiX-2M, TabICLv2, Mitra, TabFM** (`evaluation/balef_exp6.py`, `scripts/plot_balef_exp6_*.py`).
 
 - **Observation.** Removing a mid-late layer changes the final output by ~0 → **total effect ≈ 0**.
 - **The trap.** "Removed, output unchanged" has two explanations that look identical:

docs/vastai-gpu-setup.md

@@ -45,7 +45,7 @@ git checkout <branch>                       # a fresh clone is on main — see n
 Notes:
 - **Branch matters — a fresh clone defaults to `main`.** Features under active
   development live on their branch, not `main`: e.g. the GT-logit sweep metric +
-  `plot_self_repair.py --metric gt_logit` are on `feat/logit-metric`. Running the
+  `plot_balef_exp6_trajectory.py --metric gt_logit` are on `feat/logit-metric`. Running the
   sweep from `main` silently gives you only the AUC trajectory. `git checkout` the
   branch that has the code you actually want **before** installing / running.
 - Finetune only needs core + `tabicl` (synthetic prior); it does **not** need the
@@ -119,7 +119,7 @@ Exit-code triage:
 Inside tmux, and **`mkdir -p out` first** (see the redirect trap below):
 ```bash
 mkdir -p out
-tmux new -d -s sweep "/venv/main/bin/python scripts/run_self_repair_sweep.py \
+tmux new -d -s sweep "/venv/main/bin/python scripts/run_balef_exp6_sweep.py \
   --model <model> --subsample-train 1000 --subsample-test 500 \
   --skip-diffs --device cuda --out out/<model>.json > out/<model>.log 2>&1"
 ```
@@ -135,7 +135,7 @@ tmux new -d -s sweep "/venv/main/bin/python scripts/run_self_repair_sweep.py \
   `mkdir -p out` *before* launching tmux, not inside it.
 - High-feature tables may OOM on VRAM: rerun those alone with
   `--tasks <ids> --subsample-train 3000`, then merge the json.
-- Plot: `python scripts/plot_self_repair.py --model <model>`
+- Plot: `python scripts/plot_balef_exp6_trajectory.py --model <model>`
 
 ## 5. Copy files over a flaky / session-limited link (both directions)
 Same root cause as §2's "false success": the vast box drops long connections, so

scripts/validate_resample_donor_health.py → scripts/check_donor_delta_health.py

@@ -14,7 +14,7 @@
 the intervention (upstream is frozen), so one clean target forward + one forward
 per donor suffices — no per-layer re-forward, no decoders.
 
-    uv run --group eval --group viz python scripts/validate_resample_donor_health.py \
+    uv run --group eval --group viz python scripts/check_donor_delta_health.py \
         --models limix_2m mitra tabicl_v2 --tasks 363621,363671,363696
 """
 
@@ -27,7 +27,7 @@
 import torch
 
 from tfm_lens.core.capture import capture_layers
-from tfm_lens.core.resample_ablation import build_donor_delta, donor_deltas
+from tfm_lens.core.donor_delta import build_donor_delta, donor_deltas
 from tfm_lens.evaluation.datasets import TABARENA_BINARY_TASK_IDS, load_tabarena_task
 from tfm_lens.evaluation.preprocess import (
     limix_preprocess,

scripts/validate_resample_self_repair.py → scripts/compare_ablation_stability.py

@@ -1,40 +1,37 @@
-"""V2 self-repair cross-check: does the *faithful* (resample) ablation independently
-show self-repair, and more stably than zero? (#35)
-
-V1 (``validate_resample_donor_health.py``) showed resample is on-manifold +
-norm-preserving. V2 reruns the self-repair sweep under **both** ablations and
-asks, on the decision **margin**:
-
-- **Claim 1 — self-repair holds:** under resample each ablated layer dips
-  (immediate hit) then climbs back toward baseline -> a property of the model,
-  not an artifact of zero's off-manifold shock. Zero is shown **for reference**,
-  not ground truth.
-- **Claim 2 — stability:** resample perturbs *consistently* (small per-layer
+"""Which ablation perturbs better — resample or zero? (#35, V2)
+
+V1 (``check_donor_delta_health.py``) showed resample is on-manifold +
+norm-preserving. V2 reruns the Exp6 sweep under **both** ablations and compares
+them on the decision **margin**:
+
+- **Same trajectory shape:** under resample each ablated layer still dips
+  (immediate hit) then climbs back toward baseline -> the dip-recover shape is a
+  property of the model, not an artifact of zero's off-manifold shock.
+- **Stability (the criterion):** resample perturbs *consistently* (small per-layer
   spread, no overshoot); zero, off-manifold, occasionally sends the readout off
   (big swings + overshoots = ablation makes the margin *better* than baseline).
 
-We deliberately do **not** frame this as "resample reproduces zero" — resample is
-the better method that happens to land on the same self-repair, so no zero-vs-
-resample correlation stat.
+Zero is shown **for reference, not as ground truth** — so there is deliberately no
+zero-vs-resample correlation stat. The question is which intervention to trust.
 
-Inputs: ``out/v2_{model}_{zero,resample}.json`` (``run_self_repair_sweep.py``
+Inputs: ``out/v2_{model}_{zero,resample}.json`` (``run_balef_exp6_sweep.py``
 with ``--ablation zero|resample``). Each stores baseline + per-ablated-layer
 trajectories under ``sweep["skip"]`` (the generic *ablated* condition, whatever
 the ablation) for metrics {auc, gt_logit, margin}.
 
-Per ablated layer ``m`` (metric-normalized, averaged over tasks):
+The only quantity scored, per ablated layer ``m`` (metric-normalized, averaged over tasks):
+
+- **immediate** ``imm(m) = baseline[m+1] - ablated_m[m+1]`` — the direct hit at depth m+1.
 
-- **immediate** ``imm(m) = baseline[m+1] - ablated_m[m+1]`` — the direct hit.
-- **total** ``TE(m) = baseline[-1] - ablated_m[-1]`` — what survives to the final layer.
-- **self-repair** ``SR(m) = imm(m) - TE(m)`` — recovered downstream.
+Its per-layer spread and overshoot count are what rank the two ablations.
 
 Figures:
 
 - ``v2_pair_{model}.png``          margin, zero | resample side-by-side (shared y).
 - ``v2_pair_gtlogit_{model}.png``  same, gt_logit (off-manifold blow-up in raw-logit space).
 - ``v2_stability.png``             per-layer std(imm) + overshoot count, all models.
 
-    uv run --group viz python scripts/validate_resample_self_repair.py \
+    uv run --group viz python scripts/compare_ablation_stability.py \
         --models limix_2m mitra tabicl_v2 tabfm --in-dir out
 """
 
@@ -154,7 +151,7 @@ def _pair_fig(in_dir, model, metric, out):
     _draw_panel(axes[0], zb, zs, "zero (reference)", metric, ylim)
     _draw_panel(axes[1], rb, rs, "resample (faithful)", metric, ylim)
     axes[0].set_ylabel(_METRIC_YLABEL[metric])
-    fig.suptitle(f"{_MODEL_LABELS[model]} · self-repair · {metric}", fontweight="bold")
+    fig.suptitle(f"{_MODEL_LABELS[model]} · ablation trajectory · {metric}", fontweight="bold")
     fig.savefig(out, dpi=140)
     plt.close(fig)
     print(f"wrote {out}")
@@ -222,7 +219,7 @@ def main():
     args = _parse_args()
     args.out_dir.mkdir(parents=True, exist_ok=True)
 
-    # Claim 1 — self-repair holds (margin), plus the gt_logit off-manifold view.
+    # Trajectory shape (margin), plus the gt_logit off-manifold view.
     for model in args.models:
         _pair_fig(args.in_dir, model, "margin", args.out_dir / f"v2_pair_{model}.png")
         _pair_fig(args.in_dir, model, "gt_logit", args.out_dir / f"v2_pair_gtlogit_{model}.png")

scripts/draw_ce_methods.py

@@ -0,0 +1,190 @@
+"""Draw the two CE-computation methods in the Pearl/Hydra causal-graph style
+(⊕ = residual add, red = changed by ablation). For the notes, not an experiment.
+
+Residual stream:  x → (⊕ with branch a) → (⊕ with branch b) → y
+  a = the ablated layer m ;  b = a downstream layer ;  y = final logit (readout f).
+
+    uv run --group viz python scripts/draw_ce_methods.py
+"""
+
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+from matplotlib.patches import FancyArrowPatch
+
+RED = "#e8563f"
+BLACK = "#1a1a1a"
+
+# node positions (shared layout): spine at x=0.9, branch column at x=0.0
+POS = {
+    "x": (0.90, 0.20),
+    "a": (0.00, 1.05),
+    "p1": (0.90, 1.50),
+    "b": (0.00, 2.55),
+    "p2": (0.90, 3.00),
+    "y": (0.90, 3.85),
+}
+
+
+def _arrow(ax, p, q, color, rad=0.0, lw=1.6):
+    ax.add_patch(
+        FancyArrowPatch(
+            POS[p],
+            POS[q],
+            connectionstyle=f"arc3,rad={rad}",
+            arrowstyle="-|>",
+            mutation_scale=11,
+            lw=lw,
+            color=color,
+            zorder=2,
+        )
+    )
+
+
+# where each node's letter label sits, relative to the node (dx, dy, ha, va)
+_LAB = {
+    "x": (0.18, 0.0, "left", "center"),
+    "y": (0.18, 0.0, "left", "center"),
+    "a": (0.0, 0.30, "center", "bottom"),  # above (branch column, left side)
+    "b": (0.0, 0.30, "center", "bottom"),
+}
+
+
+def _node(ax, key, color, filled=True, label=None, lab_color=None):
+    x, y = POS[key]
+    if key in ("p1", "p2"):  # the ⊕ residual-add node
+        ax.scatter([x], [y], s=230, facecolor="white", edgecolor=BLACK, lw=1.4, zorder=3)
+        ax.text(x, y, "+", ha="center", va="center", fontsize=13, zorder=4)
+    else:
+        ax.scatter(
+            [x],
+            [y],
+            s=150,
+            facecolor=(color if filled else "white"),
+            edgecolor=color,
+            lw=1.8,
+            zorder=3,
+        )
+    if label:
+        dx, dy, ha, va = _LAB[key]
+        ax.text(
+            x + dx, y + dy, label, ha=ha, va=va, fontsize=12, color=lab_color or color, zorder=4
+        )
+
+
+def _skeleton(ax, red=(), read_y=False, read_edge=False, do_b=None):
+    """Draw the residual graph. red = set of nodes shown changed.
+    read_y: highlight the output read (method A). read_edge: highlight b→y (method B)."""
+
+    def col(n):
+        return RED if n in red else BLACK
+
+    # branch edges (curved) + skip edges (straight)
+    _arrow(ax, "x", "a", col("a"), rad=0.45)
+    _arrow(ax, "a", "p1", col("a"), rad=0.45)
+    _arrow(ax, "x", "p1", col("x") if "x" in red else BLACK)  # skip x→⊕
+    _arrow(ax, "p1", "b", col("b"), rad=0.45)
+    _arrow(
+        ax,
+        "b",
+        "p2",
+        RED if read_edge or "b" in red else BLACK,
+        rad=0.45,
+        lw=3.0 if read_edge else 1.6,
+    )  # b→⊕ (method B reads here)
+    _arrow(ax, "p1", "p2", col("a") if "a" in red else BLACK)  # skip ⊕→⊕
+    _arrow(ax, "p2", "y", RED if red else BLACK)
+
+    _node(ax, "x", BLACK, label="x")
+    _node(ax, "a", col("a"), label=("ã" if "a" in red else "a"))
+    _node(ax, "p1", BLACK)
+    _node(ax, "b", col("b"), label=("b'" if "b" in red else "b"))
+    _node(ax, "p2", BLACK)
+    _node(ax, "y", RED if red else BLACK, label="y")
+
+    if do_b:
+        bx, by = POS["b"]
+        ax.annotate(
+            do_b,
+            (bx, by),
+            xytext=(bx - 1.55, by - 0.05),
+            fontsize=10,
+            va="center",
+            color=BLACK,
+            arrowprops=dict(arrowstyle="-|>", color=BLACK, lw=1.2),
+        )
+    # ablation label on a
+    ax.annotate(
+        "do(m=ã)",
+        POS["a"],
+        xytext=(POS["a"][0] - 1.55, POS["a"][1] - 0.05),
+        fontsize=10,
+        color=RED,
+        va="center",
+        arrowprops=dict(arrowstyle="-|>", color=RED, lw=1.2),
+    )
+
+    if read_y:
+        yx, yy = POS["y"]
+        ax.scatter([yx], [yy], s=520, facecolor="none", edgecolor="tab:blue", lw=2.2, zorder=5)
+        ax.text(yx + 0.30, yy + 0.02, "read here", color="tab:blue", fontsize=10, va="center")
+    if read_edge:
+        ex, ey = 0.34, POS["b"][1] + 0.42  # on the b→⊕ curve, well below ⊕
+        ax.scatter([ex], [ey], s=430, facecolor="none", edgecolor="tab:blue", lw=2.2, zorder=5)
+        ax.annotate(
+            "read b's\nwrite here",
+            (ex, ey),
+            xytext=(ex - 1.9, ey),
+            fontsize=9.5,
+            color="tab:blue",
+            va="center",
+            arrowprops=dict(arrowstyle="-|>", color="tab:blue", lw=1.2),
+        )
+
+    ax.set_xlim(-1.75, 1.55)
+    ax.set_ylim(-0.15, 4.30)
+    ax.axis("off")
+    ax.set_aspect("equal")
+
+
+def main():
+    fig, axes = plt.subplots(1, 4, figsize=(16, 5.2))
+
+    # Panel 1 — clean
+    _skeleton(axes[0])
+    axes[0].set_title("① Clean\ny = x + a + b", fontsize=11)
+
+    # Panel 2 — Method A, run 1: freeze b at clean (DE side)
+    _skeleton(axes[1], red={"a"}, read_y=True, do_b="do(b = clean)")
+    axes[1].set_title("② Method A · run FROZEN\nablate m, hold b at clean → read y", fontsize=11)
+
+    # Panel 3 — Method A, run 2: b reacts (TE side)
+    _skeleton(axes[2], red={"a", "b"}, read_y=True)
+    axes[2].set_title(
+        "③ Method A · run FREE\nablate m, b reacts (b') → read y\nCE_A = y[FREE] − y[FROZEN]",
+        fontsize=11,
+    )
+
+    # Panel 4 — Method B: read b's own write, clean vs reacted
+    _skeleton(axes[3], red={"a", "b"}, read_edge=True)
+    axes[3].set_title(
+        "④ Method B · independent\nablate m, b reacts (b')\n"
+        "CE_B = (b' 's write to y) − (b's write to y)",
+        fontsize=11,
+    )
+
+    fig.suptitle(
+        "Two ways to compute CE —— A: read output y (two runs, subtract)   "
+        "vs   B: read b's own write (one run, per-layer)",
+        fontsize=13,
+        fontweight="bold",
+    )
+    fig.tight_layout(rect=(0, 0, 1, 0.94))
+    out = Path("out/ce_methods_diagram.png")
+    out.parent.mkdir(exist_ok=True)
+    fig.savefig(out, dpi=140)
+    print(f"wrote {out}")
+
+
+if __name__ == "__main__":
+    main()

scripts/plot_cross_dataset.py → scripts/plot_balef_exp6_cross_dataset.py

@@ -1,4 +1,4 @@
-"""Cross-dataset / cross-model views of the exp6 self-repair sweep.
+"""Cross-dataset / cross-model views of the balef2026 Exp6 ablation sweep.
 
 Drills the per-depth sweep down to the task level and plots the net output
 effect of skipping each layer:
@@ -9,8 +9,8 @@
 - **blue / TE < 0** — delete layer -> output *damaged* (layer does irreplaceable work).
 - **white / TE ~ 0** — output unchanged (repaired *or* redundant).
 
-Reuses ``plot_self_repair._normalize_task`` so numbers match the trajectory plots.
-Reads one sweep JSON per model (``run_self_repair_sweep.py`` output).
+Reuses ``plot_balef_exp6_trajectory._normalize_task`` so numbers match the trajectory plots.
+Reads one sweep JSON per model (``run_balef_exp6_sweep.py`` output).
 
 Draws three figures:
 - ``cross_dataset_heatmap.png``  — 2x2 task x layer TE heatmap, one panel per model,
@@ -20,7 +20,7 @@
 - ``cross_dataset_<model>_L<k>.png`` — per-task fan for one (model, layer); shows
   when a near-zero *average* TE hides a bimodal help/hurt split.
 
-    uv run --group viz python scripts/plot_cross_dataset.py
+    uv run --group viz python scripts/plot_balef_exp6_cross_dataset.py
 """
 
 import argparse
@@ -43,9 +43,9 @@
 
 
 def _load_psr():
-    """Import the sibling ``plot_self_repair.py`` script for its normalizers."""
+    """Import the sibling ``plot_balef_exp6_trajectory.py`` script for its normalizers."""
     here = Path(__file__).resolve().parent
-    spec = importlib.util.spec_from_file_location("psr", here / "plot_self_repair.py")
+    spec = importlib.util.spec_from_file_location("psr", here / "plot_balef_exp6_trajectory.py")
     psr = importlib.util.module_from_spec(spec)
     spec.loader.exec_module(psr)
     return psr