Repository navigation
explain_clean computes build_contexts twice — once on df and once on cleaned #32
Description
In explain.py:
contexts = build_contexts(df, cfg) # L160
cleaned, report = run_pipeline(df, cfg) # L161
...
post_contexts = build_contexts(cleaned, cfg) # L192build_contexts is called on the original df at L160 to compute role model selections. Then run_pipeline at L161 also calls build_contexts internally (via build_engine_cache at cleaner.py). Finally L192 calls it a third time on the cleaned output.
That's 3 full profiling passes. On wide datasets with 100+ columns, this is a measurable performance hit.
Fix: Accept an optional contexts parameter in explain_clean or refactor run_pipeline to return the cache it built so explain_clean can reuse it.
Activity
WilliamK112 commentedon Aug 14, 2026
I would like to take this. I confirmed there is no assignee or overlapping open PR. Current
mainnow computes the pre-clean role table throughinfer_roles, while the pipeline cache is built after representation repair and intentionally covers only engine-visible columns; reusing either cache wholesale would be stale when column-name, string, dtype, semantic, missing, or outlier steps change the frame. I will preserve those semantics and remove the unnecessary full post-clean profiling pass instead: profile only columns that_narrativescan actually consume (columns with rationale-bearing actions or columns still containing missing values), and skip the post-clean context build entirely when that set is empty. I will add a wide-frame regression that counts per-column profiling work, plus coverage that preserved-missing narratives remain unchanged, then run the focused and full repository checks before opening a single-purpose PR.AI assistance disclosure: I am using OpenAI Codex to inspect, implement, and validate this optimization. I will review the final diff and test evidence before submitting.
WilliamK112 commentedon Aug 14, 2026
Implemented in #181 and marked ready for review. The change preserves the original-frame role profiling and the pipeline’s post-representation engine cache, but limits the final narrative context build to columns that can actually produce a narrative; when there are no rationale-bearing actions or remaining missing values, that build is skipped entirely.
Red-first evidence: the 120-column no-op regression observed 240 per-column context builds before the fix and 120 after it. Local validation passed with 4,052 tests passed, 6 skipped, 12 deselected, and 93.16% coverage, plus Ruff, mypy (201 source files), and diff checks. All hosted CI jobs are green across Python 3.9–3.13, benchmark, truthbench, gauntlet, plugin contract, reproducibility, wheel guard, build, and the large-data collection gate.
fix(truthbench): match digit-only canaries within one numeric run (#408)