Skip to content

perf: cache lambdified equations and avoid redundant export work - #1421

Closed
M-Colley wants to merge 1 commit into
astroautomata:masterfrom
M-Colley:perf/export-caching
Closed

M-Colley wants to merge 1 commit into
astroautomata:masterfrom
M-Colley:perf/export-caching

Conversation

@M-Colley

@M-Colley M-Colley commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Summary

CallableEquation._lambda was a @property that called sympy.lambdify on every access, so every predict() and every direct equations_["lambda_format"](X) call re-generated and exec'd Python source before evaluating the expression. This caches it with functools.cached_property; __getstate__ drops the cached function, which cannot be pickled, and it is rebuilt lazily after loading.

Smaller redundancies on the same paths:

  • latex() called refresh() and then sympy(), which refreshes again, so it rebuilt every export twice.
  • idx_model_selection filtered with DataFrame.query(f"loss <= {threshold}") on every predict(). A boolean mask is about 3x cheaper and avoids round-tripping the float through a string.
  • add_export_formats and _search_output_to_callable_expressions deep-copied frames they only read, and the former rebuilt the SymPy symbol list for every row.
  • _run made a full copy of X just to inspect its dtype, and another before jl_numpy_array, which already copies into Julia memory. It now uses np.asarray, so a copy is only made when the dtype changes.

Measurements

These use a model loaded with from_file from a 6-equation hall of fame with 5 features, on Windows with Julia 1.11 and SymbolicRegression 2.7.0. The machine was shared, so absolute times are noisy.

call master this PR
equations_.iloc[-1]["lambda_format"](X) (10 rows) 3.44 ms 0.15 ms
predict(X) (10 rows) 8.8 ms 5.5 ms
latex() 25 ms 17 ms

Calling lambdify alone on a 5-variable expression takes about 1.75 ms, against about 0.008 ms to evaluate the result.

Tests

  • test_lambda_format_compiles_once_and_stays_picklable: lambdify runs once across repeated calls and predict(), and the callable still round-trips through pickle. On master lambdify runs 6 times.
  • test_latex_refreshes_once: on master refresh runs twice.

Both new tests fail on master and pass here. TestBest, TestLaTeXTable and the export pickling tests pass locally.

This came out of a broader review of the codebase. The related PRs are independent of each other and merge cleanly together.

🤖 Generated with Claude Code

`CallableEquation._lambda` was a property that re-ran `sympy.lambdify`
(code generation plus `exec`) on every call, so every `predict()` and
every direct `lambda_format(X)` call paid ~1-3 ms of compilation before
evaluating the expression. Cache it with `functools.cached_property`
and drop the cached function in `__getstate__`, since generated code
cannot be pickled; it is rebuilt lazily after loading.

Also remove smaller redundancies on the same paths:
- `latex()` refreshed the hall of fame and then called `sympy()`,
  which refreshes it again.
- `idx_model_selection` filtered with `DataFrame.query` on a formatted
  float on every `predict()`; a boolean mask is ~3x cheaper and does
  not round-trip the threshold through a string.
- `add_export_formats` and `_search_output_to_callable_expressions`
  deep-copied frames they only read, and the former rebuilt the SymPy
  symbol list for every row.
- `_run` copied `X` once just to inspect its dtype and again before
  `jl_numpy_array`, which already copies into Julia memory; use
  `np.asarray` so a copy is only made when the dtype changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MilesCranmer

Copy link
Copy Markdown
Member

Thanks for this! I don't see a user-facing problem this PR fixes, and it adds code to maintain. Closing for now; happy to revisit if you hit a real slowdown.

@M-Colley
M-Colley deleted the perf/export-caching branch October 9, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants