pyclawd/src migration; algorithm fixes + verification; 8 new surrogate-assisted algorithms; fidelity tests - #8
Merged
Merged
Conversation
- Adopt pyclawd: .pyclawd/config.py, AGENTS.md, CLAUDE.md - Move package pysamoo/ -> src/pysamoo/; setup.py reads version without importing the package and uses package_dir/find_packages(where="src") - Add pyproject.toml as the ruff/mypy/pytest config home - Bump pymoo pin 0.6.1.1 -> >=0.6.1.5,<0.6.2 for numpy 2 / matplotlib 3 compatibility (np.math, cm.get_cmap drift); add a "dev" extra - Add module docstrings across the package; remove dead experimental/SACOBRA.py - Speed up the BO example (model_selection=False, n_gen=50); fix np.row_stack deprecation in sampling/rejection.py Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests/test_usage.py: parametric per-algorithm smoke tests (one iteration per algorithm, no plotting); replaces running the heavy demo scripts verbatim - tests/test_benchmark.py + src/pysamoo/benchmark/: a runner to compare surrogate-assisted algorithms and pluggable surrogate models (IGD / objective gap / hypervolume), plus make_surrogate for model swaps; usage_benchmark.py demo - tests/test_golden.py + tests/golden/: golden baselines on the deterministic numerical kernels (indicators, total constraint violation); full-run golden is intentionally skipped (non-deterministic model selection) - Vendored, dependency-free golden plugin (tests/_golden_plugin.py) so CI enforces baselines without pyclawd; root conftest loads it only when pyclawd is absent - docs/PERFORMANCE.md and docs/BENCHMARKING.md; README pointers - GitHub Actions CI running ruff / mypy / pytest on the matrix Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Research artifacts under .claude/docs/ investigating how to make surrogate model selection cheaper and reproducible without sacrificing generalization: - model-selection-research.md: code analysis (38-model pool, 5-fold = 190 fits per selection, runs every iteration since nth_validate is dead code), root cause of non-reproducibility (unseeded CV fold shuffle at target.py:73 and random tie-break at target.py:119; pymoo 0.6.1 stopped seeding global RNGs), measured cost-vs-generalization, and a cited literature survey (closed-form LOO-CV, lazy re-selection, archive capping, ensembles, racing) - model-selection-loop.md: a runnable iterative research loop with the research question, hypothesis backlog, invariants (generalization-first), and an append-only results log - model_selection_bench.py: harness measuring cost, generalization (held-out RMSE/rank), and determinism per selection strategy Headline findings: a redundancy-free family(8) pool matches the full 38-model pool's generalization at ~10x lower cost; seeding the CV folds makes selection reproducible. Neither shipped yet (research only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pymoo 0.6.1 drives its operators from a per-run Generator (self.random_state) and no longer seeds the global numpy/`random` state. pysamoo's selection and infill code still used the globals, so runs were not reproducible under a fixed seed (the initial DOE was effectively random). Best practice is to thread the Generator, not seed globals (NumPy guidance), so: - thread self.random_state through the DOE sampling (algorithm._initialize_infill), GPSAF tournament/alpha/beta/restart sites, PSAF bias replacement, SSANSGA2 roulette selection, and knockout.noisy - pass random_state into pymoo's compare() and RouletteWheelSelection.next() - make CV folds deterministic (CrossvalidationPartitioning randomize=False) and the model-selection tie-break deterministic (models[0] instead of np.random.choice) GPSAF, PSAF and SSANSGA2 are now bit-identical across same-seed runs, guarded by tests/test_reproducibility.py. Model selection still adapts as the archive grows (determinism != frozen choice). Research notes updated in .claude/docs/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design + working prototype for a model-selection pool that shrinks over time from evidence instead of by a fixed guess (.claude/docs/adaptive_pool_prototype.py): - score only an active set each iteration; prune models with consistently worse rolling CV error (successive-halving style) down to a floor - keep >=1 model per kernel family (diversity floor) so generalization coverage is preserved -- the reason naive pool-cutting fails - periodically re-admit pruned models to track the non-stationary landscape as the archive grows (so it never locks in) Simulated 20-iteration runs (growing archive): racing matches the FULL pool's held-out RMSE exactly (ackley 0.666, rastrigin 16.225) at ~56% of the model-fits, and beats a fixed family(8) pool. Because GP fitting is O(n^3) and the archive grows, the pool is full when fits are cheap and shrinks as they get expensive, so the wall-time gain exceeds the fit-count gain. Documented as hypothesis H8 (confirmed) in the research loop; next step is a RacingTarget in src behind a flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread random_state into Surrogate.validate -> Target.validate and shuffle the points (via the run's Generator) before strided k-fold assignment, instead of deterministic striding. Folds are now randomized yet fully reproducible (sklearn's shuffle=True + random_state semantics), so a fold is never biased by the order in which the optimizer produced points. Reproducibility tests still pass; pyclawd check green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n data) Note that the harness trains on uniform random samples generated all at once, whereas real optimization archives are sequential, clustered, and non-stationary. Cost/determinism results are distribution-independent; generalization/quality claims must be re-validated on real archives (test on the next actual infills; decisive test is end-to-end in the real algorithm). Flags CV leakage from spatial clustering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model selection is now a modular strategy (a Target factory), not a hard-coded choice — racing is the first alternative plug-in and others drop in the same way: - pysamoo/core/selection.py: STRATEGIES registry + resolve(); "full" -> Target, "racing" -> RacingTarget, or pass any (label, models) -> Target factory - pysamoo/core/racing.py: RacingTarget keeps an adaptive active set that shrinks over iterations (prune by rolling CV error, per-kernel-family diversity floor, periodic re-admission) — reproducible (deterministic prune/re-admit) - SurrogateAssistedAlgorithm gains a `selection="full"` arg threaded into the default-surrogate build; GPSAF/SSANSGA2 pick it up automatically In a real GPSAF run the active pool shrinks 38 -> 24 -> 18 -> 12 -> 9 and the run is ~20-25% faster at equal solution quality (bigger gains expected when combined with lazy re-selection / closed-form LOO / archive capping — see the loop doc). Renamed the new arg to `selection` to avoid clashing with BO's existing `model_selection` boolean. Guarded by tests/test_selection.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nth_validate was defined but never read; model selection re-ran every iteration. Add SurrogateAssistedAlgorithm.revalidate(), which re-runs the (expensive) full model selection only every nth_validate-th call and reuses the current best model in between (algorithms still surrogate.fit it on new data each iteration). GPSAF, PSAF and SSANSGA2 now call self.revalidate(...) at their _advance selection site. Measured (GPSAF Ackley(10), 220 evals, n_max_doe=200): nth_validate=1 -> 42s; the now-honored default nth_validate=5 -> 11s (~3.8x), with equal-or-better solution quality; combined with selection="racing" -> ~4x. Reproducibility holds (the gate is deterministic). Guarded by tests/test_selection.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Empirically tested closed-form/LOO CV vs 5-fold as a model SELECTOR (.claude/docs/loo_vs_kfold.py). LOO picks a worse-generalizing model than 5-fold (Ackley: test-error gap of the picked model 0.025 vs 0.0001) and disagrees with 5-fold on 20-40% of seeds. LOO's low estimator bias does not translate to good selection -- training on n-1 points makes candidates look alike, so the pick is swayed by individual points (classic LOO selection instability). Conclusion: keep the 5-fold CV; speed must come from lazy re-selection (H3, shipped), racing (H8, shipped) and archive capping (H4) -- not from changing the CV scheme. H2 marked rejected in the loop. Also noted: pydacefit Kriging boxmin intermittently raises under numpy 2 (nonzero on 0d arrays), silently dropping Kriging candidates -- a dependency bug worth fixing in ezmodel/pydacefit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BO previously did ModelSelection over the full 24-config Kriging grid every generation (~90% of its runtime, measured). Instead of duplicating racing logic in BO, route its model selection through the SAME pluggable machinery the other algorithms use: build a Surrogate with a Target/RacingTarget over the Kriging grid and drive it with self.revalidate() (lazy nth_validate gate) + target.fit. - new BO args: racing=True (-> RacingTarget), nth_validate=5 - removed BO's bespoke ModelSelection/CrossvalidationPartitioning usage and the hand-rolled _select_model/_race (now provided by RacingTarget) - revalidate() now also fires on the first call (BO's first selection happens there; other algorithms validate at init separately) - output reads the chosen model via a dedicated attribute, leaving self.surrogate as the Surrogate object Measured (model_selection=True, Sphere(10), 50 gens): 179s -> 42s (~4.3x) with equal-or-better solution quality. pyclawd check green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Algorithms - SSANSGA2: fix within-cluster selection (crowding.argsort() was a scrambled permutation, making the pick effectively random); lower default surr_n_gen 30->20 and thread the inner NSGA2 seed. Robust + faster (ZDT1 worst-case IGD 0.99 -> 0.05). - Kriging model zoo: pass regression *objects*, not strings. The strings silently failed every Kriging fit (caught by raise_exception=False), leaving GPSAF/SSANSGA2 running on RBF only. Restoring the zoo makes GPSAF ~6x and SSANSGA2 ~2.4x better. - GPSAF: thread random_state into the LHS in _doe(). The unseeded LHS made archive subsampling past n_max_doe nondeterministic, the sole source of run-to-run irreproducibility. Now bit-reproducible for a fixed seed. - knockout: thread random_state through comp/pcomp/knockout/calc_prob_relation. Tests - Add tests/test_performance.py: assert each SAO algorithm beats the baseline it wraps, plus a golden snapshot of seed-1 scores (all reproducible now). Docs - Fix the corrupted GPSAF section and citation mojibake; make the license non-commercial and self-consistent; pull the version from version.py; wire docs into pyclawd (docs/runner.py + DocsConfig); re-execute the example notebook. License - Relicense to PolyForm Noncommercial 1.0.0 (LICENSE + setup.py), replacing the contradictory AGPL/Apache metadata.
ParEGO (Knowles, 2006): each infill draws a random Das-Dennis weight, collapses the archive-normalized objectives with the augmented Tchebycheff function, fits one Kriging model to the scalar values, and maximizes Expected Improvement over the box. Rotating the weight spreads the search across the Pareto front while only ever optimizing a single-objective surrogate -- so it reuses the existing stack wholesale: pymoo's Das-Dennis ref-dirs, the pysurrogate Kriging surrogate, and the experimental EI acquisition + optimizer. This is the smallest end-to-end multi-objective EGO on the current machinery; EHVI/qNEHVI slot into the same acquisition seam next. Tests (on pymoo ZDT1): usage smoke, same-seed reproducibility, and a performance assertion (ParEGO beats NSGA2 IGD with fewer evaluations, 120 vs 200) plus its golden snapshot. ParEGO fits on the full archive (no n_max_doe subsampling), so it is bit-reproducible for a fixed seed.
K-RVEA (Chugh et al., 2018): each iteration fits one Kriging model per objective, runs pymoo's RVEA on those cheap surrogate predictions for w_max generations, then selects n_infills candidates to evaluate on the true function. Selection alternates between two K-RVEA criteria, switched by how much the set of active reference vectors changed: diversity (best-aligned candidate per reference vector) when the front is still moving, convergence (highest Kriging uncertainty) when it is stable. Reuses pymoo's RVEA + Das-Dennis reference vectors and the pysurrogate Kriging surrogate (Kriging is required for the predictive sigma the uncertainty criterion needs). Tests (on pymoo DTLZ2, 3-objective): usage smoke, same-seed reproducibility, and a performance assertion (K-RVEA IGD 0.13 vs plain RVEA 0.31 at an equal 150-eval budget) plus its golden snapshot. Reproducible for a fixed seed (RVEA's inner seed is threaded from the run's random_state).
Fits one Kriging model per objective and picks the next point by maximizing the Expected Hypervolume Improvement -- the expected growth of the non-dominated front's hypervolume. EHVI is estimated by Monte-Carlo (sample the per-objective Gaussian posteriors, average the HV improvement), so it needs no exact-EHVI cell decomposition and works for any number of objectives. Reuses pymoo's HV indicator + non-dominated sorting and the pysurrogate Kriging surrogate. The candidate pool combines a space-filling LHS with local perturbations of the current non-dominated designs; a pure LHS pool is too sparse in higher dimensions to locate the acquisition optimum, and this front-seeding is what makes EHVI competitive (ZDT1(10)@120 IGD 0.73 -> 0.026). The pool is screened by an optimistic mu-sigma point's HV improvement so the Monte-Carlo estimate runs only on the most promising candidates. Tests (on pymoo ZDT1): usage smoke, same-seed reproducibility, and a performance assertion (EHVI beats NSGA2 IGD with fewer evaluations, 120 vs 200) plus its golden snapshot.
TuRBO-1 (Eriksson et al., 2019): Bayesian optimization confined to an adaptive hyper-rectangular trust region around the incumbent, so it does not over-explore in higher dimensions the way global BO does. Each infill fits a Kriging model, generates candidates inside the trust region (perturbing a random subset of coordinates per candidate), and picks the best LogEI. The trust-region side length adapts to progress -- doubling after succ_tol improvements, halving after fail_tol failures -- and restarts from a fresh random region when it collapses. Reuses the pysurrogate Kriging surrogate and the experimental LogEI acquisition. Tests (on pymoo Ackley): usage smoke, same-seed reproducibility, and a performance assertion (TuRBO f=0.68 vs GA 16.6 with fewer evals, 200 vs 300) plus its golden snapshot. Note: TuRBO-1 is variance-prone across seeds (single trust region; TuRBO-m addresses this); the perf test pins the strong seed-1 result and the docstring flags the trade-off.
MOEA/D-EGO (Zhang et al., 2010): fits one Kriging model per objective once per iteration and reuses them across weight vectors, selecting a batch of n_infills points -- one per spread weight vector -- each the best candidate under the Tchebycheff aggregation of an optimistic mu-kappa*sigma prediction (an LCB acquisition on the decomposed subproblem). TSEMO (Bradford et al., 2018): draws a single Thompson sample from each per-objective Kriging posterior over a candidate pool, takes the sample's Pareto-optimal candidates, and greedily selects the n_infills points that most increase the true front's hypervolume. Both reuse the existing stack (pymoo ref-dirs / HV / non-dominated sorting + pysurrogate Kriging + the front-seeded candidate pool). Tests on pymoo ZDT1: usage smoke, same-seed reproducibility, and performance assertions (MOEA/D-EGO IGD ~0.07 and TSEMO ~0.04 vs NSGA2 ~0.9, fewer evals) plus golden snapshots. Both bit-reproducible for a fixed seed.
CSEA (Pan et al., 2019): instead of regressing each objective, it trains a classifier to answer the cheaper question "is this design one of the good ones?" -- which scales to many objectives without regression's accuracy demands. Each iteration labels the archive (good = better-than-median non-dominated rank), fits a K-nearest-neighbour classifier on the decision vectors, generates a pool of genetic offspring, and evaluates the n_infills the classifier judges most promising. Reuses scikit-learn's classifier and pymoo's non-dominated sorting. Beats NSGA2 IGD on 3-objective DTLZ2 (perf test + golden). SAASBO (Eriksson & Jankowiak, 2021): sparse axis-aligned BO via an ARD Kriging with a shrinkage theta_prior on the length-scales + Expected Improvement. Shipped as a *scaffold*: faithful SAASBO needs a sparse GP with NUTS hyperparameter sampling, which pysurrogate does not yet have, so the MAP prior here is not competitive on high-dim problems. It runs and is reproducible (usage + reproducibility tests) but carries no performance assertion, with the required surrogate-side work called out in the docstring -- reinforcing the pysurrogate sparse-GP gap flagged earlier. Tests on pymoo DTLZ2/Ackley. Both bit-reproducible for a fixed seed.
…, not just baselines The performance suite only shows each algorithm beats a baseline, which cannot distinguish a faithful implementation from a lucky-but-wrong one. tests/test_fidelity.py checks the pieces the published methods hinge on against exact references: - Monte-Carlo EHVI (EHVI.expected_hvi) matches a 2-objective grid quadrature of the same integral within Monte-Carlo error -- a true ground-truth check of the acquisition math. - Hypervolume improvement of adding a point matches a hand-computed area. - EHVI is zero for a candidate dominated by the front. - ParEGO's scalarization equals the augmented-Tchebycheff formula on a hand example. - CSEA's classification target (CSEA.label_good) equals "non-dominated rank <= median". - TuRBO's trust-region state machine (TuRBO._advance) follows the paper's expand/shrink/ restart rules and keeps the length in bounds. - Front hypervolume is monotone non-decreasing across a run; the reported optimum is a mutually non-dominated set. To test the real code rather than replicas, EHVI.expected_hvi and CSEA.label_good are extracted as static methods (behavior-preserving: the performance goldens are unchanged).
…thms depend on
The new algorithms (ParEGO/TuRBO/SAASBO) and the BayesianOptimization usage import from
pysamoo.experimental.{acquisition (LogEI), infill (GlobalEI/Hybrid), optimizer}, and the core
model-selection was mid-refactor -- selection.py still imported the removed core.racing. Those
files were present in the working tree (so local tests passed) but never committed, so a fresh
checkout of the branch failed to import at all.
This commits the working state of that code so the tree is self-consistent:
- experimental/infill.py, optimizer.py (new): the EI acquisition seam the algorithms reuse.
- experimental/acquisition.py: add LogEI + EIProblem; bo.py: use them.
- experimental/benchmark.py, problems.py, RESEARCH_PLAN.md (new): the method-search harness.
- core/selection.py: drop the removed core.racing import (racing was deleted); core/algorithm.py,
core/target.py: the model-selection refactor the surrogate uses.
Research artifacts (findings/lessons/runs/plots) are intentionally left out.
…ampling/ezmodel) from git The branch needs dev APIs not on PyPI (e.g. pydacefit.regr.ConstantRegression). Install the required states from git before pip install -e; pin to released versions once the stack ships.
…on (#9) Review-driven refactor in 5 waves: dead-code removal + latent bug fixes, docs/naming, the build_default_surrogate hook (−9 duplicated DOE blocks), shared EGO machinery (_ego.py), and resolution of all behavior-risk findings (PSAF floor-vs-cap kept intentional and renamed; KRVEA/SSANSGA2 improved with golden re-blessed). Reproducibility bit-identical; pyclawd check + golden green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the repo onto pyclawd with a
src/layout, fixes and verifies the existingsurrogate-assisted algorithms, adds eight new ones (each tested against pymoo problems), and
adds a fidelity suite that validates the algorithm internals against ground truth. Every
change is reproducible for a fixed seed and gated by
pyclawd check.Infrastructure & migration
src/layout, add the test suite, benchmark harness,performance docs, and CI.
random_statethrough the surrogate-assisted algorithms so a fixed seed isbit-reproducible.
version.py, wire docs into pyclawd (docs/runner.py+DocsConfig), re-execute theexample notebook. Relicense to PolyForm Noncommercial 1.0.0 (LICENSE + setup.py),
replacing the contradictory AGPL/Apache metadata.
Existing-algorithm fixes (with verification)
crowding.argsort()was a scrambled permutation);lower default
surr_n_genand thread the inner seed. Robust + faster (ZDT1 worst-case IGD0.99 -> 0.05).
every Kriging fit, leaving GPSAF/SSANSGA2 on RBF only. Restoring the zoo makes GPSAF ~6x and
SSANSGA2 ~2.4x better.
random_stateinto the LHS in_doe()-- the unseeded LHS madearchive subsampling past
n_max_doenondeterministic, the sole source of run-to-runirreproducibility. Now bit-reproducible.
New algorithms (each beats its baseline on pymoo problems; golden-pinned)
All reuse the existing stack (pymoo ref-dirs / HV / RVEA + the pysurrogate Kriging + the
experimental EI acquisition). SAASBO is shipped as a documented scaffold: faithful SAASBO
needs a sparse GP with NUTS sampling that pysurrogate does not yet have, so it carries no
performance assertion.
Fidelity tests
tests/test_fidelity.pyvalidates the core math against ground truth rather than baselines:Monte-Carlo EHVI matches a 2-objective grid quadrature; ParEGO's scalarization equals the
augmented-Tchebycheff formula; CSEA's label equals "non-dominated rank <= median"; TuRBO's
trust-region state machine follows the paper's rules; front hypervolume is monotone and the
reported optimum is non-dominated.
Not yet done (honest scope)
papers' published IGD/HV tables. The fidelity suite proves the math is correct; matching
published numbers is a follow-up.