diff --git a/CLAUDE.md b/CLAUDE.md index 9736fce..86d304a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,7 +17,7 @@ This is an automated benchmark system for optimization solvers (LP, QP, SOCP, SD ## Development Environment - **Platform**: GitHub Actions CI/CD with GitHub Pages deployment -- **Languages**: Python 3.12+, MATLAB R2024+ (optional) +- **Languages**: Python 3.12+, MATLAB R2020a+ (optional) - **Storage**: SQLite database (`database/results.db`) - **Reports**: `docs/pages/` (static HTML, inline CSS) @@ -38,6 +38,7 @@ python main.py --all # Run specific library benchmarks python main.py --benchmark --library_names DIMACS python main.py --benchmark --library_names SDPLIB +python main.py --benchmark --library_names NETLIB # Generate reports only python main.py --report @@ -69,9 +70,13 @@ Fair baseline benchmarking with minimal configuration, reproducible results, and - **Solver Configurations**: Use minimal parameters (primarily `verbose: false`) for fair comparison - **Database Operations**: SQLite with comprehensive metadata and version tracking - **Result Validation**: Positive solve times, valid status codes, structure analysis -- **External Libraries**: CVXPY conversion for DIMACS/SDPLIB compatibility +- **External Libraries**: CVXPY conversion for DIMACS/SDPLIB/NETLIB compatibility - **Version Tracking**: Complete solver backend and Git commit recording -- **Problem Formats**: SeDuMi .mat (DIMACS), SDPA .dat-s (SDPLIB) +- **Problem Formats**: SeDuMi .mat (DIMACS), SDPA .dat-s (SDPLIB), MPS (NETLIB) + +## Known Issues + +- **NETLIB Python MPS loader bug**: `scripts/data_loaders/python/mps_loader.py` does not reorder free variables, producing incorrect objective values for 11 NETLIB problems under Python solvers (MATLAB's `mps_loader.m` is unaffected). Not yet fixed. See [NETLIB_KNOWN_OBJECTIVES.md](docs/guides/NETLIB_KNOWN_OBJECTIVES.md) before trusting Python-solver results on NETLIB or working on the MPS loader. ## Critical Development Constraints diff --git a/README.md b/README.md index f2ee756..ee33a52 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,12 @@ # Optimization Solver Benchmark System -A research tool for benchmarking optimization solvers across LP, QP, SOCP, and SDP problems using external problem libraries (DIMACS, SDPLIB) with automated execution and HTML report generation. +A research tool for benchmarking optimization solvers across LP, QP, SOCP, and SDP problems using external problem libraries (DIMACS, SDPLIB, NETLIB) with automated execution and HTML report generation. Benchmark results are continuously published at: https://napinoco.github.io/optimization-solver-benchmark/ > **⚠️ Development Status**: This system is currently under active development. Some solvers and problem classes have not yet been properly measured, and comprehensive benchmarking coverage across all problem types is still being completed. +> +> **⚠️ Known Issue**: The Python MPS loader (`scripts/data_loaders/python/mps_loader.py`) does not reorder free variables, which produces incorrect objective values for 11 NETLIB problems when solved with Python solvers (the MATLAB loader is unaffected). See [NETLIB_KNOWN_OBJECTIVES.md](docs/guides/NETLIB_KNOWN_OBJECTIVES.md) before relying on Python-solver results for `capri`, `cycle`, `greenbeb`, `modszk1`, `pilot_ja`, `pilot_we`, `perold`, `pilot4`, `tuff`, `stair`, or `vtp_base`. ## Overview @@ -13,7 +15,8 @@ Benchmark results are continuously published at: https://napinoco.github.io/opti **Problem Libraries**: - **DIMACS**: SeDuMi .mat format - **SDPLIB**: SDPA .dat-s format -- Roughly 120 registered problems; see `config/problem_registry.yaml` for the authoritative list +- **NETLIB**: MPS format (LP problems) +- Roughly 215 registered problems; see `config/problem_registry.yaml` for the authoritative list **Supported Solvers**: - **Python (9)**: SciPy, CVXPY backends (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS) @@ -47,6 +50,7 @@ python main.py --all # Specific libraries python main.py --benchmark --library_names DIMACS python main.py --benchmark --library_names SDPLIB +python main.py --benchmark --library_names NETLIB # Generate reports only python main.py --report @@ -106,7 +110,8 @@ See [basic_design.md](docs/development/basic_design.md) for the full design prin │ └── reporting/ # HTML generation ├── problems/ │ ├── DIMACS/ # External library (git submodule) -│ └── SDPLIB/ # External library (git submodule) +│ ├── SDPLIB/ # External library (git submodule) +│ └── NETLIB/ # External library (git submodule) ├── database/ # SQLite database files │ └── results.db # Benchmark results storage └── docs/ # Generated reports and documentation @@ -121,6 +126,8 @@ See [basic_design.md](docs/development/basic_design.md) for the full design prin **Multi-Language Support**: Unified Python/MATLAB integration via JSON data exchange and standardized result format. +See [Component Architecture](docs/development/detail_design.md#component-architecture) for the full process flow diagram. + ## Adding Components ### New Solvers @@ -134,6 +141,7 @@ Add external libraries as git submodules in `problems/` and extend loaders in `s - **[CLAUDE.md](CLAUDE.md)**: Development context and instructions - **[docs/development/](docs/development/)**: Technical design and implementation details +- **[docs/guides/](docs/guides/)**: Setup and how-to guides (local development, MATLAB configuration, CI/CD, data export, etc.) ## License @@ -141,7 +149,7 @@ MIT License - see [LICENSE](LICENSE) file. ## Acknowledgments -- **External Problem Libraries**: VSDP team for DIMACS and SDPLIB repositories +- **External Problem Libraries**: VSDP team for DIMACS and SDPLIB repositories; ozy4dm for the NETLIB LP problem set (`lp-data-netlib`) - **MATLAB Solvers**: SQLP team for maintaining Git repositories of SeDuMi and SDPT3 solvers - **Open-Source Solvers**: SciPy, CVXPY, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS communities - **Development Support**: Claude Code for coding assistance and development guidance diff --git a/docs/development/basic_design.md b/docs/development/basic_design.md index 6c6eb46..020a851 100644 --- a/docs/development/basic_design.md +++ b/docs/development/basic_design.md @@ -34,7 +34,7 @@ This system prioritizes unbiased solver comparison through minimal configuration ## System Overview -### Core Architecture +### Deployment & Execution Overview ``` LOCAL DEVELOPMENT: ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ @@ -65,8 +65,9 @@ GITHUB ACTIONS (Publishing Only): **Problem Libraries**: - **DIMACS**: SeDuMi .mat format - **SDPLIB**: SDPA .dat-s format +- **NETLIB**: MPS format (LP problems) -Roughly 120 problems are currently registered; the authoritative list (and exact count) is `config/problem_registry.yaml`. +Roughly 215 problems are currently registered; the authoritative list (and exact count) is `config/problem_registry.yaml`. **Solver Support**: - **Python (9)**: SciPy, CVXPY backends (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS) @@ -83,7 +84,7 @@ Roughly 120 problems are currently registered; the authoritative list (and exact ## Implementation Strategy ### Development Approach -**External Library Focus**: Use established problem libraries (DIMACS, SDPLIB) rather than creating internal problems to ensure research relevance and standardization. +**External Library Focus**: Use established problem libraries (DIMACS, SDPLIB, NETLIB) rather than creating internal problems to ensure research relevance and standardization. **Git Submodule Integration**: Pin problem libraries to specific commits for reproducibility while enabling updates when needed. diff --git a/docs/development/detail_design.md b/docs/development/detail_design.md index 56ba198..8af11a0 100644 --- a/docs/development/detail_design.md +++ b/docs/development/detail_design.md @@ -6,7 +6,7 @@ Detailed technical specifications for the optimization solver benchmark system s ## System Overview -**Purpose**: Benchmark optimization solvers using external problem libraries (DIMACS, SDPLIB) with minimal configuration for unbiased performance evaluation. +**Purpose**: Benchmark optimization solvers using external problem libraries (DIMACS, SDPLIB, NETLIB) with minimal configuration for unbiased performance evaluation. **Core Components**: - **Problem Loaders**: Parse MAT/DAT formats → standardized ProblemData @@ -18,82 +18,6 @@ Detailed technical specifications for the optimization solver benchmark system s --- -## Project Structure - -### Directory Layout -``` -optimization-solver-benchmark/ -├── main.py # Entry point with argument parsing -├── config/ # Configuration files -│ ├── site_config.yaml # Site metadata and overview -│ └── problem_registry.yaml # Problem metadata and file paths -├── scripts/ -│ ├── benchmark/ # Benchmark execution engine -│ │ ├── __init__.py -│ │ └── runner.py # Main BenchmarkRunner class -│ ├── solvers/ # Solver interface implementations -│ │ ├── __init__.py -│ │ ├── solver_interface.py # Abstract base classes and SolverResult -│ │ ├── python/ # Python solver implementations (subprocess) -│ │ │ ├── __init__.py -│ │ │ ├── solver_configs.py # Solver registry (single source of truth) -│ │ │ ├── python_process_interface.py # Python subprocess coordinator -│ │ │ ├── python_solver_runner.py # Subprocess entry point + solver manager -│ │ │ ├── cvxpy_runner.py # CVXPY backend handler -│ │ │ └── scipy_runner.py # SciPy linprog handler -│ │ └── matlab/ # MATLAB integration (subprocess) -│ │ ├── __init__.py -│ │ ├── matlab_process_interface.py # Python-MATLAB subprocess bridge -│ │ ├── matlab_solver_runner.m # MATLAB subprocess entry point -│ │ ├── sedumi_runner.m # SeDuMi solver wrapper -│ │ ├── sdpt3_runner.m # SDPT3 solver wrapper -│ │ ├── setup_matlab_solvers.m # MEX compilation script -│ │ ├── sedumi/ # SeDuMi solver (git submodule) -│ │ └── sdpt3/ # SDPT3 solver (git submodule) -│ ├── data_loaders/ # Problem format loaders -│ │ ├── __init__.py -│ │ ├── problem_loader.py # ProblemData class definition -│ │ ├── python/ # Python format loaders -│ │ │ ├── __init__.py -│ │ │ ├── problem_interface.py # Problem loading coordinator -│ │ │ ├── mat_loader.py # DIMACS .mat loader -│ │ │ └── dat_loader.py # SDPLIB .dat-s loader -│ │ └── matlab/ # MATLAB format loaders -│ │ ├── mat_loader.m # MATLAB .mat loader -│ │ └── dat_loader.m # MATLAB .dat-s loader -│ ├── database/ # Database management -│ │ ├── __init__.py -│ │ ├── database_manager.py # Database operations -│ │ ├── models.py # Data models -│ │ └── schema.sql # Database schema -│ ├── reporting/ # Report generation -│ │ ├── __init__.py -│ │ ├── html_generator.py # HTMLGenerator facade (public API) -│ │ ├── report_base.py # Shared base class and HTML/CSS helpers -│ │ ├── overview_report.py # Overview dashboard (index.html) -│ │ ├── results_matrix_report.py # Problems × Solvers matrix -│ │ ├── raw_data_report.py # Raw data table -│ │ ├── data_index_report.py # Data export index page -│ │ ├── result_processor.py # Result aggregation -│ │ └── data_exporter.py # JSON/CSV export -│ └── utils/ # Utility modules (logging, env info, git, temp files) -├── tests/ # pytest test suite -│ ├── unit/ # Unit tests (synthetic fixtures, no submodules needed) -│ └── integration/ # Problem registry integrity checks -├── problems/ # Problem library files -│ ├── DIMACS/ # External DIMACS library (git submodule) -│ └── SDPLIB/ # External SDPLIB library (git submodule) -├── database/ # SQLite database files (results.db, gitignored) -├── docs/ -│ ├── pages/ # Generated HTML reports and data exports -│ ├── development/ # Design documents -│ └── guides/ # User/setup guides -├── pyproject.toml # pytest and ruff configuration -└── requirements.txt # Python dependencies (single file, all pinned) -``` - ---- - ## Component Architecture ### System Data Flow @@ -130,7 +54,7 @@ graph TB %% File System Storage subgraph "File System" - PF[["Problem Libraries
DIMACS .mat.gz
SDPLIB .dat-s"]] + PF[["Problem Libraries
DIMACS .mat.gz
SDPLIB .dat-s
NETLIB .mps"]] TJ[["Temp JSON Files
/tmp/result_xxx.json"]] DBF[("Database Files
results.db")] HTML[["Generated Reports
docs/pages/"]] @@ -205,6 +129,85 @@ The parent process classifies subprocess outcomes by return code (see `_call_pyt --- +## Project Structure + +### Directory Layout +``` +optimization-solver-benchmark/ +├── main.py # Entry point with argument parsing +├── config/ # Configuration files +│ ├── site_config.yaml # Site metadata and overview +│ └── problem_registry.yaml # Problem metadata and file paths +├── scripts/ +│ ├── benchmark/ # Benchmark execution engine +│ │ ├── __init__.py +│ │ └── runner.py # Main BenchmarkRunner class +│ ├── solvers/ # Solver interface implementations +│ │ ├── __init__.py +│ │ ├── solver_interface.py # Abstract base classes and SolverResult +│ │ ├── python/ # Python solver implementations (subprocess) +│ │ │ ├── __init__.py +│ │ │ ├── solver_configs.py # Solver registry (single source of truth) +│ │ │ ├── python_process_interface.py # Python subprocess coordinator +│ │ │ ├── python_solver_runner.py # Subprocess entry point + solver manager +│ │ │ ├── cvxpy_runner.py # CVXPY backend handler +│ │ │ └── scipy_runner.py # SciPy linprog handler +│ │ └── matlab/ # MATLAB integration (subprocess) +│ │ ├── __init__.py +│ │ ├── matlab_process_interface.py # Python-MATLAB subprocess bridge +│ │ ├── matlab_solver_runner.m # MATLAB subprocess entry point +│ │ ├── sedumi_runner.m # SeDuMi solver wrapper +│ │ ├── sdpt3_runner.m # SDPT3 solver wrapper +│ │ ├── setup_matlab_solvers.m # MEX compilation script +│ │ ├── sedumi/ # SeDuMi solver (git submodule) +│ │ └── sdpt3/ # SDPT3 solver (git submodule) +│ ├── data_loaders/ # Problem format loaders +│ │ ├── __init__.py +│ │ ├── problem_loader.py # ProblemData class definition +│ │ ├── python/ # Python format loaders +│ │ │ ├── __init__.py +│ │ │ ├── problem_interface.py # Problem loading coordinator +│ │ │ ├── mat_loader.py # DIMACS .mat loader +│ │ │ ├── dat_loader.py # SDPLIB .dat-s loader +│ │ │ └── mps_loader.py # NETLIB .mps loader +│ │ └── matlab/ # MATLAB format loaders +│ │ ├── mat_loader.m # MATLAB .mat loader +│ │ ├── dat_loader.m # MATLAB .dat-s loader +│ │ └── mps_loader.m # MATLAB .mps loader +│ ├── database/ # Database management +│ │ ├── __init__.py +│ │ ├── database_manager.py # Database operations +│ │ ├── models.py # Data models +│ │ └── schema.sql # Database schema +│ ├── reporting/ # Report generation +│ │ ├── __init__.py +│ │ ├── html_generator.py # HTMLGenerator facade (public API) +│ │ ├── report_base.py # Shared base class and HTML/CSS helpers +│ │ ├── overview_report.py # Overview dashboard (index.html) +│ │ ├── results_matrix_report.py # Problems × Solvers matrix +│ │ ├── raw_data_report.py # Raw data table +│ │ ├── data_index_report.py # Data export index page +│ │ ├── result_processor.py # Result aggregation +│ │ └── data_exporter.py # JSON/CSV export +│ └── utils/ # Utility modules (logging, env info, git, temp files) +├── tests/ # pytest test suite +│ ├── unit/ # Unit tests (synthetic fixtures, no submodules needed) +│ └── integration/ # Problem registry integrity checks +├── problems/ # Problem library files +│ ├── DIMACS/ # External DIMACS library (git submodule) +│ ├── SDPLIB/ # External SDPLIB library (git submodule) +│ └── NETLIB/ # External NETLIB library (git submodule) +├── database/ # SQLite database files (results.db, gitignored) +├── docs/ +│ ├── pages/ # Generated HTML reports and data exports +│ ├── development/ # Design documents +│ └── guides/ # User/setup guides +├── pyproject.toml # pytest and ruff configuration +└── requirements.txt # Python dependencies (single file, all pinned) +``` + +--- + ## Core Data Models ### ProblemData (`scripts/data_loaders/problem_loader.py`) @@ -249,8 +252,8 @@ Standardized dataclass returned by every solver: `solve_time`, `status`, primal/ problem_name: display_name: string # Human-readable problem name file_path: string # Relative path to problem file - file_type: string # "mat" | "dat-s" (supported formats: ProblemInterface.FORMAT_LOADERS) - library_name: string # "DIMACS" | "SDPLIB" + file_type: string # "mat" | "dat-s" | "mps" (supported formats: ProblemInterface.FORMAT_LOADERS) + library_name: string # "DIMACS" | "SDPLIB" | "NETLIB" for_test_flag: boolean # Mark problem for validation testing ``` diff --git a/docs/guides/CONFIGURATION.md b/docs/guides/CONFIGURATION.md index 99f73cf..e9158e4 100644 --- a/docs/guides/CONFIGURATION.md +++ b/docs/guides/CONFIGURATION.md @@ -24,7 +24,7 @@ Contains site metadata for the generated reports and the solver `display_order`. For MATLAB solvers to work, you need: -1. **MATLAB Installation**: MATLAB R2019b or later +1. **MATLAB Installation**: MATLAB R2020a or later 2. **Solver Availability**: SeDuMi and SDPT3 must be installed and in MATLAB path 3. **Command Line Access**: `matlab` command must be available in system PATH @@ -92,15 +92,11 @@ python main.py --validate python main.py --validate-verbose ``` -Expected output for working MATLAB integration: +Working MATLAB integration reports both MATLAB solvers as available, e.g.: ``` -Solver Validation Results: - Working Solvers: 11/11 - Working Problems: 139/139 - MATLAB Solver Status: 2/2 working - ✓ matlab_sedumi: SeDuMi (version 1.3.7) - ✓ matlab_sdpt3: SDPT3 (version 4.0) + ✓ matlab_sedumi: SeDuMi + ✓ matlab_sdpt3: SDPT3 ``` ## Solver Filtering and Selection @@ -172,7 +168,7 @@ print(f"Python solvers: {python_solvers}") **Error**: `MATLAB version not supported` **Solution**: -- Use MATLAB R2019b or later +- Use MATLAB R2020a or later - Update to latest MATLAB version if possible - Check solver compatibility with MATLAB version @@ -228,7 +224,7 @@ For debugging MATLAB integration issues, see `scripts/solvers/matlab/matlab_proc ### Custom Solver Addition -See "Adding New Solvers" in the [Local Development Guide](LOCAL_DEVELOPMENT_GUIDE.md) — the entry point is `scripts/solvers/python/solver_configs.py`. +See [Development Guidelines](../development/detail_design.md#development-guidelines) in detail_design.md — the entry point is `scripts/solvers/python/solver_configs.py`. ## Related Documentation diff --git a/docs/guides/DIMACS_SDPLIB_KNOWN_OBJECTIVES.md b/docs/guides/DIMACS_SDPLIB_KNOWN_OBJECTIVES.md new file mode 100644 index 0000000..3f7d970 --- /dev/null +++ b/docs/guides/DIMACS_SDPLIB_KNOWN_OBJECTIVES.md @@ -0,0 +1,79 @@ +# DIMACS / SDPLIB Known Objective Values + +This note documents the source and caveats of the `known_objective_value` +entries for DIMACS and SDPLIB problems in `config/problem_registry.yaml`. +See [NETLIB_KNOWN_OBJECTIVES.md](NETLIB_KNOWN_OBJECTIVES.md) for the +equivalent note covering NETLIB (which also documents a currently unfixed +data quality bug — worth reading if you're comparing solver results against +known values). + +## DIMACS + +### Source + +The values come from the DIMACS library's own problem table, published by +the 7th DIMACS Implementation Challenge (compiled by Gabor Pataki and +Stefan H. Schmieta), originally hosted at +[dimacs.rutgers.edu](http://dimacs.rutgers.edu/archive/Challenges/Seventh/Instances/) +and mirrored in this project's `problems/DIMACS` submodule as `README.md`. + +### Caveats from the source + +- **torus set sign/scale convention**: the `max{c'x}` problems in this set + are solved here as `min{-c'x}`, so the table's "Opt. value" must be + multiplied by `-1` to match what a SeDuMi-form solve returns; `torusg*` + values must additionally be divided by 100,000. This is why + `torusg3-8`, `torusg3-15`, `toruspm3-8-50`, and `toruspm3-15-50` carry a + negative `known_objective_value` in the registry (corrected in commit + `b8b5ec2`). The `hamming_*` theta-function values were corrected to + negative signs in the same commit, though the source README does not + spell out an explicit min/max caveat for that set the way it does for + torus — treat the sign as empirically matched rather than derived from a + documented rule. +- **`fap` set lower bounds**: `fap25`, `fap-sup25`, and `fap-sup36` are + marked `(lb, not opt)` in the source table — the listed value is a lower + bound, not a proven optimum. `fap25` is intentionally kept in the + registry without a `known_objective_value` for this reason; `fap36` and + the `fap-sup*` problems are currently commented out of the registry + entirely ("Solver issues", commit `b8b5ec2`). +- **`hinf12` / `hinf13` uncertain accuracy**: marked `(?)` in the source + table — "the listed value is the currently known most accurate one; + nevertheless, its accuracy is still not satisfactory, and the true value + may be quite different." Both are present in the registry with numeric + `known_objective_value` fields; treat accuracy comparisons against them + with correspondingly reduced confidence. +- **`biomedP`, `industry2`**: excluded from the registry (source files are + `.dat`, not the supported `.dat-s`/`.mat` formats). + +## SDPLIB + +### Source + +The values come from the SDPLIB library's own problem table, Brian +Borchers' SDPLIB 1.2 (Borchers, B., *SDPLIB 1.2, A Library of Semidefinite +Programming Test Problems*, Optimization Methods and Software, 11(1):683-690, +1999), mirrored in this project's `problems/SDPLIB` submodule as `README.md`. + +### Caveats from the source + +- **Primal/dual sign convention**: "different authors have adopted + different conventions for the primal and dual SDP problems... thus some + objective function values have their signs changed" relative to other + conventions; the source table's values follow SDPA's convention. Loading + SDPLIB here goes through a SeDuMi-form conversion, and a batch of sign + corrections across 88 problems was needed to match (commit `cea8c62`, + "Fix SDPLIB objective value signs due to SDPA to SeDuMi format + conversion"). +- **`infd1`, `infd2`, `infp1`, `infp2`**: reported as "dual infeasible" / + "primal infeasible" rather than a numeric objective value; these are not + currently in the registry (no numeric `known_objective_value` applies). +- **Conversion precision**: "in some cases, very slight changes in the + optimal objective function value have occurred as a result of the + conversion into SDPA format" — treat the published values as reference + points for sanity-checking solver output, not exact ground truth to many + digits. + +## Related files + +- [`config/problem_registry.yaml`](../../config/problem_registry.yaml) — the `known_objective_value` fields themselves +- `problems/DIMACS/README.md` and `problems/SDPLIB/README.md` — the source tables (populated once the submodules are checked out, e.g. `git submodule update --init problems/DIMACS problems/SDPLIB`) diff --git a/docs/guides/EXTERNAL_LIBRARIES.md b/docs/guides/EXTERNAL_LIBRARIES.md index e2b5bed..4a1cfda 100644 --- a/docs/guides/EXTERNAL_LIBRARIES.md +++ b/docs/guides/EXTERNAL_LIBRARIES.md @@ -2,29 +2,29 @@ This guide explains how to set up and use external optimization problem libraries (DIMACS and SDPLIB) with the benchmark system. -## 📋 Overview +## Overview The benchmark system supports external problem libraries to test solvers on real-world optimization problems: -- **DIMACS**: 47 optimization problems in SeDuMi .mat format -- **SDPLIB**: 92 semidefinite programming problems in SDPA format +- **DIMACS**: SeDuMi `.mat` format problems +- **SDPLIB**: SDPA `.dat-s` format semidefinite programming problems - **Automatic Integration**: Problems are automatically detected and converted to CVXPY format -## 🚀 Quick Setup +Exact problem counts change over time; `config/problem_registry.yaml` is the authoritative list (see Verify Setup below for how to count them yourself). + +## Quick Setup ### 1. Clone External Libraries -The external libraries are included as Git submodules. They should already be available if you cloned the repository with the complete implementation: +The external libraries are included as Git submodules. They should already be available if you cloned the repository with `--recursive`: ```bash # Check if libraries are present ls problems/DIMACS/ ls problems/SDPLIB/ -# If missing, clone manually -cd problems/ -git clone https://github.com/vsdp/DIMACS_library.git DIMACS -git clone https://github.com/vsdp/SDPLIB.git SDPLIB +# If missing, clone manually (matches .gitmodules) +git submodule update --init problems/DIMACS problems/SDPLIB ``` ### 2. Verify Setup @@ -33,17 +33,18 @@ git clone https://github.com/vsdp/SDPLIB.git SDPLIB # Validate external libraries are properly detected python main.py --validate -# Check problem registry includes external problems +# Count registered problems per library python -c " import yaml with open('config/problem_registry.yaml') as f: - registry = yaml.safe_load(f) - print(f'DIMACS problems: {len(registry.get(\"external_libraries\", {}).get(\"DIMACS\", {}).get(\"problems\", []))}') - print(f'SDPLIB problems: {len(registry.get(\"external_libraries\", {}).get(\"SDPLIB\", {}).get(\"problems\", []))}') + registry = yaml.safe_load(f)['problem_libraries'] +for lib in ('DIMACS', 'SDPLIB'): + count = sum(1 for p in registry.values() if p['library_name'] == lib) + print(f'{lib} problems: {count}') " ``` -## 🧪 Running External Library Benchmarks +## Running External Library Benchmarks ### Basic Usage @@ -54,7 +55,7 @@ python main.py --benchmark --library_names DIMACS,SDPLIB # Run only DIMACS problems python main.py --benchmark --library_names DIMACS -# Run only SDPLIB problems +# Run only SDPLIB problems python main.py --benchmark --library_names SDPLIB # Full benchmark with reporting @@ -71,65 +72,9 @@ python main.py --benchmark --solvers cvxpy_clarabel,cvxpy_scs --library_names DI python main.py --benchmark --library_names DIMACS,SDPLIB --verbose ``` -## 📊 Understanding External Problem Results - -### Problem Structure Analysis - -The system automatically analyzes external problems and displays: - -- **Variables**: Number of decision variables -- **Constraints**: Number of constraints -- **Problem Type**: Automatic classification (LP, QP, SOCP, SDP) -- **Library Source**: Origin information (e.g., "control family", "FILTER problem set") - -Example output: -``` -control1 (SDP): 70 variables, 21 constraints -Source: control family - -filter48_socp (SDP): 3284 variables, 969 constraints -Source: FILTER problem set -``` +## Understanding External Problem Results -### Expected Performance - -External problems are significantly more challenging than synthetic test cases: - -| Library | Total Problems | Typical Success Rate | Best Solvers | -|---------|---------------|---------------------|--------------| -| DIMACS | 47 | ~40% | CLARABEL, SCS | -| SDPLIB | 92 | ~35% | CLARABEL, SCS | -| Combined | 139 | ~37% | CLARABEL primary | - -## 🔧 Configuration - -### Problem Registry Structure - -External libraries are configured in `config/problem_registry.yaml`: - -```yaml -external_libraries: - DIMACS: - path: "problems/DIMACS" - format: "sedumi_mat" - description: "DIMACS optimization problems" - problems: - - name: "arch0" - file: "arch/arch0.mat.gz" - family: "arch family" - - name: "control1" - file: "control/control1.mat.gz" - family: "control family" - - SDPLIB: - path: "problems/SDPLIB" - format: "sdpa_sparse" - description: "Semidefinite programming library" - problems: - - name: "hinf1" - file: "hinf1.dat-s" - family: "hinf family" -``` +External problems are significantly more challenging than synthetic test cases, and success rates vary by problem structure (LP/QP/SOCP/SDP) and solver. CLARABEL and SCS (via CVXPY) currently have the broadest compatibility across both libraries. ### Solver Compatibility @@ -140,34 +85,17 @@ external_libraries: *Limited support for specific problem types -## 📈 Analyzing Results +## Configuration -### HTML Reports - -External library results appear in all generated reports: - -- **Problem Analysis**: Shows structure analysis and library sources -- **Results Matrix**: Comprehensive solver performance across all problems -- **Solver Comparison**: Success rates broken down by problem source +### Problem Registry Structure -### Data Exports +External library problems are registered as flat entries in `config/problem_registry.yaml` under `problem_libraries:`. See [Problem Registry Structure](../development/detail_design.md#problem-registry-structure) in detail_design.md for the exact schema (`display_name`, `file_path`, `file_type`, `library_name`, `for_test_flag`). -JSON/CSV exports include external library metadata: +## Analyzing Results -```json -{ - "problem_name": "control1", - "problem_type": "SDP", - "n_variables": 70, - "n_constraints": 21, - "library_source": "control family", - "solver_name": "CLARABEL (via CVXPY)", - "status": "optimal", - "solve_time": 0.018 -} -``` +External library results appear in all generated reports (see [detail_design.md](../development/detail_design.md#report-generation)) and in the JSON/CSV exports (see [EXPORT_GUIDE.md](EXPORT_GUIDE.md)), tagged with their `library_name`. -## 🛠️ Troubleshooting +## Troubleshooting ### Common Issues @@ -177,10 +105,8 @@ JSON/CSV exports include external library metadata: **Problem: Libraries not found** ```bash -# Re-clone missing libraries -cd problems/ -git clone https://github.com/vsdp/DIMACS_library.git DIMACS -git clone https://github.com/vsdp/SDPLIB.git SDPLIB +# Re-clone missing submodules +git submodule update --init problems/DIMACS problems/SDPLIB ``` **Problem: Low success rates** @@ -192,56 +118,14 @@ git clone https://github.com/vsdp/SDPLIB.git SDPLIB 1. **Use CLARABEL**: Best overall performance on external problems 2. **Increase Timeouts**: Some problems require >300 seconds -3. **Filter by Type**: Use specific problem sets (dimacs/sdplib) for focused testing +3. **Filter by Library**: Use `--library_names` for focused testing 4. **Expect Failures**: Many external problems are challenging by design -## 🔍 Adding New External Libraries - -### Step-by-Step Process - -1. **Add Library Directory** -```bash -cd problems/ -git clone NEW_LIBRARY -``` - -2. **Create Loader Module** -```python -# scripts/external/new_library_loader.py -def load_new_library_problem(file_path, problem_name): - # Implement format-specific loading - return problem_data -``` - -3. **Update Problem Registry** -```yaml -external_libraries: - NEW_LIBRARY: - path: "problems/NEW_LIBRARY" - format: "custom_format" - description: "New optimization library" - problems: - - name: "problem1" - file: "problem1.ext" - family: "problem family" -``` - -4. **Update Problem Loader** -```python -# scripts/benchmark/problem_loader.py -if problem_set == "NEW_LIBRARY": - from scripts.external.new_library_loader import load_new_library_problem - return load_new_library_problem(file_path, problem_name) -``` - -### Format Requirements +## Adding New External Libraries -New libraries should provide: -- **Problem Data**: Objective function, constraints, variable bounds -- **Metadata**: Problem dimensions, type classification -- **CVXPY Compatibility**: Conversion to CVXPY format for solver compatibility +See [Adding New Problem Formats](../development/detail_design.md#adding-new-problem-formats) in detail_design.md for the current loader architecture and registration steps. -## 📚 References +## References ### External Library Sources - **DIMACS**: [DIMACS Implementation Challenges](http://dimacs.rutgers.edu/) @@ -250,4 +134,5 @@ New libraries should provide: ### Related Documentation - **[Technical Design](../development/detail_design.md)**: Problem loaders and data model -- **[Local Development Guide](LOCAL_DEVELOPMENT_GUIDE.md)**: Running benchmarks locally \ No newline at end of file +- **[Local Development Guide](LOCAL_DEVELOPMENT_GUIDE.md)**: Running benchmarks locally +- **[DIMACS/SDPLIB Known Objective Values](DIMACS_SDPLIB_KNOWN_OBJECTIVES.md)**: Source and caveats for the `known_objective_value` registry entries diff --git a/docs/guides/LOCAL_DEVELOPMENT_GUIDE.md b/docs/guides/LOCAL_DEVELOPMENT_GUIDE.md index a046379..40c4dbf 100644 --- a/docs/guides/LOCAL_DEVELOPMENT_GUIDE.md +++ b/docs/guides/LOCAL_DEVELOPMENT_GUIDE.md @@ -3,169 +3,48 @@ This guide walks you through setting up, running, and contributing to the optimization solver benchmark system on your local machine. ## Table of Contents -- [Quick Start](#quick-start) -- [Installation](#installation) -- [Running Benchmarks](#running-benchmarks) +- [Setup](#setup) - [Understanding the Output](#understanding-the-output) - [Development Workflow](#development-workflow) - [Testing](#testing) - [Troubleshooting](#troubleshooting) +- [Best Practices](#best-practices) +- [Contributing](#contributing) --- -## Quick Start +## Setup -### Prerequisites -- Python 3.9+ -- Git -- 4GB+ RAM -- 1GB+ free disk space +For installation, cloning with submodules, and running your first benchmark, see the [README Quick Start](../../README.md#quick-start). -### 30-Second Setup -```bash -# Clone the repository -git clone https://github.com/your-org/optimization-solver-benchmark.git -cd optimization-solver-benchmark - -# Install dependencies -pip install -r requirements.txt - -# Validate environment -python main.py --validate - -# Run complete benchmark and generate reports -python main.py --all -``` - -### View Results -Open `docs/index.html` in your web browser to see the generated reports. - ---- +### Alternative Python Environment Setup -## Installation - -### 1. Python Environment Setup - -**Option A: Using pyenv (Recommended)** +**Using pyenv** ```bash -# Install Python 3.12 pyenv install 3.12.2 pyenv local 3.12.2 - -# Create virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` -**Option B: Using conda** +**Using conda** ```bash conda create -n solver-benchmark python=3.12 conda activate solver-benchmark ``` -### 2. Install Dependencies - -**All Dependencies** -```bash -pip install -r requirements.txt -``` - -### 3. Verify Installation -```bash -python main.py --validate -``` - -Expected output: -``` -✅ Environment validation passed -✅ Configuration loaded successfully -✅ 5 solvers available -✅ 142 problems loaded (DIMACS + SDPLIB) -``` - ---- - -## Running Benchmarks - -### Basic Commands - -**Environment Check** -```bash -python main.py --validate -``` - -**Run Benchmarks Only** -```bash -python main.py --benchmark -``` - -**Generate Reports Only** (from existing results) -```bash -python main.py --report -``` - -**Complete Workflow** (benchmark + reports) -```bash -python main.py --all -``` - -### Advanced Options - -**Verbose Output** -```bash -python main.py --all --verbose -``` - -**Specific Solvers** -```bash -python main.py --all --solvers scipy_linprog,cvxpy_clarabel -``` - -**Verbose Output** -```bash -python main.py --all --verbose -``` - -**Quiet Mode** -```bash -python main.py --all --quiet -``` - -### Command Reference - -| Command | Description | -|---------|-------------| -| `--validate` | Check environment setup | -| `--benchmark` | Run benchmarks only | -| `--report` | Generate reports only | -| `--all` | Run benchmarks + generate reports | -| `--verbose` | Enable detailed logging | -| `--quiet` | Suppress non-error output | -| `--solvers` | Specify comma-separated solver list | -| `--dry-run` | Run benchmarks without storing results | - --- ## Understanding the Output ### Generated Files -**HTML Reports** (`docs/`) +**HTML Reports** (`docs/pages/`) - `index.html` - Main dashboard with overview statistics -- `solver_comparison.html` - Detailed solver performance comparison -- `problem_analysis.html` - Problem-specific analysis -- `results_matrix.html` - Interactive results matrix -- `statistical_analysis.html` - Statistical analysis and trends -- `performance_profiling.html` - Performance profiling report -- `environment_info.html` - System and environment information - -**Data Files** (`docs/data/`) -- `results.json` - Complete benchmark results (programmatic access) -- `results.csv` - Results in CSV format (spreadsheet analysis) -- `summary.json` - Summary statistics and solver comparison -- `metadata.json` - Environment and configuration metadata -- `statistical_analysis_report.json` - Statistical analysis data -- `performance_profiling_report.json` - Performance profiling data +- `results_matrix.html` - Problems × solvers results matrix +- `raw_data.html` - Raw results table + +**Data Files** (`docs/pages/data/`) - see [EXPORT_GUIDE.md](EXPORT_GUIDE.md) for the full format reference and analysis examples **Database** (`database/`) - `results.db` - SQLite database with all results @@ -175,7 +54,7 @@ python main.py --all --quiet **Success Rate**: Percentage of problems solved optimally **Solve Time**: Wall-clock time for solver execution **Problem Types**: LP (Linear), QP (Quadratic), SOCP (Second-Order Cone), SDP (Semidefinite) -**Solver Status**: optimal, error, timeout, infeasible, unbounded +**Solver Status**: see the full status code table in [detail_design.md](../development/detail_design.md#solverresult-scriptssolverssolver_interfacepy) ### Interpreting Results @@ -201,40 +80,12 @@ For the project structure, see [detail_design.md](../development/detail_design.m 2. **Test Changes** with `python main.py --validate` 3. **Run Benchmarks** with `python main.py --benchmark` 4. **Generate Reports** with `python main.py --report` -5. **Review Output** in `docs/index.html` +5. **Review Output** in `docs/pages/index.html` 6. **Commit Changes** including updated reports -### Adding New Solvers +### Adding New Solvers or Problems -1. **Add an entry** to `scripts/solvers/python/solver_configs.py` (the single source of truth for Python solver configurations). For a new CVXPY backend this is the only code change; otherwise create a runner class following `scripts/solvers/python/cvxpy_runner.py` and register it in `RUNNER_CLASSES` in `python_solver_runner.py`. - -2. **Add the dependency** to `requirements.txt` (pinned version) and add the solver name to `display_order` in `config/site_config.yaml`. - -3. **Test Integration** -```bash -python main.py --validate -python main.py --benchmark --solvers new_solver -``` - -### Adding New Problems - -The current system uses external problem libraries (DIMACS, SDPLIB). To add new problems: - -1. **Add to existing library structure** following DIMACS/SDPLIB format -2. **Update Registry** in `config/problem_registry.yaml` -```yaml -problem_libraries: - new_problem_name: - library_name: "DIMACS" # or appropriate library - file_path: "problems/DIMACS/data/new_file.mat.gz" - file_type: "mat" # or "dat-s" for SDPLIB - problem_type: "SDP" -``` - -3. **Test Loading** -```bash -python -c "from scripts.data_loaders.python.problem_interface import load_problem; print(load_problem('new_problem_name'))" -``` +See the [Development Guidelines](../development/detail_design.md#development-guidelines) in detail_design.md for the exact steps and files to touch. --- @@ -386,14 +237,14 @@ git push origin feature/new-solver 1. **Validate Environment**: `python main.py --validate` 2. **Run Complete Benchmark**: `python main.py --all` -3. **Check Generated Reports**: Verify `docs/index.html` looks correct +3. **Check Generated Reports**: Verify `docs/pages/index.html` looks correct 4. **Run Tests**: Execute relevant test files 5. **Update Documentation**: Update this guide if you changed workflows ### Pull Request Checklist - [ ] Environment validation passes -- [ ] All solvers execute successfully +- [ ] All solvers execute successfully - [ ] Reports generate without errors - [ ] Database contains expected results - [ ] Tests pass @@ -402,6 +253,6 @@ git push origin feature/new-solver --- -*This guide covers the essential aspects of local development. For technical details, see [detail_design.md](detail_design.md). For project overview, see the main [README.md](../../README.md).* +*This guide covers the essential aspects of local development. For technical details, see [detail_design.md](../development/detail_design.md). For project overview and installation, see the main [README.md](../../README.md).* -*Last Updated: June 2025* \ No newline at end of file +*Last Updated: July 2026* diff --git a/docs/guides/README.md b/docs/guides/README.md index bbf2bb7..845065b 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -8,6 +8,7 @@ | [PR_PREVIEW_GUIDE.md](PR_PREVIEW_GUIDE.md) | Pull request preview deployments | | [EXTERNAL_LIBRARIES.md](EXTERNAL_LIBRARIES.md) | DIMACS / SDPLIB problem library setup | | [EXPORT_GUIDE.md](EXPORT_GUIDE.md) | Accessing exported benchmark data (JSON/CSV) | -| [NETLIB_KNOWN_OBJECTIVES.md](NETLIB_KNOWN_OBJECTIVES.md) | Source and caveats for NETLIB `known_objective_value` entries | +| [DIMACS_SDPLIB_KNOWN_OBJECTIVES.md](DIMACS_SDPLIB_KNOWN_OBJECTIVES.md) | Source and caveats for DIMACS/SDPLIB `known_objective_value` entries | +| [NETLIB_KNOWN_OBJECTIVES.md](NETLIB_KNOWN_OBJECTIVES.md) | Source and caveats for NETLIB `known_objective_value` entries, including an **unfixed data quality bug** affecting 11 problems | For design documents, see [docs/development/](../development/). When updating guides, verify commands and file paths against the current implementation (`python main.py --help`).