Skip to content

Reorder detail_design.md so the architecture diagram leads Project Structure #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
napinoco merged 4 commits into from
Jul 7, 2026
Merged

Reorder detail_design.md so the architecture diagram leads Project Structure #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5292880
Reorder detail_design.md so the architecture diagram leads Project St…
claude Jul 5, 2026
13faa0e
Link docs/guides/ from README and trim duplicated/stale content in gu…
claude Jul 6, 2026
fa06fb2
Document NETLIB library and fix MATLAB version inconsistency
claude Jul 6, 2026
b7fc47e
Add DIMACS/SDPLIB known-objective-value provenance and surface the NE…
claude Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This is an automated benchmark system for optimization solvers (LP, QP, SOCP, SD
## Development Environment

- **Platform**: GitHub Actions CI/CD with GitHub Pages deployment
- **Languages**: Python 3.12+, MATLAB R2024+ (optional)
- **Languages**: Python 3.12+, MATLAB R2020a+ (optional)
- **Storage**: SQLite database (`database/results.db`)
- **Reports**: `docs/pages/` (static HTML, inline CSS)

Expand All @@ -38,6 +38,7 @@ python main.py --all
# Run specific library benchmarks
python main.py --benchmark --library_names DIMACS
python main.py --benchmark --library_names SDPLIB
python main.py --benchmark --library_names NETLIB

# Generate reports only
python main.py --report
Expand Down Expand Up @@ -69,9 +70,13 @@ Fair baseline benchmarking with minimal configuration, reproducible results, and
- **Solver Configurations**: Use minimal parameters (primarily `verbose: false`) for fair comparison
- **Database Operations**: SQLite with comprehensive metadata and version tracking
- **Result Validation**: Positive solve times, valid status codes, structure analysis
- **External Libraries**: CVXPY conversion for DIMACS/SDPLIB compatibility
- **External Libraries**: CVXPY conversion for DIMACS/SDPLIB/NETLIB compatibility
- **Version Tracking**: Complete solver backend and Git commit recording
- **Problem Formats**: SeDuMi .mat (DIMACS), SDPA .dat-s (SDPLIB)
- **Problem Formats**: SeDuMi .mat (DIMACS), SDPA .dat-s (SDPLIB), MPS (NETLIB)

## Known Issues

- **NETLIB Python MPS loader bug**: `scripts/data_loaders/python/mps_loader.py` does not reorder free variables, producing incorrect objective values for 11 NETLIB problems under Python solvers (MATLAB's `mps_loader.m` is unaffected). Not yet fixed. See [NETLIB_KNOWN_OBJECTIVES.md](docs/guides/NETLIB_KNOWN_OBJECTIVES.md) before trusting Python-solver results on NETLIB or working on the MPS loader.

## Critical Development Constraints

Expand Down
16 changes: 12 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Optimization Solver Benchmark System

A research tool for benchmarking optimization solvers across LP, QP, SOCP, and SDP problems using external problem libraries (DIMACS, SDPLIB) with automated execution and HTML report generation.
A research tool for benchmarking optimization solvers across LP, QP, SOCP, and SDP problems using external problem libraries (DIMACS, SDPLIB, NETLIB) with automated execution and HTML report generation.

Benchmark results are continuously published at: https://napinoco.github.io/optimization-solver-benchmark/

> **⚠️ Development Status**: This system is currently under active development. Some solvers and problem classes have not yet been properly measured, and comprehensive benchmarking coverage across all problem types is still being completed.
>
> **⚠️ Known Issue**: The Python MPS loader (`scripts/data_loaders/python/mps_loader.py`) does not reorder free variables, which produces incorrect objective values for 11 NETLIB problems when solved with Python solvers (the MATLAB loader is unaffected). See [NETLIB_KNOWN_OBJECTIVES.md](docs/guides/NETLIB_KNOWN_OBJECTIVES.md) before relying on Python-solver results for `capri`, `cycle`, `greenbeb`, `modszk1`, `pilot_ja`, `pilot_we`, `perold`, `pilot4`, `tuff`, `stair`, or `vtp_base`.

## Overview

Expand All @@ -13,7 +15,8 @@ Benchmark results are continuously published at: https://napinoco.github.io/opti
**Problem Libraries**:
- **DIMACS**: SeDuMi .mat format
- **SDPLIB**: SDPA .dat-s format
- Roughly 120 registered problems; see `config/problem_registry.yaml` for the authoritative list
- **NETLIB**: MPS format (LP problems)
- Roughly 215 registered problems; see `config/problem_registry.yaml` for the authoritative list

**Supported Solvers**:
- **Python (9)**: SciPy, CVXPY backends (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS)
Expand Down Expand Up @@ -47,6 +50,7 @@ python main.py --all
# Specific libraries
python main.py --benchmark --library_names DIMACS
python main.py --benchmark --library_names SDPLIB
python main.py --benchmark --library_names NETLIB

# Generate reports only
python main.py --report
Expand Down Expand Up @@ -106,7 +110,8 @@ See [basic_design.md](docs/development/basic_design.md) for the full design prin
│ └── reporting/ # HTML generation
├── problems/
│ ├── DIMACS/ # External library (git submodule)
│ └── SDPLIB/ # External library (git submodule)
│ ├── SDPLIB/ # External library (git submodule)
│ └── NETLIB/ # External library (git submodule)
├── database/ # SQLite database files
│ └── results.db # Benchmark results storage
└── docs/ # Generated reports and documentation
Expand All @@ -121,6 +126,8 @@ See [basic_design.md](docs/development/basic_design.md) for the full design prin

**Multi-Language Support**: Unified Python/MATLAB integration via JSON data exchange and standardized result format.

See [Component Architecture](docs/development/detail_design.md#component-architecture) for the full process flow diagram.

## Adding Components

### New Solvers
Expand All @@ -134,14 +141,15 @@ Add external libraries as git submodules in `problems/` and extend loaders in `s

- **[CLAUDE.md](CLAUDE.md)**: Development context and instructions
- **[docs/development/](docs/development/)**: Technical design and implementation details
- **[docs/guides/](docs/guides/)**: Setup and how-to guides (local development, MATLAB configuration, CI/CD, data export, etc.)

## License

MIT License - see [LICENSE](LICENSE) file.

## Acknowledgments

- **External Problem Libraries**: VSDP team for DIMACS and SDPLIB repositories
- **External Problem Libraries**: VSDP team for DIMACS and SDPLIB repositories; ozy4dm for the NETLIB LP problem set (`lp-data-netlib`)
- **MATLAB Solvers**: SQLP team for maintaining Git repositories of SeDuMi and SDPT3 solvers
- **Open-Source Solvers**: SciPy, CVXPY, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS communities
- **Development Support**: Claude Code for coding assistance and development guidance
7 changes: 4 additions & 3 deletions docs/development/basic_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ This system prioritizes unbiased solver comparison through minimal configuration

## System Overview

### Core Architecture
### Deployment & Execution Overview
```
LOCAL DEVELOPMENT:
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
Expand Down Expand Up @@ -65,8 +65,9 @@ GITHUB ACTIONS (Publishing Only):
**Problem Libraries**:
- **DIMACS**: SeDuMi .mat format
- **SDPLIB**: SDPA .dat-s format
- **NETLIB**: MPS format (LP problems)

Roughly 120 problems are currently registered; the authoritative list (and exact count) is `config/problem_registry.yaml`.
Roughly 215 problems are currently registered; the authoritative list (and exact count) is `config/problem_registry.yaml`.

**Solver Support**:
- **Python (9)**: SciPy, CVXPY backends (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS)
Expand All @@ -83,7 +84,7 @@ Roughly 120 problems are currently registered; the authoritative list (and exact
## Implementation Strategy

### Development Approach
**External Library Focus**: Use established problem libraries (DIMACS, SDPLIB) rather than creating internal problems to ensure research relevance and standardization.
**External Library Focus**: Use established problem libraries (DIMACS, SDPLIB, NETLIB) rather than creating internal problems to ensure research relevance and standardization.

**Git Submodule Integration**: Pin problem libraries to specific commits for reproducibility while enabling updates when needed.

Expand Down
163 changes: 83 additions & 80 deletions docs/development/detail_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Detailed technical specifications for the optimization solver benchmark system s

## System Overview

**Purpose**: Benchmark optimization solvers using external problem libraries (DIMACS, SDPLIB) with minimal configuration for unbiased performance evaluation.
**Purpose**: Benchmark optimization solvers using external problem libraries (DIMACS, SDPLIB, NETLIB) with minimal configuration for unbiased performance evaluation.

**Core Components**:
- **Problem Loaders**: Parse MAT/DAT formats → standardized ProblemData
Expand All @@ -18,82 +18,6 @@ Detailed technical specifications for the optimization solver benchmark system s

---

## Project Structure

### Directory Layout
```
optimization-solver-benchmark/
├── main.py # Entry point with argument parsing
├── config/ # Configuration files
│ ├── site_config.yaml # Site metadata and overview
│ └── problem_registry.yaml # Problem metadata and file paths
├── scripts/
│ ├── benchmark/ # Benchmark execution engine
│ │ ├── __init__.py
│ │ └── runner.py # Main BenchmarkRunner class
│ ├── solvers/ # Solver interface implementations
│ │ ├── __init__.py
│ │ ├── solver_interface.py # Abstract base classes and SolverResult
│ │ ├── python/ # Python solver implementations (subprocess)
│ │ │ ├── __init__.py
│ │ │ ├── solver_configs.py # Solver registry (single source of truth)
│ │ │ ├── python_process_interface.py # Python subprocess coordinator
│ │ │ ├── python_solver_runner.py # Subprocess entry point + solver manager
│ │ │ ├── cvxpy_runner.py # CVXPY backend handler
│ │ │ └── scipy_runner.py # SciPy linprog handler
│ │ └── matlab/ # MATLAB integration (subprocess)
│ │ ├── __init__.py
│ │ ├── matlab_process_interface.py # Python-MATLAB subprocess bridge
│ │ ├── matlab_solver_runner.m # MATLAB subprocess entry point
│ │ ├── sedumi_runner.m # SeDuMi solver wrapper
│ │ ├── sdpt3_runner.m # SDPT3 solver wrapper
│ │ ├── setup_matlab_solvers.m # MEX compilation script
│ │ ├── sedumi/ # SeDuMi solver (git submodule)
│ │ └── sdpt3/ # SDPT3 solver (git submodule)
│ ├── data_loaders/ # Problem format loaders
│ │ ├── __init__.py
│ │ ├── problem_loader.py # ProblemData class definition
│ │ ├── python/ # Python format loaders
│ │ │ ├── __init__.py
│ │ │ ├── problem_interface.py # Problem loading coordinator
│ │ │ ├── mat_loader.py # DIMACS .mat loader
│ │ │ └── dat_loader.py # SDPLIB .dat-s loader
│ │ └── matlab/ # MATLAB format loaders
│ │ ├── mat_loader.m # MATLAB .mat loader
│ │ └── dat_loader.m # MATLAB .dat-s loader
│ ├── database/ # Database management
│ │ ├── __init__.py
│ │ ├── database_manager.py # Database operations
│ │ ├── models.py # Data models
│ │ └── schema.sql # Database schema
│ ├── reporting/ # Report generation
│ │ ├── __init__.py
│ │ ├── html_generator.py # HTMLGenerator facade (public API)
│ │ ├── report_base.py # Shared base class and HTML/CSS helpers
│ │ ├── overview_report.py # Overview dashboard (index.html)
│ │ ├── results_matrix_report.py # Problems × Solvers matrix
│ │ ├── raw_data_report.py # Raw data table
│ │ ├── data_index_report.py # Data export index page
│ │ ├── result_processor.py # Result aggregation
│ │ └── data_exporter.py # JSON/CSV export
│ └── utils/ # Utility modules (logging, env info, git, temp files)
├── tests/ # pytest test suite
│ ├── unit/ # Unit tests (synthetic fixtures, no submodules needed)
│ └── integration/ # Problem registry integrity checks
├── problems/ # Problem library files
│ ├── DIMACS/ # External DIMACS library (git submodule)
│ └── SDPLIB/ # External SDPLIB library (git submodule)
├── database/ # SQLite database files (results.db, gitignored)
├── docs/
│ ├── pages/ # Generated HTML reports and data exports
│ ├── development/ # Design documents
│ └── guides/ # User/setup guides
├── pyproject.toml # pytest and ruff configuration
└── requirements.txt # Python dependencies (single file, all pinned)
```

---

## Component Architecture

### System Data Flow
Expand Down Expand Up @@ -130,7 +54,7 @@ graph TB

%% File System Storage
subgraph "File System"
PF[["Problem Libraries<br/>DIMACS .mat.gz<br/>SDPLIB .dat-s"]]
PF[["Problem Libraries<br/>DIMACS .mat.gz<br/>SDPLIB .dat-s<br/>NETLIB .mps"]]
TJ[["Temp JSON Files<br/>/tmp/result_xxx.json"]]
DBF[("Database Files<br/>results.db")]
HTML[["Generated Reports<br/>docs/pages/"]]
Expand Down Expand Up @@ -205,6 +129,85 @@ The parent process classifies subprocess outcomes by return code (see `_call_pyt

---

## Project Structure

### Directory Layout
```
optimization-solver-benchmark/
├── main.py # Entry point with argument parsing
├── config/ # Configuration files
│ ├── site_config.yaml # Site metadata and overview
│ └── problem_registry.yaml # Problem metadata and file paths
├── scripts/
│ ├── benchmark/ # Benchmark execution engine
│ │ ├── __init__.py
│ │ └── runner.py # Main BenchmarkRunner class
│ ├── solvers/ # Solver interface implementations
│ │ ├── __init__.py
│ │ ├── solver_interface.py # Abstract base classes and SolverResult
│ │ ├── python/ # Python solver implementations (subprocess)
│ │ │ ├── __init__.py
│ │ │ ├── solver_configs.py # Solver registry (single source of truth)
│ │ │ ├── python_process_interface.py # Python subprocess coordinator
│ │ │ ├── python_solver_runner.py # Subprocess entry point + solver manager
│ │ │ ├── cvxpy_runner.py # CVXPY backend handler
│ │ │ └── scipy_runner.py # SciPy linprog handler
│ │ └── matlab/ # MATLAB integration (subprocess)
│ │ ├── __init__.py
│ │ ├── matlab_process_interface.py # Python-MATLAB subprocess bridge
│ │ ├── matlab_solver_runner.m # MATLAB subprocess entry point
│ │ ├── sedumi_runner.m # SeDuMi solver wrapper
│ │ ├── sdpt3_runner.m # SDPT3 solver wrapper
│ │ ├── setup_matlab_solvers.m # MEX compilation script
│ │ ├── sedumi/ # SeDuMi solver (git submodule)
│ │ └── sdpt3/ # SDPT3 solver (git submodule)
│ ├── data_loaders/ # Problem format loaders
│ │ ├── __init__.py
│ │ ├── problem_loader.py # ProblemData class definition
│ │ ├── python/ # Python format loaders
│ │ │ ├── __init__.py
│ │ │ ├── problem_interface.py # Problem loading coordinator
│ │ │ ├── mat_loader.py # DIMACS .mat loader
│ │ │ ├── dat_loader.py # SDPLIB .dat-s loader
│ │ │ └── mps_loader.py # NETLIB .mps loader
│ │ └── matlab/ # MATLAB format loaders
│ │ ├── mat_loader.m # MATLAB .mat loader
│ │ ├── dat_loader.m # MATLAB .dat-s loader
│ │ └── mps_loader.m # MATLAB .mps loader
│ ├── database/ # Database management
│ │ ├── __init__.py
│ │ ├── database_manager.py # Database operations
│ │ ├── models.py # Data models
│ │ └── schema.sql # Database schema
│ ├── reporting/ # Report generation
│ │ ├── __init__.py
│ │ ├── html_generator.py # HTMLGenerator facade (public API)
│ │ ├── report_base.py # Shared base class and HTML/CSS helpers
│ │ ├── overview_report.py # Overview dashboard (index.html)
│ │ ├── results_matrix_report.py # Problems × Solvers matrix
│ │ ├── raw_data_report.py # Raw data table
│ │ ├── data_index_report.py # Data export index page
│ │ ├── result_processor.py # Result aggregation
│ │ └── data_exporter.py # JSON/CSV export
│ └── utils/ # Utility modules (logging, env info, git, temp files)
├── tests/ # pytest test suite
│ ├── unit/ # Unit tests (synthetic fixtures, no submodules needed)
│ └── integration/ # Problem registry integrity checks
├── problems/ # Problem library files
│ ├── DIMACS/ # External DIMACS library (git submodule)
│ ├── SDPLIB/ # External SDPLIB library (git submodule)
│ └── NETLIB/ # External NETLIB library (git submodule)
├── database/ # SQLite database files (results.db, gitignored)
├── docs/
│ ├── pages/ # Generated HTML reports and data exports
│ ├── development/ # Design documents
│ └── guides/ # User/setup guides
├── pyproject.toml # pytest and ruff configuration
└── requirements.txt # Python dependencies (single file, all pinned)
```

---

## Core Data Models

### ProblemData (`scripts/data_loaders/problem_loader.py`)
Expand Down Expand Up @@ -249,8 +252,8 @@ Standardized dataclass returned by every solver: `solve_time`, `status`, primal/
problem_name:
display_name: string # Human-readable problem name
file_path: string # Relative path to problem file
file_type: string # "mat" | "dat-s" (supported formats: ProblemInterface.FORMAT_LOADERS)
library_name: string # "DIMACS" | "SDPLIB"
file_type: string # "mat" | "dat-s" | "mps" (supported formats: ProblemInterface.FORMAT_LOADERS)
library_name: string # "DIMACS" | "SDPLIB" | "NETLIB"
for_test_flag: boolean # Mark problem for validation testing
```

Expand Down
Loading
Loading