PBench

PBench synthesizes database workloads whose aggregate CPU time, scanned data, operator mix, and temporal shape approximate a target workload profile.

This repository contains the maintained v3 engineering implementation of the method described in the PVLDB 2025 paper PBench: Workload Synthesizer with Real Statistics for Cloud Analytics Benchmarking. The paper's archival artifact is also available from ruc-datalab/PBench.

What works offline

The supported path is deterministic and does not require a database:

query metrics JSON + target workload CSV
                    |
                    v
             ILP or greedy selection
                    |
                    v
              SA interval scheduling
                    |
                    v
          pbench.plan.v1 JSON artifact

The current package provides:

• validated models with explicit CPU, scan, and duration units;
• stable query IDs derived from database plus normalized SQL;
• normalized ILP and greedy selectors;
• deterministic simulated-annealing scheduling;
• versioned plan, replay, and analysis artifacts;
• composable collect, generate, replay, and analyze pipeline stages;
• an offline CLI and unit-test suite.

Database collection and replay are external operations. The interfaces are tested with fakes, but real Databend and Prometheus behavior must be verified in the environment where an experiment runs.

Install

PBench supports Python 3.10 and newer.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
python -m pytest -q tests/unit

The default installation includes only the offline algorithm dependencies and the CBC optimizer. Optional environments are explicit:

python -m pip install -e ".[databend]"  # Databend + Prometheus adapters
python -m pip install -e ".[llm]"       # current LLM generator
python -m pip install -e ".[legacy]"    # historical experiment modules

Quick start

Generate and validate a plan with the portable example:

pbench generate \
  --metrics examples/offline/metrics.json \
  --workload examples/offline/workload.csv \
  --output results/example-plan.json \
  --selector ilp \
  --count-limit 50 \
  --time-limit 300 \
  --seed 0

pbench validate results/example-plan.json

avg_scan_bytes is interpreted as decimal GB by default. For a legacy metrics file containing raw bytes, pass --metrics-scan-unit bytes; conversion occurs exactly once at that input boundary.

Analyze measured replay output:

pbench analyze \
  --plan results/example-plan.json \
  --replay path/to/pbench.replay.v1.json \
  --output results/example-analysis.json

See artifact formats for the JSON contracts and migration notes before adapting v1 scripts.

Current, legacy, planned, and external boundaries

Status	Scope
Current	pbench/models/, modern modules in pbench/core/, pbench/artifacts.py, pbench/pipeline/, and pbench/cli.py
Legacy	historical collect/replay/baseline modules and experiment scripts retained for reproduction
Planned	additional database adapters and cross-project artifact integration
External	real Databend/Prometheus replay, LLM generation, and paper-scale experiments

The legacy scripts are not the default entry point and may require the historical optional dependencies and experiment-specific infrastructure. Machine-specific configs, generated results, notebooks, credentials, and key files are intentionally excluded from the public source tree.

Development

Read the onboarding guide before changing the project. Changes to an artifact schema must update docs/ARTIFACT_FORMATS.md and include round-trip tests.

python -m pytest -q tests/unit
python -m build
python -m compileall -q pbench tests

Citation

@article{zhou2025pbench,
  title   = {PBench: Workload Synthesizer with Real Statistics for Cloud Analytics Benchmarking},
  author  = {Yan Zhou and Chunwei Liu and Bhuvan Urgaonkar and Zhengle Wang and Magnus Mueller and Chao Zhang and Songyue Zhang and Pascal Pfeil and Dominik Horn and Zhengchun Liu and Davide Pagano and Tim Kraska and Samuel Madden and Ju Fan},
  journal = {Proceedings of the VLDB Endowment},
  volume  = {18},
  number  = {11},
  pages   = {3883--3895},
  year    = {2025},
  doi     = {10.14778/3749646.3749661}
}