Skip to content

fix(python): fix broken pylance benchmark scripts#7744

Open
mocobeta wants to merge 7 commits into
lance-format:mainfrom
mocobeta:fix/python-benchmark-tests
Open

fix(python): fix broken pylance benchmark scripts#7744
mocobeta wants to merge 7 commits into
lance-format:mainfrom
mocobeta:fix/python-benchmark-tests

Conversation

@mocobeta

@mocobeta mocobeta commented Jul 12, 2026

Copy link
Copy Markdown

The benchmarks in python/benchmarks seem broken or obsolete. This PR fixes various minor issues in the benchmark scripts.

  • Add necessary dependencies to benchmarks group.
  • Fix a typo in a test method signature in test_index.py.
  • Add mandatory num_sub_vectors param when creating IVF_PQ index in test_index.py.
  • Fix an assertion in test_file.py. The right side of the assertion should be the number of sampled rows, not NUM_ROWS.

Run benchmarks on main branch

❯ python -V
Python 3.13.13

❯ uv run pytest python/benchmarks/ -m "not slow" --ignore python/benchmarks/test_random_access.py

...
==================================================================================================== short test summary info =====================================================================================================
FAILED python/benchmarks/test_file.py::test_sample_integer[2_0] - assert 10 == 10000000
FAILED python/benchmarks/test_file.py::test_sample_integer[2_1] - assert 10 == 10000000
FAILED python/benchmarks/test_file.py::test_sample_integer[2_2] - assert 10 == 10000000
FAILED python/benchmarks/test_file.py::test_sample_nested_integer - assert 10 == 10000000
FAILED python/benchmarks/test_index.py::test_create_ivf_pq_torch_cpu - ModuleNotFoundError: torch.device requires 'torch' module to be installed
FAILED python/benchmarks/test_index.py::test_create_ivf_pq_torch_cpu_one_pass - ModuleNotFoundError: torch.device requires 'torch' module to be installed
FAILED python/benchmarks/test_index.py::test_optimize_index[12000-5-256] - ValueError: num_partitions and num_sub_vectors are required for IVF_PQ
FAILED python/benchmarks/test_index.py::test_optimize_index[12000-5-512] - ValueError: num_partitions and num_sub_vectors are required for IVF_PQ
ERROR python/benchmarks/test_index.py::test_transform_vectors_no_precomputed_parts
ERROR python/benchmarks/test_search.py::test_geo_rtree[no_index] - ModuleNotFoundError: No module named 'geoarrow'
ERROR python/benchmarks/test_search.py::test_geo_rtree[with_index] - ModuleNotFoundError: No module named 'geoarrow'
==================================================================== 8 failed, 280 passed, 2 skipped, 2 deselected, 6 warnings, 3 errors in 431.23s (0:07:11) ====================================================================

Run benchmarks on this branch

❯ python -V
Python 3.13.13

❯ uv run pytest python/benchmarks/ -m "not slow" --ignore python/benchmarks/test_random_access.py

...
Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
======================================================================================== 291 passed, 2 skipped, 2 deselected, 5 warnings in 731.11s (0:12:11) =========================================================================================

Summary by CodeRabbit

Summary by CodeRabbit

  • Chores

    • Added benchmark-related dependencies for PyTorch and GeoArrow Rust support.
  • Bug Fixes

    • Corrected sampling benchmark assertions to verify the actual sampled row count.
    • Updated index optimization benchmarks to pass the intended vector configuration.
    • Fixed benchmark test fixture wiring for vector transformation benchmark setup.

@github-actions github-actions Bot added A-python Python bindings A-deps Dependency updates bug Something isn't working labels Jul 12, 2026
@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: df112dee-f06a-429f-8b44-a2ed1d009885

📥 Commits

Reviewing files that changed from the base of the PR and between 65582a0 and dbdc9a6.

📒 Files selected for processing (1)
  • python/python/benchmarks/test_index.py

📝 Walkthrough

Walkthrough

Benchmark dependencies are expanded, sampling benchmarks assert sampled row counts, IVF-PQ optimization specifies sub-vector configuration, and vector transformation uses the dataset fixture.

Changes

Benchmark updates

Layer / File(s) Summary
Sampling result assertions
python/python/benchmarks/test_file.py
Sampling tests now compare result row counts with the number of sampled indices.
Index benchmark setup and environment
python/python/benchmarks/test_index.py, python/pyproject.toml
The IVF-PQ benchmark adds an explicit sub-vector parameter, the transformation benchmark uses test_dataset, and benchmark dependencies include PyTorch and GeoArrow Rust packages.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: fixing broken benchmark scripts in the Python package.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-python Python bindings bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant