perf(python): accelerate Dataset Index metadata lookups

[ColinLeeo]

Summary

This PR reduces CPython overhead in the mmap-backed Python Dataset Index and runtime metadata paths without changing the Dataset Index (.tsidx) format or introducing dataset-sized worker maps: the new kernel owns no index-sized state and only reads the existing read-only mmap.

Changes

• Added tsfile.dataset._index (IndexLookup), a Cython kernel over the existing mmap, to remove Python tuple/bytes allocation from the hot paths:
  • device/column/series lookup;
  • series descriptor, span, and locator expansion;
  • device/table/column metadata;
  • device time bounds;
  • direct string-pool decoding.
• MappedDatasetIndex now exposes scalar accessors (describe_series, find_series_span, locator_metadata, prepared_locator_metadata, device_route, table_name_id, column_name_id, device_time_bounds, series_identity) and routes find_device_id / find_column_id / find_series_id / string through the kernel. The extension is built by python/setup.py alongside the existing Cython extensions, and the now-unused pure-Python _find_child is gone.
• Hardened the kernel for damaged or hand-crafted index files: the device/column/series child ranges are bounds-checked before they are walked (the same guard describe_series and find_series_span already used), and ids are read as signed values. A corrupt .tsidx used to be able to walk past the mapping and crash the interpreter instead of raising, and negative ids raised OverflowError rather than IndexError.
• RuntimeSeriesShard and RuntimeSeriesDescriptor use slots on Python 3.10+ to cut per-object memory.
• DatasetRuntime and RuntimeSeriesReader use the fast metadata paths.
• No new configuration surface: the descriptor cache keeps its existing fixed 4096-entry budget, and the temporary hit/miss/eviction counters and their environment variables were dropped.

Results

Synthetic microbenchmarks compared with the previous pure-Python lookups:

operation	speedup
find_device_id	16.4×
find_column_id	14.9×
find_series_id	5.4×
describe_series	5.8×
_span	4.9×
_identity	2.3×
prepared locator metadata	3.2×
string decoding	5.2×

These are local operation-level results. End-to-end DDP training throughput still requires a remote A/B benchmark with the rebuilt wheel.

Validation

python/tests/test_dataset_index.py + python/tests/test_tsfile_dataset.py: 152 passed
black --check passed
git diff --check passed
all CI checks green (ubuntu/macos/windows x Python 3.9/3.11/3.14, plus MSVC)

New regression tests cover behavior that was previously untested: a corrupt index whose child range points outside its section must raise IndexError (the pre-fix build crashed on the same input), negative ids must raise IndexError, and a stored hash match with different bytes must still be rejected by the full-byte comparison.

[Copilot]

🟡 Changes recommended

Unchecked string-pool references can overflow and cause out-of-bounds mmap access on damaged indexes.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR accelerates Dataset Index metadata lookups with a Cython mmap kernel while preserving the existing index format.

Changes:

• Added optimized lookup and string-decoding paths.
• Updated runtime metadata consumers and descriptor memory usage.
• Added build integration and corruption regression tests.

File summaries

File	Description
python/tsfile/dataset/runtime.py	Uses optimized metadata accessors.
python/tsfile/dataset/index.py	Routes index access through the kernel.
python/tsfile/dataset/_index.pyx	Implements the mmap lookup kernel.
python/tests/test_dataset_index.py	Adds lookup and corruption regression tests.
python/setup.py	Builds the new Cython extension.

Review details

Suppressed comments (1)

python/tsfile/dataset/_index.pyx:532

• string_id + 1 is evaluated using the Py_ssize_t argument, so passing sys.maxsize can overflow before the bounds check on platforms where that is the maximum signed size. That can bypass the check and lead to an out-of-bounds mmap read; use a subtraction-free upper-bound comparison such as string_id >= string_count - 1.

        if string_id < 0 or string_id + 1 >= string_count:

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.