Skip to content

[Python] Expose dataset read limits and bound prepared-series caching - #1005

Closed
ColinLeeo wants to merge 1 commit into
apache:rc/2.5.0from
ColinLeeo:codex/dataset-cache-options
Closed

ColinLeeo wants to merge 1 commit into
apache:rc/2.5.0from
ColinLeeo:codex/dataset-cache-options

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Indexed TsFileDataFrame reads currently retain every prepared series until the runtime closes and use a fixed descriptor-cache capacity. Applications cannot set these cache limits through the public constructor.

This adds keyword-only max_prepared_series, descriptor_cache_size, max_open_files, query_workers, and query_parallel_min_rows options for use_index=True. Explicit arguments override the corresponding TSFILE_DATAFRAME_* environment variables, followed by built-in defaults. Options are validated before opening dataset resources and fixed for each runtime; subsets share their parent's configuration.

Prepared-series metadata now uses an LRU cache with a default capacity of 4096. Query leases and dependency tracking protect active handles and shared aligned-time metadata during eviction. Closing the cache waits for preparation, active queries, and native-handle cleanup. Cache capacities of zero disable retention; active queries can temporarily exceed the configured entry limit.

The English and Chinese README sections document the options, defaults, indexed-mode requirement, and per-runtime resource semantics.

Validation

  • Built the C++ library and Cython extensions from rc/2.5.0 on macOS arm64 with Python 3.12.
  • PYTHONPATH=. python -m pytest tests/test_dataset_read_options.py tests/test_dataset_index.py tests/test_tsfile_dataset.py -q: 193 passed, including 53 new cases for configuration precedence, validation, small/disabled caches, aligned reads, concurrent access, and shutdown races.
  • Black 26.3.1 checks and git diff --check passed.

@ColinLeeo ColinLeeo closed this Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant