[cwklurks]

Streaming skip(n) currently decodes every preceding Parquet source. This change uses _generate_num_examples to advance past complete sources using footer row counts, then keeps the existing scan/slice logic for the remainder.

Addresses #5380, following the maintainer’s suggested approach.

Scope and behavior

• Sources are files, or selected row groups after explicit .reshard(). No automatic resharding or pre_buffer changes.
• Preserve source coordinates, worker/rank skip allocation, checkpoint state and repeated epochs. Adjacent row-group sources reuse their footer within an iteration.
• Keep the scan path for unsupported counting, upstream transforms/shuffling, filters, explicit feature casts/projections, restored progress, and explicit page checksum verification. Unsupported Parquet counts raise NotImplementedError; other builders remain unchanged.
• Keep Fix loading parquet files with empty row groups #8742 separate. Empty-first-row-group errors on current main are preserved, including when PyArrow normalizes a noncanonical row-group selection.

Validation tradeoff: wholly skipped data pages are not read or validated. With a valid footer, corruption confined to those pages can therefore be bypassed even under the default on_bad_files="error". This was reproduced locally and is documented in skip(). Explicit page_checksum_verification=True keeps the scan path. Invalid footers and unsupported metadata fall back to ordinary iteration and its error handling.

Local validation

Against main 9e7496a4d11caf68da2846d84c9596cda4a4f984, Python 3.11 and PyArrow 25.0.1:

• Aggregate suite: 839 passed, 81 skipped, 5 deselected across test_iterable_dataset.py, packaged_modules/test_parquet.py, test_distributed.py, io/test_parquet.py, and test_streaming_download_manager.py. Optional-framework/slow tests were skipped. The selection excluded the remote GeoParquet fixture and four unguarded PyTorch interleave cases.
• Expanded CPU-focused suite using existing PyTorch 2.11.0: 174 passed, 650 deselected, including those four interleave cases, actual DataLoader workers (including more workers than files), the new Parquet tests, and existing skip/sharding/epoch regressions. No GPU or multi-node training run.
• Repository-wide Ruff lint and formatting, plus git diff --check, passed.

The tests cover source boundaries, partial and selected groups, exact/beyond exhaustion, first-row and cross-source restoration, batched checkpoint replay, repeated epochs, worker/rank partitioning, unknown metadata, required fields/schema changes, bad footers, and fallback paths. Mechanism tests assert that completely skipped sources are not decoded; these failed before the implementation.

Aggregate command (with the needed test dependencies installed):

PYTHONPATH=src HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 python -m pytest \
  tests/test_iterable_dataset.py tests/packaged_modules/test_parquet.py \
  tests/test_distributed.py tests/io/test_parquet.py \
  tests/test_streaming_download_manager.py -q -ra \
  -k 'not test_parquet_read_geoparquet and not test_interleave_dataset_with_sharding'

Local benchmark

New CPU-only measurements on three generated local Parquet files, 100,000 rows/file, 10,000 rows/group, with an integer ID and seeded random 64-byte payload. Files are about 7.56 MB each. The scan baseline disables the count callback. Dataset construction and explicit resharding are outside the timer; figures are medians of three first-result iterations. Both Python and Arrow output were checked; the timings below are for Python output.

skip(290001)	Rows materialized before first result, scan → optimized	Median latency, scan → optimized
File sources	300,000 → 100,000	8.81 → 3.47 ms
Explicitly resharded groups	300,000 → 10,000	13.96 → 1.33 ms

Skipping beyond the 300,000-row end materialized zero rows on the optimized path. These are local measurements, not remote-network or bytes-transferred benchmarks, and are separate from the earlier October 8 issue measurements. Within a retained unresharded file, preceding groups still use the scan path.

AI assistance: Codex assisted with implementation and local validation. Claude provided independent static reviews; reproduced findings were addressed, and its final review reported no blocking correctness findings.

[cwklurks]

@lhoestq, could you approve the CI workflows when you get a chance? they've been waiting since this opened, so it hasn't had a run yet. it still merges cleanly with main.

this follows your suggestion on #5380: skip complete sources using _generate_num_examples, and fall back to the current implementation when it isn't available.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cwklurks]

the two red jobs are both deps-latest integration runs, and every failure is a bucket test in tests/test_upstream_hub.py (test_push_*_to_hub_bucket, FileNotFoundError on the test bucket). this pr doesn't touch push_to_hub or buckets, and main passed the same jobs about half an hour later.

the branch was 13 commits behind, so i merged main to pick up the newer fsspec cap from #8825. @lhoestq, could you approve the run again when you get a chance?