Refactor indexing API around enums and argsort

  - replace legacy indexing names with a cleaner public API
  - add strict IndexKind enum and rename tiers to SUMMARY/BUCKET/PARTIAL/FULL
  - require enum kinds and build= in create_index()
  - fold expression indexing into create_index()
  - remove create_csindex() and create_expr_index()
  - rename itersorted() to iter_sorted()
  - standardize on argsort() for arrays and lazy expressions
  - remove public indices()-style indexing entry points
  - add tmpdir support for OOC full-index builds
  - fix stale sidecar-handle invalidation on Windows
  - update docstrings, tutorial, examples, and benchmarks to match
  - refresh indexing tests for the new public surface

Author: FrancescAlted

bench/indexing/blosc2-vs-duckdb-indexes.md

@@ -32,10 +32,10 @@ and 24 GB of RAM.
 
 - Script: `index_query_bench.py`
 - Index kinds:
-  - `ultralight`
-  - `light`
-  - `medium`
-  - `full`
+  - `summary`
+  - `bucket`
+  - `exact`
+  - `sorted`
 - Default geometry in these runs:
   - `chunks=1,250,000`
   - `blocks=10,000`
@@ -104,30 +104,30 @@ Command:
 python index_query_bench.py \
   --size 10M \
   --outdir /tmp/indexes-10M \
-  --kind light \
+  --kind bucket \
   --query-width 50 \
-  --in-mem \
+  --build memory \
   --dist random
 ```
 
-Observed `light` results:
+Observed `bucket` results:
 
 - build: `705.193 ms`
 - cold lookup: `6.370 ms`
 - warm lookup: `6.250 ms`
 - base array size: about `31 MB`
-- `light` index sidecars: about `27 MB`
+- `bucket` index sidecars: about `27 MB`
 - total footprint: about `58 MB`
 
 ### Interpretation
 
 For this moderately selective random workload:
 
-- Blosc2 `light` is about `2x` faster than DuckDB `zonemap`
-- Blosc2 `light` has a total footprint similar to DuckDB `zonemap`
+- Blosc2 `bucket` is about `2x` faster than DuckDB `zonemap`
+- Blosc2 `bucket` has a total footprint similar to DuckDB `zonemap`
 - DuckDB `art-index` is only slightly faster than `zonemap` here, but much larger
 
-This suggests that Blosc2 `light` is more than a simple zonemap. It behaves like an active lossy lookup
+This suggests that Blosc2 `bucket` is more than a simple zonemap. It behaves like an active lossy lookup
 structure rather than only coarse pruning metadata.
 
 
@@ -169,23 +169,23 @@ python index_query_bench.py \
 
 Observed results:
 
-- `light`
+- `bucket`
   - cold lookup: `0.841 ms`
   - warm lookup: `0.184 ms`
-- `medium`
+- `exact`
   - cold lookup: `0.564 ms`
   - warm lookup: `0.168 ms`
-- `full`
+- `sorted`
   - cold lookup: `0.554 ms`
   - warm lookup: `0.167 ms`
 
 ### Interpretation
 
 With the generic width-1 range form, Blosc2 is much faster than DuckDB:
 
-- Blosc2 `light` is already much faster than DuckDB `zonemap`, and comfortably faster than the
+- Blosc2 `bucket` is already much faster than DuckDB `zonemap`, and comfortably faster than the
   generic-range DuckDB `art-index` behavior
-- Blosc2 `medium` and `full` are in a different regime on warm hits, at about `0.17 ms`
+- Blosc2 `exact` and `sorted` are in a different regime on warm hits, at about `0.17 ms`
 - DuckDB `art-index` does not show its real point-lookup behavior in this predicate form
 - Blosc2 warm reuse changes the picture substantially for repeated lookups
 
@@ -236,17 +236,17 @@ python index_query_bench.py \
 
 Observed results:
 
-- `light`
+- `bucket`
   - build: `960.048 ms`
   - cold lookup: `2.489 ms`
   - warm lookup: `0.172 ms`
   - index sidecars: `27,497,393` bytes
-- `medium`
+- `exact`
   - build: `4745.880 ms`
   - cold lookup: `2.202 ms`
   - warm lookup: `0.147 ms`
   - index sidecars: `37,645,201` bytes
-- `full`
+- `sorted`
   - build: `9539.843 ms`
   - cold lookup: `1.753 ms`
   - warm lookup: `0.144 ms`
@@ -258,21 +258,21 @@ Once DuckDB is allowed to use the more planner-friendly single-value predicate:
 
 - `art-index` becomes very fast
 - `art-index` is clearly faster than Blosc2 on cold point lookups in this run
-- Blosc2 is clearly faster on warm repeated point lookups across `light`, `medium`, and `full`
+- Blosc2 is clearly faster on warm repeated point lookups across `bucket`, `exact`, and `sorted`
 
 However, the storage costs are very different:
 
 - DuckDB `art-index` database size: about `478.4 MB`
 - DuckDB zonemap baseline size: about `56.1 MB`
 - estimated ART overhead over baseline: about `422.3 MB`
-- Blosc2 `full` base + index footprint: about `31 MB + 29.9 MB = 60.9 MB`
+- Blosc2 `sorted` base + index footprint: about `31 MB + 29.9 MB = 60.9 MB`
 
 So for true point lookups:
 
 - DuckDB `art-index` wins on cold point-lookup latency in this measurement
-- Blosc2 `full` remains much smaller overall
-- Blosc2 `light`, `medium`, and `full` all become faster than DuckDB `art-index` on warm repeated hits
-- DuckDB `art-index` still has a very large storage premium over both Blosc2 `light` and `full`
+- Blosc2 `sorted` remains much smaller overall
+- Blosc2 `bucket`, `exact`, and `sorted` all become faster than DuckDB `art-index` on warm repeated hits
+- DuckDB `art-index` still has a very large storage premium over both Blosc2 `bucket` and `sorted`
 
 
 ## Blosc2 Light vs DuckDB Zonemap
@@ -284,16 +284,16 @@ Main observations:
 
 - storage footprint is in roughly the same ballpark
   - DuckDB zonemap DB: about `56 MB`
-  - Blosc2 base + `light`: about `58 MB`
-- Blosc2 `light` lookup speed is much better
+  - Blosc2 base + `bucket`: about `58 MB`
+- Blosc2 `bucket` lookup speed is much better
   - width `50`: about `6.25 ms` vs `13.33 ms`
   - width `1` range: about `0.18 ms` warm vs `12.61 ms` generic-range DuckDB
   - width `1` equality: about `0.17 ms` warm vs `2.94 ms` DuckDB zonemap warm
 
 Conclusion:
 
-- DuckDB zonemap is closer in spirit to Blosc2 `light` than DuckDB ART is
-- but Blosc2 `light` is a materially stronger lookup structure on these workloads
+- DuckDB zonemap is closer in spirit to Blosc2 `bucket` than DuckDB ART is
+- but Blosc2 `bucket` is a materially stronger lookup structure on these workloads
 
 
 ## Blosc2 Full vs DuckDB ART
@@ -304,20 +304,20 @@ Main observations:
 
 - point-lookup latency
   - DuckDB `art-index`: `0.613 ms` cold, `0.245 ms` warm
-  - Blosc2 `full`: `1.753 ms` cold, `0.144 ms` warm
+  - Blosc2 `sorted`: `1.753 ms` cold, `0.144 ms` warm
 - build time
   - DuckDB `art-index`: `2000.316 ms`
-  - Blosc2 `full`: `9539.843 ms`
+  - Blosc2 `sorted`: `9539.843 ms`
 - footprint
   - DuckDB `art-index` DB: about `478.4 MB`
-  - Blosc2 `full` base + index: about `60.9 MB`
+  - Blosc2 `sorted` base + index: about `60.9 MB`
 
 Conclusion:
 
-- Blosc2 `full` wins on storage efficiency
+- Blosc2 `sorted` wins on storage efficiency
 - DuckDB `art-index` wins on cold point-lookup latency
-- Warm repeated point lookups favor Blosc2 `full` more clearly
-- DuckDB `art-index` is much faster to build than Blosc2 `full`
+- Warm repeated point lookups favor Blosc2 `sorted` more clearly
+- DuckDB `art-index` is much faster to build than Blosc2 `sorted`
 - DuckDB ART is much more sensitive to predicate shape
 
 
@@ -349,8 +349,8 @@ Practical implication:
 
 ## Current Takeaways
 
-1. Blosc2 `light` is very competitive against DuckDB zonemap-like pruning.
-2. Blosc2 `light` offers much faster selective lookups than DuckDB zonemap at a similar total storage cost.
+1. Blosc2 `bucket` is very competitive against DuckDB zonemap-like pruning.
+2. Blosc2 `bucket` offers much faster selective lookups than DuckDB zonemap at a similar total storage cost.
 3. DuckDB `art-index` becomes strong only when queries are written as true equality predicates.
 4. On true point lookups, DuckDB `art-index` wins on cold latency in the current M4 Pro run, but
    Blosc2 exact indexes are markedly better on warm repeated lookups.

bench/indexing/index_query_bench.py

@@ -24,18 +24,19 @@
 
 SIZES = (1_000_000, 2_000_000, 5_000_000, 10_000_000)
 DEFAULT_REPEATS = 3
-KINDS = ("ultralight", "light", "medium", "full")
-DEFAULT_KIND = "light"
+KINDS = ("summary", "bucket", "partial", "full")
+DEFAULT_KIND = "bucket"
 DISTS = ("sorted", "block-shuffled", "permuted", "random")
 RNG_SEED = 0
 DEFAULT_OPLEVEL = 5
 FULL_QUERY_MODES = ("auto", "selective-ooc", "whole-load")
 DATASET_LAYOUT_VERSION = "payload-ramp-v1"
+BUILD_MODES = ("auto", "memory", "ooc")
 
 COLD_COLUMNS = [
     ("rows", lambda result: f"{result['size']:,}"),
     ("dist", lambda result: result["dist"]),
-    ("builder", lambda result: "mem" if result["in_mem"] else "ooc"),
+    ("builder", lambda result: "mem" if result["build"] == "memory" else "ooc"),
     ("kind", lambda result: result["kind"]),
     ("create_idx_ms", lambda result: f"{result['create_idx_ms']:.3f}"),
     ("scan_ms", lambda result: f"{result['scan_ms']:.3f}"),
@@ -50,7 +51,7 @@
 WARM_COLUMNS = [
     ("rows", lambda result: f"{result['size']:,}"),
     ("dist", lambda result: result["dist"]),
-    ("builder", lambda result: "mem" if result["in_mem"] else "ooc"),
+    ("builder", lambda result: "mem" if result["build"] == "memory" else "ooc"),
     ("kind", lambda result: result["kind"]),
     ("create_idx_ms", lambda result: f"{result['create_idx_ms']:.3f}"),
     ("scan_ms", lambda result: f"{result['scan_ms']:.3f}"),
@@ -277,7 +278,7 @@ def build_persistent_array(
     for start in range(0, size, chunk_len):
         stop = min(start + chunk_len, size)
         chunk = np.zeros(stop - start, dtype=dtype)
-        if dist == "sorted":
+        if dist == "full":
             chunk["id"] = ordered_id_slice(size, start, stop, id_dtype)
         elif dist == "block-shuffled":
             _fill_block_shuffled_ids(chunk["id"], size, start, stop, block_len, block_order)
@@ -308,14 +309,14 @@ def indexed_array_path(
     kind: str,
     optlevel: int,
     id_dtype: np.dtype,
-    in_mem: bool,
+    build: str,
     chunks: int | None,
     blocks: int | None,
     codec: blosc2.Codec | None,
     clevel: int | None,
     nthreads: int | None,
 ) -> Path:
-    mode = "mem" if in_mem else "ooc"
+    mode = "mem" if build == "memory" else "ooc"
     codec_token = "codec-auto" if codec is None else f"codec-{codec.name}"
     clevel_token = "clevel-auto" if clevel is None else f"clevel-{clevel}"
     thread_token = "threads-auto" if nthreads is None else f"threads-{nthreads}"
@@ -442,11 +443,11 @@ def _condition_expr(lo: object, hi: object, dtype: np.dtype, *, query_single_val
     return f"(id >= {lo_literal}) & (id <= {hi_literal})"
 
 
-def _valid_index_descriptor(arr: blosc2.NDArray, kind: str, optlevel: int, in_mem: bool) -> dict | None:
+def _valid_index_descriptor(arr: blosc2.NDArray, kind: str, optlevel: int, build: str) -> dict | None:
     for descriptor in arr.indexes:
         if descriptor.get("version") != blosc2_indexing.INDEX_FORMAT_VERSION:
             continue
-        expected_ooc = descriptor.get("ooc", False) if kind == "ultralight" else (not bool(in_mem))
+        expected_ooc = build != "memory"
         if (
             descriptor.get("field") == "id"
             and descriptor.get("kind") == kind
@@ -474,7 +475,7 @@ def _open_or_build_indexed_array(
     id_dtype: np.dtype,
     kind: str,
     optlevel: int,
-    in_mem: bool,
+    build: str,
     chunks: int | None,
     blocks: int | None,
     codec: blosc2.Codec | None,
@@ -484,15 +485,20 @@ def _open_or_build_indexed_array(
 ) -> tuple[blosc2.NDArray, float]:
     if path.exists():
         arr = blosc2.open(path, mode="a")
-        if _valid_index_descriptor(arr, kind, optlevel, in_mem) is not None:
+        if _valid_index_descriptor(arr, kind, optlevel, build) is not None:
             return arr, 0.0
         if arr.indexes:
             arr.drop_index(field="id")
         blosc2.remove_urlpath(path)
 
     arr = build_persistent_array(size, dist, id_dtype, path, chunks, blocks)
     build_start = time.perf_counter()
-    kwargs = {"field": "id", "kind": kind, "optlevel": optlevel, "in_mem": in_mem}
+    kwargs = {
+        "field": "id",
+        "kind": blosc2.IndexKind[kind.upper()],
+        "optlevel": optlevel,
+        "build": build,
+    }
     cparams = {}
     if codec is not None:
         cparams["codec"] = codec
@@ -515,7 +521,7 @@ def benchmark_size(
     query_single_value: bool,
     optlevel: int,
     id_dtype: np.dtype,
-    in_mem: bool,
+    build: str,
     full_query_mode: str,
     chunks: int | None,
     blocks: int | None,
@@ -549,7 +555,7 @@ def benchmark_size(
                 kind,
                 optlevel,
                 id_dtype,
-                in_mem,
+                build,
                 chunks,
                 blocks,
                 codec,
@@ -561,7 +567,7 @@ def benchmark_size(
             id_dtype,
             kind,
             optlevel,
-            in_mem,
+            build,
             chunks,
             blocks,
             codec,
@@ -588,7 +594,7 @@ def benchmark_size(
             "dist": dist,
             "kind": kind,
             "optlevel": optlevel,
-            "in_mem": in_mem,
+            "build": build,
             "query_rows": index_len,
             "build_s": build_time,
             "create_idx_ms": build_time * 1_000,
@@ -714,10 +720,10 @@ def parse_args() -> argparse.Namespace:
         help=f"Index kind to benchmark. Use 'all' to benchmark every kind. Default: {DEFAULT_KIND}.",
     )
     parser.add_argument(
-        "--in-mem",
-        action=argparse.BooleanOptionalAction,
-        default=False,
-        help="Use the in-memory index builders. Disabled by default; pass --in-mem to force them.",
+        "--build",
+        choices=BUILD_MODES,
+        default="auto",
+        help="Index builder policy: auto, memory, or ooc. Default: auto.",
     )
     parser.add_argument(
         "--full-query-mode",
@@ -787,7 +793,7 @@ def main() -> None:
                 args.repeats,
                 args.optlevel,
                 id_dtype,
-                args.in_mem,
+                args.build,
                 args.full_query_mode,
                 args.chunks,
                 args.blocks,
@@ -809,7 +815,7 @@ def main() -> None:
             args.repeats,
             args.optlevel,
             id_dtype,
-            args.in_mem,
+            args.build,
             args.full_query_mode,
             args.chunks,
             args.blocks,
@@ -831,7 +837,7 @@ def run_benchmarks(
     repeats: int,
     optlevel: int,
     id_dtype: np.dtype,
-    in_mem: bool,
+    build: str,
     full_query_mode: str,
     chunks: int | None,
     blocks: int | None,
@@ -852,7 +858,7 @@ def run_benchmarks(
     print("Structured range-query benchmark across index kinds")
     print(
         f"{geometry_label}, repeats={repeats}, dist={dist_label}, "
-        f"query_width={query_width:,}, optlevel={optlevel}, dtype={id_dtype.name}, in_mem={in_mem}, "
+        f"query_width={query_width:,}, optlevel={optlevel}, dtype={id_dtype.name}, build={build}, "
         f"query_single_value={query_single_value}, "
         f"full_query_mode={full_query_mode}, index_codec={'auto' if codec is None else codec.name}, "
         f"index_clevel={'auto' if clevel is None else clevel}, "
@@ -878,7 +884,7 @@ def cold_progress_callback(row: dict) -> None:
                 query_single_value,
                 optlevel,
                 id_dtype,
-                in_mem,
+                build,
                 full_query_mode,
                 chunks,
                 blocks,