From fca747c3a03edc237b56c68aa33d4673bf1f3305 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 08:16:19 +0000 Subject: [PATCH 01/12] Add extension type research notes Research notes on why extension types must be opaque, what is wrong on develop, a proposed design, urgent changes, and an implementation plan. The agents directory holds the detailed findings catalog, design notes, and the throwaway probe that reproduced the findings. Signed-off-by: Claude Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb --- research/README.md | 12 + research/extension-types/01-why-opaque.md | 96 ++++ .../extension-types/02-current-problems.md | 114 ++++ research/extension-types/03-design.md | 170 ++++++ research/extension-types/04-urgent-changes.md | 107 ++++ research/extension-types/05-implementation.md | 116 ++++ research/extension-types/README.md | 40 ++ .../extension-types/agents/design-notes.md | 355 ++++++++++++ research/extension-types/agents/findings.md | 535 ++++++++++++++++++ research/extension-types/agents/probe.rs | 520 +++++++++++++++++ 10 files changed, 2065 insertions(+) create mode 100644 research/README.md create mode 100644 research/extension-types/01-why-opaque.md create mode 100644 research/extension-types/02-current-problems.md create mode 100644 research/extension-types/03-design.md create mode 100644 research/extension-types/04-urgent-changes.md create mode 100644 research/extension-types/05-implementation.md create mode 100644 research/extension-types/README.md create mode 100644 research/extension-types/agents/design-notes.md create mode 100644 research/extension-types/agents/findings.md create mode 100644 research/extension-types/agents/probe.rs diff --git a/research/README.md b/research/README.md new file mode 100644 index 00000000000..959c80bcef2 --- /dev/null +++ b/research/README.md @@ -0,0 +1,12 @@ +# Type system research + +This directory holds research notes about the Vortex type system. The notes are based on `develop` +at `5fc5cb8` (2026-10-17). They live on the `ct/ext-types-research` branch and are not meant to be +merged. + +| Topic | Summary for people | Detail for agents | +| --- | --- | --- | +| Extension types | [`extension-types/`](extension-types/README.md) | [`extension-types/agents/`](extension-types/agents/) | + +Each summary document links to the agent documents. The agent documents hold file and line +references, reproductions, fix sketches and file lists. diff --git a/research/extension-types/01-why-opaque.md b/research/extension-types/01-why-opaque.md new file mode 100644 index 00000000000..f417ede029b --- /dev/null +++ b/research/extension-types/01-why-opaque.md @@ -0,0 +1,96 @@ + + +# 1. Why extension types must be opaque + +## Terms + +An **extension type** gives new meaning to the values of a **storage dtype**. `Timestamp` stores an +`i64`, and the extension type says that the `i64` counts time units since the Unix epoch. `Json` +stores `Utf8`, and the extension type says that the text is a JSON document. + +A **structural operation** does not read values. Slice, take, filter, mask, validity, reading a row, +serialization and compression of the storage are structural. A **semantic operation** depends on +what the values mean. Equality, ordering, arithmetic, casts, aggregates over values, zero values and +export to other systems are semantic. + +## The argument + +A storage operation computes a result for the storage meaning. A semantic operation on an +extension type can run on storage. The result is then correct only if the storage result and the +extension result agree for every input. Agreement depends on the extension type, on the operation, +and often on the metadata: + +| Extension type | Operation | What storage computes | Agrees | +| --- | --- | --- | --- | +| `Timestamp(ms)` | `<` with `Timestamp(ms)` | `i64` order | Yes | +| `Timestamp(ms)` | `<` with `Timestamp(ns)` | Order of counts in different units | No | +| `Decimal(10, 2)` | `<` with `Decimal(10, 2)` | Order of unscaled integers | Yes | +| `Decimal(10, 2)` | `*` with `Decimal(10, 2)` | A product with scale 4 | No, the result type changes | +| `Json` | `=` | Text equality | No, key order and spaces differ | +| WKB geometry | `=` | Byte equality | Only in one direction (see below) | +| Unit vector | `+` | Element sums | No, the sum is not a unit vector | +| `Uuid` | `<` | Byte order | A design decision | + +The meaning lives in the extension type, so the core cannot know which rows of this table are +"yes". The core has two choices: + +- **Transparent:** inherit storage behavior unless the type opts out. When an author forgets to opt + out, the result is a silent wrong value. +- **Opaque:** inherit no semantic behavior unless the type opts in. When an author forgets to opt + in, the result is an "unsupported" error. + +Only the second failure is safe. It also covers types that nobody can audit: third-party types and +unknown types read from a file. Structural operations are the exception. They never read values, so +they are correct for every extension type and always stay available. + +Opaque does not mean inaccessible. `ext_storage` remains the explicit way to read the storage, like +`.0` on a public newtype field. + +## The newtype model + +A Rust newtype has the same split: + +| Rust | Vortex extension type | +| --- | --- | +| `struct Meters(f64)` is a separate type | Identity is the ID, the metadata and the storage dtype | +| `#[repr(transparent)]` costs nothing | One storage child, compressed as storage | +| `Vec` can slice, filter and index | Structural operations work on the storage child | +| `m.0` reads the field | `ext_storage` | +| `Meters::new(x)` can check invariants | Building an extension value checks the rules of the type | +| Nothing is inherited without `#[derive]` | No semantic operation is inherited without an opt-in | +| `impl From for Meters` converts on purpose | The extension type owns its casts | + +## Equality needs care + +Within one extension dtype, equal storage values are equal extension values. The reverse holds only +when each value has exactly one storage form. This document calls that **canonical storage**. +`Timestamp` and `Decimal` storage are canonical. JSON text and WKB bytes are not, because one +document or geometry has many encodings. + +So storage equality is safe for compression and for constant detection. `!=`, `DISTINCT`, `GROUP BY` +and joins are safe only when the type declares that its storage is canonical. + +## Refinement types are a special case + +A **refinement type** restricts a parent type to a subset of its values. Examples are UTF-8 text +over binary, and a unit vector over a vector. Every value of the refinement is a valid value of the +parent, so reads and predicates of the parent stay correct. Operations that produce new values +return the parent type, because the result can leave the subset. In the opaque model, a refinement +type opts in to all of the behavior of its parent and adds a value check. + +## Prior art + +- **Postgres** has both models. A base type made with `CREATE TYPE` gets ordering only from an + operator class. A `DOMAIN` is a refinement: it inherits the operators of its base type and adds + `CHECK` constraints. The `json` type stores text and has no `=` operator. The `jsonb` type stores + a canonical form, and it defines `=`, ordering and hashing on that form. +- **Parquet** first wrote min and max statistics in the physical sort order. That order was wrong + for some logical types, and the format later added a declared column order. +- **Haskell** newtypes derive nothing by default. `deriving newtype` and `DerivingVia` opt in. + Type roles decide when a value can be converted between a newtype and its representation. +- The draft [RFC 0062](https://github.com/vortex-data/rfcs/blob/ngates/types/rfcs/0062-value-types-and-representations.md) + states the same rule: "An extension does not inherit storage comparison, hashing, arithmetic, or + parsing by default." diff --git a/research/extension-types/02-current-problems.md b/research/extension-types/02-current-problems.md new file mode 100644 index 00000000000..dd531302702 --- /dev/null +++ b/research/extension-types/02-current-problems.md @@ -0,0 +1,114 @@ + + +# 2. Current problems on `develop` + +All findings are for `develop` at `5fc5cb8`. Each ID refers to an entry in +[`agents/findings.md`](agents/findings.md), which has file and line references, reproductions and +fix sketches. "Confirmed" means that a throwaway test ([`agents/probe.rs`](agents/probe.rs)) +reproduced the behavior. + +## Results depend on the encoding and the read path (EXT-P1) + +The same cast or comparison returns a correct value, a wrong value, or an error. The outcome depends +on the encoding of the input, and on whether one row or the whole array is read. Confirmed: + +| Expression | Whole array | One row (`execute_scalar`) | +| --- | --- | --- | +| `cast(Timestamp(ms) to Timestamp(ns))` | Error | Wrong: one second becomes one microsecond | +| Same cast, constant input | Wrong: one second becomes one microsecond | Wrong | +| `cast(i64 to Timestamp(ms))` | Error | Wraps the raw `i64` | +| `Timestamp(ms) < i64` | Compares raw storage | Error | + +Cast and compare each have two implementations. Constant inputs use the scalar implementation, and +other encodings use the array kernels. The two implementations apply different rules to extension +types. Reading one row wraps each input row in a one-row constant, so it always takes the scalar +implementation. + +## Wrong results + +- **Casts reinterpret storage (EXT-C1, EXT-C2).** A scalar cast into an extension type casts to the + storage dtype and wraps the result. The source type plays no part. An array cast out of an + extension type returns the storage cast, so `Timestamp(ms)` casts to `i64`, `f64` and `u8`. Issue + [#7504](https://github.com/vortex-data/vortex/issues/7504) decided to remove the second behavior, + but it is still on `develop`. +- **Comparisons against raw storage (EXT-Q1).** The compare type check is skipped when either side + is an extension. A lone extension side then compares its storage, so `ts < 1500` compares raw + `i64` values. `between` checks types strictly, so `ts BETWEEN 0 AND 1500` fails while + `ts >= 0 AND ts <= 1500` runs. +- **Storage equality and order for every type (EXT-Q2).** Any two values of one extension dtype + compare by storage. This happens in the compare kernel, the nested comparator and `ExtScalar`. A + test type with no ordering still supports `<`. The built-in kernel runs first, and the registry + cannot override it, so an extension type has no way to replace it. +- **Engine boundaries (EXT-B1 to EXT-B5).** DataFusion converts non-temporal extension scalars to + storage scalars. Arrow export to a storage-like target type exports storage through the cast + fallback. Python `as_py()` returns the storage value. DuckDB decides aggregate pushdown from the + storage type. JNI write summaries report storage values. + +## Panics and invalid values + +- **Arrays are never value-checked (EXT-V1).** `ExtensionArray::new` and `ExtensionBuilder` accept + values that the type rejects. Reading such a row panics. Confirmed with a type that rejects zero. +- **Nullability changes can panic (EXT-N1).** `ExtDType::with_nullability` validates the type again + and panics if validation fails. A type that accepts only non-nullable storage panics in `take` + with nullable indices. Confirmed. +- **Zero values skip validation (EXT-V3).** `Scalar::zero_value` uses the storage zero. For a type + that rejects zero, debug builds panic and release builds return an invalid scalar. Confirmed. +- **Built-in temporal types (EXT-V4, EXT-V5).** `Date` and `Time` panic on non-primitive storage, so + a malformed file panics instead of returning an error. `Timestamp` accepts the `Days` unit and + unknown time zones, and then rejects every value. +- **Array validation is a no-op (EXT-V2).** `Extension::validate` compares the dtype with itself. + It accepts `Utf8` storage under a `Timestamp` dtype. Confirmed. +- **Lead: `DateTimeParts` compare (EXT-Q3).** The kernel calls `as_extension()` on its constant + operand before it checks the type, and it does not compare time units. The compare adaptor does + not check operand types. A `DateTimeParts` column compared with an `i64` constant can panic. This + needs a direct test. + +## Disagreements + +- **Ordering (EXT-A1).** `min_max` uses storage order for every extension type. `is_sorted` returns + `false` for every extension type, so a sorted timestamp column reports "not sorted". `vortex-row` + rejects extension types. `sum` rejects them. Three open PRs add three different opt-in mechanisms: + [#10246](https://github.com/vortex-data/vortex/pull/10246), + [#10566](https://github.com/vortex-data/vortex/pull/10566) and + [#10408](https://github.com/vortex-data/vortex/pull/10408). +- **Zone maps (EXT-A2).** When a statistic rejects an extension dtype, the zone map uses the storage + dtype. JSON and WKB columns store exact, unbounded bounds, so WKB zone maps hold whole geometry + blobs. +- **Two shapes for one constant (EXT-S1).** A constant extension column is either + `Constant(extension scalar)` or `Extension(Constant(storage))`. Code must handle both. +- **Encodings with an extension dtype (EXT-S2).** `DateTimeParts` carries the `Timestamp` dtype + itself, so an extension-typed array is not always the wrapper. +- **Unknown types (EXT-R3).** A reader without the type gets a placeholder that compares and orders + values by storage. + +## Root causes + +1. **One dispatch key.** Every extension type shares the `vortex.ext` encoding, so a kernel written + for one extension type runs for all of them. +2. **No declaration of inherited behavior.** `ExtVTable` cannot say which semantic operations a type + shares with its storage, so each subsystem guesses. +3. **Two implementations.** Cast and compare each have a scalar and an array implementation with + different extension rules. +4. **Trusted arrays.** Values are checked only when a scalar is built, and that check panics. +5. **Nullability in the storage dtype.** Any nullability change validates the type again. +6. **Manual authoring.** The core has no typed array view, no explicit wrap expression, and no + export hooks for DuckDB, DataFusion or Python. Extension crates fill the gaps by hand. + `vortex-spatial` has eight typed array wrappers, and its Arrow code is 4.7 times the size of its + type definitions. + +
+Smaller problems + +- A same-type extension cast hides the storage error behind "no `CastKernel`" (EXT-C3). +- Eight extension types never check values, because `unpack_native` returns the raw storage + (EXT-V6). +- `Debug for ExtDTypeRef` prints the metadata before the struct, then `metadata: Ok(())` (EXT-R1). +- `DTypeSession::register` silently replaces a type with the same ID, and Arrow plugins register + twice if `initialize` runs twice (EXT-R2). +- The developer guide page for extension dtypes is a stub, and the vtables page shows an old API + (EXT-R4). + +
diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md new file mode 100644 index 00000000000..83b859318b7 --- /dev/null +++ b/research/extension-types/03-design.md @@ -0,0 +1,170 @@ + + +# 3. Design + +## Goal + +Extension types behave like Rust newtypes. Structural operations always work. A semantic operation +works only where the extension type opts in or provides its own rule. No code path applies storage +semantics to an extension value without that opt-in. + +## Rules + +1. A semantic operation on an extension type is rejected unless the type opts in. +2. Structural operations are always allowed. +3. One place decides semantics: the binder, which runs when an expression or a scalar function + array is built. +4. Below the binder, everything is storage. Kernels, encodings, statistics, pruning and compression + see storage dtypes only. +5. The extension type owns every conversion that changes meaning. There are no implicit casts. +6. An unknown extension type supports structural operations and `ext_storage` only. + +## Models considered + +| Model | Summary | Verdict | +| --- | --- | --- | +| Transparent | Every operation runs on storage | Rejected: wrong results for most types | +| Refinement for every type | Inherit reads and predicates, return the parent type for new values | Correct only for true subtypes, kept as an opt-in | +| Opaque, flags in each subsystem | Each consumer reads a declaration and decides | Every consumer must check, and new consumers drift | +| Opaque, kernels per extension ID | Look up `(function, extension ID)` before the generic kernel | Computes results, but statistics and pruning learn nothing | +| Type-owned encodings | Each extension type gets its own array encoding | Mixes logical type and encoding, and generic encodings still carry the type | +| Full dtype plugin | The type defines its own arrays, scalars and compute | The most power and the most work, and storage becomes optional | +| **Opaque, bind-time rewrite** | The binder rewrites semantic operations into storage expressions | **Proposed** | + +The deciding property is the number of places that must understand extension semantics. With +flags, every consumer must read the flag correctly. Today, three groups of consumers give three +different answers to "is this type ordered?" (EXT-A1, EXT-B4): + +- `compare` and `min_max` treat every extension type as ordered. +- `is_sorted` and `vortex-row` treat no extension type as ordered. +- DuckDB pushdown treats every extension type except `Uuid` as ordered. + +With bind-time rewrites, only the binder needs to understand extension semantics. + +## Proposal + +### Bind-time rewrite + +When an operation is bound and an operand has an extension type, the binder asks the extension type +how to answer. The answer is a rewrite into an expression over storage, or a rejection: + +```text +ts_ms < lit(Timestamp(ms) 1500) => ext_storage(ts_ms) < 1500i64 +min(ts_ms) => ext_wrap(min(ext_storage(ts_ms)), Timestamp(ms)) +cast(ts_ms, Timestamp(ns)) => ext_wrap(ext_storage(ts_ms) * 1_000_000, Timestamp(ns)) +json_a = json_b => rejected, because Json derives no equality +``` + +The scalar path and the array path then evaluate the same expression, so the results of EXT-P1 +cannot differ. Pushdown, pruning and compression keep working, because the rewritten expression is +an ordinary storage expression. + +### Pairwise derives + +A derive says that a semantic operation on two extension values is the same operation on their +storage values. It depends on the metadata of both operands: + +```rust +/// Storage behavior that values of `lhs` and `rhs` share, like `#[derive]` on a newtype. +/// +/// The default shares nothing, so every semantic operation on this type is rejected. +fn derives(lhs: &ExtDType, rhs: &ExtDType) -> Derives { + Derives::NONE +} +``` + +`Derives` has two flags. `EQ` means that storage equality is value equality, which also makes the +storage hash valid. `ORD` means that storage order is value order. + +| Type | Pair | Derives | +| --- | --- | --- | +| `Timestamp` | Same unit, and both with a time zone or both without | `EQ`, `ORD` | +| `Date`, `Time` | Same unit | `EQ`, `ORD` | +| `Decimal` | Same scale and same storage width | `EQ`, `ORD` | +| `Uuid` | Any | `EQ`, plus `ORD` if byte order is the chosen order | +| `Vector`, `FixedShapeTensor` | Same shape | `EQ` | +| `Json`, WKB, native geometry | Any | None | +| Unknown type | Any | None | + +A derive is a preset rewrite. A type that needs something else writes its own rewrite. An example is +a comparison of two decimals with different scales, which needs a rescale first. + +### Casts + +The extension type owns its casts. The binder tries the cast hook of the source type, then the cast +hook of the target type, then rejects the cast. The hooks return expressions, not closures, so the +result can be pushed down and evaluated in the same way on both paths. This is the design in issue +[#8570](https://github.com/vortex-data/vortex/issues/8570), with expressions in place of closures. + +Two explicit expressions cover everything else: + +- `ext_storage(x)` reads the storage. It exists today. +- `ext_wrap(x, dtype)` builds an extension value from storage and checks it. It is new. + +### One normal form + +Only the wrapper carries an extension dtype. Generic encodings (`Constant`, `Dict`, `Chunked`) and +type-specific encodings (`DateTimeParts`) live under the wrapper and see the storage dtype. This +removes the two shapes of EXT-S1 and EXT-S2. The compressor chooses a type-specific encoding from +the extension metadata. + +### Statistics + +Zone maps and file statistics store storage statistics for extension columns. The rewrites in the +binder produce predicates on `ext_storage(col)`, and pruning uses the storage statistics for them. +`vortex-spatial` already prunes in this way, by hand. + +### Validation + +An extension type can define `validate_array`. The core calls it where data enters: `try_new`, +Arrow import, `ext_wrap`, and optionally file reads. Kernels that keep the invariants use an +unchecked constructor. Reads return errors and never panic. Validated arrays also remove the +per-row check that each read does today. + +### Nullability + +The short-term rule: `validate_dtype` sees the storage dtype without nullability, so a nullability +change never validates the type again. The long-term option: store nullability on the extension +dtype, like every other `DType` variant, and keep the storage dtype non-nullable. + +### Authoring and boundaries + +The core provides a typed array view, a typed accessor for native values, metadata helpers, and a +conformance test harness for `ExtVTable` implementations. DuckDB, DataFusion and Python get export +plugins like the Arrow registry. Then a new extension type does not need code in each engine crate. + +## Costs and risks + +- Binding must be mandatory. The array API builds scalar function arrays directly, so the rewrite + must happen when such an array is built, not in an optional optimizer pass. +- `DateTimeParts` changes from a `Timestamp` encoding to an `i64` encoding. That is a file format + change for the encoding, and it needs an edition. +- Engines must send extension literals for extension columns. Today the lone-extension compare rule + can hide literals that have the storage type. +- After binding, plans show storage operations, so plans are harder to read. + +## Open questions + +- Open question: where does binding happen for the array API? One option is a rewrite when the + array is built, with hooks on `ExtVTable`, so no session is necessary. The other option is a + mandatory rewrite pass that also covers the row path. +- Open question: are rewrites owned only by the extension type? Or can a function owner register a + rewrite for a type that it does not own? +- Open question: are operations on values with different metadata rejected, or coerced to a common + type? Rust and the built-in `Decimal` reject them today. SQL coerces them. +- Open question: are `EQ` and `ORD` enough? A refinement over a storage dtype can need a third flag + that allows comparisons with plain storage values. +- Open question: builders pad non-nullable children under null parents with a zero. Does the + padding have to be a valid extension value? +- Open question: existing files store min and max for extension columns as extension scalars. + Readers can convert them to storage values, but the switch needs a compatibility decision. +- Open question: is `validate_array` a no-op by default, or a check of each value through + `validate_scalar_value`? +- Open question: is the nullability contract enough, or is the structural change worth its cost? +- Open question: does `Uuid` order by bytes, or have no order? +- Open question: `Scalar` implements `PartialEq` and `Hash` for use in Rust code. Can these stay + structural (same dtype and same storage value) if they are documented as identity, not SQL + equality? `PartialOrd` must follow the derives. diff --git a/research/extension-types/04-urgent-changes.md b/research/extension-types/04-urgent-changes.md new file mode 100644 index 00000000000..68549625ef4 --- /dev/null +++ b/research/extension-types/04-urgent-changes.md @@ -0,0 +1,107 @@ + + +# 4. Urgent changes + +These changes fix wrong results and panics. They do not depend on the design in part 3, and each one +fits in a small PR with a regression test. Finding IDs refer to +[`agents/findings.md`](agents/findings.md). + +| ID | Change | Fixes | Visible effect | +| --- | --- | --- | --- | +| U1 | Give cast one rule on both paths | EXT-P1, EXT-C1, EXT-C2, EXT-C3 | A cast that changes meaning returns an error | +| U2 | Give compare one type rule | EXT-P1, EXT-Q1, EXT-Q3 | `ts < 1500` fails when it is built | +| U3 | Return errors from reads | EXT-V1 | No panic when a row is read | +| U4 | Make nullability changes infallible | EXT-N1 | No panic in `take`, `mask` or `min_max` | +| U5 | Fix the built-in temporal validation | EXT-V4, EXT-V5 | A malformed dtype returns an error | +| U6 | Make `Extension::validate` check the storage | EXT-V2 | An array with the wrong storage is rejected | +| U7 | Stop invalid zero values | EXT-V3 | No invalid scalars | +| U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output | + +## Test requirement + +U1 and U2 need a shared test helper. It evaluates each case three ways: on the whole array, on one +row with `execute_scalar`, and with constant inputs. The three results must be equal, or all three +must be errors. The table in EXT-P1 lists the first cases. + +## U1. One cast rule + +- Remove the wrap in `Scalar::cast` for extension targets (`vortex-array/src/scalar/cast.rs:56-61`). + Keep only casts between one extension dtype and the same dtype with another nullability. +- Remove the storage fallback in the extension cast kernel + (`vortex-array/src/arrays/extension/compute/cast.rs:18-22`). +- Return the storage error instead of "no `CastKernel`" (EXT-C3). +- Change the Arrow exporters that call `array.cast(target)` (EXT-B2). An exporter that exports + storage on purpose calls `ext_storage`. Every other exporter returns an error. + +Visible changes: `cast(Timestamp, i64)` returns an error, and `ext_storage` replaces it. Python +`vx.scalar(5, dtype=timestamp)` returns an error. The dedicated temporal constructors keep working. + +## U2. One compare type rule + +- `Binary::return_dtype` requires equal dtypes (ignoring nullability) for every comparison, with no + exception for extension types (`vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`). +- Remove the lone-extension case in `compare_arrays` + (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`). +- `CompareExecuteAdaptor` checks that the operand dtypes match before it calls an encoding kernel + (`compare/mod.rs:73-121`). This also closes the `DateTimeParts` lead (EXT-Q3). + +Coupling: an engine converter that sends a storage-typed literal for an extension column must send +an extension literal. DuckDB and DataFusion already convert temporal literals to extension scalars. +A test for each engine confirms the other cases. + +## U3. Errors from reads + +- Row reads of an extension array and `ExtensionConstantRule` build the extension scalar with + `try_new` and return its error (`vortex-array/src/arrays/extension/vtable/operations.rs:29`, + `vortex-array/src/arrays/extension/compute/rules.rs:39`). +- `Scalar::extension_ref` gets a fallible variant, and read paths use it. + +## U4. Nullability changes cannot fail + +- `ExtDType::try_with_vtable` validates the dtype with both nullabilities. After that, + `with_nullability` cannot fail and does not validate again. +- The documentation of `ExtVTable::validate_dtype` states that its result must not depend on + nullability. + +A fallible `with_nullability` is the alternative. It moves the error into every caller, so a check +when the dtype is built is the better place. + +## U5. Built-in temporal validation + +- `Date` and `Time` match on `DType::Primitive` and return an error for any other storage + (`vortex-array/src/extension/datetime/date.rs:104`, `time.rs:105`). +- `Timestamp::validate_dtype` rejects the `Days` unit and unknown time zones + (`vortex-array/src/extension/datetime/timestamp.rs:198-204`). + +## U6. `Extension::validate` + +- Compare the dtype of the storage child with `ext_dtype.storage_dtype()` + (`vortex-array/src/arrays/extension/vtable/mod.rs:96-123`). + +## U7. Zero values + +- `Scalar::zero_value` validates extension values and does not use `new_unchecked` + (`vortex-array/src/scalar/scalar_impl.rs:165-169`). An extension type that rejects the storage + zero has no zero value. + +## U8. Debug output + +- Format the metadata through a `Debug` adapter, like the existing `MetadataDisplay` + (`vortex-array/src/dtype/extension/erased.rs:181-191`). + +## Coordination with open work + +Three open PRs add mechanisms for extension semantics: + +- [#10246](https://github.com/vortex-data/vortex/pull/10246) adds a per-type aggregate kernel + registry. +- [#10566](https://github.com/vortex-data/vortex/pull/10566) adds an aggregate adapter with a + composite ID. +- [#10408](https://github.com/vortex-data/vortex/pull/10408) adds `between` pushdown for + `AnyTemporal`. + +Each one is a reasonable local fix, but each one fixes a different mechanism in place. Settle the +derive model in part 3 first, or merge each PR with a follow-up that moves it onto derives. diff --git a/research/extension-types/05-implementation.md b/research/extension-types/05-implementation.md new file mode 100644 index 00000000000..f9bbcc0ec6e --- /dev/null +++ b/research/extension-types/05-implementation.md @@ -0,0 +1,116 @@ + + +# 5. Implementation + +## Phases + +| Phase | Content | Depends on | +| --- | --- | --- | +| 0 | The urgent changes in part 4, and the path-consistency test helper | Nothing | +| 1 | The `ExtVTable` contract, a conformance harness, and three reference test types | Nothing | +| 2 | The binder and derives: `Derives`, `ext_wrap`, and rewrites for compare, min, max and sort | 0, 1 | +| 3 | Casts through type-owned hooks, and an audit of engine literals | 2 | +| 4 | The normal form, and storage statistics for extension columns | 2 | +| 5 | Array validation, and nullability on the extension dtype | 0 | +| 6 | Authoring helpers, engine export plugins, and documentation | 2, 3 | + +```mermaid +flowchart LR + P0["0. Urgent changes"] --> P2["2. Binder and derives"] + P1["1. Contract and harness"] --> P2 + P2 --> P3["3. Casts"] + P2 --> P4["4. Normal form and statistics"] + P0 --> P5["5. Validation and nullability"] + P2 --> P6["6. Authoring and boundaries"] + P3 --> P6 +``` + +Phases 0, 1 and 5 can run in parallel. Phase 2 is the critical step, because phases 3, 4 and 6 +build on the binder. + +## Phase 0: urgent changes + +- Write the path-consistency helper first, because U1 and U2 use it. +- Land U1 to U8 as separate PRs. +- Some existing code depends on a leak, for example a comparison of an extension column with a + storage literal. The full test suite finds these callers. Fix each one with an explicit + `ext_storage`. + +## Phase 1: contract and harness + +- Document the contract on `ExtVTable`: structural and semantic operations, the derive rules, + validation, and nullability independence. +- Add a conformance harness for `ExtVTable` implementations. It checks the metadata round trip, that + `validate_dtype` ignores nullability, that `ORD` agrees with `EQ`, cast round trips, and path + consistency. +- Add three reference test types: + - An opaque type that derives nothing. Every semantic operation on it must fail at bind time. + - A decimal-like type over `i64` that derives `EQ` and `ORD` for the same scale. A differential + test compares its results with the built-in `Decimal`. + - A refinement type that derives everything and adds a value check. + +## Phase 2: binder and derives + +- Add `Derives`, `ExtVTable::derives`, and the erased forwarder on `ExtDTypeRef`. +- Add the `ext_wrap` expression. +- Rewrite compare, `between`, min, max, `is_sorted` and sort keys at bind time. +- Remove the code that applies storage semantics. That code is the extension compare kernel, the + extension branch of the nested comparator, the extension path of `min_max`, and storage order in + `ExtScalar`. +- Give each type in the repository its derives (the table in part 3). +- Move the open PRs onto derives. #10408 becomes a derive check. #10246 and #10566 become aggregate + rewrites. + +## Phase 3: casts + +- Implement the hooks from #8570 as rewrites. +- Add `Timestamp` unit casts with overflow checks, and unit casts for `Date` and `Time`. +- Audit the DuckDB and DataFusion converters: every literal for an extension column must be an + extension scalar. + +## Phase 4: normal form and statistics + +- Add rewrite rules that move the wrapper above `Constant`, `Dict` and `Chunked`. +- Store storage statistics for extension columns. Readers convert existing extension-typed + statistics to storage values. +- Move `DateTimeParts` under the wrapper as an `i64` encoding. An edition gates the new form, and + readers keep the old form. + +## Phase 5: validation and nullability + +- Add `validate_array`, a checked `try_new`, and an unchecked constructor for kernels. +- Trust validated arrays when a row is read, and remove the per-row check. +- Optionally move nullability onto the extension dtype. + +## Phase 6: authoring and boundaries + +- Add a typed array view, a native value accessor, and metadata helpers. +- Add export plugin traits for DuckDB, DataFusion and Python, modeled on the Arrow registry. Move + the hard-coded type lists into plugins. +- Write the developer guide page for extension types. + +## Test strategy + +- **Path consistency.** Every semantic operation gives the same answer on the whole array, on one + row, and with constant inputs. +- **Opaque rejection.** The opaque reference type fails at bind time for every semantic operation. +- **Differential parity.** The decimal-like reference type matches the built-in `Decimal` for every + operation that both support, including errors. +- **Engines.** DuckDB and DataFusion pushdown tests use extension literals for temporal, `Uuid` and + `Json` columns. + +## Compatibility + +- Phases 0 to 3 change behavior, not the file format. The release notes must list the behavior + changes from part 4. +- Phase 4 changes the `DateTimeParts` format and the statistics dtype of extension columns. Both + need editions and read compatibility. +- Phase 5 adds an optional check when a file is read. It does not change the format. + +## Relation to the Decimal and type roadmap + +The Decimal work in [`../decimal/`](../decimal/README.md) depends on phases 2 and 3. An opaque +Decimal wrapper uses the same derives, casts and normal form. diff --git a/research/extension-types/README.md b/research/extension-types/README.md new file mode 100644 index 00000000000..cead7f700fa --- /dev/null +++ b/research/extension-types/README.md @@ -0,0 +1,40 @@ +# Extension types + +Extension types are meant to work like Rust newtypes: an opaque wrapper over a storage type, with +behavior added on purpose. On `develop`, extension types inherit storage behavior in many places. +That causes wrong results, panics, and code paths that disagree with each other. + +| Part | Content | +| --- | --- | +| [1. Why extension types must be opaque](01-why-opaque.md) | The argument for an opaque default | +| [2. Current problems](02-current-problems.md) | What is wrong on `develop` | +| [3. Design](03-design.md) | The proposed model, the alternatives, and the open questions | +| [4. Urgent changes](04-urgent-changes.md) | Fixes to make now, before the larger design | +| [5. Implementation](05-implementation.md) | Phases, order, tests and compatibility | + +## Summary + +- An extension type gives new meaning to storage values. A storage operation is correct for an + extension type only when it agrees with that meaning, and only the extension type knows when it + agrees. So the safe default is to inherit no semantic behavior. +- On `develop`, the same cast or comparison can return a correct value, a wrong value, or an error. + The outcome depends on the encoding of the input, and on whether one row or the whole array is + read. +- The proposal: extension types are opaque by default. Semantics are decided once, when an operation + is bound, by a rewrite into storage operations. Kernels, encodings, statistics and compression see + only storage. +- Some fixes are urgent, because they cause wrong results or panics. They cover implicit casts + through storage, comparisons against raw storage, and panics on reads and on nullability changes. + +## Agent documents + +- [`agents/findings.md`](agents/findings.md): every finding, with file and line references, + reproductions, observed and expected behavior, and fix sketches. +- [`agents/design-notes.md`](agents/design-notes.md): API sketches, invariants, rewrite rules, the + derive table for each type, and file lists for each phase. +- [`agents/probe.rs`](agents/probe.rs): the throwaway test that reproduced the confirmed findings. + +## Evidence + +The findings come from reading `develop` at `5fc5cb8`, and from a throwaway test against +`vortex-array` in a debug build. Findings marked "lead" in the agent catalog are not confirmed. diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md new file mode 100644 index 00000000000..06079d0aa1a --- /dev/null +++ b/research/extension-types/agents/design-notes.md @@ -0,0 +1,355 @@ + + +# Extension type design notes (agent reference) + +This file is the detailed design behind `../03-design.md` and `../05-implementation.md`. It is a +proposal, not an accepted design. "Open" marks a decision that is not made. Finding IDs (`EXT-*`) +refer to `findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. + +## 1. Invariants + +The design holds these invariants. Every change in the plan must keep them. + +- **I1.** A semantic operation with an extension operand is rejected, unless the extension type + declares a derive for it or provides a rewrite for it. +- **I2.** A structural operation on an extension value is always allowed and keeps the extension + dtype. +- **I3.** The binder is the only code that decides extension semantics. It runs in + `Expression::bind` (`vortex-array/src/expr/bound_expression.rs:304`) and when a scalar function + array is built (`vortex-array/src/arrays/scalar_fn/array.rs:79`). +- **I4.** After binding, no semantic operation has an extension operand. Kernels, encodings, + statistics, pruning and compression see storage dtypes only. +- **I5.** The only ways to move between an extension dtype and another dtype are `ext_storage`, + `ext_wrap`, and cast hooks that the extension type owns. +- **I6.** The row path and the array path evaluate the same bound expression, so they return the + same result (EXT-P1). +- **I7.** Reading a value never panics. A value that the type rejects produces an error. +- **I8.** A nullability change never fails and never validates the type again. +- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and no rewrites. + +## 2. Operation classes + +Structural operations do not read values. Semantic operations depend on what values mean. + +| Operation | Class | Note | +| --- | --- | --- | +| slice, take, filter, mask | Structural | Run on storage, keep the wrapper | +| `is_null`, `is_not_null`, validity, `null_count`, `count` | Structural | | +| `fill_null` with a value of the same dtype | Structural | Selects between existing values | +| `zip`, `case_when` with branches of the same dtype | Structural | Selects between existing values | +| `first`, `last` | Structural | Selects an existing value | +| `uncompressed_size_in_bytes` | Structural | Size of the storage | +| `pack`, `merge`, `select`, `get_item` on a struct that holds extension fields | Structural | The extension field stays opaque | +| `scalar_at`, serialization, compression of storage | Structural | | +| `ext_storage` | Structural | Explicit unwrap | +| `=`, `!=` | Semantic | Needs `EQ` or a rewrite | +| `<`, `<=`, `>`, `>=`, `between` | Semantic | Needs `ORD` or a rewrite | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | Semantic | Needs `ORD` or a rewrite | +| `is_constant`, `all_non_distinct`, `list_contains`, distinct, group by, join keys | Semantic | Needs `EQ` or a rewrite | +| Arithmetic, `sum`, `sum_v2`, `mean` | Semantic | Needs a rewrite, because the result type usually changes | +| `cast` other than a nullability change | Semantic | Needs a cast hook | +| `is_nan`, `nan_count`, `like`, `byte_length` | Semantic | Needs a rewrite | +| Zero and default values | Semantic | Needs a `zero_value` hook | +| Export to Arrow, DuckDB, DataFusion, Python, JNI | Semantic | Needs an export plugin | + +## 3. Derives + +### Definition + +```rust +/// Storage behavior that two values of one extension type share. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Derives { + /// Storage equality is value equality, in both directions. The storage hash is then a valid + /// hash for value equality. + pub eq: bool, + /// Storage order is value order. `ord` requires `eq`. + pub ord: bool, +} + +pub trait ExtVTable { + // The existing methods stay. + + /// Returns the storage behavior that values of `lhs` and `rhs` share. + /// + /// Both dtypes have this extension type. The default shares nothing. + fn derives(lhs: &ExtDType, rhs: &ExtDType) -> Derives { + Derives::default() + } +} +``` + +`ExtDTypeRef` gets an erased forwarder: `fn derives(&self, other: &ExtDTypeRef) -> Derives`. It +returns `Derives::default()` when the two extension IDs differ. + +### Rules + +- **D1.** A derive applies only when the two storage dtypes are equal, ignoring nullability. For + other pairs, the type must provide a rewrite, for example a cast of one storage to the other. +- **D2.** `derives(a, b)` must equal `derives(b, a)`. The conformance harness checks this. +- **D3.** `eq` requires canonical storage: each value has exactly one storage form under both + metadata values. +- **D4.** `ord` requires `eq`, and the storage order must be a total order on values. +- **D5.** A type with `eq` must not let two different metadata values give different meanings to + the same storage value. If they do, `derives` returns no `eq` for that pair. + +### Table for the types in the repository + +| Type | Pair | `eq` | `ord` | Reason | +| --- | --- | --- | --- | --- | +| `Timestamp` | Same unit, both with a time zone | Yes | Yes | Both store UTC instants | +| `Timestamp` | Same unit, both without a time zone | Yes | Yes | Both store wall-clock counts | +| `Timestamp` | One with and one without a time zone | No | No | Different meanings | +| `Timestamp` | Different units | No | No | Needs a cast first | +| `Date`, `Time` | Same unit | Yes | Yes | | +| `Uuid` | Any version metadata | Yes | Open | Byte order is the Postgres order | +| `Json` | Any | No | No | Text is not canonical | +| `Vector` | Same dimension and element type | Yes | No | No natural order | +| `FixedShapeTensor` | Equal metadata | Yes | No | Shape and permutation change the meaning | +| WKB and native geometry types | Any | No | No | Two encodings can hold one geometry | +| `ForeignExtDType` | Any | No | No | I9 | + +The float element types of `Vector` give storage equality IEEE semantics (`NaN != NaN`). That is +correct for a derive, because the derive inherits the storage semantics on purpose. + +## 4. Binder + +### Resolution order + +For a scalar function call that has at least one extension operand: + +1. If the function is structural (section 2), bind it without a change. +2. The function can be a comparison, `between`, or an order-based aggregate. If the extension type + declares the matching derive for the operand pair, rewrite the call to the same operation on + `ext_storage` of each operand. +3. If the extension type provides a rewrite (`ExtVTable::rewrite`), use it. +4. If the function itself handles the extension type (for example `replace_time_zone` for + `Timestamp`), bind it without a change. The function owner then owns the semantics. +5. Otherwise, return an error that names the function, the extension ID, and the missing derive. + +For a cast, the binder tries these steps in order: + +1. A nullability change of the same dtype. +2. `cast_to` of the source type. +3. `cast_from` of the target type. +4. An error. + +### Proposed hooks + +```rust +pub trait ExtVTable { + /// Rewrites a cast from this type into an expression, or returns `None`. + fn cast_to( + ext: &ExtDType, + input: Expression, + target: &DType, + ) -> VortexResult> { + Ok(None) + } + + /// Rewrites a cast into this type into an expression, or returns `None`. + fn cast_from( + ext: &ExtDType, + input: Expression, + source: &DType, + ) -> VortexResult> { + Ok(None) + } + + /// Rewrites a scalar function call that has an operand of this type, or returns `None`. + fn rewrite(ext: &ExtDType, call: &ScalarFnCall) -> VortexResult> { + Ok(None) + } +} +``` + +`ScalarFnCall` is a new view of the function, its options and its bound arguments. Aggregates +need the same pair of hooks (`rewrite_aggregate`). Open: whether a session registry keyed by +`(function ID, extension ID)` also exists, so that a function owner can support a type that it does +not own. + +### Examples + +```text +Timestamp(ms) a < Timestamp(ms) b => ext_storage(a) < ext_storage(b) +Timestamp(ms) a < lit(Timestamp(ms) 5) => ext_storage(a) < lit(5i64) +min(Timestamp(ms) a) => ext_wrap(min(ext_storage(a)), Timestamp(ms), unchecked) +cast(Timestamp(ms) a, Timestamp(ns)) => ext_wrap(checked_mul(ext_storage(a), 1_000_000), + Timestamp(ns), unchecked) +cast(Timestamp(ns) a, Timestamp(ms)) => rejected unless the type defines rounding +DecimalLike(10, 1) a < DecimalLike(10, 2) b + => checked_mul(ext_storage(a), 10) < ext_storage(b) + (a rewrite that the type owns) +Json a = Json b => rejected +``` + +`ext_wrap(..., unchecked)` is correct in these examples, because `min` returns an existing value and +the unit cast keeps the range check in `checked_mul`. + +### Array API + +The array API builds `ScalarFnArray` values directly. The binder must run in +`ScalarFnArray::try_new` (or in the typed constructors such as `Binary::try_new`). It returns the +rewritten tree, so the new array holds `ext_storage` children and an `ext_wrap` parent where needed. +The row path (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42`) then evaluates the +rewritten tree, and I6 holds without other changes. + +Open: some rewrites can need session state. The hooks above use only the `ExtDType`, which travels +with the dtype, so the binder does not need a session. + +### Literals + +- `ext_storage(lit(s))` simplifies to `lit(s.to_storage_scalar())`. +- `ext_wrap(ext_storage(x), dtype(x))` simplifies to `x`. +- `ext_storage(ext_wrap(y, d))` simplifies to `y`. + +## 5. `ext_wrap` + +- Options: the target `ExtDTypeRef`, and a flag that is `checked` or `unchecked`. +- `return_dtype`: the target dtype, with the nullability of the input. +- Argument: must have the storage dtype of the target, ignoring nullability. +- Checked execution: calls `validate_array` (section 8), then builds the wrapper. +- Unchecked execution: builds the wrapper. Only the binder and kernels that keep the invariants of + the type use it. +- Row path: `Scalar::try_new` for checked, `Scalar::new_unchecked` for unchecked. + +## 6. Normal form + +Only the wrapper (`vortex.ext`) carries an extension dtype. These reduce rules move the wrapper to +the top: + +| From | To | +| --- | --- | +| `Constant(extension scalar s)` | `Extension(Constant(s.to_storage_scalar()))` | +| `Chunked[Extension(d, a_i)]` | `Extension(d, Chunked[a_i])` | +| `Dict(codes, Extension(d, values))` | `Extension(d, Dict(codes, values))` | +| `Masked(Extension(d, x), m)` | `Extension(d with the result nullability, Masked(x, m))` | +| `Sparse` with extension values | `Extension(Sparse` over storage`)` | + +Consequences: + +- `ExtensionConstantRule` (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`) reverses + direction. +- `ArrayRef::as_constant()` must look through the wrapper, or a new helper does it. Many kernels + call `as_constant()` on an operand. +- `encodings/sparse/src/canonical.rs:360-362` no longer needs to reject extension arrays. +- `DateTimeParts` becomes an `i64` encoding with the time unit in its metadata. Today its dtype is + `Timestamp` (`encodings/datetime-parts/src/canonical.rs:28`). Readers convert the old form on + read. An edition gates writes of the new form. `vortex-btrblocks/src/schemes/temporal.rs` chooses + the encoding from the extension metadata, and compresses the storage child. +- The same applies to any future encoding for one extension type. + +## 7. Statistics + +- **S1.** Zone maps and file statistics for an extension column store statistics of the storage, + with the storage dtype. The fallback in `vortex-layout/src/layouts/zoned/schema.rs:112-121` + becomes the rule. +- **S2.** A new stats rewrite rule maps `stat(ext_storage(x), s)` to the storage statistic `s` of + `x`. Bound predicates reference `ext_storage(col)` after binding, so pruning needs only this rule. +- **S3.** Readers convert existing extension-typed min and max to storage scalars with + `to_storage_scalar()`. The values are already storage values. +- **S4.** Engine statistics export (DuckDB `column_statistics.rs`, DataFusion `convert/stats`, JNI + write summaries) goes through the export plugin of the type. A type without `ord` exports no min + and max. +- **S5.** Bounded min and max (truncation) apply to the storage dtype, so they also apply to + extensions over `Utf8` and `Binary`. + +## 8. Validation + +```rust +pub trait ExtVTable { + /// Checks that every non-null value of `storage` is valid for this type. + /// + /// The default accepts every value. A type with value-level rules overrides it. + fn validate_array( + ext: &ExtDType, + storage: &ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult<()> { + Ok(()) + } +} +``` + +- Callers: `ExtensionArray::try_new`, `ext_wrap` (checked), Arrow import, and file reads when a read + option enables it. +- Kernels that keep the invariants use `ExtensionArray::new_unchecked`. +- A row read from an array uses `Scalar::new_unchecked`. The per-row validation in the current read + path (`vortex-array/src/arrays/extension/vtable/operations.rs:29`) then goes away. +- Examples: + - `Timestamp` computes the storage min and max, then checks only those two values with `jiff`. + - `Uuid` checks the version nibble of each value against the metadata. + - `NormalizedVector` checks the norm of each row within a tolerance. +- Open: is the default a no-op, or a check of each value through `validate_scalar_value`? + +## 9. Nullability + +- **N1, short term.** `ExtDType::try_with_vtable` calls `validate_dtype` with the non-nullable form + of the storage dtype. The documentation states that `validate_dtype` receives a non-nullable + storage dtype. `with_nullability` then never validates again. +- **N2, long term.** `ExtDType` stores a non-nullable storage dtype and a `Nullability`, like every + other `DType` variant. The storage child has the dtype `storage_dtype.with_nullability(n)`. The + wire format does not change, because the serialized storage dtype already carries nullability. + +## 10. Zero values + +```rust +pub trait ExtVTable { + /// Returns the zero of this type, or `None` if the type has no zero. + fn zero_value(ext: &ExtDType) -> Option { + None + } +} +``` + +- `Scalar::zero_value` and `ScalarValue::try_zero_value` use this hook for extension dtypes. +- Open: builders pad a non-nullable child under a null parent + (`vortex-array/src/builders/fixed_size_list.rs:288` and `:303`). A type with no zero has no valid + padding value. One option is that `validate_array` checks only rows that are valid in every + ancestor. Another option is that such a child must be nullable. + +## 11. Scalar identity + +`Scalar` implements `PartialEq` and `Hash` for Rust code: tests, maps of constants, and +deduplication. Keep these structural: two scalars are equal when the dtypes are equal (ignoring +nullability) and the storage values are equal. Document this as identity, not as SQL equality. +`PartialOrd` for extension scalars must follow `derives`, because ordering is semantic. The +compare paths (`scalar_cmp`) must go through the binder rules, not through `PartialOrd`. + +## 12. Files to change, by phase + +| Phase | Files | +| --- | --- | +| 0, U1 | `vortex-array/src/scalar/cast.rs`, `vortex-array/src/arrays/extension/compute/cast.rs`, `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | +| 0, U2 | `vortex-array/src/scalar_fn/fns/binary/mod.rs`, `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs`, `encodings/datetime-parts/src/compute/compare.rs` | +| 0, U3 | `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/scalar/constructor.rs` | +| 0, U4 | `vortex-array/src/dtype/extension/typed.rs`, `vortex-array/src/dtype/extension/vtable.rs` | +| 0, U5 | `vortex-array/src/extension/datetime/{date,time,timestamp}.rs` | +| 0, U6 | `vortex-array/src/arrays/extension/vtable/mod.rs` | +| 0, U7 | `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/scalar_value.rs` | +| 0, U8 | `vortex-array/src/dtype/extension/erased.rs` | +| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/scalar_fn/fns/{binary,between}/`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | +| 3 | `vortex-array/src/scalar_fn/fns/cast/mod.rs`, `vortex-array/src/extension/datetime/`, `vortex-duckdb/src/convert/expr.rs`, `vortex-datafusion/src/convert/exprs/mod.rs` | +| 4 | `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/`, `encodings/sparse/src/canonical.rs`, `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/`, `vortex-edition/` | +| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/builders/extension.rs`, `vortex-array/src/dtype/extension/` | +| 6 | `vortex-arrow/src/session.rs`, `vortex-duckdb/src/convert/`, `vortex-duckdb/src/exporter/`, `vortex-datafusion/src/convert/`, `vortex-python/src/scalar/`, `docs/developer-guide/extending/extension-dtypes.md` | + +## 13. Tests + +- **Path consistency.** A helper in `vortex-array/src/compute/conformance/` takes an expression + builder and inputs. It evaluates the expression on the array path, on the row path, and with + constant-encoded inputs. It asserts equal values or an error on all three. +- **Reference types.** `vortex-array/src/extension/tests/` already has `DivisibleInt`. Add: + - `Opaque` over `i64`, with no derives. Every semantic operation fails when it is bound. + - `DecimalLike` over `i64`, with `eq` and `ord` for the same scale and a rescale rewrite. A + differential test runs each operation on `DecimalLike` and on the built-in `Decimal`, and + compares values or errors. + - `Refined` over `i64` (for example, positive values only), with every derive and a + `validate_array` check. +- **Conformance harness for `ExtVTable`.** Checks the metadata round trip, that `validate_dtype` + ignores nullability, rules D1 to D4, cast round trips, and path consistency. +- **Engines.** DuckDB and DataFusion pushdown tests with extension literals for `Timestamp`, `Date`, + `Uuid` and `Json` columns. A `Json` comparison must not be pushed down. diff --git a/research/extension-types/agents/findings.md b/research/extension-types/agents/findings.md new file mode 100644 index 00000000000..af9e5d22ca8 --- /dev/null +++ b/research/extension-types/agents/findings.md @@ -0,0 +1,535 @@ + + +# Extension type findings (agent reference) + +This file is the detailed catalog behind `../02-current-problems.md`. Each finding has an ID that +the human documents use. All paths are relative to the repository root. All line numbers are for +`develop` at commit `5fc5cb8` (2026-10-17). + +## Conventions + +Severity: + +- **S1, wrong result.** An operation returns a value without an error, and the value is wrong for + the extension type. +- **S2, panic or invalid state.** An operation panics, or an invalid value enters an array or a + scalar. +- **S3, inconsistency.** Two parts of the system give different answers to the same question. +- **S4, hygiene.** Debug output, registration, documentation. + +Status: + +- **probe** means that the throwaway test in `probe.rs` reproduced the behavior. The probe ran + against `vortex-array` in a debug build. +- **code** means that the behavior follows directly from the cited code. Nothing ran. +- **lead** means that the code suggests a problem, but the problem is not confirmed. A lead names + the check that confirms or rejects it. + +Terms: + +- **Row path:** `ArrayRef::execute_scalar(i)` on a lazy array. For a `ScalarFn` array, this + evaluates the function on one-row constant inputs. +- **Array path:** `execute::` on the lazy array, then read the stored values. +- **Wrapper:** the `Extension` array (encoding ID `vortex.ext`) that holds the storage array as its + only child. + +## Summary + +| ID | Title | Severity | Status | +| --- | --- | --- | --- | +| EXT-P1 | Encoding and read path change the result of cast and compare | S1 | probe | +| EXT-C1 | Scalar cast into an extension reinterprets the storage value | S1 | probe | +| EXT-C2 | Array cast out of an extension returns the storage cast | S1 | probe | +| EXT-C3 | Same-type extension cast hides the storage error | S3 | code | +| EXT-Q1 | Compare skips the type check when one side is an extension | S1 | probe | +| EXT-Q2 | Every extension type gets storage equality and order, with no override | S1 | probe | +| EXT-Q3 | `DateTimeParts` compare kernel trusts its operand types | S2 | lead | +| EXT-A1 | Ordering-based operations disagree about extension types | S3 | probe | +| EXT-A2 | Zone maps fall back to storage statistics | S3 | code | +| EXT-V1 | Arrays are never value-checked, and reads panic on bad values | S2 | probe | +| EXT-V2 | `Extension::validate` never checks the storage dtype | S2 | probe | +| EXT-V3 | Zero values come from storage and skip extension validation | S2 | probe | +| EXT-V4 | `Date` and `Time` panic on non-primitive storage | S2 | code | +| EXT-V5 | `Timestamp` accepts dtypes that reject every value | S2 | code | +| EXT-V6 | Eight extension types never check values | S3 | code | +| EXT-N1 | A nullability change can panic | S2 | probe | +| EXT-S1 | A constant extension column has two shapes | S3 | code | +| EXT-S2 | Some encodings carry the extension dtype themselves | S3 | code | +| EXT-B1 | DataFusion converts extension values to storage values | S1 | code | +| EXT-B2 | Arrow export can silently export storage | S1 | code | +| EXT-B3 | Python returns and builds raw storage values | S1 | code | +| EXT-B4 | DuckDB decides pushdown from the storage type | S3 | code | +| EXT-B5 | JNI write summaries report storage values | S3 | code | +| EXT-B6 | Pruning looks through casts out of extensions | S3 | lead | +| EXT-R1 | `Debug for ExtDTypeRef` prints the wrong output | S4 | probe | +| EXT-R2 | Registration overwrites and duplicates silently | S4 | code | +| EXT-R3 | Unknown extension IDs inherit storage semantics | S3 | code | +| EXT-R4 | Documentation is a stub or stale | S4 | code | + +## Findings + +### EXT-P1: Encoding and read path change the result of cast and compare + +- **Locations:** + - Row path: `vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42` wraps each input row in + a one-row `ConstantArray` and executes the function on those constants. + - Constant path: `vortex-array/src/arrays/constant/compute/cast.rs:14-20` calls `Scalar::cast`. + `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:158-163` calls `scalar_cmp` when both + sides are constant. + - Array path: `vortex-array/src/arrays/extension/compute/cast.rs:17-43` and + `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`. +- **Mechanism:** Cast and compare each have a scalar implementation and an array implementation. + Constant inputs use the scalar implementation. Other encodings use the array implementation. The + two implementations apply different rules to extension types. The row path converts every input + to a constant, so it always takes the scalar implementation. +- **Observed** (probe IDs in brackets): + + | Expression | Array path | Row path | + | --- | --- | --- | + | `cast(Timestamp(ms) to Timestamp(ns))`, primitive storage | Error: no `CastKernel` [D1'] | `1970-01-01T00:00:01Z` becomes `00:00:00.000001Z` [D1] | + | Same cast, constant storage | `00:00:01Z` becomes `00:00:00.000001Z` [D2'] | Same wrong value [D2] | + | `cast(i64 to Timestamp(ms))` | Error: no `CastKernel` [B'] | Wraps the raw `i64` [B] | + | `cast(Opaque to NonZero)`, two unrelated test types | Error: no `CastKernel` [E4'] | Wraps the raw value [E4] | + | `Timestamp(ms) < i64` | Compares raw storage: `[true, false]` [F1'] | Error: incompatible types [F1] | + +- **Expected:** One answer per expression, whatever the encoding and the read path. The type + system RFC draft (`vortex-data/rfcs#55`) states this as a rule for every operation. +- **Fix:** Give cast and compare one rule. Short term, make both implementations reject the cases + in EXT-C1, EXT-C2 and EXT-Q1. Long term, rewrite semantic operations at bind time so that both + paths evaluate the same expression (see `design-notes.md`). +- **Regression test:** Add a helper that evaluates an expression three ways: on the array path, on + the row path, and with constant-encoded inputs. The helper asserts equal values or an error in all + three. Run it for every case in the table above. + +### EXT-C1: Scalar cast into an extension reinterprets the storage value + +- **Location:** `vortex-array/src/scalar/cast.rs:56-61`, which has a `TODO(connor)` that says the + code is not correct for extension types. +- **Mechanism:** When the target dtype is an extension, `Scalar::cast` casts the value to the + storage dtype of the target. It then wraps the result in the target dtype. The source type and its + metadata play no part. +- **Observed:** `Timestamp(ms)=1000` casts to `Timestamp(ns)=1000`, so one second becomes one + microsecond [A]. A plain `i64` casts to `Timestamp` [C]. A value of one extension type casts to an + unrelated extension type [E4]. +- **Reach:** All constant inputs (EXT-P1), expression simplification of literals + (`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`), and Python + `vx.scalar(v, dtype=ext)` (`vortex-python/src/scalar/factory.rs:106`). +- **Expected:** An error, unless the source and target are the same extension dtype with a + different nullability. A conversion that changes meaning must go through a cast that the + extension type owns (issue #8570), or through explicit `ext_storage` and a future `ext_wrap`. +- **Fix:** Remove the wrap. Keep only the nullability case. +- **Regression test:** The three cases above must return an error on the row path, the array path, + and with constant inputs. + +### EXT-C2: Array cast out of an extension returns the storage cast + +- **Location:** `vortex-array/src/arrays/extension/compute/cast.rs:18-22`. +- **Mechanism:** If the target is not the same extension dtype (ignoring nullability), the kernel + returns `storage_array().cast(target)`. +- **Observed:** `Timestamp(ms)` casts to `i64`, `f64` and `u8` [E1, E2, E3, E1']. +- **History:** PR #7469 added this behavior. Issue #7504 decided to remove it, and was closed as a + duplicate of #8570, which is still open. The conformance suite says the opposite of the kernel: + `vortex-array/src/compute/conformance/consistency.rs:1051` ("Extension types typically only cast + to themselves"). +- **Coupling:** The Arrow exporters call `array.cast(target)` for primitive, byte-view and struct + targets (`vortex-arrow/src/executor/primitive.rs:45`, `byte_view.rs:64`, `struct_.rs:115`). When + they receive an extension array, they export its storage through this fallback (EXT-B2). Removing + the fallback without changing these callers turns a silent export into an error. +- **Expected:** An error, unless the extension type defines the cast. +- **Fix:** Remove the fallback. Callers that intend to read storage must call `ext_storage`. +- **Regression test:** Casting `Timestamp(ms)` to `i64` returns an error on both paths. + `ext_storage` on the same input returns the `i64` storage. + +### EXT-C3: Same-type extension cast hides the storage error + +- **Location:** `vortex-array/src/arrays/extension/compute/cast.rs:28-36`. +- **Mechanism:** When the target is the same extension dtype, the kernel casts the storage. If that + cast fails, the kernel logs `tracing::warn!` and returns `Ok(None)`. The caller then reports "no + CastKernel", and the real error is lost. +- **Fix:** Return the error. + +### EXT-Q1: Compare skips the type check when one side is an extension + +- **Locations:** + - `vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`: `Binary::return_dtype` rejects + different dtypes for comparisons only when neither side is an extension. + - `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`: `compare_arrays` replaces a + lone extension side with its storage (`extension_storage`, line 226) and compares. + - `vortex-array/src/scalar_fn/fns/between/mod.rs:282-305`: `Between::return_dtype` requires the + same dtype for all three operands. +- **Observed:** On the array path, `Timestamp(ms) < 1500i64` compares the raw `i64` storage + [F1', F2']. On the row path, the same expression is an error [F1, F2]. `Timestamp(ms) < 'a'` is + accepted when the expression is built, and fails at execution with `Cannot compare different + DTypes i64 and utf8` [F3']. `between(ts, 0i64, 1500i64)` is rejected when it is built [F4]. So + `ts >= 0 AND ts <= 1500` runs while `ts BETWEEN 0 AND 1500` does not. +- **Expected:** Comparison requires the same dtype, ignoring nullability, like every other dtype. +- **Fix:** Remove the extension exception in `Binary::return_dtype`. Remove the lone extension case + in `compare_arrays`. Engines that send storage-typed literals for extension columns must send + extension literals. +- **Regression test:** All three comparisons are rejected when the expression is built. + +### EXT-Q2: Every extension type gets storage equality and order, with no override + +- **Locations:** + - `vortex-array/src/arrays/extension/compute/compare.rs:18-55`: the compare kernel compares + storage whenever both sides have the same extension dtype. + - `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`: the comparator for nested + values compares storage. + - `vortex-array/src/scalar/typed_view/extension/mod.rs:107-137`: `ExtScalar` `PartialEq`, + `PartialOrd` and `Hash` use storage (`TODO(connor)` at line 107). + - `vortex-array/src/arrays/extension/vtable/kernel.rs:15-22`: the compare kernel is registered on + the shared `Extension` encoding, so it applies to every extension type. + - `vortex-array/src/optimizer/kernels.rs:262`: kernels registered later for the same key run + after the built-in kernel and cannot override it. +- **Observed:** A test type with no ordering semantics (`Opaque`) supports `<` and returns storage + order [G]. +- **Impact:** The result is wrong for any type whose storage order or storage equality differs from + its value order or value equality. Examples in the repository: JSON (key order and whitespace), + WKB geometry (two encodings of one geometry), vectors and tensors (no natural order). +- **Fix:** See `design-notes.md` (pairwise derives that the binder reads). + +### EXT-Q3: `DateTimeParts` compare kernel trusts its operand types (lead) + +- **Locations:** + - `encodings/datetime-parts/src/compute/compare.rs:24-61`: the kernel calls + `rhs_const.as_extension()` before it checks the dtype. It then splits the constant with the time + unit of the constant. It does not compare that unit with the time unit of the column. + - `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:73-121`: `CompareExecuteAdaptor` calls + the kernel of the encoding without checking that the operand dtypes match. + - `vortex-array/src/scalar/downcast.rs:161-164`: `Scalar::as_extension` panics if the scalar is + not an extension. +- **Suspected behavior:** A `DateTimeParts`-encoded timestamp column compared with an `i64` + constant panics in `as_extension()`. EXT-Q1 lets that expression through. A comparison with a + timestamp constant in a different unit compares sub-second parts in different units. +- **Counter-evidence:** `vortex-file/src/tests.rs:2369` and `:2410` assert that a unit mismatch + returns an error in a file scan. The tests do not show which layer raises the error. Zone-map + pruning can raise it before the kernel runs. +- **Check:** Build a `DateTimePartsArray` directly, compare it with an `i64` constant, and execute + it on the array path. Then repeat with a timestamp constant in another unit. +- **Fix:** Make `CompareExecuteAdaptor` require equal dtypes (ignoring nullability) before it calls + any kernel. Make the kernel check the unit. + +### EXT-A1: Ordering-based operations disagree about extension types + +- **Locations:** + - `vortex-array/src/aggregate_fn/fns/min_max/mod.rs:284-313` accepts every extension dtype. + `min_max/extension.rs:17-35` computes min and max on storage. + - `vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs:255-283` rejects every extension dtype, and + `is_sorted_impl` then returns `false` (`mod.rs:104-109`). The extension branch that calls + `check_extension_sorted` can never run. + - `vortex-row/src/lib.rs:24-28` and `vortex-row/src/encoder.rs:128` reject extension types + because the crate "does not define a total ordering for them". + - `vortex-array/src/aggregate_fn/fns/sum/mod.rs:131` rejects every extension dtype. +- **Observed:** `min_max` on the `Opaque` test type returns storage min and max [H]. `is_sorted` on + a sorted `Timestamp` array returns `false` [I]. +- **Open work that touches this.** Each of these PRs adds a different opt-in mechanism: + - PR #10246 adds a per-type aggregate kernel registry and `StorageAggregateKernel`. + - PR #10566 adds an `ExtensionAggregate` adapter with a composite aggregate ID. + - PR #10408 adds a `between` pushdown that matches `AnyTemporal` inside the rule. +- **Fix:** Make every ordering-based operation read one declaration (see `design-notes.md`). + +### EXT-A2: Zone maps fall back to storage statistics + +- **Locations:** + - `vortex-layout/src/layouts/zoned/schema.rs:112-121`: when a statistic rejects the extension + dtype, the stats schema uses the storage dtype (comment at line 114). + - `vortex-layout/src/layouts/zoned/writer.rs:217-231`: bounded min and max apply only to + top-level `Utf8` and `Binary`. An extension over `Utf8` or `Binary` (JSON, WKB) stores exact, + unbounded bounds. +- **Impact:** WKB zone maps store whole geometry blobs as their min and max, ordered by bytes. +- **Fix:** Store storage statistics explicitly for every extension column, and have pruning use them + through `ext_storage` (see `design-notes.md`). + +### EXT-V1: Arrays are never value-checked, and reads panic on bad values + +- **Locations:** + - `vortex-array/src/arrays/extension/array.rs:52-57`: `try_new` checks the storage dtype only. + - `vortex-array/src/builders/extension.rs:110-112`: `append_zeros` writes storage zeros. + - `vortex-array/src/arrays/extension/vtable/operations.rs:29`: reading a row calls + `Scalar::extension_ref`. + - `vortex-array/src/arrays/extension/compute/rules.rs:39`: `ExtensionConstantRule` calls + `Scalar::extension_ref`. + - `vortex-array/src/scalar/constructor.rs:255-260`: `Scalar::extension_ref` panics with + `vortex_expect` if the value is invalid. `Scalar::extension::` (line 243) also panics. +- **Observed:** With a `NonZero` test type, `ExtensionArray::new(non_zero, [0])` succeeds [L3]. + `append_zeros(1)` succeeds [L1]. Reading the row panics with `unable to construct an extension + Scalar` [L2]. +- **Fix:** Return an error from these read paths. Add value validation at trust boundaries + (`design-notes.md`, "Validation"). + +### EXT-V2: `Extension::validate` never checks the storage dtype + +- **Location:** `vortex-array/src/arrays/extension/vtable/mod.rs:96-123`. Line 116 builds + `DType::Extension(ext_dtype.clone())` from `dtype` and compares the result with `dtype`. The two + are always equal. +- **Observed:** `ExtensionArray::try_from_parts` accepts a `Utf8` storage child under a + `Timestamp(ms)` dtype [O]. +- **Fix:** Compare the dtype of the storage child with `ext_dtype.storage_dtype()`. + +### EXT-V3: Zero values come from storage and skip extension validation + +- **Locations:** + - `vortex-array/src/scalar/scalar_value.rs:90-95`: the zero of an extension is the zero of its + storage. + - `vortex-array/src/scalar/scalar_impl.rs:165-169`: `Scalar::zero_value` calls `new_unchecked`. + Its `SAFETY` comment says that the value is valid for the dtype. + - `vortex-array/src/scalar/scalar_impl.rs:97-104`: `new_unchecked` validates only in debug + builds. + - `vortex-array/src/scalar/downcast.rs:172`: `as_extension_opt` assumes that every `Scalar` is + valid. + - `vortex-array/src/scalar/scalar_impl.rs:254`: `is_zero` on an extension is `is_zero` on its + storage. +- **Observed:** For the `NonZero` test type, `Scalar::zero_value` panics in a debug build [K1]. + In a release build, it returns an invalid scalar (from code). +- **Fix:** Make the zero of an extension type an opt-in. Without it, `try_zero_value` returns + `None`. + +### EXT-V4: `Date` and `Time` panic on non-primitive storage + +- **Locations:** `vortex-array/src/extension/datetime/date.rs:104` and + `vortex-array/src/extension/datetime/time.rs:105` call `storage_dtype().as_ptype()`. + `vortex-array/src/dtype/dtype_impl.rs:209-215` panics if the dtype is not primitive. +- **Impact:** A file whose `vortex.date` or `vortex.time` dtype has non-primitive storage panics + while its dtype is read. It does not return an error. +- **Fix:** Match on `DType::Primitive` and return an error otherwise. + +### EXT-V5: `Timestamp` accepts dtypes that reject every value + +- **Locations:** `vortex-array/src/extension/datetime/timestamp.rs:198-204` checks only for `i64` + storage. `unpack_native` rejects the `Days` unit (line 231) and unknown time zones (line 241). +- **Impact:** `Timestamp(Days)` and a timestamp with an unknown time zone are valid dtypes, but + every non-null value fails. `vortex-array/src/dtype/extension/vtable.rs:20-24` says that + `validate_dtype` must prevent this. +- **Fix:** Reject the `Days` unit and unknown time zones in `validate_dtype`. + +### EXT-V6: Eight extension types never check values + +- **Locations:** These types return the raw `&ScalarValue` from `unpack_native` and do not + override `validate_scalar_value`: + - `vortex-tensor/src/types/vector/vtable.rs` + - `vortex-tensor/src/types/fixed_shape_tensor/vtable.rs` + - `vortex-spatial/src/extension/linestring.rs`, `multipoint.rs`, `polygon.rs`, + `multilinestring.rs`, `multipolygon.rs` and `rect.rs` +- **Impact:** These types check nothing at the value level, and `Display` shows raw storage. This is + allowed by the trait, but it means that "scalars are checked" is not true for these types. + +### EXT-N1: A nullability change can panic + +- **Locations:** + - `vortex-array/src/dtype/extension/typed.rs:91-99`: `ExtDType::with_nullability` runs + `validate_dtype` again and panics if it fails. + - Callers: `take` (`vortex-array/src/arrays/extension/compute/take.rs:23` and `:42`), `mask` + (`compute/mask.rs:23`), masked execution (`vortex-array/src/arrays/masked/execute.rs:187-190`), + `min_max` (`vortex-array/src/aggregate_fn/fns/min_max/extension.rs:23`), and + `DType::with_nullability` (`vortex-array/src/dtype/dtype_impl.rs:108`), which + `DType::as_nullable` uses. +- **Observed:** A test type that accepts only non-nullable storage panics in `take` with nullable + indices [M1]. +- **Cause:** Nullability is part of the storage dtype, and nothing says that `validate_dtype` must + ignore it. +- **Fix:** Short term, validate both nullabilities when the dtype is built, so `with_nullability` + cannot fail later. Long term, see `design-notes.md` ("Nullability"). + +### EXT-S1: A constant extension column has two shapes + +- **Locations:** `vortex-array/src/arrays/extension/compute/rules.rs:28-46` rewrites + `Extension(Constant(storage))` to `Constant(extension scalar)`. + `vortex-array/src/scalar_fn/unstable/row/types/element/tuple/element_tuple.rs:126` has + `TODO(connor): This is maybe incorrect unless this is a refinement type?`. +- **Impact:** Kernels and extension crates must handle both shapes. `vortex-tensor` handles both by + hand. The same choice exists for `Dict` and `Chunked`. +- **Fix:** Choose one normal form (see `design-notes.md`, "Normal form"). + +### EXT-S2: Some encodings carry the extension dtype themselves + +- **Locations:** `encodings/datetime-parts/src/canonical.rs:28` requires a `Timestamp` dtype on the + `DateTimeParts` array. `encodings/parquet-variant/src/kernel.rs:196` registers a kernel on the + shared `Extension` encoding, and the kernel checks `is::()` itself. +- **Impact:** An array with an extension dtype is not always the wrapper. Extension-specific kernels + cannot be keyed on the wrapper encoding alone. + +### EXT-B1: DataFusion converts extension values to storage values + +- **Locations:** + - `vortex-datafusion/src/convert/scalars.rs:138-145`: a non-temporal extension scalar converts as + its storage scalar. + - `vortex-datafusion/src/convert/scalars.rs:324`: `struct_to_df` converts extension children to + storage values. + - `vortex-datafusion/src/convert/schema.rs:105`: schema conversion walks through extension dtypes + to the storage struct. + - `vortex-datafusion/src/convert/stats/mod.rs:66`: `is_nan_free` checks `dtype.is_float()`, which + is false for an extension with float storage. +- **Guard to keep:** `vortex-datafusion/src/convert/exprs/mod.rs:355-361` refuses to push down a + cast that involves an extension type. + +### EXT-B2: Arrow export can silently export storage + +- **Locations:** + - `vortex-arrow/src/executor/primitive.rs:45`, `byte_view.rs:64`, `struct_.rs:115`: exporters + call `array.cast(target)`, which reaches EXT-C2 for an extension input. + - `vortex-arrow/src/session.rs:329`: `to_arrow_field` handles `AnyTemporal` by itself and needs a + plugin for every other extension type. + - `vortex-arrow/src/session.rs:873-876`: export falls back to `execute_arrow_naive` with the bare + Arrow data type. + - `vortex-arrow/src/scalar.rs:242`: scalar export supports temporal extension types only. +- **Impact:** A caller can ask for a storage-like Arrow type, for example `Int64` for a timestamp + column through `to_arrow_array(arrow_type=...)` in Python. The caller then gets the storage + values with no error. + +### EXT-B3: Python returns and builds raw storage values + +- **Locations:** `vortex-python/src/scalar/into_py.rs:93-95` (`as_py()` returns the storage value). + `vortex-python/src/scalar/factory.rs:106` (`vx.scalar(v, dtype=ext)` calls `Scalar::cast`, so + EXT-C1 applies). + +### EXT-B4: DuckDB decides pushdown from the storage type + +- **Locations:** + - `vortex-duckdb/src/table_function.rs:532-537`: aggregate pushdown excludes `Uuid` only, because + its storage is a `FixedSizeList`. Every other extension type gets min and max pushed down in + storage order. + - `vortex-duckdb/src/column_statistics.rs:37`: the NaN check uses `dtype.is_float()`. + - `vortex-duckdb/src/convert/dtype.rs:260-284` and `vortex-duckdb/src/exporter/extension.rs:36`: + a hard-coded list of extension types. There is no plugin hook. + +### EXT-B5: JNI write summaries report storage values + +- **Location:** `vortex-jni/src/writer.rs:217` converts extension bounds with + `to_storage_scalar()`. Java receives a raw `i64` for a timestamp bound, with no unit. + +### EXT-B6: Pruning looks through casts out of extensions (lead) + +- **Locations:** `vortex-file/src/pruning.rs:105-116` (`direct_field_path` looks through `Cast`) + and `vortex-array/src/stats/rewrite/builtins.rs:797-803` (`cast_stat` casts min and max). +- **Suspected behavior:** With EXT-C2, `cast(ext_col, i64)` reuses the statistics of the extension + column through a storage cast. This is correct only for casts that keep order. +- **Check:** Build a pruning predicate on `cast(timestamp_col, u8)` and compare the pruning result + with a full scan. + +### EXT-R1: `Debug for ExtDTypeRef` prints the wrong output + +- **Location:** `vortex-array/src/dtype/extension/erased.rs:181-191`. Line 183 writes the metadata + into the formatter and keeps the returned `fmt::Result` as the metadata field. +- **Observed** [N]: + + ```text + TimestampOptions { unit: Milliseconds, tz: None }ExtDType { id: Id("vortex.timestamp"), + metadata: Ok(()), storage_dtype: Primitive(I64, NonNullable) } + ``` + + The output is one line. It is split here to fit the page. + +### EXT-R2: Registration overwrites and duplicates silently + +- **Locations:** `vortex-array/src/dtype/session.rs:57-60` (`register` replaces an existing ID with + no error). `vortex-arrow/src/session.rs:257-269` (`register_exporter` and `register_importer` + push onto a list, so a second `initialize` call registers each plugin twice). + +### EXT-R3: Unknown extension IDs inherit storage semantics + +- **Locations:** + - `vortex-array/src/dtype/serde/flatbuffers.rs:321`, `proto.rs:145` and `serde.rs:806`: three + copies of the lookup, with three different error messages. + - `vortex-array/src/dtype/extension/foreign.rs:28-75`: with `allow_unknown()`, an unknown ID + becomes a `ForeignExtDType` that accepts any storage and any value. + - `vortex-array/src/dtype/extension/mod.rs:31-32`: `ForeignExtDType` is crate-private, so other + crates cannot detect a placeholder. +- **Impact:** A placeholder gets all of EXT-Q2 and EXT-A1, so a reader without the type compares + and orders values by storage. + +### EXT-R4: Documentation is a stub or stale + +- `docs/developer-guide/extending/extension-dtypes.md` is a stub. +- `docs/developer-guide/internals/vtables.md:84-113` shows the old `ExtDTypeVTable` shape. +- `vortex-array/src/arrays/extension/vtable/mod.rs:39-74` says that extension arrays delegate most + operations to storage. That statement conflicts with an opaque design. + +## Appendix A: probe results + +`probe.rs` defines three test extension types: `NonZero` (`u64`, rejects zero), `NonNullableOnly` +(rejects nullable storage), and `Opaque` (`i64`, no meaning for order). The probe ran in a debug +build of `vortex-array`. "Row" means `execute_scalar` per row on the lazy result. "Array" means +`execute::`. + +| Probe | Expression | Path | Result | +| --- | --- | --- | --- | +| A | scalar `cast(Timestamp(ms)=1000, Timestamp(ns))` | scalar | `1970-01-01T00:00:01Z => 1970-01-01T00:00:00.000001Z` | +| B | `cast(i64 [1000, 2000], Timestamp(ms))` | row | wraps: `[00:00:01Z, 00:00:02Z]` | +| B' | same | array | `No CastKernel to cast canonical array vortex.primitive from i64 to vortex.timestamp[ms](i64)` | +| C | `cast(Constant(1000i64), Timestamp(ms))` | row | wraps | +| D1 | `cast(Timestamp(ms) [1000, 2000], Timestamp(ns))` | row | `[00:00:00.000001Z, 00:00:00.000002Z]` | +| D1' | same | array | no `CastKernel` error | +| D2 | same cast, constant storage | row | `00:00:00.000001Z` | +| D2' | same | array | `00:00:00.000001Z` | +| E1, E1' | `cast(Timestamp(ms), i64)` | both | `[1000, 2000]` | +| E2 | `cast(Timestamp(ms), f64)` | row | `[1000.0, 2000.0]` | +| E3 | `cast(Timestamp(ms) [1, 2], u8)` | row | `[1, 2]` | +| E4 | `cast(Opaque, NonZero)` | row | `scratch.non_zero(u64) [1, 2]` | +| E4' | same | array | no `CastKernel` error | +| F1, F2 | `Timestamp(ms) < i64` (array, then constant) | row | `Cannot compare scalars with incompatible types` | +| F1', F2' | same | array | `[true, false]` | +| F3' | `Timestamp(ms) < utf8` | array | `Cannot compare different DTypes i64 and utf8` | +| F4 | `between(Timestamp(ms), 0i64, 1500i64)` | build | `Array dtype ... does not match lower dtype i64` | +| F5' | `Timestamp(ms) < Timestamp(ns)` | array | `Cannot compare extension dtypes ...` | +| G | `Opaque < Opaque` | row | `[true, false]` | +| H | `min_max(Opaque [5, 1, 3])` | call | `min=1 max=5` | +| I | `is_sorted(Timestamp(ms) [1, 2, 3])` | call | `false` | +| K1 | `Scalar::zero_value(NonZero)` | call | panic: `Scalar::new_unchecked called with incompatible dtype and value` | +| L1 | `ExtensionBuilder(NonZero).append_zeros(1).finish()` | call | succeeds | +| L2 | then `execute_scalar(0)` | row | panic: ``unable to construct an extension `Scalar` `` | +| L3 | `ExtensionArray::new(NonZero, [0])` | call | succeeds | +| M1 | `take(NonNullableOnly, nullable indices)` | call | panic: `Extension DType should not fail validation ...` | +| N | `format!("{:?}", Timestamp(ms))` | call | see EXT-R1 | +| O | `try_from_parts(Timestamp(ms), utf8 storage)` | call | succeeds | +| P | `Timestamp(ms) + Timestamp(ms)` | build | `incompatible types for arithmetic operation` | +| Q | `Scalar::is_zero(Timestamp(ms)=0)` | call | `Some(true)` | + +## Appendix B: where storage semantics are applied + +This table lists every place where code outside an extension crate applies storage semantics to an +extension value. "Structural" means that the operation does not read values, so it is correct for +any extension type. + +| Location | Behavior | Kind | +| --- | --- | --- | +| `vortex-array/src/arrays/extension/compute/{filter,slice,take,mask}.rs` | Run on storage and wrap again | Structural | +| `vortex-array/src/arrays/dict/execute.rs:182`, `arrays/masked/execute.rs:178` | Run on storage and wrap again | Structural | +| `vortex-compressor/src/compressor/cascade.rs:175-205` | Compress storage and wrap again | Structural | +| `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/extension.rs` | Size of storage | Structural | +| `vortex-array/src/aggregate_fn/fns/is_constant/extension.rs` | Storage equality | Same storage implies same value. A `false` can be wrong for types with non-canonical storage. | +| `vortex-array/src/aggregate_fn/fns/all_non_distinct/extension.rs` | Storage equality | Same note as `is_constant` | +| `vortex-array/src/arrays/extension/compute/compare.rs` | Storage compare | Semantic (EXT-Q2) | +| `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193` | Storage compare, lone side | Semantic (EXT-Q1) | +| `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229` | Storage compare | Semantic (EXT-Q2) | +| `vortex-array/src/scalar/typed_view/extension/mod.rs:107-137` | Storage eq, order, hash | Semantic (EXT-Q2) | +| `vortex-array/src/aggregate_fn/fns/min_max/extension.rs` | Storage min and max | Semantic (EXT-A1) | +| `vortex-array/src/arrays/extension/compute/cast.rs:18-22` | Storage cast | Semantic (EXT-C2) | +| `vortex-array/src/scalar/cast.rs:56-61` | Storage cast and wrap | Semantic (EXT-C1) | +| `vortex-array/src/scalar/scalar_value.rs:90-95`, `builders/extension.rs:110` | Storage zero | Semantic (EXT-V3) | +| `vortex-layout/src/layouts/zoned/schema.rs:112-121` | Storage stats dtype | Semantic (EXT-A2) | +| `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | Storage export through cast | Semantic (EXT-B2) | +| `vortex-datafusion/src/convert/scalars.rs:138-145`, `schema.rs:105` | Storage conversion | Semantic (EXT-B1) | +| `vortex-python/src/scalar/into_py.rs:93-95` | Storage value | Semantic (EXT-B3) | +| `vortex-duckdb/src/table_function.rs:532-537` | Storage-based pushdown decision | Semantic (EXT-B4) | +| `vortex-jni/src/writer.rs:217` | Storage bounds | Semantic (EXT-B5) | +| `vortex-spatial/src/prune/mod.rs:100-104` | `ext_storage` on a statistic, in the owning crate | Correct use (explicit unwrap by the type owner) | + +## Appendix C: authoring cost + +These numbers come from a survey of every `ExtVTable` implementation. They are evidence for the +design, not defects. + +- The repository has 15 extension types outside tests (plus `ForeignExtDType`). +- `vortex-spatial` has about 5,800 lines of non-test source. Its eight `ExtVTable` implementations + are 241 lines (about 4%). Its Arrow import and export plugins are 1,132 lines. +- Authors wrote eight typed wrappers around `ExtensionArray` (`TemporalData` in + `vortex-array/src/arrays/datetime/mod.rs:44`, and `WellKnownBinaryData`, `PointData`, + `LineStringData`, `MultiPointData`, `PolygonData`, `MultiLineStringData`, `MultiPolygonData` in + `vortex-spatial`). +- Authors wrote several matchers (`AnyTemporal`, `AnyTensor`, `AnyVector`, `AnyFixedShapeTensor`) + and row input elements (`TensorRow`, `GeometryRow`). +- DuckDB, DataFusion and Python have no plugin hooks for extension types, so each new type needs + code in those crates. diff --git a/research/extension-types/agents/probe.rs b/research/extension-types/agents/probe.rs new file mode 100644 index 00000000000..35588c2233a --- /dev/null +++ b/research/extension-types/agents/probe.rs @@ -0,0 +1,520 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Throwaway probes for extension type behavior on `develop` at `5fc5cb8`. +//! +//! This file is not part of any crate, so nothing compiles it. To run it, copy it into +//! `vortex-array/tests/` and run: +//! +//! ```text +//! cargo test -p vortex-array --test probe -- --nocapture +//! ``` +//! +//! The build needs several GB of disk space. Appendix A of `findings.md` records the output of the +//! last run. The `probes` test reads rows with `execute_scalar` (the row path). The +//! `probes_array_path` test executes whole arrays with `RecursiveCanonical` (the array path). + +#![allow(clippy::all, clippy::disallowed_methods, unused)] + +use std::panic::AssertUnwindSafe; +use std::panic::catch_unwind; + +use vortex_array::ArrayParts; +use vortex_array::ArrayRef; +use vortex_array::EmptyArrayData; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::aggregate_fn::NumericalAggregateOpts; +use vortex_array::aggregate_fn::fns::is_sorted::is_sorted; +use vortex_array::aggregate_fn::fns::min_max::min_max; +use vortex_array::array_session; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::Extension; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::VarBinViewArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::extension::ExtensionSlots; +use vortex_array::builders::ArrayBuilder; +use vortex_array::builders::ExtensionBuilder; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::dtype::extension::ExtDTypeRef; +use vortex_array::dtype::extension::ExtId; +use vortex_array::dtype::extension::ExtVTable; +use vortex_array::extension::datetime::TimeUnit; +use vortex_array::extension::datetime::Timestamp; +use vortex_array::scalar::Scalar; +use vortex_array::scalar::ScalarValue; +use vortex_array::scalar_fn::fns::between::BetweenOptions; +use vortex_array::scalar_fn::fns::between::StrictComparison; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_buffer::BufferAllocatorRef; +use vortex_buffer::buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; + +/// u64 values where zero is not a valid value. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonZero; + +impl ExtVTable for NonZero { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_zero") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::U64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + let v = storage_value.as_primitive().cast::()?; + if v == 0 { + vortex_bail!("zero is not a valid NonZero value"); + } + Ok(v) + } +} + +/// An extension type that only accepts non-nullable storage. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonNullableOnly; + +impl ExtVTable for NonNullableOnly { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_nullable_only") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!( + ext_dtype.storage_dtype() == &DType::Primitive(PType::U64, Nullability::NonNullable) + ); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +/// An extension type with no meaningful order (think "opaque handle"). +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct Opaque; + +impl ExtVTable for Opaque { + type Metadata = EmptyMetadata; + type NativeValue<'a> = i64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.opaque") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::I64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +fn report(name: &str, f: impl FnOnce() -> VortexResult) { + match catch_unwind(AssertUnwindSafe(f)) { + Ok(Ok(v)) => println!("[{name}] OK: {v:?}"), + Ok(Err(e)) => println!( + "[{name}] ERR: {}", + e.to_string().lines().next().unwrap_or_default() + ), + Err(p) => { + let msg = p + .downcast_ref::() + .cloned() + .or_else(|| p.downcast_ref::<&str>().map(|s| s.to_string())) + .unwrap_or_default(); + println!( + "[{name}] PANIC: {}", + msg.lines().next().unwrap_or_default() + ) + } + } +} + +fn ts(unit: TimeUnit) -> ExtDTypeRef { + Timestamp::new(unit, Nullability::NonNullable).erased() +} + +fn ts_array(unit: TimeUnit, values: ArrayRef) -> ArrayRef { + ExtensionArray::new(ts(unit), values).into_array() +} + +fn display(a: &ArrayRef) -> VortexResult { + let mut ctx = array_session().create_execution_ctx(); + let mut out = vec![]; + for i in 0..a.len() { + out.push(a.execute_scalar(i, &mut ctx)?.to_string()); + } + Ok(format!("{} [{}]", a.dtype(), out.join(", "))) +} + +/// Fully materialize through the array kernels, then read the stored values. +fn display_array(a: &ArrayRef) -> VortexResult { + let mut ctx = array_session().create_execution_ctx(); + let canonical = a + .clone() + .execute::(&mut ctx)? + .0 + .into_array(); + display(&canonical) +} + +#[test] +fn probes_array_path() { + report("B' array path: PrimitiveArray i64 -> ts(ms)", || { + let a = buffer![1000i64, 2000].into_array(); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?) + }); + + report("D1' array path: ts(ms) -> ts(ns)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?) + }); + + report("D2' array path: constant ts(ms) -> ts(ns)", || { + let a = ConstantArray::new( + Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)), + 2, + ) + .into_array(); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?) + }); + + report("E1' array path: ts(ms) -> i64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display_array(&a.cast(DType::Primitive(PType::I64, Nullability::NonNullable))?) + }); + + report("E4' array path: Opaque -> NonZero", || { + let opaque = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(opaque, buffer![1i64, 2].into_array()).into_array(); + display_array(&a.cast(DType::Extension(nz))?) + }); + + report("E5' array path: i64 [0] -> NonZero (invalid value)", || { + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = buffer![0i64, 1].into_array(); + display_array(&a.cast(DType::Extension(nz))?) + }); + + report("F1' array path: ts(ms) array < i64 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = buffer![1500i64, 1500].into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F2' array path: ts(ms) array < i64 constant", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F3' array path: ts(ms) array < utf8 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F5' array path: ts(ms) array < ts(ns) array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ts_array(TimeUnit::Nanoseconds, buffer![1500i64, 1500].into_array()); + display_array(&a.binary(b, Operator::Lt)?) + }); +} + +#[test] +fn probes() { + let session = array_session(); + + // --- Casts -------------------------------------------------------------------------------- + + report("A scalar cast ts(ms)=1000 -> ts(ns)", || { + let s = Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)); + let c = s.cast(&DType::Extension(ts(TimeUnit::Nanoseconds)))?; + Ok(format!("{s} => {c}")) + }); + + report("B array cast PrimitiveArray i64 -> ts(ms)", || { + let a = buffer![1000i64, 2000].into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?; + display(&c) + }); + + report("C constant cast i64 -> ts(ms)", || { + let a = ConstantArray::new(Scalar::from(1000i64), 2).into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?; + display(&c) + }); + + report("D1 array cast ts(ms) -> ts(ns) (non-constant)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let c = a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?; + display(&c) + }); + + report("D2 array cast ts(ms) -> ts(ns) (constant)", || { + let a = ConstantArray::new( + Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)), + 2, + ) + .into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?; + display(&c) + }); + + report("E1 array cast ts(ms) -> i64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display(&a.cast(DType::Primitive(PType::I64, Nullability::NonNullable))?) + }); + + report("E2 array cast ts(ms) -> f64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display(&a.cast(DType::Primitive(PType::F64, Nullability::NonNullable))?) + }); + + report("E3 array cast ts(ms) -> u8 (narrowing)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + display(&a.cast(DType::Primitive(PType::U8, Nullability::NonNullable))?) + }); + + report("E4 array cast Opaque -> NonZero (different ext ids)", || { + let opaque = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(opaque, buffer![1i64, 2].into_array()).into_array(); + display(&a.cast(DType::Extension(nz))?) + }); + + // --- Comparisons -------------------------------------------------------------------------- + + report("F1 compare ts(ms) array < i64 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = buffer![1500i64, 1500].into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + report("F2 compare ts(ms) array < i64 constant", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + report("F3 compare ts(ms) array < utf8 array (typing)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + // Building the expression succeeds or fails here; executing it is in display(). + let r = a.binary(b, Operator::Lt)?; + display(&r) + }); + + report("F4 between ts(ms) with i64 bounds", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let lo = ConstantArray::new(Scalar::from(0i64), 2).into_array(); + let hi = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display(&a.between( + lo, + hi, + BetweenOptions { + lower_strict: StrictComparison::NonStrict, + upper_strict: StrictComparison::NonStrict, + }, + )?) + }); + + report("G compare Opaque < Opaque (no opt-in needed)", || { + let dt = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(dt.clone(), buffer![1i64, 5].into_array()).into_array(); + let b = ExtensionArray::new(dt, buffer![3i64, 3].into_array()).into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + // --- Aggregates --------------------------------------------------------------------------- + + report("H min_max Opaque (no opt-in needed)", || { + let dt = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(dt, buffer![5i64, 1, 3].into_array()).into_array(); + let mut ctx = session.create_execution_ctx(); + let r = min_max(&a, &mut ctx, NumericalAggregateOpts::default())?; + Ok(r.map(|r| format!("min={} max={}", r.min, r.max))) + }); + + report("I is_sorted on sorted ts(ms)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2, 3].into_array()); + let mut ctx = session.create_execution_ctx(); + is_sorted(&a, &mut ctx) + }); + + // --- Invalid values from "zero" / "default" paths ----------------------------------------- + + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased(); + + report("K1 Scalar::zero_value(NonZero)", || { + let z = Scalar::zero_value(&DType::Extension(nz.clone())); + Ok(format!("created {z}")) + }); + + report("K2 Scalar::try_new(NonZero, zero_value)", || { + let z = Scalar::zero_value(&DType::Extension(nz.clone())); + let (dtype, value) = z.into_parts(); + Ok(Scalar::try_new(dtype, value)?.to_string()) + }); + + report("L1 ExtensionBuilder(NonZero).append_zeros(1).finish()", || { + let mut b = ExtensionBuilder::new_in(nz.clone(), BufferAllocatorRef::static_ref()); + b.append_zeros(1); + let a = b.finish(); + Ok(format!("built array of len {}", a.len())) + }); + + report("L2 ... then execute_scalar(0)", || { + let mut b = ExtensionBuilder::new_in(nz.clone(), BufferAllocatorRef::static_ref()); + b.append_zeros(1); + let a = b.finish(); + display(&a) + }); + + report("L3 ExtensionArray::new(NonZero, [0]) (no value validation)", || { + let a = ExtensionArray::new(nz.clone(), buffer![0u64].into_array()).into_array(); + Ok(format!("built array of len {}", a.len())) + }); + + // --- Nullability changes ------------------------------------------------------------------ + + let nn = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased(); + + report("M1 take(NonNullableOnly, nullable indices)", || { + let a = ExtensionArray::new(nn.clone(), buffer![1u64, 2].into_array()).into_array(); + let idx = PrimitiveArray::from_option_iter([Some(0u32), None]).into_array(); + display(&a.take(idx)?) + }); + + report("M2 with_nullability(NonNullableOnly, Nullable)", || { + Ok(nn.with_nullability(Nullability::Nullable).to_string()) + }); + + // --- Misc --------------------------------------------------------------------------------- + + report("N Debug of ExtDTypeRef", || Ok(format!("{:?}", ts(TimeUnit::Milliseconds)))); + + report("O try_from_parts with utf8 storage under ts(ms)", || { + let storage = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + let parts = ArrayParts::new( + Extension, + DType::Extension(ts(TimeUnit::Milliseconds)), + 2, + EmptyArrayData, + ExtensionSlots { storage }.into_slots(), + ); + let a = ExtensionArray::try_from_parts(parts)?; + Ok(format!("accepted: dtype={} storage dtype={}", a.dtype(), a.storage_array().dtype())) + }); + + report("P arithmetic ts + ts", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + let b = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + display(&a.binary(b, Operator::Add)?) + }); + + report("Q Scalar::is_zero on ts(ms)=0", || { + let s = Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(0i64)); + Ok(s.is_zero()) + }); +} From 9b5c344118b207cca12043f6413f773aa8318fed Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 08:18:06 +0000 Subject: [PATCH 02/12] Tighten extension type research after a read-through Aligns the short-term nullability rule across the design documents, fixes the count of typed wrappers in vortex-spatial, and uses a checked multiply in the unit cast example. Signed-off-by: Claude Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb --- research/extension-types/01-why-opaque.md | 3 +++ research/extension-types/02-current-problems.md | 2 +- research/extension-types/03-design.md | 17 ++++++++++------- research/extension-types/agents/design-notes.md | 8 +++++--- 4 files changed, 19 insertions(+), 11 deletions(-) diff --git a/research/extension-types/01-why-opaque.md b/research/extension-types/01-why-opaque.md index f417ede029b..47eeac136ab 100644 --- a/research/extension-types/01-why-opaque.md +++ b/research/extension-types/01-why-opaque.md @@ -34,6 +34,9 @@ and often on the metadata: | Unit vector | `+` | Element sums | No, the sum is not a unit vector | | `Uuid` | `<` | Byte order | A design decision | +`Decimal` is a built-in dtype, but it has the same shape: an integer storage value plus a scale. The +[Decimal notes](../decimal/README.md) use it as a test of this design. + The meaning lives in the extension type, so the core cannot know which rows of this table are "yes". The core has two choices: diff --git a/research/extension-types/02-current-problems.md b/research/extension-types/02-current-problems.md index dd531302702..3dc1df83f4a 100644 --- a/research/extension-types/02-current-problems.md +++ b/research/extension-types/02-current-problems.md @@ -96,7 +96,7 @@ implementation. 5. **Nullability in the storage dtype.** Any nullability change validates the type again. 6. **Manual authoring.** The core has no typed array view, no explicit wrap expression, and no export hooks for DuckDB, DataFusion or Python. Extension crates fill the gaps by hand. - `vortex-spatial` has eight typed array wrappers, and its Arrow code is 4.7 times the size of its + `vortex-spatial` has seven typed array wrappers, and its Arrow code is 4.7 times the size of its type definitions.
diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md index 83b859318b7..e7331d3511e 100644 --- a/research/extension-types/03-design.md +++ b/research/extension-types/03-design.md @@ -54,7 +54,8 @@ how to answer. The answer is a rewrite into an expression over storage, or a rej ```text ts_ms < lit(Timestamp(ms) 1500) => ext_storage(ts_ms) < 1500i64 min(ts_ms) => ext_wrap(min(ext_storage(ts_ms)), Timestamp(ms)) -cast(ts_ms, Timestamp(ns)) => ext_wrap(ext_storage(ts_ms) * 1_000_000, Timestamp(ns)) +cast(ts_ms, Timestamp(ns)) => ext_wrap(checked_mul(ext_storage(ts_ms), 1_000_000), + Timestamp(ns)) json_a = json_b => rejected, because Json derives no equality ``` @@ -83,9 +84,9 @@ storage hash valid. `ORD` means that storage order is value order. | --- | --- | --- | | `Timestamp` | Same unit, and both with a time zone or both without | `EQ`, `ORD` | | `Date`, `Time` | Same unit | `EQ`, `ORD` | -| `Decimal` | Same scale and same storage width | `EQ`, `ORD` | +| `Decimal`, if it follows this design | Same scale and same storage width | `EQ`, `ORD` | | `Uuid` | Any | `EQ`, plus `ORD` if byte order is the chosen order | -| `Vector`, `FixedShapeTensor` | Same shape | `EQ` | +| `Vector`, `FixedShapeTensor` | Equal metadata | `EQ` | | `Json`, WKB, native geometry | Any | None | | Unknown type | Any | None | @@ -126,9 +127,10 @@ per-row check that each read does today. ### Nullability -The short-term rule: `validate_dtype` sees the storage dtype without nullability, so a nullability -change never validates the type again. The long-term option: store nullability on the extension -dtype, like every other `DType` variant, and keep the storage dtype non-nullable. +The short-term rule: the result of `validate_dtype` must not depend on nullability. Building a dtype +checks both nullabilities, so a later nullability change never validates the type again. The +long-term option: store nullability on the extension dtype, like every other `DType` variant, and +keep the storage dtype non-nullable. ### Authoring and boundaries @@ -163,7 +165,8 @@ plugins like the Arrow registry. Then a new extension type does not need code in Readers can convert them to storage values, but the switch needs a compatibility decision. - Open question: is `validate_array` a no-op by default, or a check of each value through `validate_scalar_value`? -- Open question: is the nullability contract enough, or is the structural change worth its cost? +- Open question: is the nullability contract enough, or does nullability move onto the extension + dtype? - Open question: does `Uuid` order by bytes, or have no order? - Open question: `Scalar` implements `PartialEq` and `Hash` for use in Rust code. Can these stay structural (same dtype and same storage value) if they are documented as identity, not SQL diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md index 06079d0aa1a..96dba437ba8 100644 --- a/research/extension-types/agents/design-notes.md +++ b/research/extension-types/agents/design-notes.md @@ -287,9 +287,11 @@ pub trait ExtVTable { ## 9. Nullability -- **N1, short term.** `ExtDType::try_with_vtable` calls `validate_dtype` with the non-nullable form - of the storage dtype. The documentation states that `validate_dtype` receives a non-nullable - storage dtype. `with_nullability` then never validates again. +- **N1, short term.** `ExtDType::try_with_vtable` calls `validate_dtype` for both nullabilities of + the storage dtype, and fails if either call fails. The documentation states that the result of + `validate_dtype` must not depend on nullability. `with_nullability` then never validates again. A + type that breaks the rule fails when its dtype is built, not later in a kernel. Calling + `validate_dtype` only with the non-nullable form is the alternative, but it hides such a type. - **N2, long term.** `ExtDType` stores a non-nullable storage dtype and a `Nullability`, like every other `DType` variant. The storage child has the dtype `storage_dtype.with_nullability(n)`. The wire format does not change, because the serialized storage dtype already carries nullability. From cf3699487f144d3bca5c562fd29b30203ed7e9cf Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 09:17:59 +0000 Subject: [PATCH 03/12] Add Decimal and type roadmap research Research notes on making Decimal an opaque wrapper over an integer child, the choice between an integer child and a FixedSizeBinary child, a wide-integer tier, FixedSizeBinary as its own type, and a dependency graph for the type work. The agents directory holds the full report and the call-site and PType inventories. The extension type notes now link to the Decimal notes, record the position of the wrapper relative to generic encodings as an open question, and state which parts of the Decimal work wait for the extension phases. Signed-off-by: Claude Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb --- research/README.md | 1 + research/decimal/01-current-state.md | 67 + research/decimal/02-design.md | 274 ++++ research/decimal/03-roadmap.md | 162 ++ research/decimal/README.md | 44 + research/decimal/agents/decimal-call-sites.md | 350 ++++ .../decimal/agents/ptype-match-inventory.md | 618 +++++++ research/decimal/agents/report.md | 1445 +++++++++++++++++ research/extension-types/03-design.md | 10 + research/extension-types/05-implementation.md | 15 +- research/extension-types/README.md | 5 + .../extension-types/agents/design-notes.md | 9 + 12 files changed, 2998 insertions(+), 2 deletions(-) create mode 100644 research/decimal/01-current-state.md create mode 100644 research/decimal/02-design.md create mode 100644 research/decimal/03-roadmap.md create mode 100644 research/decimal/README.md create mode 100644 research/decimal/agents/decimal-call-sites.md create mode 100644 research/decimal/agents/ptype-match-inventory.md create mode 100644 research/decimal/agents/report.md diff --git a/research/README.md b/research/README.md index 959c80bcef2..32fca7750f5 100644 --- a/research/README.md +++ b/research/README.md @@ -7,6 +7,7 @@ merged. | Topic | Summary for people | Detail for agents | | --- | --- | --- | | Extension types | [`extension-types/`](extension-types/README.md) | [`extension-types/agents/`](extension-types/agents/) | +| Decimal and the type roadmap | [`decimal/`](decimal/README.md) | [`decimal/agents/`](decimal/agents/) | Each summary document links to the agent documents. The agent documents hold file and line references, reproductions, fix sketches and file lists. diff --git a/research/decimal/01-current-state.md b/research/decimal/01-current-state.md new file mode 100644 index 00000000000..fe11dda73f9 --- /dev/null +++ b/research/decimal/01-current-state.md @@ -0,0 +1,67 @@ + + +# 1. Current state of `Decimal` + +All statements are for `develop` at `5fc5cb8`. Sections 2 and 3 of +[`agents/report.md`](agents/report.md) give file and line references for each one. + +## How it works + +- The dtype is `DType::Decimal(DecimalDType { precision, scale }, nullability)`. The precision is + 1 to 76. +- A decimal value is stored as an unscaled integer. The value of a row is + `unscaled * 10^(-scale)`. +- `DecimalArray` stores the unscaled integers in one buffer. The metadata field `values_type` + (`I8` to `I256`) gives the width of each value in the buffer. +- The width does not depend on the precision. The array docs allow precision 39 in an `i8` buffer, + and precision 2 in an `i256` buffer. +- The usual width is the smallest integer type that holds every value of the precision. It is `I8` + for precision 1 and 2, and `I256` for precision 39 to 76. These notes call it the canonical + width. + +## What is wrong + +| Problem | Effect | +| --- | --- | +| One dtype has up to six physical forms | Kernels must handle mixed widths, widen on the fly, or test whether a constant fits the storage. `DecimalArray::patch` panics when the width cannot hold the dtype. Issue [#5820](https://github.com/vortex-data/vortex/issues/5820) collects the fuzzer crashes | +| The canonical width is a convention, not a contract | 25 call sites compute it. The row codec uses it so that keys "from differently compressed chunks stay memcmp-comparable" | +| Arrays never check the precision | Scalars check it, but arrays do not. So out-of-range values can reach kernels, and a comment in the arithmetic kernel says so | +| The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel" | +| `Decimal` duplicates `Primitive` | It has its own canonical variant (51 dispatch sites), builder, compare, arithmetic and aggregates. Decimal-specific files hold about 7,600 non-test lines | +| DecimalByteParts carries the decimal dtype | It splits wide integers into 64-bit parts, but only decimals can use it | +| Unknown enum values decode silently | An unknown `values_type` decodes as `I8`. An unknown protobuf `PType` decodes as `U8`, and Variant metadata stores protobuf dtypes in files | +| Shared code has decimal special cases | Bloom filters, `IN` lists, size statistics, sparse arrays, run-end decoding and the row codec each have a decimal path | +| No 128-bit integer columns | DuckDB `HUGEINT` cannot be imported, although decimal storage already holds 128-bit values | + +The silent decoding causes no wrong reads today, because no writer emits unknown values. But it +blocks safe additions to both enums, so a fix is part of the urgent changes in +[part 3](03-roadmap.md#urgent-changes). + +## `Decimal` as a test of the extension design + +Each decimal operation is an integer operation plus bookkeeping for the scale. So `Decimal` has the +shape of an extension type over integer storage, and it is a good test of the +[extension design](../extension-types/03-design.md). It needs both kinds of behavior from that +design: operations that derive from the storage, and operations that the type must rewrite. + +| Operation | Same precision and scale | Different scales | +| --- | --- | --- | +| `=`, `<`, hash | The same operation on the unscaled integers | Rescale to a common scale first | +| `+`, `-` | An integer operation with overflow and precision checks | Rescale to a common scale first | +| `*`, `/` | A new result type, for example scale `s1 + s2` for `*` | A new result type | +| `min`, `max`, sort | The same operation on the unscaled integers | Not applicable | +| `sum` | An integer sum with a wider result precision | Not applicable | +| Casts | Owned by the type | Owned by the type | + +Equality and hashing need care. On `develop`, one value can have several physical widths, so the +storage bytes do not identify a value. For this reason, `DecimalValue` converts to `i256` before it +compares or hashes. With one storage width for each precision, a value has exactly one storage form +within one precision and scale. Then storage equality and storage hashing are value equality and +value hashing. + +Other pairs need a rewrite. `1.0` at scale 1 is stored as `10`, and `1.00` at scale 2 is stored as +`100`. So a comparison across scales rescales one side first. Two precisions with the same scale +can have different storage widths, so their comparison widens one side first. diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md new file mode 100644 index 00000000000..a65435c3d93 --- /dev/null +++ b/research/decimal/02-design.md @@ -0,0 +1,274 @@ + + +# 2. Design + +## Goal + +`Decimal` behaves like an extension type over integer storage, and it stays a built-in type. +Built-in means that every reader knows the type and that its wire tag never changes. It does not +fix the Rust representation. + +## The wrapper + +A decimal array becomes a wrapper with no buffers and one child: the unscaled integers. The child +dtype is a function of the precision, with the current canonical widths: + +```rust +impl DecimalDType { + /// The integer type that holds every unscaled value of this precision. + pub const fn storage_ptype(&self) -> PType { + match self.precision.get() { + 1..=2 => PType::I8, + 3..=4 => PType::I16, + 5..=9 => PType::I32, + 10..=18 => PType::I64, + 19..=38 => PType::I128, + _ => PType::I256, + } + } +} +``` + +The wrapper keeps three invariants: + +1. **Type.** The child dtype is `Primitive(storage_ptype(p), nullability)`. Construction checks it, + and the check costs nothing for each row. +2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. Imports from Arrow, + DuckDB, FFI and Python check it. Kernels that make new values check their own results. +3. **No data.** The wrapper owns no buffer. Validity comes from the child. + +A narrower physical width becomes an encoding of the child. For precision 1 to 18, the integer +compression schemes narrow the values better than a fixed smaller width does. Examples are frame of +reference, bit packing and dictionaries. For precision 19 and up, the child is a lazy widening of +an `i64` array when every value fits. Otherwise the child is a split into 64-bit parts, which is +DecimalByteParts in a general form. + +So a kernel never receives a width that it does not expect. The `patch` panic and the "does the +constant fit" special cases go away. + +## Decimal operations become integer operations + +Every decimal operation becomes an expression over the child: + +| Operation | Rewrite onto the unscaled integers | +| --- | --- | +| `=`, `<`, hash, with the same precision and scale | The same operation | +| `<` across scales | Rescale one side to the larger scale, then compare | +| `x < 1.005` with scale 2 | `unscaled(x) <= 100` | +| `+`, `-`, `*`, `/` | Widen, do a checked integer operation, then check the precision | +| Cast to `Decimal(p2, s2)` | Rescale with an exactness check, then check the precision | +| Cast to an integer | Divide by `10^s` toward zero, then do a range-checked integer cast | +| Cast to a float | Convert to `f64`, divide by `10^s`, then narrow. Scalars and arrays use one path | +| `sum` | Sum the integers at a wider width, then check the precision | +| `min`, `max`, `is_sorted` | Run the storage aggregate, then wrap the result | + +These rewrites need two integer kernels that do not exist yet. `rescale` multiplies or divides by a +power of ten with a rounding mode, and `check_range` fails on values outside a range. The unit +casts of `Timestamp`, `Time` and `Duration` need the same kernels, because their units are powers +of ten apart (except days). The current decimal arithmetic kernel does the operation and the +precision check in one pass. A fused integer kernel keeps that speed. + +## One wrapper abstraction for `Decimal` and extension types + +`Decimal` and `Extension` need the same structural behavior, but each one implements it separately +today. One trait can hold the shared part: + +```rust +/// A logical wrapper with no data over one storage child. +pub trait StorageWrapper: VTable { + fn storage(array: ArrayView<'_, Self>) -> &ArrayRef; + fn storage_dtype(dtype: &DType) -> VortexResult; + fn rewrap(array: ArrayView<'_, Self>, storage: ArrayRef) -> VortexResult; + fn wrap_scalar(dtype: &DType, storage: Scalar) -> VortexResult; +} +``` + +Generic rules then cover both types: + +- Slice, filter, take and mask push into the child and wrap the result again. +- `scalar_at` wraps the scalar of the child. +- Compression compresses the child with the ordinary schemes. +- Sparse, run-end, dictionary and chunked arrays work over the storage dtype. Today a sparse array + of an extension type cannot canonicalize at all, so this rule also closes a gap for extension + types. + +The wrapper stays on top of generic encodings, as in the +[extension design](../extension-types/03-design.md#one-normal-form). Type-specific encodings, such +as DecimalByteParts and `DateTimeParts`, live under the wrapper and see only the storage dtype. The +position of the wrapper is still an open question (see below). + +## Integer child or FixedSizeBinary child + +### Options + +| Option | Child | +| --- | --- | +| (a) Integer child | `Primitive(I8)` to `Primitive(I256)`, with two new primitive types | +| (b) FixedSizeBinary child | `FixedSizeBinary(n)`, a new core type | +| (c) Hybrid | Integers up to 64 bits, FixedSizeBinary above | +| (d) Separate wide-integer family | A new dtype and canonical array for 128 and 256 bits | +| (e) Current layout, opaque API | The current buffer, with the width fixed to the canonical width | + +### Byte order is not integer order + +FixedSizeBinary values order by byte comparison (memcmp). Arrow, DuckDB and Vortex store decimal +values as little-endian two's complement integers. Byte comparison of those bytes does not give +integer order: + +- `1` and `256` as 16-byte little-endian values: byte comparison says `256 < 1`. +- `-1` and `0`: byte comparison says `-1 > 0`. +- On 19,999 random pairs of 128-bit values, little-endian byte comparison agreed with integer order + 10,060 times. That is chance. + +Byte comparison gives integer order only for big-endian bytes with the sign bit flipped. The row +codec already uses that form. But that layout breaks zero-copy exchange with Arrow and DuckDB +decimals, and it needs a conversion before each arithmetic operation. With either layout, a +FixedSizeBinary child keeps every decimal kernel and adds a reinterpretation step. + +### Comparison + +| | (a) Integer child | (b) FixedSizeBinary child | (e) Current layout | +| --- | --- | --- | --- | +| Compare, arithmetic, aggregates | Generic integer kernels | Decimal-specific kernels | Decimal-specific kernels | +| Compression | All integer schemes up to 64 bits. Narrowing or a 64-bit split for wide values | Dictionaries and general-purpose schemes only | DecimalByteParts only | +| Min and max statistics | Correct | Wrong for little-endian bytes | Correct | +| Arrow and DuckDB decimals | Zero-copy | Zero-copy | Zero-copy | +| DuckDB `HUGEINT` | Possible | Not possible | Not possible | +| Decimal file format | No change | Needs FixedSizeBinary first | No change | +| Size of the change | Largest, because of the `PType` migration. The compiler finds most sites | A new core type | Smallest | +| Duplicated logic | Lowest | High | High, about 7,600 lines stay | + +### Verdict: an integer child + +Option (a) is the proposal, built as a separate wide-integer tier. The tier keeps the useful part of +option (d). Wide integers opt into a small set of kernels, and they do not act like 64-bit +integers. + +Wide integers do not need FixedSizeBinary: + +- Rust stores `i128` natively, and Vortex already has an `i256` type with checked arithmetic. +- The shared fixed-width layer (`FixedWidthArray`) already moves records of 16 and 32 bytes for + take and filter. +- Compression of wide integers is a split into 64-bit parts, which DecimalByteParts already does. +- Arrow export of decimals uses the Arrow decimal types. Arrow has no 128-bit integer type, so only + a top-level `I128` or `I256` column needs an export rule. `FixedSizeBinary(16)` is one option for + that rule, not a prerequisite. + +Three findings can change the verdict: + +- If maintainers reject wide integers as a Vortex type, option (e) is the fallback for one release. + It fixes the crashes, but `Decimal` stays a separate world. +- If benchmarks show that generic 128-bit and 256-bit kernels are clearly slower, the decimal + kernels stay as specializations behind the rewrites. This result is unlikely, because the current + decimal kernels already run on `i128` and `i256` values. +- If Vortex needs more than 76 digits or arbitrary precision, no fixed-width child works. + +## The wide-integer tier + +- `PType` gets `I128 = 11` and `I256 = 12`. The values 13 and 14 stay reserved for `U128` and + `U256`, which wait until a consumer needs them. +- Two typed sub-enums split the integers. `LaneIntPType` holds the eight integer types up to 64 + bits, and `WideIntPType` holds `I128` and `I256`. Offsets, codes, take indices and lane encodings + such as FastLanes accept only lane integers. +- `is_int()` goes away, so each of its 53 call sites must choose between "lane integer" and "any + integer". Most of them guard offsets, codes or indices. +- The dispatch macros that end in `other => panic!` retire, because a new type turns them into + run-time panics. Typed macros replace them. +- The compiler drives the migration. The sub-enums and macros come first, and the removal of + `is_int()` comes next. Then the macros migrate one crate at a time. The two new variants come + last, so every exhaustive match fails to compile until it handles them. +- The flatbuffer `PType` enum gets the two values, and old readers fail loudly on them. The + protobuf `PType` enum stays as it is, because old readers decode an unknown value as `U8` without + an error. A new protobuf `oneof` variant carries wide integers instead. An edition gates + top-level wide columns. +- Both wide types use 16-byte alignment. Two places assume that the alignment equals the byte + width, and they need a fix for `I256`. +- Wide integers get no FastLanes encodings, never act as offsets or codes, and get no CUDA dynamic + dispatch at first. + +Top-level wide columns need a rule in each engine: + +| Engine | `I128` | `I256` | +| --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)` | +| DuckDB | `HUGEINT` | Error | +| DataFusion | Error | Error | +| Python | Python `int` scalars | Python `int` scalars | + +## FixedSizeBinary on its own track + +FixedSizeBinary is a core type for opaque bytes of a fixed length, as RFC 0062 proposes. It refines +`Binary` in the same way that `FixedSizeList` refines `List`. + +- **Layout.** One buffer of `len * n` bytes, validity for whole values, and an explicit length so + that `n = 0` works. Take and filter come from `FixedWidthArray`. +- **Operations.** Equality and hashing by bytes, and byte order like `Binary` and Arrow. Explicit + casts to and from `Binary` and `FixedSizeList`. No arithmetic and no integer semantics. +- **UUID.** `Uuid` storage becomes `FixedSizeBinary(16)` in memory. Files keep the current + `FixedSizeList` storage until an edition allows the new form. A 128-bit lane can implement UUID + equality and hashing, but UUID order needs a separate decision. + +`Decimal`, wide integers and FixedSizeBinary share only the fixed-width structural layer. That +layer is the right place for shared code. PR [#8864](https://github.com/vortex-data/vortex/pull/8864) +started it. + +## The file format does not change + +The wrapper reads every existing decimal wire form: + +| Wire node | In memory | +| --- | --- | +| `vortex.decimal` at the canonical width | A wrapper over the buffer, zero-copy | +| `vortex.decimal` at a narrower width | A wrapper over a lazy widening, zero-copy | +| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at low precision | A wrapper over a lazy checked narrowing | +| `vortex.decimal` with an unknown width | An error, not a silent `I8` | +| DecimalByteParts v1 and v2 | DecimalByteParts at first, later a split of the wide child | + +Writes produce forms that old readers understand. A flat child becomes `vortex.decimal`. A +compressed child with a lane-integer type becomes DecimalByteParts v1, which the first core edition +already contains. A dictionary child becomes a dictionary of `vortex.decimal` values. So wide +`PType` values never reach the wire for decimal columns, and decimal statistics keep the `Decimal` +dtype. + +Before the reader changes, the compatibility fixtures need narrow, wide and Arrow-style widths. +Today they cover only `i32` and `i64`. + +## Does `Decimal` become an extension type? + +RFC 0062 proposes `Decimal` as a well-known extension type over signed-integer storage, with the +old wire tag kept. The wrapper above is the array side of that proposal. Only the location of the +Rust type is still open. + +The move is not safe today, because extension types inherit storage semantics. If `Decimal` +becomes an extension type today, a cast from `Decimal(10, 2)` to `i64` returns the unscaled +integers, and `price < 5` compares `500` with `5`. So the move waits for the extension fixes, the +cast hooks and the derives. After the decimal rewrites land, the move changes the API, not the +behavior. About 270 lines in 97 files mention `DType::Decimal`. + +The proposal builds the wrapper with `DType::Decimal` first, and it uses `Decimal` as an +acceptance test for the extension hooks. The decision on the move comes after the rewrites land. + +## Open questions + +- Open question: are `I128` and `I256` user-visible columns from the start, or storage only until + an engine needs them? DuckDB `HUGEINT` is the only current consumer. +- Open question: is the wrapper always on top of generic encodings such as `Constant` and `Dict`? + This design and the extension design say yes. Today `ExtensionConstantRule` puts `Constant` + above the extension wrapper, and the old decimal wire form puts `Dict` above `vortex.decimal`. +- Open question: does Vortex compare and compute across different precisions and scales? The + alternative keeps the rule of equal dtypes, and engines coerce first. +- Open question: does FixedSizeBinary get a new core wire tag, or an extension descriptor over + `FixedSizeList` that old readers can read? +- Open question: are the lazy widening and narrowing nodes cast arrays, or a small dedicated + encoding? +- Open question: do file reads check the decimal precision, or only imports? Zone-map minimum and + maximum statistics can make the check on file reads cheap. +- Open question: what is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does + not exist, and tests use -128. DuckDB rejects negative scales, and Arrow allows them. +- Open question: does the canonical width keep its `I8` minimum, or match an engine? The choice + affects only memory, so it can change later. +- Open question: after the rewrites land, is the move of `Decimal` into a Rust extension type worth + its API cost? diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md new file mode 100644 index 00000000000..4062e9ac6d1 --- /dev/null +++ b/research/decimal/03-roadmap.md @@ -0,0 +1,162 @@ + + +# 3. Roadmap + +## The type work + +Many planned type changes depend on each other. The IDs below match the full graph in section 7 +of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. + +| ID | Work | Kind | +| --- | --- | --- | +| N1 | Extension opacity fixes ([part 4](../extension-types/04-urgent-changes.md) of the extension notes) | Urgent | +| N2 | Bind-time rewrites for extension types | Research, then implementation | +| N3 | Pairwise derives | Implementation | +| N4 | Cast hooks for extension types ([#8570](https://github.com/vortex-data/vortex/issues/8570)) | Implementation | +| N5 | Array validation where data enters | Implementation | +| N6 | Nullability on the extension dtype | Research, optional | +| N7 | Shared fixed-width structural kernels | Implementation | +| N8 | Scaled-integer kernels: `rescale` and `check_range` | Implementation | +| N9 | Wide integers: `I128` and `I256` | Implementation | +| N10 | Wire safety: strict enum decoding, and edition gates for core dtypes | Implementation | +| N11 | Decimal storage at least as wide as the canonical width | Urgent | +| N12 | Decimal wrapper over an integer child | Implementation | +| N13 | Decimal operations as integer rewrites | Implementation | +| N14 | DecimalByteParts as a split of wide integers | Implementation, optional | +| N15 | `Decimal` as a well-known extension type in Rust | Implementation, later | +| N16 | Wire translation for well-known types | Research, then implementation | +| N17 | FixedSizeBinary core type | Implementation | +| N18 | `Uuid` on FixedSizeBinary | Implementation | +| N19 | `Utf8` as a well-known extension type | Research, low priority | +| N20 | `FixedSizeList` and `Map` as well-known extension types | Research, low priority | +| N21 | `Duration` and `Interval` extension types ([#10467](https://github.com/vortex-data/vortex/pull/10467), [#10468](https://github.com/vortex-data/vortex/pull/10468)) | In progress | +| N22 | Execution layouts (RFC 0062) | Research | +| N23 | Engine type plugins for DuckDB and DataFusion | Implementation | +| N24 | Refinement types (RFC 0055) | Research | +| N26 | Scalar value model (RFC 0056) | Research, then implementation | +| N27 | DuckDB `HUGEINT` import and export | Implementation | + +## Graph + +A solid line is a hard dependency: without the earlier work, the later work is wrong or unsafe. A +dashed line is a soft dependency: the earlier work makes the later work cheaper. This graph shows +all hard dependencies and the three most useful soft ones. The report lists every edge. + +```mermaid +flowchart TD + subgraph EXT["Extension types"] + N1["N1 Opacity fixes"] + N2["N2 Bind-time rewrites"] + N3["N3 Pairwise derives"] + N4["N4 Cast hooks"] + end + subgraph DEC["Decimal"] + N11["N11 Storage at least canonical width"] + N7["N7 Fixed-width kernels"] + N8["N8 Scaled-integer kernels"] + N9["N9 Wide integers"] + N12["N12 Wrapper over integer child"] + N13["N13 Operations as rewrites"] + N14["N14 Wide-integer split"] + end + subgraph BYTES["Bytes"] + N17["N17 FixedSizeBinary"] + N18["N18 Uuid on FixedSizeBinary"] + end + subgraph WIRE["Wire and engines"] + N10["N10 Wire safety"] + N16["N16 Wire translation"] + N27["N27 DuckDB HUGEINT"] + end + subgraph LATER["Later"] + N15["N15 Decimal as extension type"] + N19["N19 Utf8 as extension type"] + N20["N20 FixedSizeList and Map as extension types"] + N22["N22 Execution layouts"] + end + N21["N21 Duration and Interval"] + + N2 --> N3 + N1 --> N21 + N4 --> N21 + N11 --> N12 + N9 --> N12 + N12 --> N13 + N8 --> N13 + N9 --> N14 + N12 --> N14 + N9 --> N27 + N10 --> N27 + N17 --> N18 + N1 --> N15 + N3 --> N15 + N4 --> N15 + N13 --> N15 + N16 --> N15 + N16 --> N19 + N16 --> N20 + N22 --> N19 + N22 --> N20 + N7 -.-> N9 + N7 -.-> N17 + N8 -.-> N4 +``` + +The nodes N5, N6, N23, N24 and N26 have only soft edges, so the graph leaves them out. + +## Urgent changes + +These changes are small and independent. They are safe to make now: + +1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and + the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a + silent `I8` or `U8`. Readers must have this change before any writer emits a new value. +2. **Decimal storage at least as wide as the canonical width (N11).** `DecimalData::validate` + rejects narrower storage. DecimalByteParts assembly widens a narrow part, the compressor stops + writing narrowed `vortex.decimal` nodes, and readers widen old narrow nodes. Then the `patch` + panic and the "does the constant fit" special cases are unreachable for inputs with the same + dtype. This is option 3 of issue [#5820](https://github.com/vortex-data/vortex/issues/5820). + The compatibility fixtures for narrow, wide and Arrow-style widths come first. +3. **One owner for the width function.** `DecimalDType::storage_type()` replaces the 25 direct + calls to `smallest_decimal_value_type`. +4. **Extension opacity fixes (N1).** These are the urgent changes in + [part 4 of the extension notes](../extension-types/04-urgent-changes.md). + +## Order of work + +| Wave | Work | +| --- | --- | +| 0, urgent | Strict enum decoding, N11, one owner for the width function, and N1 | +| 1, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast hooks, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rewrites as acceptance tests. N23 engine plugins | +| 2, new types | N9 wide integers, without user-visible wide columns. N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary. N5 array validation | +| 3, decimal restructure | N12 wrapper and wire plugin. N13 rewrites, which delete most decimal kernels. N14, which is optional. N18 and N27 | +| 4, API unification | N16, then N15. N19 and N20 after N22 | + +The array-level fix for `Decimal` (N11, N12 and N13) does not wait for the extension work, because +the built-in wrapper owns its rewrites. Only the move to an extension type (N15) waits for N1 to N4. + +## Shared work between the tracks + +- The scaled-integer kernels (N8) serve decimal rescales and the unit casts of `Timestamp`, `Time` + and `Duration`. +- The decimal rewrites are acceptance tests for the extension binder (N2 and N3). Phase 1 of the + extension notes adds a decimal-like extension type over `i64`, and that type can run the same + tests against the built-in `Decimal`. +- The fixed-width structural kernels (N7) serve wide integers and FixedSizeBinary. +- The wrapper normal form and the `StorageWrapper` rules serve `Decimal` and every extension type. + `DateTimeParts` under the `Timestamp` wrapper is the temporal twin of DecimalByteParts under the + `Decimal` wrapper. + +## Risks + +| Risk | Mitigation | +| --- | --- | +| The `PType` migration touches 36 crates and several languages | Typed sub-enums, compiler-driven steps, and one crate in each PR | +| A leftover `other => panic!` arm panics on a wide type | Retire those macros, and fuzz decimals at every width | +| An old reader reads a protobuf wide `PType` as `U8` | A new protobuf `oneof` variant, and strict readers early | +| Rewritten arithmetic makes several passes over the data | Fused scaled-integer kernels, checked with the existing decimal and compare benchmarks | +| The public API breaks: `DecimalArray::buffer::()`, `values_type()` | Deprecation shims. The in-memory ID stays `vortex.decimal`, which the Python bindings use | +| Wide integers reach users before engines can export them | Edition gates, and an explicit export rule for each engine | diff --git a/research/decimal/README.md b/research/decimal/README.md new file mode 100644 index 00000000000..93910b0e75f --- /dev/null +++ b/research/decimal/README.md @@ -0,0 +1,44 @@ +# Decimal and the type roadmap + +`Decimal` is a built-in dtype with the shape of an extension type: an integer storage value plus a +scale. On `develop`, the decimal array does not keep to that shape. One decimal dtype can have six +physical forms, and `Decimal` duplicates much of `Primitive`. These notes propose an opaque +`Decimal` wrapper over an integer child, and they put the type work that it needs in order. + +| Part | Content | +| --- | --- | +| [1. Current state](01-current-state.md) | How `Decimal` works on `develop`, and what is wrong | +| [2. Design](02-design.md) | The wrapper, the child type, wide integers, FixedSizeBinary, and the open questions | +| [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the urgent changes, and the order of work | + +## Summary + +- The width of the decimal buffer does not depend on the precision. So one dtype has up to six + physical forms, every kernel must handle mixed widths, and some paths panic. +- Proposal: a decimal array is a wrapper with one child, the unscaled integers. The child dtype is + a function of the precision. A narrower physical width becomes an encoding of the child. +- The child is an integer array, not a FixedSizeBinary array. Each decimal operation is an integer + operation plus bookkeeping for the scale, and byte order is not integer order. +- Integer storage for precision 19 to 76 needs two new primitive types, `I128` and `I256`. They do + not need FixedSizeBinary. +- FixedSizeBinary is still a good addition, on its own track, for opaque bytes such as UUIDs. +- The decimal file format does not change. The wrapper reads and writes the existing wire forms. +- `Decimal` stays a built-in type. A move into a well-known extension type in Rust is possible + after the extension fixes land, and then it changes only the API. + +## Agent documents + +- [`agents/report.md`](agents/report.md): the full report. It has file and line references, API + sketches, the wire plan, the complete dependency graph with a reason for each edge, and the + evidence log. +- [`agents/decimal-call-sites.md`](agents/decimal-call-sites.md): every decimal-specific file with + line counts, every use of the width function, and every canonical dispatch site. +- [`agents/ptype-match-inventory.md`](agents/ptype-match-inventory.md): every `PType` dispatch + macro use, every exhaustive `PType` match, and every copy of `PType` in other languages and + formats. + +## Evidence + +The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. No Rust +code was built or run for this part. One short Python script checked byte order. Section 10 of the +report lists the claims that are not verified, which include every performance claim. diff --git a/research/decimal/agents/decimal-call-sites.md b/research/decimal/agents/decimal-call-sites.md new file mode 100644 index 00000000000..fd1cf9a40ec --- /dev/null +++ b/research/decimal/agents/decimal-call-sites.md @@ -0,0 +1,350 @@ + + +# Decimal call-site inventory + +Generated from `develop` at `5fc5cb8`. It supports `report.md` in the same directory. Line +counts split code from tests. The lines of a file after its first `#[cfg(test)]` count as tests, +and files named `tests.rs` count as tests. + +## 1. Decimal-specific files + +The first column is non-test lines, the second is test lines. "Fate under the wrapper design" uses +these labels: + +- **delete**: the logic becomes a rewrite to integer kernels or a shared wrapper rule. +- **shrink**: a smaller decimal-specific piece remains (dtype rules, rescaling, legacy wire). +- **keep**: the logic stays, possibly moved. +- **generalize**: the logic becomes a generic wide-integer component. + +```text + 410 353 encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs + 199 262 encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs + 106 149 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs + 36 98 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs + 151 173 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs + 21 67 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/filter.rs + 63 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs + 26 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/kernel.rs + 22 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/mask.rs + 13 121 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/mod.rs + 21 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/slice.rs + 30 73 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/take.rs + 25 38 encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs + 21 77 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/mod.rs + 0 324 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs + 85 0 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v1.rs + 112 0 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs + 0 177 encodings/decimal-byte-parts/src/decimal_byte_parts/prop_tests.rs + 19 0 encodings/decimal-byte-parts/src/decimal_byte_parts/rules.rs + 238 73 encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs + 34 0 encodings/decimal-byte-parts/src/decimal_byte_parts/testing.rs + 35 0 encodings/decimal-byte-parts/src/lib.rs + 42 0 vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs + 13 0 vortex-array/src/aggregate_fn/fns/is_constant/decimal.rs + 63 0 vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs + 82 0 vortex-array/src/aggregate_fn/fns/min_max/decimal.rs + 112 353 vortex-array/src/aggregate_fn/fns/sum/decimal.rs + 36 0 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs + 609 0 vortex-array/src/arrays/decimal/array.rs + 154 0 vortex-array/src/arrays/decimal/compute/between.rs + 490 751 vortex-array/src/arrays/decimal/compute/cast.rs + 86 135 vortex-array/src/arrays/decimal/compute/fill_null.rs + 36 0 vortex-array/src/arrays/decimal/compute/fixed_width.rs + 29 0 vortex-array/src/arrays/decimal/compute/mask.rs + 10 111 vortex-array/src/arrays/decimal/compute/mod.rs + 85 129 vortex-array/src/arrays/decimal/compute/rules.rs + 23 0 vortex-array/src/arrays/decimal/mod.rs + 144 0 vortex-array/src/arrays/decimal/utils.rs + 25 0 vortex-array/src/arrays/decimal/vtable/kernel.rs + 218 84 vortex-array/src/arrays/decimal/vtable/mod.rs + 31 70 vortex-array/src/arrays/decimal/vtable/operations.rs + 16 0 vortex-array/src/arrays/decimal/vtable/validity.rs + 326 114 vortex-array/src/builders/decimal.rs + 60 0 vortex-array/src/dtype/decimal/macros.rs + 623 0 vortex-array/src/dtype/decimal/max_precision.rs + 137 178 vortex-array/src/dtype/decimal/mod.rs + 135 16 vortex-array/src/dtype/decimal/precision.rs + 262 0 vortex-array/src/dtype/decimal/types.rs + 171 0 vortex-array/src/scalar/typed_view/decimal/arithmetic.rs + 294 0 vortex-array/src/scalar/typed_view/decimal/dvalue.rs + 40 0 vortex-array/src/scalar/typed_view/decimal/float_cast.rs + 63 0 vortex-array/src/scalar/typed_view/decimal/integer_cast.rs + 18 2 vortex-array/src/scalar/typed_view/decimal/mod.rs + 258 0 vortex-array/src/scalar/typed_view/decimal/scalar.rs + 0 1486 vortex-array/src/scalar/typed_view/decimal/tests.rs + 185 0 vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs + 468 0 vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs + 222 196 vortex-arrow/src/executor/decimal.rs + 167 38 vortex-btrblocks/src/schemes/decimal.rs + 68 46 vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs + 38 101 vortex-cuda/src/kernel/filter/decimal.rs + 131 120 vortex-duckdb/src/exporter/decimal.rs + 7637 5915 TOTAL (code, tests) +``` + +| File or group | Role today | Fate under the wrapper design | +| --- | --- | --- | +| `vortex-array/src/arrays/decimal/array.rs` | `DecimalData { decimal_dtype, values, values_type }`, constructors, `patch` | shrink: wrapper over one child | +| `vortex-array/src/arrays/decimal/vtable/*` | `vortex.decimal` vtable, `DecimalMetadata { values_type }` serde | shrink: wrapper vtable plus legacy plugin | +| `vortex-array/src/arrays/decimal/compute/cast.rs` | decimal to decimal, integer, and f64 casts, plus `upcast_decimal_values` | shrink: rescale rules over integer casts | +| `vortex-array/src/arrays/decimal/compute/{between,fill_null,mask,rules,fixed_width}.rs` | per-width kernels and storage-fit special cases | delete: wrapper rules plus integer kernels | +| `vortex-array/src/arrays/decimal/utils.rs` | `widened_buffer`, `converted_buffer`, `narrowed_decimal` | delete: no mixed widths in canonical form | +| `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs` | same-dtype compare with width widening | delete: storage compare | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs` | checked arithmetic at a working width | shrink: rescale and precision check over integer ops | +| `vortex-array/src/aggregate_fn/fns/*/decimal.rs` | sum, min/max, is_sorted, is_constant, all_non_distinct, size | delete: storage aggregates plus result wrapping | +| `vortex-array/src/builders/decimal.rs` | `DecimalBuilder` with six buffer variants | shrink: wrapper over `PrimitiveBuilder` | +| `vortex-array/src/dtype/decimal/*` | `DecimalDType`, `DecimalType`, `NativeDecimalType`, i256 bound tables | keep `DecimalDType`, and keep `DecimalType` only for legacy metadata | +| `vortex-array/src/scalar/typed_view/decimal/*` | `DecimalValue`, `DecimalScalar`, Arrow arithmetic rules | shrink: scalar view over a storage integer | +| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | generalize: wide-integer limb split under the wrapper | +| `vortex-arrow/src/executor/decimal.rs` | width adaptation to Decimal32/64/128/256 | keep, reads the child | +| `vortex-duckdb/src/exporter/decimal.rs` | width adaptation to DuckDB widths | keep, reads the child | +| `vortex-btrblocks/src/schemes/decimal.rs` | narrow then split | generalize: wide-integer scheme for the child | +| `vortex-cuda/src/kernel/*decimal*` | device filter and DBP | keep or generalize to wide-integer lanes | + +## 2. Uses of `DecimalType::smallest_decimal_value_type` + +This function is the de facto canonical width. Every site below already treats the width as a +function of precision. The wrapper design makes that rule the contract instead of a convention. +The first `types.rs` entry is the definition, so the list holds 25 call sites. + +```text +vortex-duckdb/src/convert/vector.rs:314 +vortex-array/src/dtype/dtype_impl.rs:320 +vortex-array/src/dtype/decimal/types.rs:46 +vortex-array/src/dtype/decimal/types.rs:73 +vortex-array/src/array/haystack/mod.rs:119 +vortex-array/src/builders/mod.rs:414 +vortex-array/src/scalar/arbitrary.rs:164 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209 +vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112 +vortex-array/src/arrays/constant/vtable/canonical.rs:97 +vortex-array/src/arrays/constant/vtable/canonical.rs:356 +vortex-array/src/arrays/decimal/compute/cast.rs:155 +vortex-array/src/arrays/arbitrary.rs:146 +vortex-array/src/arrays/primitive/compute/cast.rs:144 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416 +vortex-array/src/canonical.rs:188 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49 +vortex-cuda/src/kernel/arrays/constant.rs:73 +fuzz/src/array/sum/mod.rs:54 +vortex-row/src/codec.rs:583 +encodings/sparse/src/canonical.rs:316 +``` + +## 3. Sources that produce a `values_type` other than the canonical width + +| Source | Location | Width produced | +| --- | --- | --- | +| Compressor narrowing | `vortex-btrblocks/src/schemes/decimal.rs:133` calls `narrowed_decimal` | narrower than canonical. V1 mode returns the narrowed `vortex.decimal` array for I128 and I256 values (`:134-137`) | +| DBP assembly | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:83-93` | the MSP ptype (I8 to I64) for any precision | +| Arrow import | `vortex-arrow/src/convert.rs:204-277` | the Arrow width (32, 64, 128, 256 bits) for any precision | +| DuckDB import | `vortex-duckdb/src/convert/vector.rs:303-330` | DuckDB width, with I16 narrowed to I8 for p <= 2 | +| Zero-copy precision-widening cast | `vortex-array/src/arrays/decimal/compute/cast.rs:128-141` | the source width, which can exceed the target canonical width | +| User constructors | `DecimalArray::new::` (`vortex-array/src/arrays/decimal/array.rs:406`) | any `T` | + +## 4. Consumers that must handle mixed widths today + +| Consumer | Location | Workaround | +| --- | --- | --- | +| Compare | `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:114-126` | widen both sides to the wider width | +| Compare with constant | `.../compare/decimal.rs:128-153` | sign test when the constant does not fit the storage | +| Between | `vortex-array/src/arrays/decimal/compute/between.rs:48-126` | sign test when a bound does not fit the storage | +| Arithmetic | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:446-450` | widen mismatched pairs | +| Fill null | `vortex-array/src/arrays/decimal/compute/fill_null.rs:55-72` | upcast the whole array when the fill value does not fit | +| Patch | `vortex-array/src/arrays/decimal/array.rs:594-600` | panics when the storage cannot hold the dtype | +| all_non_distinct | `vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs` | triple-nested width dispatch | +| Row codec | `vortex-row/src/codec.rs:579-584` | keys use the canonical width so chunks stay comparable | +| Haystack (IN list) | `vortex-array/src/array/haystack/mod.rs:118-125` | converts to the canonical width | +| Size statistic | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | reports canonical width, not the stored width | +| Bloom filter | `vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:34-38` | unsupported, citing issue 5820 | + +## 5. Canonical dispatch sites that name `Canonical::Decimal` (non-test) + +Each site needs a decision when the decimal canonical form changes. With a wrapper, most of +them reduce to "push into the child" through a shared wrapper rule. + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:38 +vortex-duckdb/src/exporter/canonical.rs:30 +vortex-array/src/builders/decimal.rs:255 +vortex-array/src/arrays/constant/vtable/canonical.rs:110 +vortex-array/src/arrays/constant/vtable/canonical.rs:123 +vortex-array/src/arrays/masked/execute.rs:54 +vortex-array/src/arrays/dict/execute.rs:53 +vortex-array/src/arrays/filter/execute/mod.rs:141 +vortex-array/src/arrays/filter/execute/mod.rs:142 +vortex-array/src/scalar_fn/fns/cast/mod.rs:218 +vortex-array/src/scalar_fn/fns/fill_null/mod.rs:197 +vortex-array/src/canonical.rs:158 +vortex-array/src/canonical.rs:190 +vortex-array/src/canonical.rs:360 +vortex-array/src/canonical.rs:368 +vortex-array/src/canonical.rs:667 +vortex-array/src/canonical.rs:674 +vortex-array/src/canonical.rs:844 +vortex-array/src/canonical.rs:851 +vortex-array/src/canonical.rs:1202 +vortex-array/src/canonical.rs:1222 +vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs:516 +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:291 +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:387 +vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs:284 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:332 +vortex-array/src/aggregate_fn/fns/is_constant/mod.rs:421 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:457 +vortex-array/src/aggregate_fn/fns/sum/mod.rs:285 +vortex-array/src/aggregate_fn/fns/min_max/mod.rs:474 +vortex-cuda/src/kernel/arrays/dict.rs:294 +vortex-cuda/src/kernel/arrays/constant.rs:146 +vortex-cuda/src/kernel/arrays/constant.rs:182 +vortex-cuda/src/kernel/slice/mod.rs:53 +vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:63 +vortex-cuda/src/kernel/filter/decimal.rs:31 +vortex-cuda/src/kernel/filter/mod.rs:78 +vortex-cuda/src/arrow/canonical.rs:306 +vortex-cuda/src/canonical.rs:145 +vortex-cuda/src/canonical.rs:154 +vortex-btrblocks/src/schemes/decimal.rs:87 +fuzz/src/array/scalar_at.rs:49 +fuzz/src/array/sum/mod.rs:135 +fuzz/src/array/mask.rs:90 +fuzz/src/array/fill_null.rs:45 +fuzz/src/row.rs:370 +vortex-row/src/codec.rs:299 +vortex-row/src/codec.rs:410 +vortex-compressor/src/compressor/cascade.rs:123 +vortex-compressor/src/compressor/cascade.rs:124 +vortex-compressor/src/compressor/constant/mod.rs:80 +``` + +## 6. Shared files with decimal arms (non-test), with reference counts + +The count is the number of lines that mention a decimal type, array, value, or macro. + +```text +74 vortex-array/src/arrays/primitive/compute/cast.rs +31 vortex-array/src/canonical.rs +29 vortex-datafusion/src/convert/scalars.rs +28 vortex-cuda/src/arrow/canonical.rs +26 vortex-array/src/aggregate_fn/fns/sum/mod.rs +26 vortex-array/src/aggregate_fn/fns/mean/mod.rs +26 fuzz/src/array/fill_null.rs +19 vortex-array/src/scalar/proto.rs +19 vortex-array/src/arrays/constant/vtable/canonical.rs +19 vortex-array/src/aggregate_fn/fns/sum/constant.rs +18 vortex-arrow/src/scalar.rs +18 encodings/runend/src/array.rs +16 vortex-python/src/scalar/factory.rs +15 vortex-row/src/codec.rs +15 vortex-cuda/src/kernel/arrays/dict.rs +13 vortex-ffi/src/scalar.rs +13 encodings/sparse/src/canonical.rs +12 vortex-duckdb/src/convert/vector.rs +12 vortex-arrow/src/convert.rs +12 vortex-array/src/scalar_fn/fns/between/mod.rs +12 vortex-array/src/scalar/convert/into_scalar.rs +11 vortex-jni/src/expression.rs +10 vortex-duckdb/src/convert/scalar.rs +9 vortex-cuda/src/arrow/mod.rs +9 vortex-array/src/aggregate_fn/fns/min_max/mod.rs +9 vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs +8 vortex-cuda/src/kernel/arrays/constant.rs +8 vortex-array/src/compute/conformance/binary_numeric.rs +8 fuzz/src/array/sum/mod.rs +8 fuzz/src/array/mask.rs +7 vortex-array/src/stats/stats_set.rs +7 vortex-array/src/scalar/downcast.rs +7 vortex-array/src/dtype/dtype_impl.rs +7 vortex-array/src/array/haystack/mod.rs +7 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs +7 fuzz/src/array/take.rs +6 vortex-ffi/src/dtype.rs +6 vortex-array/src/scalar/validate.rs +6 vortex-array/src/scalar/arbitrary.rs +6 vortex-array/src/arrays/filter/execute/take/fixed_width.rs +6 vortex-array/benches/binary_ops.rs +6 fuzz/src/array/sort.rs +6 fuzz/src/array/slice.rs +6 fuzz/src/array/search_sorted.rs +6 fuzz/src/array/filter.rs +6 fuzz/src/array/compare.rs +6 encodings/runend/src/compress.rs +5 vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_v2.rs +5 vortex-duckdb/src/convert/dtype.rs +5 vortex-array/src/scalar_fn/fns/binary/compare/nested.rs +5 vortex-array/src/scalar/typed_view/primitive/scalar.rs +5 fuzz/src/array/scalar_at.rs +4 vortex-test/e2e-cuda/src/lib.rs +4 vortex-python/src/scalar/into_py.rs +4 vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs +4 vortex-cuda/src/canonical.rs +4 vortex-array/src/scalar_fn/fns/fill_null/mod.rs +4 vortex-array/src/scalar/scalar_value.rs +4 vortex-array/src/dtype/serde/proto.rs +4 vortex-array/src/builders/mod.rs +4 vortex-array/src/arrays/masked/execute.rs +4 vortex-array/src/arrays/dict/execute.rs +4 vortex-array/src/arrays/constant/compute/cast.rs +4 vortex-array/src/arrays/chunked/compute/mod.rs +4 vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs +4 vortex-array/src/aggregate_fn/fns/is_constant/mod.rs +4 fuzz/src/row.rs +3 vortex-duckdb/src/duckdb/logical_type.rs +3 vortex-cuda/src/kernel/filter/mod.rs +3 vortex-compressor/src/compressor/constant/mod.rs +3 vortex-arrow/src/dtype.rs +3 vortex-arrow/benches/to_arrow.rs +3 vortex-array/src/scalar_fn/fns/cast/mod.rs +3 vortex-array/src/scalar_fn/fns/case_when.rs +3 vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs +3 vortex-array/src/scalar/constructor.rs +3 vortex-array/src/arrays/constant/vtable/mod.rs +3 vortex-array/src/arrays/arbitrary.rs +3 vortex-array/benches/filter_fixed_width.rs +3 vortex-array/benches/expr/case_when_bench.rs +3 vortex-array/benches/cast_primitive.rs +2 vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs +2 vortex-python/src/arrays/native.rs +2 vortex-layout/src/layouts/zoned/aggregates/bloom_filter/scalar_fn.rs +2 vortex-compressor/src/compressor/cascade.rs +2 vortex-array/src/scalar_fn/fns/binary/mod.rs +2 vortex-array/src/scalar/scalar_impl.rs +2 vortex-array/src/flatbuffers/generated/dtype.rs +2 vortex-array/src/dtype/serde/serde.rs +2 vortex-array/src/compute/conformance/consistency.rs +2 vortex-array/src/arrays/validation_tests.rs +2 vortex-array/src/arrays/struct_/compute/cast.rs +2 vortex-array/src/arrays/mod.rs +2 vortex-array/src/arrays/filter/execute/mod.rs +2 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs +2 vortex-array/benches/compare.rs +1 vortex-python/src/scalar/mod.rs +1 vortex-python/src/python_repr.rs +1 vortex-python/src/dtype/mod.rs +1 vortex-python/src/dtype/factory.rs +1 vortex-layout/src/layouts/zoned/zone_map.rs +1 vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs +1 vortex-jni/src/writer.rs +1 vortex-file/src/writer.rs +1 vortex-duckdb/src/table_function.rs +1 vortex-duckdb/src/exporter/canonical.rs +1 vortex-datafusion/src/convert/stats/mod.rs +1 vortex-cuda/src/kernel/slice/mod.rs +1 vortex-array/src/scalar_fn/fns/binary/compare/mod.rs +1 vortex-array/src/scalar/typed_view/mod.rs +1 vortex-array/src/scalar/display.rs +1 vortex-array/src/scalar/convert/from_scalar.rs +1 vortex-array/src/scalar/cast.rs +1 vortex-array/src/dtype/serde/flatbuffers.rs +1 vortex-array/src/dtype/arbitrary/mod.rs +1 vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs +1 fuzz/src/array/mod.rs +``` diff --git a/research/decimal/agents/ptype-match-inventory.md b/research/decimal/agents/ptype-match-inventory.md new file mode 100644 index 00000000000..9bef4ef1fbd --- /dev/null +++ b/research/decimal/agents/ptype-match-inventory.md @@ -0,0 +1,618 @@ + + +# PType blast-radius inventory + +Generated from `develop` at `5fc5cb8` with `grep`. Paths are relative to the repository root. A +site marked `[test]` is in a test file or a `tests` module file. This file supports `report.md` +in the same directory. + +## Summary of dispatch macros + +| Macro | Uses | Files | Crates | +| --- | ---: | ---: | --- | +| `match_each_native_ptype!` | 70 | 54 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb vortex-row | +| `match_each_integer_ptype!` | 169 | 94 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb vortex-layout vortex-python vortex-row | +| `match_each_unsigned_integer_ptype!` | 118 | 53 | encodings vortex-array vortex-cuda vortex-duckdb | +| `match_each_signed_integer_ptype!` | 3 | 2 | encodings vortex-array | +| `match_each_float_ptype!` | 9 | 9 | vortex-array vortex-duckdb vortex-layout vortex-tensor | +| `match_each_native_simd_ptype!` | 5 | 5 | vortex-cuda vortex-duckdb | +| `match_each_alp_float_ptype!` | 7 | 6 | encodings vortex-cuda | +| `match_each_pvalue!` | 5 | 3 | encodings | +| `match_each_decimal_value_type!` | 77 | 49 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb | +| `match_each_decimal_value!` | 9 | 8 | encodings vortex-array vortex-python | + +## Trait bound mentions + +| Trait | Mentions | Files | +| --- | ---: | ---: | +| `NativePType` | 598 | 169 | +| `IntegerPType` | 169 | 49 | +| `UnsignedPType` | 124 | 23 | +| `PhysicalPType` | 29 | 6 | +| `OffsetBuilderPType` | 68 | 18 | +| `NativeDecimalType` | 155 | 40 | + +## PType helper call sites + +| Method | Call sites | +| --- | ---: | +| `is_int()` | 53 | +| `is_signed_int()` | 25 | +| `is_unsigned_int()` | 31 | +| `is_float()` | 44 | +| `to_signed()` | 14 | +| `to_unsigned()` | 63 | +| `max_value_as_u64()` | 14 | + +## Wire, FFI, and language mirrors of the PType enum + +Each of these must change, or explicitly reject the new variants, when PType gains variants. + +```text +vortex-array/flatbuffers/vortex-dtype/dtype.fbs:15: F64, +vortex-array/proto/dtype.proto:22: F64 = 10; +vortex-ffi/cinclude/vortex.h:168: PTYPE_F64 = 10, +vortex-cuda/kernels/src/dynamic_dispatch.h:55: PTYPE_F64 = 9, +vortex-cuda/kernels/src/dynamic_dispatch.h:75: case PTYPE_F64: +vortex-cuda/kernels/src/dynamic_dispatch.h:96: case PTYPE_F64: +lang/cpp/include/vortex/dtype.hpp:55: F64 = PTYPE_F64, +lang/cpp/include/vortex/common.hpp:102: return PTYPE_F64; +vortex-python/python/vortex/_lib/dtype.pyi:65: F64: PType +vortex-cuda/src/dynamic_dispatch/mod.rs:57:pub fn ptype_to_tag(ptype: PType) -> PTypeTag { +vortex-cuda/src/dynamic_dispatch/mod.rs:74:pub fn tag_to_ptype(tag: PTypeTag) -> PType { +vortex-ffi/src/ptype.rs:35:impl From for PType { +vortex-ffi/src/ptype.rs:53:impl From for vx_ptype { +vortex-python/src/dtype/ptype.rs:64:impl From for PyPType { +vortex-python/src/dtype/ptype.rs:82:impl From for PType { +vortex-array/src/dtype/serde/flatbuffers.rs:525:impl From for fb::PType { +vortex-array/src/dtype/serde/flatbuffers.rs:543:impl TryFrom for PType { +vortex-array/src/dtype/serde/proto.rs:229:impl From for PType { +vortex-array/src/dtype/serde/proto.rs:248:impl From for pb::PType { +vortex-duckdb/src/convert/dtype.rs:100: DUCKDB_TYPE::DUCKDB_TYPE_HUGEINT => vortex_bail!("I128 is not in Vortex type system"), +vortex-duckdb/src/convert/dtype.rs:101: DUCKDB_TYPE::DUCKDB_TYPE_UHUGEINT => vortex_bail!("U128 is not in Vortex type system"), +``` + +## Exhaustive PType matches outside ptype.rs (proxy: an arm naming PType::F64) + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:39: PType::F16 | PType::F32 | PType::F64 => { +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:27: PType::F16 | PType::F32 | PType::F64 => accumulate_primitive_float(array, partial, ctx)?, +vortex-array/src/dtype/serde/proto.rs:262: PType::F64 => F64, +vortex-array/src/dtype/serde/flatbuffers.rs:538: PType::F64 => Self::F64, +vortex-array/src/dtype/serde/flatbuffers.rs:558: fb::PType::F64 => Self::F64, +vortex-array/src/dtype/decimal/types.rs:259: | PType::F64) => vortex_bail!("cannot convert ptype {p} to DecimalType"), +vortex-array/src/compute/conformance/consistency.rs:894: PType::F64 => return None, [test] +vortex-array/src/compute/conformance/consistency.rs:955: PType::F64 => { [test] +vortex-array/src/compute/conformance/cast.rs:64: PType::F16 | PType::F32 | PType::F64 => test_cast_from_floating_point_types(array, ctx), [test] +vortex-array/src/compute/conformance/binary_numeric.rs:137: PType::F64 => test_binary_numeric_conformance::(array, ctx), [test] +vortex-array/src/compute/conformance/binary_numeric.rs:376: PType::F64 => test_binary_numeric_edge_cases_float::(array, ctx), [test] +vortex-array/src/array/haystack/mod.rs:412: PType::F64 => PType::U64, +vortex-array/src/builders/tests.rs:804: PType::F64 => Scalar::primitive(i as f64 * 1.5, *n), [test] +vortex-array/src/scalar/proto.rs:312: ftype @ (PType::F16 | PType::F32 | PType::F64) => vortex_bail!( +vortex-array/src/scalar/proto.rs:345: ftype @ (PType::F32 | PType::F64) => vortex_bail!( +vortex-array/src/scalar/arbitrary.rs:152: PType::F64 => PValue::F64(u.arbitrary()?), +vortex-array/src/scalar/typed_view/decimal/tests.rs:542: PType::F16 | PType::F32 | PType::F64 => panic!("Unexpected type {ptype}"), [test] +vortex-array/src/scalar/typed_view/decimal/scalar.rs:116: PType::F64 => Scalar::primitive(actual_value, *nullability), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:256: PType::F64 => PValue::F64(0.0), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:347: PType::F64 => f64::from_bits(*v).into(), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:363: PType::F64 => f64::from_bits(i64::cast_unsigned(*v)).into(), +vortex-array/src/arrays/listview/tests/operations.rs:281: PType::F64 => buffer![1f64, 2., 3., 4.].into_array(), [test] +vortex-array/src/arrays/listview/tests/operations.rs:290: PType::F64 => (buffer![0u32, 2].into_array(), buffer![2u32, 2].into_array()), [test] +vortex-array/src/arrays/bool/compute/cast.rs:155: PType::F64 => buffer![1f64, 0., 1.].into_array(), +vortex-array/src/arrays/arbitrary.rs:142: PType::F64 => random_primitive::(u, *n, chunk_len), +vortex-array/src/arrays/primitive/compute/cast.rs:346: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/arrays/primitive/compute/cast.rs:360: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/arrays/primitive/compute/cast.rs:374: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:96: DType::Primitive(PType::U64 | PType::I64 | PType::F64, _) +vortex-array/src/aggregate_fn/fns/sum/mod.rs:123: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/aggregate_fn/fns/sum/mod.rs:343: PType::F16 | PType::F32 | PType::F64 => SumState::Float(0.0), +vortex-array/src/aggregate_fn/fns/min_max/mod.rs:169: PType::F64 => Scalar::primitive(f64::NAN, Nullability::NonNullable), +vortex-cuda/src/dynamic_dispatch/mod.rs:69: PType::F64 => PTypeTag_PTYPE_F64, +vortex-cuda/src/dynamic_dispatch/mod.rs:393: PType::U64 | PType::I64 | PType::F64 => PType::U64, +vortex-cuda/src/dynamic_dispatch/plan_builder.rs:91: return matches!(arr.dtype().as_ptype(), PType::F32 | PType::F64); +vortex-btrblocks/src/schemes/float/alprd.rs:72: PType::F64 => RDEncoder::new(primitive_array.as_slice::()), +vortex-ffi/src/ptype.rs:66: PType::F64 => vx_ptype::PTYPE_F64, +fuzz/src/row.rs:351: PType::F64 => a +vortex-python/src/dtype/ptype.rs:77: PType::F64 => Self::F64, +vortex-python/src/dtype/ptype.rs:95: PyPType::F64 => PType::F64, +vortex-python/src/scalar/into_py.rs:54: PType::F64 => p.typed_value::().into_pyobject(py), +vortex-python/src/python_repr.rs:51: PType::F16 | PType::F32 | PType::F64 => { +vortex-arrow/src/scalar.rs:109: PType::F64 => value_to_arrow_scalar!(scalar.typed_value(), Float64Array), +vortex-arrow/src/dtype.rs:331: PType::F64 => DataType::Float64, +vortex-arrow/src/convert.rs:1111: PType::F64 => ( +vortex-datafusion/src/convert/scalars.rs:54: PType::F64 => ScalarValue::Float64(pscalar.typed_value::()), +vortex-compressor/src/stats/float.rs:116: PType::F64 => typed_float_stats::(input, opts.count_distinct_values, ctx), +encodings/pco/src/array.rs:288: PType::F64 => NumberType::F64, +encodings/alp/src/alp/compress.rs:38: PType::F64 => { +encodings/alp/src/alp/compress.rs:54: PType::F64 => alp_encode_components_typed::(parray, exponents, ctx)?, +``` + +## Sites of `match_each_native_ptype!` + +```text +vortex-duckdb/src/exporter/primitive.rs:34 +vortex-array/src/dtype/ptype.rs:768 +vortex-array/src/dtype/ptype.rs:781 +vortex-array/src/builders/mod.rs:406 +vortex-array/src/builders/dict/mod.rs:65 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:96 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:175 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:357 +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:212 +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:220 +vortex-array/src/arrays/patched/compute/compare.rs:68 +vortex-array/src/arrays/patched/compute/take.rs:49 +vortex-array/src/arrays/patched/array.rs:284 +vortex-array/src/arrays/patched/vtable/mod.rs:232 +vortex-array/src/arrays/patched/vtable/mod.rs:290 +vortex-array/src/arrays/constant/vtable/mod.rs:209 +vortex-array/src/arrays/constant/vtable/canonical.rs:81 +vortex-array/src/arrays/constant/vtable/canonical.rs:341 +vortex-array/src/arrays/list/array.rs:304 +vortex-array/src/arrays/bool/compute/cast.rs:68 +vortex-array/src/arrays/interleave/execute/primitive.rs:42 +vortex-array/src/arrays/primitive/compute/cast.rs:119 +vortex-array/src/arrays/primitive/compute/cast.rs:120 +vortex-array/src/arrays/primitive/compute/between.rs:39 +vortex-array/src/arrays/primitive/compute/zip.rs:64 +vortex-array/src/arrays/primitive/compute/fill_null.rs:32 +vortex-array/src/arrays/primitive/array/patch.rs:34 +vortex-array/src/arrays/primitive/array/mod.rs:491 +vortex-array/src/arrays/primitive/array/top_value.rs:31 +vortex-array/src/arrays/primitive/vtable/operations.rs:26 +vortex-array/src/arrays/primitive/vtable/mod.rs:197 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:40 +vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs:43 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:165 +vortex-array/src/scalar_fn/fns/binary/numeric/row.rs:77 +vortex-array/src/canonical.rs:180 +vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs:20 +vortex-array/src/aggregate_fn/fns/sum_v2/grouped.rs:89 +vortex-array/src/aggregate_fn/fns/all_non_distinct/primitive.rs:15 +vortex-array/src/aggregate_fn/fns/is_constant/primitive.rs:54 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:44 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:49 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:54 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:140 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:153 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:166 +vortex-array/src/aggregate_fn/fns/sum/grouped.rs:83 +vortex-array/src/aggregate_fn/fns/min_max/primitive.rs:27 +vortex-cuda/src/kernel/encodings/runend.rs:75 +vortex-cuda/src/kernel/encodings/runend.rs:90 +fuzz/src/array/scalar_at.rs:45 +fuzz/src/array/filter.rs:61 +fuzz/src/array/sort.rs:45 +fuzz/src/array/slice.rs:50 +fuzz/src/array/search_sorted.rs:86 +fuzz/src/array/compare.rs:74 +fuzz/src/array/fill_null.rs:115 +fuzz/src/array/take.rs:83 +vortex-row/src/codec.rs:774 +vortex-row/src/codec.rs:1297 +vortex-compressor/src/compressor/constant/mod.rs:62 +encodings/sequence/src/compute/is_sorted.rs:39 +encodings/sparse/src/canonical.rs:310 +encodings/runend/src/compress.rs:70 +encodings/runend/src/compress.rs:79 +encodings/runend/src/compress.rs:203 +encodings/runend/src/compute/sum/kernel.rs:100 +encodings/fastlanes/src/rle/array/rle_decompress.rs:40 +encodings/fastlanes/src/rle/array/rle_compress.rs:37 +encodings/pco/src/probe.rs:135 +``` + +## Sites of `match_each_integer_ptype!` + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:48 +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:67 +vortex-duckdb/src/exporter/dict.rs:99 +vortex-duckdb/src/exporter/run_end.rs:57 +vortex-duckdb/src/exporter/list.rs:91 +vortex-duckdb/src/exporter/list_view.rs:129 +vortex-duckdb/src/exporter/list_view.rs:130 +vortex-array/src/array/haystack/mod.rs:113 +vortex-array/src/builders/listview.rs:342 +vortex-array/src/builders/listview.rs:428 +vortex-array/src/builders/listview.rs:446 +vortex-array/src/builders/listview.rs:598 +vortex-array/src/builders/list.rs:224 +vortex-array/src/builders/list.rs:285 +vortex-array/src/builders/list.rs:286 +vortex-array/src/builders/dict/bytes.rs:262 +vortex-array/src/scalar/typed_view/decimal/scalar.rs:85 +vortex-array/src/arrays/listview/rebuild.rs:349 +vortex-array/src/arrays/listview/compute/take.rs:56 +vortex-array/src/arrays/listview/compute/take.rs:59 +vortex-array/src/arrays/listview/array.rs:380 +vortex-array/src/arrays/listview/array.rs:401 +vortex-array/src/arrays/listview/conversion.rs:54 +vortex-array/src/arrays/listview/conversion.rs:123 +vortex-array/src/arrays/list/compute/filter.rs:165 +vortex-array/src/arrays/list/array.rs:225 +vortex-array/src/arrays/bool/compute/take.rs:70 +vortex-array/src/arrays/decimal/compute/cast.rs:100 +vortex-array/src/arrays/decimal/compute/cast.rs:633 +vortex-array/src/arrays/decimal/compute/cast.rs:661 +vortex-array/src/arrays/varbin/compute/filter.rs:67 +vortex-array/src/arrays/varbin/compute/filter.rs:174 +vortex-array/src/arrays/varbin/compute/compare.rs:64 +vortex-array/src/arrays/varbin/compute/compare.rs:70 +vortex-array/src/arrays/varbin/compute/compare.rs:77 +vortex-array/src/arrays/varbin/compute/take.rs:176 +vortex-array/src/arrays/varbin/array.rs:300 +vortex-array/src/arrays/varbin/array.rs:364 +vortex-array/src/arrays/varbin/vtable/mod.rs:255 +vortex-array/src/arrays/varbin/vtable/canonical.rs:29 +vortex-array/src/arrays/varbin/builder.rs:258 +vortex-array/src/arrays/varbin/builder.rs:359 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:122 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:240 +vortex-array/src/arrays/primitive/compute/cast.rs:168 +vortex-array/src/arrays/primitive/array/patch.rs:33 +vortex-array/src/arrays/dict/compute/fill_null.rs:67 +vortex-array/src/arrays/dict/array.rs:159 +vortex-array/src/arrays/dict/array.rs:163 +vortex-array/src/arrays/dict/vtable/mod.rs:296 +vortex-array/src/arrays/dict/vtable/mod.rs:330 +vortex-array/src/arrays/varbinview/compute/take.rs:55 +vortex-array/src/arrays/null/compute/take.rs:27 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:42 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:67 +vortex-array/src/arrays/filter/execute/take/rank.rs:26 +vortex-array/src/arrays/filter/execute/take/rank.rs:36 +vortex-array/src/arrays/filter/execute/take/rank.rs:46 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:264 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:779 +vortex-array/src/scalar_fn/unstable/row/types/element/utf8.rs:202 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:81 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:82 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:117 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:118 +vortex-array/src/aggregate_fn/accumulator_grouped.rs:386 +vortex-cuda/benches/fsst_cuda.rs:76 [test] +vortex-cuda/src/kernel/arrays/dict.rs:88 +vortex-cuda/src/kernel/arrays/dict.rs:110 +vortex-cuda/src/kernel/arrays/dict.rs:237 +vortex-cuda/src/kernel/arrays/dict.rs:342 +vortex-cuda/src/kernel/arrays/list.rs:87 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:113 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:114 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:115 +vortex-cuda/src/kernel/encodings/onpair.rs:192 +vortex-cuda/src/kernel/encodings/onpair.rs:421 +vortex-cuda/src/kernel/encodings/onpair.rs:649 +vortex-cuda/src/kernel/encodings/bitpacked.rs:129 +vortex-cuda/src/kernel/encodings/fsst.rs:144 +vortex-cuda/src/kernel/encodings/fsst.rs:390 +vortex-cuda/src/kernel/encodings/for_.rs:74 +vortex-cuda/src/kernel/encodings/for_.rs:86 +vortex-cuda/src/arrow/offsets.rs:44 +vortex-cuda/src/arrow/list_view.rs:143 +vortex-cuda/src/arrow/list_view.rs:144 +vortex-cuda/src/arrow/list_view.rs:470 +vortex-cuda/src/arrow/list_view.rs:471 +fuzz/src/array/cast.rs:34 +fuzz/src/array/cast.rs:37 +fuzz/src/array/sum/mod.rs:126 +fuzz/src/row.rs:356 +vortex-python/src/arrays/mod.rs:516 +vortex-python/src/arrays/range_to_sequence.rs:149 +vortex-row/src/codec.rs:475 +vortex-row/src/codec.rs:486 +vortex-compressor/src/stats/integer.rs:260 +encodings/sequence/src/compress.rs:72 +encodings/sequence/src/compress.rs:103 +encodings/sequence/src/compute/filter.rs:28 +encodings/sequence/src/compute/take.rs:75 +encodings/sequence/src/compute/take.rs:94 +encodings/sequence/src/array.rs:195 +encodings/sequence/src/array.rs:269 +encodings/sequence/src/model_tests.rs:45 +encodings/sequence/src/model_tests.rs:91 +encodings/sequence/src/model_tests.rs:105 +encodings/sparse/src/canonical.rs:161 +encodings/sparse/src/canonical.rs:204 +encodings/sparse/src/canonical.rs:532 +encodings/sparse/src/canonical.rs:822 +encodings/runend/src/compute/take.rs:50 +encodings/bytebool/src/compute.rs:101 +encodings/fastlanes/src/bitpacking/compute/is_constant.rs:50 +encodings/fastlanes/src/bitpacking/compute/compare.rs:61 +encodings/fastlanes/src/bitpacking/compute/cast.rs:110 +encodings/fastlanes/src/bitpacking/compute/cast.rs:116 +encodings/fastlanes/src/bitpacking/compute/cast.rs:239 +encodings/fastlanes/src/bitpacking/compute/between.rs:59 +encodings/fastlanes/src/bitpacking/compute/take.rs:90 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:41 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:255 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:313 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:586 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:616 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:656 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:35 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:72 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:211 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/mod.rs:40 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:206 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:216 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:226 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:236 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:296 +encodings/fastlanes/src/bitpacking/vtable/mod.rs:194 +encodings/fastlanes/src/for_/compute/compare.rs:40 +encodings/fastlanes/src/for_/array/for_compress.rs:44 +encodings/fastlanes/src/for_/array/for_compress.rs:55 +encodings/fastlanes/src/for_/array/for_decompress.rs:76 +encodings/fastlanes/src/for_/array/for_decompress.rs:105 +encodings/fastlanes/src/for_/array/for_decompress.rs:181 +encodings/fastlanes/src/for_/array/for_decompress.rs:264 +encodings/fastlanes/src/for_/vtable/operations.rs:45 +encodings/fastlanes/src/delta/tests.rs:62 [test] +encodings/fastlanes/src/delta/tests.rs:163 [test] +encodings/datetime-parts/src/canonical.rs:48 +encodings/datetime-parts/src/canonical.rs:67 +encodings/datetime-parts/src/canonical.rs:85 +encodings/alp/src/alp_rd/mod.rs:160 +encodings/fsst/src/compress.rs:118 +encodings/fsst/src/compress.rs:169 +encodings/fsst/src/compress.rs:251 +encodings/fsst/src/compute/mod.rs:143 +encodings/fsst/src/compute/like.rs:63 +encodings/fsst/src/array.rs:367 +encodings/fsst/src/array.rs:407 +encodings/fsst/src/canonical.rs:34 +encodings/fsst/src/canonical.rs:84 +encodings/fsst/src/canonical.rs:110 +encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs:117 +encodings/onpair/src/tests.rs:417 [test] +encodings/onpair/src/tests.rs:419 [test] +encodings/onpair/src/tests.rs:449 [test] +encodings/onpair/src/array.rs:666 +encodings/onpair/src/array.rs:705 +encodings/onpair/src/decode.rs:51 +encodings/onpair/src/canonical.rs:42 +encodings/onpair/src/canonical.rs:73 +``` + +## Sites of `match_each_unsigned_integer_ptype!` + +```text +vortex-duckdb/src/exporter/rle.rs:64 +vortex-duckdb/src/exporter/rle.rs:65 +vortex-array/src/patches.rs:659 +vortex-array/src/patches.rs:694 +vortex-array/src/patches.rs:862 +vortex-array/src/patches.rs:864 +vortex-array/src/patches.rs:868 +vortex-array/src/patches.rs:948 +vortex-array/src/patches.rs:949 +vortex-array/src/patches.rs:1020 +vortex-array/src/patches.rs:1042 +vortex-array/src/arrays/patched/compute/take.rs:48 +vortex-array/src/arrays/patched/array.rs:283 +vortex-array/src/arrays/listview/rebuild.rs:146 +vortex-array/src/arrays/listview/rebuild.rs:277 +vortex-array/src/arrays/listview/array.rs:278 +vortex-array/src/arrays/listview/array.rs:279 +vortex-array/src/arrays/listview/array.rs:442 +vortex-array/src/arrays/listview/array.rs:443 +vortex-array/src/arrays/listview/array.rs:773 +vortex-array/src/arrays/listview/array.rs:774 +vortex-array/src/arrays/listview/array.rs:807 +vortex-array/src/arrays/listview/array.rs:808 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:130 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:131 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:132 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:153 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:154 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:155 +vortex-array/src/arrays/list/compute/take.rs:66 +vortex-array/src/arrays/list/compute/take.rs:67 +vortex-array/src/arrays/list/compute/take.rs:203 +vortex-array/src/arrays/list/compute/take.rs:228 +vortex-array/src/arrays/list/compute/take.rs:250 +vortex-array/src/arrays/list/compute/take.rs:275 +vortex-array/src/arrays/list/compute/take.rs:301 +vortex-array/src/arrays/bool/compute/take.rs:97 +vortex-array/src/arrays/bool/compute/take.rs:108 +vortex-array/src/arrays/bool/compute/take.rs:109 +vortex-array/src/arrays/bool/patch.rs:30 +vortex-array/src/arrays/interleave/execute/bool.rs:53 +vortex-array/src/arrays/interleave/execute/bool.rs:54 +vortex-array/src/arrays/interleave/execute/primitive.rs:90 +vortex-array/src/arrays/interleave/execute/primitive.rs:91 +vortex-array/src/arrays/decimal/array.rs:559 +vortex-array/src/arrays/fixed_width/take/mod.rs:156 +vortex-array/src/arrays/fixed_width/take/mod.rs:186 +vortex-array/src/arrays/fixed_width/take/mod.rs:200 +vortex-array/src/arrays/fixed_width/take/mod.rs:201 +vortex-array/src/arrays/fixed_width/take/avx2/mod.rs:61 +vortex-array/src/arrays/fixed_width/take/avx2/mod.rs:64 +vortex-array/src/arrays/varbin/compute/take.rs:152 +vortex-array/src/arrays/varbin/compute/take.rs:175 +vortex-array/src/arrays/varbin/compute/take.rs:272 +vortex-array/src/arrays/varbin/compute/take.rs:273 +vortex-array/src/arrays/varbin/compute/take.rs:424 +vortex-array/src/arrays/varbin/compute/take.rs:492 +vortex-array/src/arrays/varbin/compute/take.rs:515 +vortex-array/src/arrays/chunked/compute/take.rs:395 +vortex-array/src/arrays/chunked/compute/take.rs:406 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:148 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:159 +vortex-array/src/arrays/varbinview/compute/take.rs:90 +vortex-array/src/arrays/varbinview/compute/take.rs:101 +vortex-array/src/arrays/varbinview/compute/take.rs:102 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:531 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:532 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:681 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:682 +vortex-cuda/src/kernel/encodings/runend.rs:91 +vortex-cuda/src/kernel/encodings/fsst.rs:158 +vortex-cuda/src/kernel/encodings/fsst.rs:202 +vortex-cuda/src/kernel/encodings/zigzag.rs:64 +vortex-cuda/src/kernel/encodings/sequence.rs:51 +vortex-cuda/src/kernel/encodings/delta.rs:135 +vortex-cuda/src/kernel/patches/types.rs:85 +vortex-cuda/src/dynamic_dispatch/mod.rs:396 +encodings/sparse/src/canonical.rs:397 +encodings/runend/src/compress.rs:204 +encodings/runend/src/compress.rs:377 +encodings/runend/src/compress.rs:409 +encodings/runend/src/compute/filter.rs:69 +encodings/runend/src/compute/sum/kernel.rs:97 +encodings/runend/src/compute/take.rs:96 +encodings/runend/src/ops.rs:48 +encodings/runend/src/ops.rs:70 +encodings/runend/src/decompress_bool.rs:51 +encodings/runend/src/decompress_bool.rs:62 +encodings/zigzag/src/array.rs:254 +encodings/fastlanes/src/bitpacking/compute/filter.rs:55 +encodings/fastlanes/src/bitpacking/compute/stream_predicate.rs:64 +encodings/fastlanes/src/bitpacking/compute/compare_fused.rs:133 +encodings/fastlanes/src/bitpacking/compute/is_constant.rs:163 +encodings/fastlanes/src/bitpacking/compute/take.rs:68 +encodings/fastlanes/src/bitpacking/compute/take.rs:89 +encodings/fastlanes/src/bitpacking/compute/take.rs:124 +encodings/fastlanes/src/bitpacking/array/tests.rs:107 [test] +encodings/fastlanes/src/bitpacking/array/tests.rs:142 [test] +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:128 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:229 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:400 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:158 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:177 +encodings/fastlanes/src/bitpacking/array/mod.rs:101 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:157 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:212 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:270 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:372 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:129 +encodings/fastlanes/src/rle/array/rle_decompress.rs:32 +encodings/fastlanes/src/for_/array/for_decompress.rs:411 +encodings/fastlanes/src/delta/array/delta_decompress.rs:42 +encodings/fastlanes/src/delta/array/delta_compress.rs:33 +encodings/fastlanes/src/delta/array/mod.rs:118 +encodings/alp/src/alp/decompress.rs:117 +encodings/alp/src/alp/decompress.rs:120 +encodings/fsst/src/compute/mod.rs:99 +encodings/fsst/src/compute/mod.rs:101 +``` + +## Sites of `match_each_signed_integer_ptype!` + +```text +vortex-array/src/arrays/primitive/compute/cast.rs:149 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:90 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:118 +``` + +## Sites of `match_each_float_ptype!` + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:40 +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:39 +vortex-duckdb/src/column_statistics.rs:40 +vortex-array/src/scalar_fn/fns/is_nan.rs:43 +vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs:17 +vortex-tensor/src/scalar_fns/l2_norm.rs:93 +vortex-tensor/src/scalar_fns/inner_product.rs:89 +vortex-tensor/src/scalar_fns/cosine_similarity.rs:89 +vortex-tensor/src/scalar_fns/l2_normalize.rs:128 +``` + +## Sites of `match_each_native_simd_ptype!` + +```text +vortex-duckdb/src/convert/scalar.rs:108 +vortex-cuda/src/kernel/arrays/dict.rs:87 +vortex-cuda/src/kernel/arrays/constant.rs:66 +vortex-cuda/src/kernel/encodings/for_.rs:98 +vortex-cuda/src/kernel/filter/mod.rs:74 +``` + +## Sites of `match_each_decimal_value_type!` + +```text +vortex-duckdb/src/exporter/decimal.rs:55 +vortex-duckdb/src/exporter/decimal.rs:63 +vortex-duckdb/src/exporter/decimal.rs:64 +vortex-array/src/array/haystack/mod.rs:122 +vortex-array/src/builders/decimal.rs:136 +vortex-array/src/builders/decimal.rs:165 +vortex-array/src/builders/mod.rs:413 +vortex-array/src/scalar/arbitrary.rs:163 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:31 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:210 +vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112 +vortex-array/src/arrays/constant/vtable/canonical.rs:100 +vortex-array/src/arrays/constant/vtable/canonical.rs:357 +vortex-array/src/arrays/decimal/compute/cast.rs:157 +vortex-array/src/arrays/decimal/compute/cast.rs:158 +vortex-array/src/arrays/decimal/compute/cast.rs:188 +vortex-array/src/arrays/decimal/compute/cast.rs:219 +vortex-array/src/arrays/decimal/compute/cast.rs:228 +vortex-array/src/arrays/decimal/compute/cast.rs:474 +vortex-array/src/arrays/decimal/compute/cast.rs:476 +vortex-array/src/arrays/decimal/compute/cast.rs:625 +vortex-array/src/arrays/decimal/compute/rules.rs:47 +vortex-array/src/arrays/decimal/compute/rules.rs:167 +vortex-array/src/arrays/decimal/compute/between.rs:42 +vortex-array/src/arrays/decimal/compute/fill_null.rs:41 +vortex-array/src/arrays/decimal/compute/fill_null.rs:66 +vortex-array/src/arrays/decimal/array.rs:309 +vortex-array/src/arrays/decimal/array.rs:561 +vortex-array/src/arrays/decimal/array.rs:563 +vortex-array/src/arrays/decimal/utils.rs:25 +vortex-array/src/arrays/decimal/utils.rs:48 +vortex-array/src/arrays/decimal/vtable/operations.rs:22 +vortex-array/src/arrays/decimal/vtable/mod.rs:170 +vortex-array/src/arrays/arbitrary.rs:146 +vortex-array/src/arrays/primitive/compute/cast.rs:169 +vortex-array/src/arrays/primitive/compute/cast.rs:312 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:65 +vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:121 +vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:134 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:171 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:357 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:369 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:421 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:446 +vortex-array/src/scalar_fn/fns/list_contains/prepared/tests.rs:195 [test] +vortex-array/src/canonical.rs:187 +vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs:19 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:21 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:27 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:28 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:29 +vortex-array/src/aggregate_fn/fns/is_constant/decimal.rs:10 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:50 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:51 +vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:27 +vortex-cuda/src/kernel/arrays/dict.rs:236 +vortex-cuda/src/kernel/arrays/constant.rs:74 +vortex-cuda/src/kernel/filter/mod.rs:79 +vortex-cuda/src/arrow/canonical.rs:478 +fuzz/src/array/scalar_at.rs:50 +fuzz/src/array/filter.rs:76 +fuzz/src/array/sort.rs:66 +fuzz/src/array/slice.rs:61 +fuzz/src/array/search_sorted.rs:111 +fuzz/src/array/compare.rs:105 +fuzz/src/array/sum/tests.rs:144 [test] +fuzz/src/array/sum/mod.rs:55 +fuzz/src/array/sum/mod.rs:135 +fuzz/src/array/mask.rs:92 +fuzz/src/array/fill_null.rs:161 +fuzz/src/array/take.rs:94 +fuzz/src/row.rs:372 +vortex-compressor/src/compressor/constant/mod.rs:81 +encodings/sparse/src/canonical.rs:318 +encodings/runend/src/compress.rs:376 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:246 +``` + diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md new file mode 100644 index 00000000000..50f53f70d01 --- /dev/null +++ b/research/decimal/agents/report.md @@ -0,0 +1,1445 @@ + + +# Decimal and the Vortex type roadmap (agent reference) + +This file is the detailed report behind the summary in `../README.md`. It is research, not an +accepted design, and it changes no code. Paths and line numbers are for `develop` at `5fc5cb8` +(2026-10-17), the same commit as the extension-type notes in `../../extension-types/`. + +Audience: Vortex maintainers and coding agents who implement the type-system work. + +Companion files in this directory: + +- `ptype-match-inventory.md` lists every PType dispatch macro use, every exhaustive PType match, + and every place where another language or wire format mirrors the PType enum. +- `decimal-call-sites.md` lists every decimal-specific file with code and test line counts, every + use of the canonical-width function, and every `Canonical::Decimal` dispatch site. + +Conventions: + +- Paths are relative to the repository root. A `file:line` reference points at the commit above. +- "Storage" means the integer child of a decimal, or the storage child of an extension array. +- "Canonical width" means `DecimalType::smallest_decimal_value_type` + (`vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, + 5-9 to I32, 10-18 to I64, 19-38 to I128, and 39-76 to I256. +- "Lane integer" means U8 to U64 and I8 to I64. "Wide integer" means I128 and I256. +- Issue and pull request numbers refer to `vortex-data/vortex` on GitHub. Section 10 links + each one. +- No claim in this report comes from running Rust code. The claims come from reading code, RFC + drafts, issues, and pull requests. Appendix A holds the one script that was run. Section 10 + lists what is not verified. + +## 1. Summary + +### 1.1 What is wrong today + +`DecimalArray` stores unscaled integers in one typed buffer. The buffer width (`values_type`) is +independent of the precision (`vortex-array/src/arrays/decimal/array.rs:70-75`). One logical +dtype therefore has up to six physical forms. That single fact explains most of the strangeness: + +- Kernels must accept mixed widths, widen on the fly, or test whether a constant fits the + storage. Some paths still panic, for example `DecimalArray::patch` + (`vortex-array/src/arrays/decimal/array.rs:594-600`). Issue #5820 collects the fuzzer crashes. +- The width that every consumer actually wants is a function of precision. The code computes it + at 25 call sites, so it is a convention, not a contract (`decimal-call-sites.md`, section 2). +- Decimal duplicates much of Primitive. It has its own canonical variant (51 dispatch sites), + builder, compare, arithmetic, and aggregates. DecimalByteParts is a wide-integer limb split that + only decimals can use. Decimal-specific files hold about 7,600 non-test lines. +- Arrays never validate precision (`array.rs:301-319`), while scalars do + (`vortex-array/src/scalar/validate.rs:159-167`). +- The scalar path and the array path disagree. A decimal scalar casts to `f32` and `f16` + (`vortex-array/src/scalar/typed_view/decimal/scalar.rs:111-115`). The array kernel only handles + `f64` (`vortex-array/src/arrays/decimal/compute/cast.rs:104`), so the array cast fails with + "No CastKernel" (`vortex-array/src/scalar_fn/fns/cast/mod.rs:143`). + +### 1.2 Recommendation in short + +1. Make the decimal array a zero-data wrapper with exactly one child, the unscaled integers. The + child dtype is a pure function of the precision: `Primitive(storage_ptype(p), nullability)`. + Narrow physical storage becomes an encoding of the child, never a property of the canonical + form. +2. Use an integer child, not a FixedSizeBinary child. Add `PType::I128` and `PType::I256` as a + distinct "wide integer" tier. The tier opts into a small kernel set: compare, arithmetic, cast, + sum, min and max, hash, and structural operations. Do not add `U128` or `U256` until a + consumer needs them. Section 4 gives the full comparison. +3. Keep every decimal wire format. The new in-memory wrapper reads the legacy `vortex.decimal` + and DecimalByteParts nodes and writes them back. No new array id and no new dtype tag is + needed for decimals. Wide PTypes never reach the wire for decimal columns. +4. Express decimal semantics as rewrites onto integer kernels plus one shared "scaled integer" + kernel family. The same family serves temporal unit conversions (Timestamp, Duration, Time). +5. Treat FixedSizeBinary as an independent type for opaque bytes (UUID, hashes). It shares the + fixed-width structural layer with Primitive, but it never carries integer semantics. Decimal + does not depend on it, and wide integers do not depend on it. +6. Move Decimal to a well-known extension in the Rust API (RFC 0062) only after the extension + opacity fixes, cast hooks, and pairwise derives land. The array-level wrapper design is the + same either way, so the early work carries over. + +Section 7 gives the dependency graph. Section 8 gives the recommended order of work. + +## 2. Current state of Decimal + +### 2.1 Logical dtype + +- `DType::Decimal(DecimalDType, Nullability)` (`vortex-array/src/dtype/mod.rs:85`). +- `DecimalDType { precision: NonZero, scale: i8 }` + (`vortex-array/src/dtype/decimal/mod.rs:35-40`). +- `MAX_PRECISION` and `MAX_SCALE` both come from the i256 implementation and equal 76 + (`vortex-array/src/dtype/decimal/mod.rs:26-29`, `types.rs:215-218`). +- `DecimalDType::try_new` checks precision in 1..=76, scale at most 76, and a positive scale at + most the precision (`mod.rs:48-77`). Nothing bounds a negative scale except `i8::MIN`. The doc + comment names a `MIN_SCALE` that does not exist (`mod.rs:47`). Tests use scale -128 + (`vortex-array/src/arrays/decimal/compute/cast.rs`, case `zero_extreme_scale`). +- Wire: flatbuffer `table Decimal { precision: uint8, scale: int8, nullable: bool }` with union + tag 4 (`vortex-array/flatbuffers/vortex-dtype/dtype.fbs:29,88`). Protobuf `message Decimal` + with `uint32 precision`, `int32 scale`, and oneof field 4 + (`vortex-array/proto/dtype.proto:36,96`). + +### 2.2 Physical width types + +- `DecimalType { I8 = 0, I16, I32, I64, I128, I256 = 5 }` derives `prost::Enumeration` + (`vortex-array/src/dtype/decimal/types.rs:25-42`). +- `DecimalType::smallest_decimal_value_type` is the canonical width (`types.rs:46-57`). + `is_compatible_decimal_value_type` accepts any width at least that wide (`types.rs:72-74`). +- `NativeDecimalType` is implemented for `i8` through `i128` and for `i256`. It carries the + per-precision bound tables `MIN_BY_PRECISION` and `MAX_BY_PRECISION` and visitor hooks + (`types.rs:80-229`). The i256 tables are a 623-line literal table because the bounds cannot be + computed in a `const` context (`vortex-array/src/dtype/decimal/max_precision.rs:4-5`). +- `TryFrom for DecimalType` only covers I8 to I64 (`types.rs:244-262`). There is no + PType for 128 or 256 bits, so decimals cannot share the Primitive machinery at those widths. +- Dispatch macros: `match_each_decimal_value_type!` (77 uses in 49 files) and + `match_each_decimal_value!` (9 uses) (`vortex-array/src/dtype/decimal/macros.rs:13-60`). + +### 2.3 The i256 type + +- `#[repr(transparent)] pub struct i256(arrow_buffer::i256)` + (`vortex-array/src/dtype/bigint/mod.rs:34-40`). The Arrow type is + `#[repr(C)] struct i256 { low: u128, high: i128 }`, and its `Ord` compares `high` and then + `low` (arrow-buffer 59.3.0, `src/bigint/mod.rs:59-66` and `:171-183`). +- Size 32 bytes, alignment equal to `u128` alignment, which is 16 bytes on x86_64 and aarch64 + with the pinned toolchain (`rust-toolchain.toml`: 1.98.0, workspace `rust-version` 1.95). +- It implements the arithmetic operators, `CheckedAdd`, `CheckedSub`, `CheckedMul`, and + `CheckedDiv`. It also implements wrapping add and sub, `Zero`, `One`, `ToPrimitive` with a + full-range `to_f64`, and `AsPrimitive` to and from `i8` through `i128` + (`bigint/mod.rs:153-350`). `BigCast` and `ToI256` give checked conversions up to 256 bits + (`bigint/bigcast.rs`). It is a CUDA `DeviceRepr` (`bigint/mod.rs:42-43`). +- It does not implement `Num`, `NumCast`, `FromPrimitive`, `Bounded`, or `PrimInt`. Those are + the traits that `NativePType` and `IntegerPType` require (`vortex-array/src/dtype/ptype.rs:68` + and `:124-140`). + +### 2.4 DecimalArray layout and vtable + +- `DecimalData { decimal_dtype, values: BufferHandle, values_type: DecimalType }` + (`vortex-array/src/arrays/decimal/array.rs:110-115`). One slot holds validity (`:46-51`). +- The docs allow any width for any precision. Precision 39 can be stored in `i8`, and + precision 2 can be stored in `i256` (`array.rs:70-75`). +- `DecimalData::validate` checks only that the buffer length is a multiple of the width and + that the buffer is aligned (`array.rs:301-319`). The safety docs of the unchecked + constructors say values must fit the precision (`array.rs:256-262`), but nothing checks it. + The arithmetic kernel documents the consequence: out-of-precision values can reach kernels + (`vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:13-16`). +- `DecimalData::buffer::()` panics when `T` is not the stored width (`array.rs:360-369`). +- Encoding id `vortex.decimal` (`vortex-array/src/arrays/decimal/vtable/mod.rs:78-81`). The + metadata is `DecimalMetadata { values_type }` with the comment "The type of the values can be + determined by looking at the type info...right?" (`vtable/mod.rs:52-57`). +- `ArrayEq` and `ArrayHash` include `values_type` (`vtable/mod.rs:59-70`). Two arrays with the + same logical values at different widths are not `ArrayEq`. +- Deserialization reads the width with the prost getter `metadata.values_type()` + (`vtable/mod.rs:170`). prost-derive 0.14.4 generates that getter as + `TryFrom::try_from(raw).unwrap_or(default)`, and the default of a prost enumeration is its + first variant (prost-derive `src/field/scalar.rs:286-300`, `src/lib.rs:308`). An unknown + width therefore decodes silently as `I8`. No such value exists today, but the pattern blocks + safe extension of this metadata. +- Take and filter are shared with Primitive through the `FixedWidthArray` adapter + (`vortex-array/src/arrays/fixed_width/array.rs:20-41`, + `vortex-array/src/arrays/decimal/compute/fixed_width.rs:14-36`). PR #8864 introduced it. The + record-width dispatch already covers 1, 2, 4, 8, 16, and 32 bytes + (`vortex-array/src/arrays/fixed_width/mod.rs:16-47`). + +### 2.5 Scalar values + +- `DecimalValue { I8(i8), I16, I32, I64, I128, I256(i256) }` + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:48-61`). Its `PartialEq`, + `PartialOrd`, and `Hash` upcast to i256 (`dvalue.rs:255-281`) because one dtype allows several + widths. `normalize` and `zero` produce the canonical width (`dvalue.rs:185-216`). +- `ScalarValue::Decimal(DecimalValue)` (`vortex-array/src/scalar/scalar_value.rs:31-32`). +- `DecimalScalar` equality and ordering require equal dtypes apart from nullability + (`vortex-array/src/scalar/typed_view/decimal/scalar.rs:188-204`). +- Decimal scalars serialize as protobuf `bytes_value` with the byte length of the variant + (`vortex-array/src/scalar/proto.rs:87-98`). Readers infer the width from the byte length and + do not check precision, which a TODO notes (`scalar/proto.rs:402-445`, TODO at `:407`). +- Scalar construction validates precision (`vortex-array/src/scalar/validate.rs:58-60`, + `:159-167`). Array construction does not. + +### 2.6 Builders + +`DecimalBuilder` holds an enum of six `BufferMut` variants and coerces appended values into the +builder width (`vortex-array/src/builders/decimal.rs:38-56`, `:160-180`). The generic builder +factory picks the canonical width (`vortex-array/src/builders/mod.rs:412-416`). + +### 2.7 Compute kernels and aggregates + +| Operation | Location | Behavior | +| --- | --- | --- | +| Cast, nullability only | `arrays/decimal/compute/cast.rs:45-80` | Reduce rule, same decimal dtype | +| Cast to integer | `cast.rs:97-103`, `:172-196` | Truncate toward zero, range checked | +| Cast to float | `cast.rs:104-107`, `:198-245` | `f64` only, one rounding step | +| Cast decimal to decimal | `cast.rs:108-170`, plan at `:348` | Zero-copy when the scale is equal, the precision widens, and the width already fits. Otherwise rescale, exact scale-down only | +| Integer to decimal | `arrays/primitive/compute/cast.rs:87-89`, `:127-173` | Floats rejected. Zero-copy when the width equals the canonical width and the scale is 0 | +| Compare | `scalar_fn/fns/binary/compare/decimal.rs` | Equal dtypes only. Widens mismatched widths (`:114-126`). Sign test for constants that do not fit (`:128-153`) | +| Arithmetic | `scalar_fn/fns/binary/numeric/decimal.rs` | Equal dtypes only (`scalar_fn/fns/binary/mod.rs:146-153`). Arrow result rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`). Checked at a working width, then narrowed (`numeric/decimal.rs:410-430`) | +| Between | `arrays/decimal/compute/between.rs` | Constant bounds only, storage-fit logic | +| Fill null | `arrays/decimal/compute/fill_null.rs:55-72` | Upcasts the whole array when the fill value does not fit | +| Mask | `arrays/decimal/compute/mask.rs` | Metadata-only | +| Patch | `arrays/decimal/array.rs:537-609` | Panics when the width cannot hold the dtype | +| Sum | `aggregate_fn/fns/sum/mod.rs:78-86`, `sum/decimal.rs:49` | Result precision p+10, capped at 76. Overflow gives null | +| Min, max | `aggregate_fn/fns/min_max/decimal.rs` | Per-width | +| is_sorted, is_constant | `aggregate_fn/fns/is_sorted/decimal.rs`, `is_constant/decimal.rs` | Per-width | +| all_non_distinct | `aggregate_fn/fns/all_non_distinct/decimal.rs` | Triple-nested width dispatch | +| Uncompressed size | `aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | Reports the canonical width, not the stored width | +| IN list | `array/haystack/mod.rs:118-125` | Converts to the canonical width | + +Paths in the table are relative to `vortex-array/src/`. Shared code also has decimal arms: + +- The row codec (`vortex-row/src/codec.rs:579-584`, `:822-860`). +- Sparse canonicalization (`encodings/sparse/src/canonical.rs:314-327`). +- Run-end decoding (`encodings/runend/src/compress.rs:358-388`). +- Constant canonicalization (`vortex-array/src/arrays/constant/vtable/canonical.rs:96-125`). +- Dictionary execution (`vortex-array/src/arrays/dict/execute.rs:53`, `:102-111`). +- Masked execution (`vortex-array/src/arrays/masked/execute.rs:54`, `:89-95`). + +### 2.8 Encodings and compression + +- DecimalByteParts (DBP) stores a signed most significant part (MSP) and up to three unsigned + 64-bit lower parts (`encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:6-15`). The + MSP holds validity (`decimal_byte_parts/array.rs:404-409`). Each part is a normal integer array + that any integer scheme can compress. +- DBP carries `DType::Decimal` itself (`decimal_byte_parts/array.rs:102-104`). It mixes a + logical type with a physical layout. +- Two wire ids exist. `vortex.decimal_byte_parts` (v1, MSP only) is in the first core edition. + `vortex.decimal_byte_parts.v2` (with lower parts) is in no edition yet + (`decimal_byte_parts/plugin/mod.rs:30-40`, + `vortex-edition/src/declarations/core/v2025_05.rs:30-31`). The plugin writes v1 whenever there + are no lower parts (`decimal_byte_parts/plugin/mod.rs:62-79`). +- DBP assembly with no lower parts reuses the MSP buffer, so the result width is the MSP width + for any precision (`decimal_byte_parts/assemble.rs:82-93`). +- The DBP compare kernel only handles the MSP-only case. A TODO describes the missing limb-wise + comparison (`decimal_byte_parts/compute/compare.rs`, top of `compare`). +- The btrblocks `DecimalScheme` narrows first (`vortex-btrblocks/src/schemes/decimal.rs:133`). + In v1 mode it returns the narrowed `vortex.decimal` array unchanged when values still need + 128 or 256 bits (`:134-137`). Files written today can therefore contain `vortex.decimal` + nodes whose width is smaller than the canonical width. + +### 2.9 Statistics and zone maps + +- Array statistics store `min`, `max`, and `sum` as serialized `ScalarValue` bytes + (`vortex-array/flatbuffers/vortex-array/array.fbs`, `table ArrayStats`). +- Sum statistics merge with a precision check (`vortex-array/src/stats/stats_set.rs:480-492`). +- Zone maps use the column dtype, falling back to the extension storage dtype + (`vortex-layout/src/layouts/zoned/schema.rs:112-121`, `:177-188`). +- The bloom filter aggregate rejects decimals and cites issue #5820 + (`vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:34-49`). + +### 2.10 Serialization and editions + +- Editions gate four component kinds: array ids, layout ids, extension dtype ids, and + aggregates (`vortex-edition/src/lib.rs:161-178`). +- Core dtype tags and PType values are not gated. The writer only checks extension ids inside + the schema (`vortex-file/src/writer.rs:429-475`). Union, Variant, and Map dtype tags reach the + wire without an edition check, although their array ids are gated. +- An unknown flatbuffer dtype tag or PType is a loud error for old readers + (`vortex-array/src/dtype/serde/flatbuffers.rs:326`, `:559`). An unknown protobuf oneof is + also loud ("Unrecognized DType", `vortex-array/src/dtype/serde/proto.rs:32`). An unknown + protobuf PType value is silent: `p.r#type()` returns `U8` (`proto.rs:36`). Protobuf dtypes are + persisted on disk in Variant shredding metadata + (`vortex-array/src/arrays/variant/vtable/mod.rs:54`) and Parquet-variant metadata + (`encodings/parquet-variant/src/vtable.rs:73`). +- Unknown extension dtype ids read as `ForeignExtDType`, which keeps the storage dtype and the + raw metadata (`vortex-array/src/dtype/extension/foreign.rs:26-75`). Old readers can read new + extension types as their storage. +- `ArrayPlugin` separates the in-memory id from the serialized ids. A plugin can read several + historical ids into one in-memory array. It can also write an older form as a "lossless + structural downgrade" (`vortex-array/src/array/plugin.rs:26-41`, `:101-157`). DBP uses this + today. +- Compatibility fixtures exist for decimals, DBP v1, and DBP v2 + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/`). The decimal fixture only covers + canonical widths (i32 and i64). + +### 2.11 Engine mappings + +| Engine | Decimal mapping | Wide integers | +| --- | --- | --- | +| Arrow | Export Decimal128 for p <= 38 and Decimal256 above. Decimal32 and Decimal64 are commented out for DataFusion (`vortex-arrow/src/dtype.rs:333-347`). Import keeps the Arrow width zero-copy (`vortex-arrow/src/convert.rs:204-277`) | Arrow has no Int128 or Int256 | +| DuckDB | DECIMAL both ways (`vortex-duckdb/src/convert/dtype.rs:106-110`, `:237-241`). Widths 16, 32, 64, and 128 bits, zero-copy when equal (`vortex-duckdb/src/exporter/decimal.rs:122-130`). Import narrows i16 to i8 for p <= 2 (`convert/vector.rs:303-330`) | HUGEINT and UHUGEINT are rejected (`convert/dtype.rs:100-101`) | +| DataFusion | Scalars to Decimal32, 64, 128, or 256 by precision (`vortex-datafusion/src/convert/scalars.rs:57-110`) | No Int128 scalar | +| Python | `DecimalDType` class, Python `Decimal` scalars built from the unscaled value and exponent (`vortex-python/src/scalar/into_py.rs:153-171`). `DecimalArray` subclass is keyed by encoding id (`vortex-python/src/arrays/native.rs:174-175`) | NumPy has no 128-bit integer dtype | +| FFI | `DTYPE_DECIMAL = 8`. Scalar constructors and getters for i8 to i64 and 16-byte little-endian i128 (`vortex-ffi/src/scalar.rs:265-330`) | `vx_ptype` is a `#[non_exhaustive]` C enum (`vortex-ffi/src/ptype.rs:6-31`) | +| JNI | Statistics as `java.math.BigDecimal` (`vortex-jni/src/writer.rs:254-270`). Literals from big-endian bytes (`vortex-jni/src/expression.rs:511-545`) | Not applicable | +| CUDA | Arrow device widths 32, 64, 128, 256 (`vortex-cuda/src/arrow/mod.rs:840-849`). Decimal filter and DBP kernels | Separate C `PTypeTag` enum (`vortex-cuda/kernels/src/dynamic_dispatch.h:45-56`) | + +Four different width functions exist: + +- The Vortex canonical width, with an I8 minimum. +- DuckDB, with an I16 minimum. +- Arrow device export, with an I32 minimum. +- Arrow dtype export, with an I128 minimum below precision 39. + +### 2.12 The PType enum and its blast radius + +- `PType { U8 = 0, U16, U32, U64, I8, I16, I32, I64, F16, F32, F64 = 10 }`, `#[repr(u8)]`, + `prost::Enumeration` (`vortex-array/src/dtype/ptype.rs:33-60`). +- `NativePType` requires `Num`, `NumCast`, `FromPrimitiveOrF16`, byte conversions, and a seal + (`ptype.rs:124-140`). `IntegerPType` adds `PrimInt` and `AsPrimitive` (`ptype.rs:68`). +- `match_each_native_ptype!` is exhaustive over 11 arms. `match_each_integer_ptype!` and its + unsigned, signed, and float siblings end with `other => panic!` (`ptype.rs:355-737`). +- Counts at this commit (details in `ptype-match-inventory.md`): + +| Item | Count | +| --- | ---: | +| `match_each_native_ptype!` uses | 70 in 54 files | +| `match_each_integer_ptype!` uses | 169 in 94 files | +| `match_each_unsigned_integer_ptype!` uses | 118 in 53 files | +| Other PType macros (signed, float, SIMD) | 17 | +| Exhaustive PType matches outside `ptype.rs` (proxy) | about 45, 10 of them in tests | +| `is_int()` call sites | 53, about 40 of them guard offsets, codes, or indices | +| `NativePType` mentions | 598 in 169 files | +| Mirrors in other languages and formats | flatbuffer, protobuf, C header, C++, Python stub, Python enum, CUDA C enum and mapping, FFI enum, JNI codes, DuckDB, Arrow, DataFusion | + +- `is_int()` routes a type into 64-bit-only code in many places. Examples are list, list-view, + and varbin offsets, dictionary codes, take indices, and every btrblocks integer scheme + (`vortex-btrblocks/src/schemes/integer/*.rs`, where `matches` uses `is_int()`). If `is_int()` + returns true for I128, those guards admit wide integers, which then reach a + `match_each_integer_ptype!` panic. +- Primitive assumes alignment equals byte width in deserialization + (`vortex-array/src/arrays/primitive/vtable/mod.rs:158`) and in `from_values_byte_buffer` + (`vortex-array/src/arrays/primitive/array/mod.rs:505`). That is wrong for I256, whose Rust + alignment is 16, not 32. + +### 2.13 What is strange, in one list + +1. The canonical form has up to six physical shapes per dtype. The narrowing is compression that + leaked into the canonical form (issue #5820, Epic #7706). +2. The width that consumers need is a function of precision, but it is a convention repeated at + 25 call sites, not a contract. The row codec says so directly: keys use the declared dtype "so + keys from differently compressed chunks stay memcmp-comparable" + (`vortex-row/src/codec.rs:579-584`). +3. Kernels carry storage-fit special cases: constant sign tests in compare and between, the + fill-null upcast, the patch panic, widening in compare and arithmetic. +4. Arrays skip precision validation, scalars enforce it. Kernels must defend. +5. The scalar and array cast paths disagree for `f32` and `f16` targets. +6. Decimal duplicates Primitive in builders, kernels, aggregates, canonical dispatch, and + compression. Epic #7706 lists this as an open problem. +7. DBP is a wide-integer limb split that only decimals can use, and it carries a decimal dtype. +8. Decimal metadata decodes with a prost getter that turns unknown values into `I8`. +9. Bloom filters, IN lists, size statistics, and the row codec each have decimal special cases. +10. DuckDB HUGEINT cannot be imported although Vortex already stores i128 decimal lanes. +11. Decimal scalars on the wire carry the width of whatever variant the writer used, and + readers do not check precision. +12. Sparse canonicalization supports decimals with a dedicated path but rejects all extension + arrays (`encodings/sparse/src/canonical.rs:361`). The wrapper idea is not applied anywhere + as a shared rule. + +## 3. Target: Decimal as an opaque wrapper over an integer child + +### 3.1 Definition + +The logical dtype does not change. `DType::Decimal(DecimalDType, Nullability)` keeps its Rust +variant in this phase and keeps its wire tag forever (section 3.8 covers the later move to a +well-known extension). + +The storage dtype becomes a pure function of the precision. It reuses the current canonical +width: + +```rust +impl DecimalDType { + /// The integer type that holds every unscaled value of this precision. + pub const fn storage_ptype(&self) -> PType { + match self.precision.get() { + 1..=2 => PType::I8, + 3..=4 => PType::I16, + 5..=9 => PType::I32, + 10..=18 => PType::I64, + 19..=38 => PType::I128, + _ => PType::I256, + } + } + + /// The dtype of the unscaled child for this precision and the given nullability. + pub fn storage_dtype(&self, nullability: Nullability) -> DType { + DType::Primitive(self.storage_ptype(), nullability) + } +} +``` + +The array becomes a zero-data wrapper with one slot: + +```rust +#[array_slots(Decimal)] +pub struct DecimalSlots { + /// The unscaled integers. The dtype is `decimal_dtype.storage_dtype(nullability)`. + #[slot(0)] + pub unscaled: ArrayRef, +} + +impl VTable for Decimal { + type TypedArrayData = EmptyArrayData; + type OperationsVTable = Self; // scalar_at wraps the child scalar + type ValidityVTable = ValidityVTableFromChild; + + fn id(&self) -> ArrayId { + // The in-memory id stays `vortex.decimal`. Section 3.5 covers the wire. + static ID: CachedId = CachedId::new("vortex.decimal"); + *ID + } + + fn nbuffers(_array: ArrayView<'_, Self>) -> usize { + 0 + } + + // validate: check that the child dtype equals the storage dtype and that lengths match. + // execute: return the wrapper unchanged. The child stays as it is. +} + +impl Array { + /// Wraps unscaled integers whose dtype is the storage dtype of `decimal_dtype`. + pub fn try_new(decimal_dtype: DecimalDType, unscaled: ArrayRef) -> VortexResult; + + /// The unscaled integers. This is the only way to see the storage. + pub fn unscaled(&self) -> &ArrayRef; +} +``` + +Invariants: + +- **I1, type.** `unscaled.dtype() == decimal_dtype.storage_dtype(nullability)`. `validate` + checks this. It costs nothing at run time. +- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. Trust boundaries + establish it (Arrow, DuckDB, FFI, and Python imports, and the array `validate_array` hook of + node N5). Kernels that produce new values (arithmetic, rescaling casts, sums) check their own + outputs, as `numeric/decimal.rs` does today with `DecimalValueBounds`. +- **I3, no data.** The wrapper owns no buffer. Validity comes from the child, as for + `ExtensionArray` (`vortex-array/src/arrays/extension/vtable/mod.rs:85-89`) and DBP + (`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:404-409`). + +The value of a row is `unscaled * 10^(-scale)`. Nothing else is stored. + +### 3.2 Canonical form + +`Canonical::Decimal(DecimalArray)` remains a variant in this phase, but the wrapped child does +not have to be canonical. That matches the existing rule that canonical form is not recursive +(`vortex-array/src/canonical.rs:85-88`) and matches `ExtensionArray`, whose `execute` returns +itself without touching the storage (`vortex-array/src/arrays/extension/vtable/mod.rs:189-191`). +A kernel that needs flat integers executes the child explicitly: + +```rust +let unscaled = decimal.unscaled().clone().execute::(ctx)?; +``` + +This is the parent and child relation proposed in issue #5820 ("an integer array as a child that +is separately decompressed, like list offsets"). It also answers the objection in the same thread +that the two designs are equivalent. They are not equivalent. With a fixed child dtype, a kernel +never receives a width that it does not expect. Also, the decimal-specific kernels disappear +instead of being duplicated (section 3.4). + +### 3.3 Shared wrapper machinery with extensions + +Decimal and extensions need the same structural behavior. Today each type implements it +separately, and some cases exist for only one of them. One trait can carry the shared part: + +```rust +/// A zero-data logical wrapper over one storage child. +/// +/// Structural operations push into the storage child and rewrap the result. Semantic +/// operations never read the storage unless a type-owned rewrite says so. +pub trait StorageWrapper: VTable { + /// The storage child. + fn storage(array: ArrayView<'_, Self>) -> &ArrayRef; + + /// The storage dtype implied by a wrapper dtype. It must be a pure function of the dtype. + fn storage_dtype(dtype: &DType) -> VortexResult; + + /// Rebuilds a wrapper with the same logical dtype around a new storage child. + fn rewrap(array: ArrayView<'_, Self>, storage: ArrayRef) -> VortexResult; + + /// Wraps a storage scalar into a logical scalar of `dtype`. + fn wrap_scalar(dtype: &DType, storage: Scalar) -> VortexResult; +} +``` + +`Extension` implements it with `ExtensionArray::new` and `Scalar::extension_ref`. `Decimal` +implements it with `DecimalArray::try_new` and a decimal scalar built from the storage value. +Generic rules then cover both: + +| Behavior | Extension today | Decimal today | Shared rule | +| --- | --- | --- | --- | +| slice, filter, take, mask | Pushdown rules (`arrays/extension/compute/rules.rs:48-79`) | Buffer kernels through `FixedWidthArray` | Push into the child and rewrap | +| Constant child | `ExtensionConstantRule` (`rules.rs:28-46`) turns `Ext(Constant(s))` into `Constant(ext s)` | `Constant(decimal scalar)` | The wrapper stays on top: `Constant(wrapper s)` becomes `Wrapper(Constant(storage s))` | +| scalar_at | Wraps the storage scalar | Reads the buffer at the stored width | `wrap_scalar(child.scalar_at(i))` | +| Validity | From the child | Validity slot | From the child | +| Compression | Compress storage and rewrap (`vortex-compressor/src/compressor/cascade.rs:175-205`) | `DecimalScheme` | Compress the child with ordinary schemes and rewrap | +| Sparse, run-end, dictionary, chunked | Sparse bails (`encodings/sparse/src/canonical.rs:361`) | Dedicated decimal paths | The wrapper stays on top: `Dict(codes, Wrapper(v))` becomes `Wrapper(Dict(codes, v))`, so the structural encoding sees only storage | + +The last row fixes a real gap for extensions as a side effect: a sparse array of an extension +type currently cannot canonicalize at all. + +This normal form is the one in part 3 of the extension notes, and it reverses +`ExtensionConstantRule`. It is still an open question (section 9). The alternative keeps generic +encodings above the wrapper, as the current code does for constants. Then each generic encoding +needs a pushdown rule for `ext_storage`, or a fallback that executes it. Old wire forms that put +the wrapper lower, such as `Dict(codes, vortex.decimal)`, come from the plugin at write time +(section 3.5). + +Type-specific encodings move under the wrapper and see only the storage dtype. For decimals that +means DBP stops carrying `DType::Decimal`. It becomes an encoding of `Primitive(I128)` or +`Primitive(I256)` (node N14). The same normal form applies to DateTimeParts under temporal +wrappers, which today carries the extension dtype (`encodings/datetime-parts/src/array.rs:294-305`). + +### 3.4 Decimal semantics as storage rewrites + +Every decimal operation is an integer operation plus scale bookkeeping. With a fixed storage +dtype, the rewrites below are exact. + +| Operation on `Decimal(p, s)` | Rewrite onto the unscaled child | Notes | +| --- | --- | --- | +| Compare, same `(p, s)` | `compare(unscaled(a), unscaled(b))` | Valid because the unscaled value is unique within one `(p, s)` | +| Compare across `(p1, s1)` and `(p2, s2)` | Rescale both to `s = max(s1, s2)` at a width that holds `max(p1 - s1, p2 - s2) + s` digits, then compare | Pairwise derive. Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | +| Compare with a constant that has more fractional digits than `s` | Fold the constant and adjust the operator. `x < 1.005` at `s = 2` becomes `unscaled <= 100` | Needs floor and ceiling rounding | +| Add, subtract | Widen to the result width, rescale to the larger scale, checked operation, precision check | Arrow rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`) | +| Multiply | Widen, checked multiply, precision check | Result scale `s1 + s2` | +| Divide | Multiply the dividend by `10^k`, checked divide toward zero, precision check | As today | +| Cast to `Decimal(p2, s2)` | Rescale with exactness, integer width cast, precision check | Replaces `DecimalCastPlan` | +| Cast to an integer | Divide by `10^s` toward zero, then range-checked integer cast | Replaces `DecimalToIntegerCast` | +| Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays | +| Cast from an integer | Widen, multiply by `10^s`, precision check | Replaces `cast_to_decimal` | +| Sum | Integer sum with accumulator `storage_ptype(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | +| Min, max, is_sorted, is_constant | Storage aggregate, then wrap | Matches `min_max/extension.rs:17-35` | +| Hash, group by, distinct, join keys | Storage hash and equality | Valid within one `(p, s)` | +| Zone-map pruning | Rewrite the predicate onto storage statistics | Old files keep decimal-typed statistics, which hold the same integers | +| IN list | Storage set over rescaled needles | Replaces `haystack/mod.rs:118-125` | +| Row encoding | Storage at the canonical width, big-endian with the sign bit flipped | Already done (`vortex-row/src/codec.rs:19`, `:1466-1480`) | + +These rewrites need a small set of integer kernels that do not exist as reusable pieces today. +They are the scaled-integer family (node N8): + +```rust +/// Rounding when a rescale divides by a power of ten. +pub enum RescaleRounding { + /// Fail when a nonzero remainder would be dropped. + Exact, + /// Round toward zero. Used by decimal to integer casts and by division. + TowardZero, + /// Round toward negative infinity. Used for upper bounds of folded predicates. + Floor, + /// Round toward positive infinity. Used for lower bounds of folded predicates. + Ceil, +} + +/// Options of the `vortex.int.rescale` scalar function. +pub struct RescaleOptions { + /// Positive to multiply by 10^delta, negative to divide by 10^-delta. + pub delta: i16, + pub rounding: RescaleRounding, + /// The integer type of the result. The kernel fails on overflow. + pub target: PType, +} + +/// Options of the `vortex.int.check_range` scalar function, which fails when a valid value +/// falls outside `[min, max]` and otherwise returns its input unchanged. +pub struct CheckRangeOptions { + pub min: Scalar, + pub max: Scalar, +} +``` + +Two points matter for the design: + +- **Temporal types need the same kernels.** Timestamp, Time, and Duration units are powers of + ten apart, except Days. A Timestamp cast from milliseconds to nanoseconds (issue #8570) is + `rescale(delta = 6)`. A Duration comparison across units (PR #10467) is the same pairwise + rescale as a decimal comparison across scales. One kernel family serves both. +- **Fusion keeps performance.** The current decimal arithmetic kernel checks the operation and + the precision bound in one pass (`numeric/decimal.rs`, `DecimalOpConstants`). A chain of + separate array operations makes several passes. So generalize the fused kernel from + `DecimalValue` widths to integer PTypes, with optional pre-scale factors and bounds. Plain + integer arithmetic is then the special case with type bounds and unit factors. + +### 3.5 Wire compatibility + +No new wire id and no new dtype tag is needed. The `ArrayPlugin` mechanism already lets one +in-memory array read and write several historical wire forms +(`vortex-array/src/array/plugin.rs:102-157`). + +Reading: + +| Wire node | In-memory result | +| --- | --- | +| `vortex.decimal`, `values_type` equals the canonical width | `Decimal { Primitive(I_w) }`, zero-copy | +| `vortex.decimal`, `values_type` narrower (compressor narrowing) | `Decimal { widen(Primitive(I_v)) }`, zero-copy and lazy | +| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | `Decimal { checked_narrow(Primitive(I_v)) }`, lazy. Fails at execution only if a value violates the precision | +| `vortex.decimal`, unknown `values_type` | Error. Decode with `DecimalType::try_from(raw)`, not the prost getter | +| `vortex.decimal_byte_parts` (v1) | Phase 2: unchanged DBP that executes into the wrapper. Later: `Decimal { widen(msp) }` | +| `vortex.decimal_byte_parts.v2` | Phase 2: unchanged DBP. Later: `Decimal { WideIntParts { msp, lower } }` | + +Writing: + +| In-memory shape | Wire node | Readable by old readers | +| --- | --- | --- | +| `Decimal { flat Primitive(I_w) }` | `vortex.decimal`, `values_type = w`, the buffer and validity of the child | Yes | +| `Decimal { widen(flat Primitive(I_v)) }` | `vortex.decimal`, `values_type = v` | Yes, same bytes as today | +| `Decimal { compressed child with a lane-integer ptype }` | `vortex.decimal_byte_parts` v1 with the child as MSP | Yes, core 2025.05 | +| `Decimal { WideIntParts }` | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2 | +| `Decimal { Dict, RunEnd, Chunked, ... }` | Push the wrapper below the structural encoding, for example `Dict(codes, Decimal { values })` | Yes | +| Any other child | Execute the child to a flat array and write `vortex.decimal` | Yes | + +Consequences: + +- Old readers read every file that a new writer produces, with the same edition rules as today. +- Wide PTypes never appear on the wire for decimal columns. The legacy metadata carries + `DecimalType`, and DBP carries lane-integer PTypes. Only top-level `Primitive(I128)` columns + (DuckDB HUGEINT, node N27) need new wire values and edition gating. +- Decimal statistics and zone-map entries keep the `Decimal` dtype tag. They stay readable. +- Writing DBP from the decimal plugin needs one owner for the decimal and DBP serialized ids. In + phase 2 the simplest path keeps DBP as its own encoding and lets the compressor produce it. + Moving DBP under the wrapper (node N14) is a later step. Section 9 lists the ownership choice. +- Extend the compat fixtures before the reader changes. Add narrowed widths (I8 at p 10), wide + widths (I128 at p 20, I256 at p 40), and Arrow-style wider widths (I128 at p 5). + +### 3.6 What happens to `values_type` narrowing + +Narrowing becomes an encoding of the child: + +- **Lane-integer children (p <= 18).** The existing integer schemes (FoR with BitPacking, + ZigZag, Delta, RLE, Dict) already narrow integers better than a fixed smaller width does. No + decimal-specific scheme is needed. +- **Wide children (p >= 19).** If every value fits `i64`, the child is `widen(i64 array)` and + the inner array uses the lane-integer schemes. Otherwise the child is a limb split, which is + DBP generalized (node N14). It has a signed MSP and unsigned 64-bit lower parts, and the + lane-integer schemes compress each part. +- **Legacy reads** produce `widen(...)` or `checked_narrow(...)` nodes as shown in section 3.5. + +`widen` has two possible representations. One is the existing lazy `Cast` scalar-function array, +which already has reduce and kernel pushdown machinery. The other is a small dedicated in-memory +encoding. Either way, the encoding needs one generic rule that already exists in DBP form +(`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). A comparison of +`widen(x)` with a constant becomes a comparison of `x` with the constant when the constant fits +the narrow type. Otherwise the result is a constant. The generalized rule helps every widened +integer column, not only decimals. + +### 3.7 Builders, scalars, engines, and the public API + +- **Builder.** `DecimalBuilder` becomes the decimal dtype plus a `PrimitiveBuilder` of the + storage type. `append_value::` converts with a checked cast and fails when the value is + outside the precision. The six-variant buffer enum + (`vortex-array/src/builders/decimal.rs:38-56`) goes away. +- **Scalars.** `DecimalScalar` keeps its API. `DecimalValue` is normalized to the storage width + at construction with the existing `normalize` (`dvalue.rs:208-216`). `scalar_at` wraps the + child scalar. +- **Deprecated accessors.** `DecimalArray::buffer::()` becomes + `unscaled().execute::(ctx)?.as_slice::()` with `T` equal to the storage + type. `values_type()` returns the storage width. `DecimalArray::new::` converts a buffer of + another width, or wraps it in a lazy `widen` or `checked_narrow` child. +- **Engines.** The Arrow, DuckDB, and CUDA exporters read the child. An Arrow `Decimal128` + export of a `checked_narrow` child can reuse the original 128-bit buffer without a copy. This + is a small case of the execution layouts of RFC 0062. +- **Python.** The binding keys its `DecimalArray` subclass on the in-memory encoding id + (`vortex-python/src/arrays/native.rs:174-175`). Keeping the id `vortex.decimal` keeps it working. + +### 3.8 Does Decimal become a well-known extension in Rust? + +RFC 0062 (branch `ngates/types`, `rfcs/0062-value-types-and-representations.md`) proposes +`vortex.decimal` as a well-known extension with signed-integer storage, while the existing wire +tag keeps working. It also states the precondition directly: "Moving Decimal requires integer +storage capable of its currently supported precisions, including wide lanes." + +The wrapper in sections 3.1 to 3.7 is exactly the array side of that proposal. The remaining +question is only where the Rust type lives: + +| Aspect | Keep `DType::Decimal` (phase 2) | Well-known extension (phase 4) | +| --- | --- | --- | +| Array | `DecimalArray` wrapper | `ExtensionArray` with the decimal ext dtype | +| Shared machinery | Through `StorageWrapper` | Directly | +| Semantics | Decimal-owned rewrites | `ExtVTable` hooks: cast (#8570), pairwise derives, validation | +| API change | Small | 269 lines in 97 files mention `DType::Decimal` | +| Wire | Unchanged | Tag translation in both directions, recursively | +| Risk | Low | High until extension semantics are opaque by default | + +Today the extension machinery leaks storage semantics. An extension array cast to any other +dtype casts its storage (`vortex-array/src/arrays/extension/compute/cast.rs:17-22`). A scalar cast +into an extension casts to the storage and rewraps (`vortex-array/src/scalar/cast.rs:56-61`). A +comparison with a lone extension side compares raw storage +(`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-194`). If Decimal moves now, a cast +from `decimal(10, 2)` to `i64` returns the unscaled integers, and `price < 5` compares `500` +with `5`. So the move must wait for nodes N1 to N4. Built-in registration is not an +obstacle: Date, Time, and Timestamp are already default extensions +(`vortex-array/src/dtype/session.rs:36-39`). + +Recommendation: build the wrapper with `DType::Decimal` first. Use Decimal as the acceptance test +for the extension hooks while they are designed. Decide on the Rust move after node N13, when it +has become a mechanical API change. + +## 4. The child representation question + +### 4.1 The options + +- **(a) Integer child with new PTypes.** The child is `Primitive(I8 .. I256)`. Add `I128` and + `I256` to `PType`. `U128` and `U256` are a separate decision (section 5.7). +- **(b) FixedSizeBinary child.** The child is `FixedSizeBinary(n)` with `n` in 1, 2, 4, 8, 16, 32, + or only 16 and 32 for the wide widths. FixedSizeBinary does not exist yet. +- **(c) Hybrid.** `Primitive` child up to 64 bits, `FixedSizeBinary(16)` or `(32)` above. +- **(d) Separate wide-integer family.** A new canonical family for 128 and 256 bits, outside + `Primitive`. An example is a new dtype variant with a new canonical array. +- **(e) Status quo layout, opaque API.** Keep the typed buffer inside `DecimalArray`, pin the + width to the canonical width, and hide `values_type`. + +### 4.2 Byte order is not integer order + +A FixedSizeBinary value orders by byte-wise comparison (memcmp). Arrow, DuckDB, and Vortex store +decimal lanes as little-endian two's complement. Memcmp of those bytes is not integer order: + +- `1` and `256` as 16-byte little-endian values: memcmp says `256 < 1`. +- `-1` and `0`: memcmp says `-1 > 0`. +- On 19,999 random pairs of 128-bit values, little-endian memcmp agreed with integer order + 10,060 times, which is chance. Big-endian bytes with the sign bit flipped agreed 19,999 times. + Appendix A has the script. + +Vortex already uses the sign-flip transform where it needs byte order. The row codec writes +"fixed-width integers big-endian, with the sign bit flipped for signed types" +(`vortex-row/src/codec.rs:19`). The DuckDB UUID exporter flips the top bit of a big-endian `u128` +to get the ordered HUGEINT form of DuckDB (`vortex-duckdb/src/exporter/uuid.rs`, export loop). + +So a FixedSizeBinary child can reuse memcmp order only if it stores big-endian sign-flipped +bytes. That layout breaks zero-copy exchange with Arrow `Decimal128` and `Decimal256` and with +DuckDB `DECIMAL`. It also forces a conversion before every arithmetic operation. With the native +little-endian layout, every ordered operation on the child needs a decimal-specific +reinterpretation. Either way, the decimal logic stays duplicated. + +### 4.3 Comparison + +Compute: + +| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| --- | --- | --- | --- | --- | --- | +| Compare | Generic primitive kernel at 128 and 256 bits | Decimal reinterpretation | Integer to 64 bits, reinterpretation above | New kernels, or shared generic code as in (a) | Existing decimal kernel | +| Checked arithmetic | `i128` native. `i256` through num-traits `Checked*` | Reinterpretation | Mixed | As (a) | Existing | +| Sum, min, max, sorted | Integer aggregates with a wider accumulator | Decimal-specific | Mixed | As (a) | Existing | +| Casts | Integer width casts plus rescale | Decimal-specific | Mixed | As (a) | Existing | +| Hash | Native | Byte hash, correct at a fixed width | Both | Native | Upcast to i256 | +| Lane layout | Array of little-endian lanes. A limb split gives a SIMD-friendly layout | Same bytes, no typed lanes | Mixed | As (a) | As today | + +Data: + +| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| --- | --- | --- | --- | --- | --- | +| Compression | All lane schemes to 64 bits. Wide: narrow if it fits, limb split, Dict, RLE, run-end, constant, sparse, zstd | Dict, zstd, byte split. No FoR or BitPacking without reinterpretation | Mixed | As (a) | DBP stays decimal-only | +| Statistics | Integer min, max, and sum are meaningful | Memcmp min and max are wrong for numbers | Mixed | As (a) | Existing | +| Ordering | Native | Wrong for little-endian bytes | Wrong above 64 bits | Native | Native | + +Interop: + +| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| --- | --- | --- | --- | --- | --- | +| Arrow | Decimal export zero-copy at 128 and 256 bits. A top-level `Primitive(I128)` needs a policy, because Arrow has no Int128 | Decimal export zero-copy | Zero-copy | As (a) | As today | +| DuckDB | DECIMAL as today. HUGEINT becomes possible | DECIMAL needs reinterpretation. No HUGEINT | No HUGEINT | HUGEINT possible | No HUGEINT | +| DataFusion | Decimals as today. No Int128 scalar exists | As today | As today | As today | As today | +| Python, NumPy | Python `int` scalars. NumPy has no 128-bit integer | As today | As today | As (a) | As today | +| Scalar value | `PValue::I128`, `PValue::I256` | Binary payload | Two shapes | New scalar variant | `DecimalValue` | + +Cost: + +| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| --- | --- | --- | --- | --- | --- | +| Decimal wire | No change | Needs FSB first | Needs FSB first | No change | No change | +| New wire values | Flatbuffer enum values, a new protobuf oneof, edition gating. Only for top-level wide columns | FSB tag or extension descriptor | As (b) | New dtype tag if user-visible | None | +| Blast radius | Largest: about 380 macro uses, about 45 exhaustive matches, 53 `is_int()` sites, 12 mirrors. The compiler finds most of them | An FSB type: the closed prototype PR #8865 changed 118 files | As (b), plus two decimal paths | A dtype variant and a canonical variant: 51 canonical sites plus dtype matches | Smallest | +| Alignment | 16 bytes for both wide widths. Fix two `byte_width == alignment` assumptions | 1 byte by default. Typed access needs 16 | Mixed | As (a) | Unchanged | +| Duplicated logic | Lowest. Decimal kernels become rewrites | High | Medium | Medium | High, about 7,600 lines stay | + +### 4.4 Verdict + +Choose (a), built as a distinct wide-integer tier (section 5). The tier keeps the useful part of +(d): wide integers opt into a small, explicit kernel set instead of pretending to be lane +integers. + +The suspected dependency in the request does not hold. Wide PTypes do not need FixedSizeBinary: + +- Rust stores `i128` natively, and Vortex already has `i256` with checked arithmetic. +- The structural layer that both need already exists. `FixedWidthArray` moves records of 1, 2, + 4, 8, 16, and 32 bytes for take and filter (`vortex-array/src/arrays/fixed_width/mod.rs:16-47`). +- Arrow export of decimals uses the Arrow decimal types. Only a top-level wide-integer column + needs an export policy, and `FixedSizeBinary(16)` is one possible policy, not a prerequisite. +- Compression of wide integers is a limb split into lane integers, which DBP already implements. + +The two share the fixed-width structural layer and, later, the "execution layouts" idea of +RFC 0062. A UUID can use 128-bit lanes for equality and hashing, and a wide integer can use a +limb split for comparisons. Neither type carries the semantics of the other. + +Evidence that changes the decision: + +- Maintainers can decide that wide integers never become a Vortex type, or that the PType + migration is too large. Then fall back to (e) for one release. Require storage at least as wide + as the precision (node N11), and share generic lane kernels at the function level. This fixes + the crash class of issue #5820, but Decimal stays a separate world. +- Benchmarks can show that the generic integer kernels at 128 and 256 bits are clearly slower + than the current decimal kernels. Then keep the decimal kernels as specializations behind the + rewrites. This result is unlikely, because the current decimal kernels already run on `i128` + and `i256` lanes. +- If a requirement appears for precisions above 76 or arbitrary precision (for example PostgreSQL + NUMERIC), no fixed-width child works, and a variable-width representation is needed. + +## 5. Design of the wide-integer tier + +### 5.1 Types + +```rust +#[repr(u8)] +pub enum PType { + U8 = 0, U16 = 1, U32 = 2, U64 = 3, + I8 = 4, I16 = 5, I32 = 6, I64 = 7, + F16 = 8, F32 = 9, F64 = 10, + I128 = 11, + I256 = 12, + // 13 and 14 stay reserved for U128 and U256. +} + +/// Integers that fit a 64-bit lane. Only these can be offsets, codes, sizes, or take indices, +/// and only these reach FastLanes and other lane kernels. +pub enum LaneIntPType { U8, U16, U32, U64, I8, I16, I32, I64 } + +/// Signed integers wider than 64 bits. +pub enum WideIntPType { I128, I256 } + +impl PType { + pub const fn as_lane_int(self) -> Option; + pub const fn as_wide_int(self) -> Option; + /// True for every integer, lane or wide. + pub const fn is_integer(self) -> bool; + /// The required buffer alignment. Equal to the byte width up to 16 bytes, then 16. + pub const fn alignment(self) -> Alignment; +} +``` + +Trait plan: + +- Implement `NativePType` for `i128` and `i256`. `i128` already has every supertrait. `i256` + needs `Num`, `NumCast`, `FromPrimitive`, `Bounded`, and the byte conversion traits, about two + hundred lines over the Arrow type. Most `match_each_native_ptype!` bodies are generic over + `NativePType`, so they then compile and run correctly at the new widths. +- Keep `IntegerPType` (which requires `PrimInt`) for lane integers only. +- Add one trait for any-width integer kernels, for example `NativeIntPType: NativePType + Ord + + Hash + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv + BigCast`, implemented for the eight + lane integers and the two wide ones. Compare, arithmetic, sum, min and max, casts, and the + scaled-integer family bound on it. +- Fold `NativeDecimalType` into it: the precision bound tables become a `Pow10` trait on the + signed integers, and `DecimalType` survives only to decode legacy metadata. + +### 5.2 Dispatch macros + +| Macro | Arms | Use | +| --- | --- | --- | +| `match_each_native_ptype!` | All 13 | Generic code over `NativePType` | +| `match_each_lane_int_ptype!` | 8, takes `LaneIntPType` | Offsets, codes, indices, lane encodings. Exhaustive, no panic arm | +| `match_each_int_ptype!` | 10 | Any-width integer kernels | +| `match_each_wide_int_ptype!` | 2 | Wide-only code, such as the limb split | + +Retire `match_each_integer_ptype!` and its siblings after the migration, because their +`other => panic!` arm turns a type mistake into a run-time panic +(`vortex-array/src/dtype/ptype.rs:562`, `:589`, `:616`, `:640`). A caller that holds a `PType` +converts with `as_lane_int()` and returns a typed error. + +### 5.3 Migration technique + +Let the compiler enumerate the sites: + +1. Add `LaneIntPType`, `WideIntPType`, and the new macros while `PType` still has 11 variants. +2. Remove `PType::is_int` and add `is_lane_int` and `is_integer`. Every one of the 53 call sites + stops compiling and must choose. About 40 of them guard offsets, codes, indices, or lane + encodings and choose `is_lane_int`. +3. Migrate `match_each_integer_ptype!` uses (169) and `match_each_unsigned_integer_ptype!` uses + (118) to the typed macros, crate by crate. +4. Add the two variants last. Every exhaustive match (about 45 outside `ptype.rs`) and every + mirror then fails to compile until it handles them. +5. Teach the fuzzer to generate decimal children at every width before exposing wide columns. + +### 5.4 Wire encoding and gating + +- **Flatbuffers.** Append `I128` and `I256` to `enum PType: uint8` + (`vortex-array/flatbuffers/vortex-dtype/dtype.fbs:4-16`). Old readers fail loudly with + "Unknown PType variant" (`vortex-array/src/dtype/serde/flatbuffers.rs:559`). +- **Protobuf.** Do not append to `enum PType` in `dtype.proto`. Old readers decode an unknown + value as `U8` without an error (`vortex-array/src/dtype/serde/proto.rs:36`, prost-derive + getter semantics in section 2.4). Variant shredding and Parquet-variant metadata persist + protobuf dtypes on disk, so a silent misread corrupts data. Add a new oneof variant + instead, for example `WideInteger wide_integer = 14`, which old readers reject with + "Unrecognized DType" (`proto.rs:32`). Also change new readers to decode `PType` with + `TryFrom` and fail on unknown values. +- **Editions.** Add a component kind for core dtype features, or treat `vortex.primitive.i128` + and `vortex.primitive.i256` as dtype members, and check them in `validate_dtype_editions` + (`vortex-file/src/writer.rs:429-475`). Use the same mechanism for any later core dtype tag, + including FixedSizeBinary if it gets a tag. Decimal columns do not need it, because their + storage never reaches the wire (section 3.5). + +### 5.5 Scalars + +- Add `PValue::I128(i128)` and `PValue::I256(i256)`. `PValue` equality and hashing upcast within + each signedness family today (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-178`). + Extend the upcast target to `i256` for signed values. +- Serialize them as protobuf `bytes_value` holding 16 or 32 little-endian bytes, decoded by the + dtype, the same way decimals work (`vortex-array/src/scalar/proto.rs:402-445`). Old readers + reject `bytes_value` for a primitive dtype (`proto.rs:442-444`), which is a loud failure. +- Decide later whether `DecimalValue` becomes a storage `PValue` plus the dtype. In phase 2 it + can stay, normalized to the canonical width at construction. + +### 5.6 Engine policy for top-level wide columns + +| Engine | `Primitive(I128)` | `Primitive(I256)` | +| --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `i128` needs 39 digits) or `FixedSizeBinary(16)` through an export plugin | Error by default. Opt-in `FixedSizeBinary(32)`. `Decimal256` holds at most 76 digits, and `i256` needs 77 | +| DuckDB | HUGEINT, zero-copy little-endian | No type. Error | +| DataFusion | Error | Error | +| Python | Python `int` scalars. Arrays follow the Arrow policy | Same | +| FFI, C++, Java | New enum values in `vx_ptype` and mirrors | Same | +| CUDA | Not supported in dynamic dispatch at first | Same | + +### 5.7 U128 and U256 + +Defer them. Decimal needs only signed widths. The only current consumer is DuckDB UHUGEINT +(`vortex-duckdb/src/convert/dtype.rs:101`). Reserve the wire values. Until they exist, +`to_unsigned()` and `to_signed()` must not be called on wide types. Audit their 77 call sites +during step 3 of section 5.3, because they reinterpret lane integers for indices and codes. + +### 5.8 What wide integers deliberately do not get + +- No FastLanes BitPacking, FoR, or Delta on 128-bit lanes. The `fastlanes` lanes stop at 64 + bits. Compression goes through narrowing or the limb split. +- Never offsets, sizes, codes, or take indices. +- No CUDA dynamic dispatch at first. The decimal device kernels keep their own width handling. + +## 6. FixedSizeBinary design + +### 6.1 Where the discussion stands + +- RFC 0055 (branch `ct/types`) and its earlier draft RFC 0029 (branch `ngates/rfc-0029-types`) + apply a two-step test. The first step asks if the type is semantically distinct, or if a + refinement predicate gates operations. The second step asks if the type is core or an + extension. Both drafts leave FixedSizeBinary unresolved. +- RFC 0062 makes `FixedSizeBinary(n)` a core variant with contiguous fixed-width canonical + storage and whole-value validity, and makes UUID an extension over `FixedSizeBinary(16)`. It + states that "Any UUID ordering must be specified separately before a byte or integer kernel can + implement it." +- Issue #9902 asks for first-class support. One user stores trigrams as `FixedSizeBinary(3)`. + The discussion there raises power-of-two over-alignment and the cost of filtering a contiguous + layout. +- PR #8865 (closed, not merged) prototyped it in 118 files (+4,021, -2,004), stacked on the + merged `FixedWidthArray` refactor (PR #8864). It used `DType::FixedSizeBinary(u32, Nullability)`, + a contiguous buffer with an explicit length for zero-width values, a binary scalar payload, + byte-lexicographic ordering, and a new flatbuffer tag. +- PR #9886 (closed) shows the current gap: a DataFusion `FixedSizeBinary` literal had to become + a `FixedSizeList` scalar to compare against a UUID column. +- UUID today is `vortex.uuid` over `FixedSizeList(Primitive(U8, non-null), 16)` + (`encodings/uuid/src/vtable.rs:53-78`), frozen in edition core 2026.08.3 + (`vortex-edition/src/declarations/core/v2026_08_3.rs:23`). Its scalar unpacking walks sixteen + `PValue::U8` list elements (`vtable.rs:78-115`). + +### 6.2 Is it justified, and is it core? + +Apply the RFC 0055 test: + +- **Refinement predicate.** `FixedSizeBinary(n) = { b : Binary | len(b) = n }`. The predicate + gates a fixed-width canonical layout, constant-time row addressing, and scalar-compatible + values, exactly as `FixedSizeList` refines `List`, and `FixedSizeList` is already core. +- **Semantic distinctness from `FixedSizeList`.** Opaque bytes have byte-string + operations (memcmp order, prefix, conversion to `Binary`). A list of bytes has element + operations. The distinction is real but weak. +- **Core or extension.** Core Vortex owns the operations that matter: the canonical layout, + equality, hashing, ordering, Arrow and Parquet mapping, and scalar representation. RFC 0056 + (branch `ngates/scalar-values`) keeps `Scalar` only for scalar-compatible values and moves + nested values, including extensions over nested storage, to arrays. Under that rule a UUID + over `FixedSizeList` has no cheap scalar, while a UUID over `FixedSizeBinary(16)` does. + +Conclusion: make it core, as RFC 0062 proposes. + +### 6.3 Canonical layout + +- One buffer of `len * n` bytes, whole-value validity, and an explicit length so that `n = 0` + works. This is the PR #8865 layout. +- Implement `FixedWidthArray` so take and filter come from the shared kernels. Those kernels + already have typed paths for 1, 2, 4, 8, 16, and 32 bytes and a fallback for other widths. +- Alignment: 1 byte by default. Allow a writer to request power-of-two alignment for + power-of-two widths. The file format records alignment per buffer + (`vortex-array/flatbuffers/vortex-array/array.fbs`, `struct Buffer`), so this is free on the wire. + +### 6.4 Operations + +| Operation | Definition | +| --- | --- | +| Equality, hashing | Bytes | +| Ordering | Byte-lexicographic (memcmp), the same as `Binary` and Arrow | +| Cast to and from `Binary` | Explicit. From `Binary` validates every length | +| Cast to and from `FixedSizeList` | Zero-copy when the elements are a flat non-nullable `u8` array | +| Arithmetic, numeric casts | None. Never implicit | + +FixedSizeBinary must never carry integer semantics. A type that wants integer behavior over +fixed bytes uses an integer type, or it asks for an explicit reinterpretation with a declared +byte order. A 128-bit lane can implement equality and hashing for UUID, because those do not +depend on byte order. It must not implement ordering without the big-endian sign-flip transform. + +### 6.5 Scalars and compression + +- Scalars: `ScalarValue::Binary` with a validated length. The value is scalar-compatible under + RFC 0056. +- Compression: Dict, constant, run-end, sparse, zstd, and a byte-lane split. The split + transposes the bytes into `n` lane columns, or into 64-bit words, and integer schemes compress + the lanes. The limb split for wide integers (node N14) and the lane split for FixedSizeBinary + can share code at the word level. They keep different dtypes. + +### 6.6 Wire representation + +| Option | Old readers | Gating | +| --- | --- | --- | +| New core tag (the next free flatbuffer union value and protobuf oneof field) and a new `vortex.fixed_size_binary` array id | Fail loudly on the schema | Needs the core dtype feature gate of section 5.4 | +| Extension descriptor `vortex.fixed_size_binary` over `FixedSizeList` storage, and the array written in that storage form | Read it as a foreign extension over `FixedSizeList`, so the data stays readable (`vortex-array/src/dtype/extension/foreign.rs:26-75`) | The existing dtype edition kind | + +The second option is the inverse of the decimal plan: the Rust API has a core variant, and the +wire uses an extension descriptor. The rule behind both is the same: choose the wire form for +reader compatibility and the Rust form for kernels and API. Translate at the serialization +boundary. Section 9 keeps the choice open. + +### 6.7 UUID on FixedSizeBinary + +- In memory, `vortex.uuid` storage becomes `FixedSizeBinary(16)`. +- `Uuid::validate_dtype` accepts both `FixedSizeBinary(16)` and the frozen + `FixedSizeList` storage, and the reader normalizes the second form zero-copy. +- The writer keeps writing the frozen `FixedSizeList` storage until a new edition declares the + new storage. The conversion is zero-copy for flat data. The existing Arrow plugin already does + the same conversion (`encodings/uuid/src/arrow.rs`, `try_fsl_to_fsb`). +- The DuckDB exporter keeps its big-endian sign-flip transform. + +### 6.8 Relation to decimals and wide integers + +None is required. FixedSizeBinary, Primitive (with wide integers), and Decimal storage share only +the fixed-width structural layer. That layer is the right abstraction to grow. PR #8864 started +it, and Epic #7706 suggests it ("a `Flat` encoding that allows for any aligned fixed-size binary +values"). + +## 7. The type-system dependency graph + +### 7.1 Nodes + +Kind: **impl** means implementation work with a known shape. **research** means a design decision +must come first. **urgent** means a correctness bug that exists today. + +| Id | Node | Kind | Notes | +| --- | --- | --- | --- | +| N1 | Extension opacity fixes | impl, urgent | Stop storage casts, storage-and-wrap scalar casts, lone-side storage compares, storage zero values | +| N2 | Bind-time rewrite infrastructure | research, then impl | `ext_storage` and `ext_wrap` style nodes, rewrites keyed by function and type ids | +| N3 | Pairwise derives | impl | Eq, Ord, Hash as rewrites that see both operand dtypes | +| N4 | Cast hooks (#8570) | impl | `cast_to` and `cast_from` on `ExtVTable`, bound `CastFn` | +| N5 | Array validation at trust boundaries | impl | `validate_array`: decimal precision, UTF-8, FixedSizeBinary length | +| N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes | +| N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width | +| N8 | Scaled-integer kernels | impl | `rescale`, `check_range`, fused scaled arithmetic and compare | +| N9 | Wide-integer tier (I128, I256) | impl | Section 5 | +| N10 | Wire safety and core dtype gating | impl | Strict enum decoding, protobuf oneof for wide PTypes, edition kind for core dtype features | +| N11 | Decimal storage at least as wide as the precision | impl, urgent | Option 3 of issue #5820. Removes the crash class | +| N12 | Decimal wrapper over an integer child | impl | Section 3, with the legacy wire plugin | +| N13 | Decimal semantics as storage rewrites | impl | Delete the decimal-specific kernels | +| N14 | DBP generalized as a wide-integer limb split | impl, optional | DBP moves under the wrapper and loses its decimal dtype | +| N15 | Decimal as a well-known extension in Rust | impl, later | RFC 0062. Wire tag unchanged | +| N16 | Wire translation layer for well-known types | research, then impl | Rust representation and wire representation chosen separately | +| N17 | FixedSizeBinary core type | impl | Section 6 | +| N18 | UUID on FixedSizeBinary | impl | Accept frozen storage, write frozen storage | +| N19 | Utf8 as a well-known extension | research, low priority | 776 lines in 212 files mention `DType::Utf8` | +| N20 | FixedSizeList and Map as well-known extensions | research, low priority | Needs requested layouts for the offset-free FixedSizeList form | +| N21 | Duration and Interval extensions (PRs #10467, #10468) | impl, in flight | Both PRs leave edition membership open | +| N22 | Execution layouts (RFC 0062) | research | Kernels request acceptable layouts, for example a limb split or 128-bit lanes | +| N23 | Engine type plugins for DuckDB and DataFusion | impl | Arrow already has plugin registries (`vortex-arrow/src/session.rs`) | +| N24 | Refinement types (RFC 0055) | research only | UUID within FixedSizeBinary(16), Utf8 within Binary | +| N26 | Scalar value model (RFC 0056) | research, then impl | Decides which types keep a host `Scalar` | +| N27 | DuckDB HUGEINT import and export | impl | First user-visible wide-integer column | + +(There is no N25. The Variant interaction is an edge on N10.) + +Not in the graph: Union completion. Union casts and several aggregates are still TODOs +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:223-228`). That work is independent of this graph. +It gains from the shared structural rules of section 3.3. + +### 7.2 Graph + +```mermaid +flowchart TD + classDef urgent fill:#f8d7da,stroke:#842029,color:#000 + classDef critical fill:#fff3cd,stroke:#997404,color:#000 + classDef impl fill:#d1e7dd,stroke:#0f5132,color:#000 + classDef research fill:#e2e3e5,stroke:#41464b,color:#000,stroke-dasharray: 4 3 + + N1["N1 Extension opacity fixes"]:::urgent + N11["N11 Decimal storage at least canonical width"]:::urgent + N2["N2 Bind-time rewrite infrastructure"]:::research + N3["N3 Pairwise derives"]:::impl + N4["N4 Cast hooks #8570"]:::impl + N5["N5 Array validation"]:::impl + N6["N6 Nullability restructure"]:::research + N7["N7 Fixed-width structural layer"]:::critical + N8["N8 Scaled-integer kernels"]:::impl + N9["N9 Wide-integer tier I128 I256"]:::critical + N10["N10 Wire safety and core dtype gating"]:::impl + N12["N12 Decimal wrapper over integer child"]:::critical + N13["N13 Decimal semantics as rewrites"]:::critical + N14["N14 DBP as wide-integer limb split"]:::impl + N15["N15 Decimal as well-known extension"]:::critical + N16["N16 Wire translation layer"]:::research + N17["N17 FixedSizeBinary core type"]:::impl + N18["N18 UUID on FixedSizeBinary"]:::impl + N19["N19 Utf8 as extension"]:::research + N20["N20 FixedSizeList and Map as extensions"]:::research + N21["N21 Duration and Interval"]:::impl + N22["N22 Execution layouts"]:::research + N23["N23 Engine type plugins"]:::impl + N24["N24 Refinement types"]:::research + N26["N26 Scalar value model"]:::research + N27["N27 DuckDB HUGEINT"]:::impl + + N1 --> N15 + N1 --> N21 + N1 -.-> N18 + N2 --> N3 + N2 -.-> N13 + N3 --> N15 + N3 -.-> N21 + N4 --> N15 + N4 --> N21 + N8 -.-> N4 + N8 --> N13 + N7 -.-> N9 + N7 -.-> N17 + N11 --> N12 + N9 --> N12 + N9 --> N14 + N9 --> N27 + N10 --> N27 + N10 -.-> N9 + N10 -.-> N17 + N12 --> N13 + N12 --> N14 + N13 --> N15 + N16 --> N15 + N16 -.-> N18 + N16 --> N19 + N16 --> N20 + N17 --> N18 + N22 --> N19 + N22 --> N20 + N23 -.-> N21 + N23 -.-> N27 + N23 -.-> N18 + N5 -.-> N12 + N5 -.-> N17 + N26 -.-> N17 + N24 -.-> N18 + N24 -.-> N19 + N6 -.-> N15 +``` + +Solid arrows are hard dependencies. Dashed arrows are soft: the target works without the source, +but it is cheaper, safer, or faster with it. + +### 7.3 Edges with reasons + +1. N1 to N15: as an extension on `develop`, Decimal inherits storage casts and lone-side storage + compares (`arrays/extension/compute/cast.rs:17-22`, + `scalar_fn/fns/binary/compare/mod.rs:177-194`). +2. N1 to N21: without N1, `Duration(ms)` and `Duration(ns)` compare and cast raw `i64` storage. + New types must not ship with a known leak that users can come to depend on. +3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The + engine literal paths then change once, before the storage changes. PR #9886 shows the + current workaround. +4. N2 to N3: pairwise derives are rewrites over both operands. +5. N2 to N13 (soft): expression-level rewrites let zone maps and compressed integer kernels + evaluate decimal predicates on the integer child. +6. N3 to N15: decimal comparison across scales is a pairwise rescale. +7. N3 to N21 (soft): Duration comparison across units is the same pairwise rescale. Until N3 + lands, a comparison across units is rejected, which is correct but limited. +8. N4 to N15: rescaling casts and casts to and from integers and floats must be type-owned. +9. N4 to N21: unit conversions are casts. +10. N8 to N4 (soft): Timestamp, Time, and Duration unit casts are powers of ten, except Days. +11. N8 to N13: decimal rewrites need `rescale` and `check_range`. +12. N7 to N9 (soft): width-keyed structural kernels give wide integers take, filter, run-end, + sparse, dict, and constant support without new per-type code. +13. N7 to N17 (soft): width-keyed structural kernels give FixedSizeBinary run-end, sparse, dict, + and constant support without new per-type code. Take and filter already come from + `FixedWidthArray`. +14. N11 to N12: N11 forbids narrow storage, which removes the crash class and the "does not fit" + special cases. The wrapper refactor then starts from a consistent base. +15. N9 to N12: the wrapper child for precision 19 to 76 is `Primitive(I128)` or `Primitive(I256)`. +16. N9 to N14: the limb split is an encoding of wide integers. +17. N9 to N27: HUGEINT maps to `Primitive(I128)`. +18. N10 to N27: a top-level wide column writes new dtype values. An edition must gate them, and + the protobuf encoding must fail loudly in old readers. +19. N10 to N9 (soft): ship the strict enum decoding in readers before any writer can emit wide + PTypes. Variant shredding stores protobuf dtypes on disk + (`vortex-array/src/arrays/variant/vtable/mod.rs:54`). +20. N10 to N17 (soft): needed only if FixedSizeBinary gets a new core wire tag. +21. N12 to N13: the rewrites rely on the fixed storage dtype. +22. N12 to N14: the limb split lives under the wrapper. +23. N13 to N15: once decimal semantics are storage rewrites, the move to an extension changes + the API, not the behavior. +24. N16 to N15: the `Decimal` wire tag must translate to and from the extension, recursively. +25. N16 to N18 (soft): UUID storage is `FixedSizeBinary(16)` in memory and `FixedSizeList` on + the wire. +26. N16 to N19 and N16 to N20: the `Utf8`, `FixedSizeList`, and `Map` tags need the same + translation. +27. N17 to N18: UUID storage needs FixedSizeBinary. +28. N22 to N19 and N22 to N20: Utf8 views and the offset-free FixedSizeList layout must stay + available when those types become extensions. +29. N23 to N21, N27, and N18 (soft): with plugins, new types get DuckDB and DataFusion mappings + without edits to central `match` statements (`vortex-duckdb/src/convert/dtype.rs`, + `vortex-datafusion/src/convert/scalars.rs`). Edits to the central statements also work. +30. N5 to N12 (soft): import boundaries establish invariant I2 (values within the precision). +31. N5 to N17 (soft): a cast from `Binary` must check every length. FixedSizeBinary can check + lengths in its own constructors and casts, so the shared hook only avoids a second mechanism. +32. N26 to N17 (soft): a FixedSizeBinary scalar is a binary payload, which RFC 0056 treats as + scalar-compatible. +33. N24 to N18 and N24 to N19 (soft): a refinement framework can express these types, but + neither one depends on it. +34. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every + nullability change rebuilds the extension dtype + (`vortex-array/src/dtype/extension/typed.rs:91-99`). + +### 7.4 Critical path and parallel work + +- **Critical path for fixing Decimal:** N11 runs first and alone. The hard chain is N9, N12, + N13. N8 must land before N13. N7 is soft, but it shrinks N9, so do it first. +- **Critical path for the RFC 0062 end state:** N9, N12, N13, N15, with N1, N2 then N3, N4, and + N16 joining at N15. +- **Parallel tracks:** + - Extension correctness: N1, N2, N3, N4, N5. + - Decimal: N11, N7, N8, N9, N12, N13, N14. + - Bytes: N17, N18. + - Temporal: N21, after N1 and N4. + - Wire and engines: N10, N16, N23, N27. +- **Research-only nodes:** N6, N22, N24. **Research before implementation:** N2, N16, N26, N19, + N20. + +### 7.5 Mapping to the extension-type research + +The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to +U8) and phases (0 to 6). This table maps them onto the graph. + +| Extension research item | Node here | +| --- | --- | +| U1 one cast rule, U2 one compare rule, U7 zero values | N1 | +| U3 errors from reads | Not a separate node. It belongs with N1 | +| U4 infallible nullability changes | Not a separate node. It removes the panic in the rebuild that edge 34 describes | +| U5 temporal validation, U6 `Extension::validate` checks the storage | N5 | +| U8 debug output | Not in this graph | +| Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 | +| Phase 2, binder and derives | N2 and N3 | +| Phase 3, casts | N4 | +| Phase 4, normal form and storage statistics | The shared rules of section 3.3, used by N12. DateTimeParts under the wrapper is the temporal twin of N14 | +| Phase 5, validation and nullability | N5 and N6 | +| Phase 6, authoring helpers and engine export plugins | N23 | + +The array-level decimal fix (N11, N12, N13) does not wait for the extension phases, because the +built-in Decimal wrapper owns its rewrites (section 3.4). The RFC 0062 end state (N15) needs +phases 0, 2 and 3 first (N1, N3 and N4). Part 5 of the extension notes gives the same order. + +## 8. Recommendations + +### 8.1 Decision + +Give Decimal an integer child. Add `I128` and `I256` as a wide-integer tier with typed dispatch. +Keep the decimal wire format and the `DType::Decimal` variant for now. Build FixedSizeBinary on its +own track for opaque bytes. Revisit the Rust-level move to a well-known extension after the +decimal rewrites land. + +Reasons, in order of weight: + +1. Every decimal operation is an integer operation plus scale bookkeeping. An integer child turns + the decimal kernels into rewrites over shared integer kernels. A FixedSizeBinary child keeps + every decimal kernel and adds a reinterpretation step. +2. Byte order is not integer order for the layout that Arrow and DuckDB use (section 4.2). +3. Integer statistics, integer compression, and zero-copy Arrow and DuckDB decimal exchange all + work unchanged with an integer child. +4. The decimal wire format needs no change, because the wrapper reads and writes the legacy forms. +5. The cost is a large but mechanical PType migration that the compiler can drive. + +### 8.2 Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| The PType migration touches 36 crates (counting each encoding crate) and several languages | Typed sub-enums and macros, compiler-driven steps (section 5.3), one crate per PR | +| A leftover `other => panic!` arm panics on a wide type | Retire the panic macros. Fuzz decimals at every width | +| Old readers silently read a protobuf wide PType as `U8` | Use a new protobuf oneof. Ship strict readers early (N10) | +| Rewritten arithmetic makes several passes | Fused scaled-integer kernels (N8). Compare before and after with the existing `vortex-array/benches/binary_ops.rs`, `cast_decimal.rs`, and `compare.rs` | +| Arrow import of `Decimal128` at low precision loses zero-copy | Lazy `checked_narrow` child. The Arrow exporter can reuse the original buffer | +| Public API breaks: `DecimalArray::buffer::()`, `values_type()`, `DecimalBuilder::new::` | Deprecation shims that convert. Keep the in-memory id `vortex.decimal`, which the Python bindings key on (`vortex-python/src/arrays/native.rs:174-175`) | +| Wide integers become visible to users before engines can export them | Writer gating (N10) and an explicit engine policy (section 5.6) | + +### 8.3 Sequence + +Wave 0, small and independent: + +1. Strict decoding: `DecimalType::try_from(raw)` in the decimal deserializer + (`vortex-array/src/arrays/decimal/vtable/mod.rs:170`) and `PType::try_from(raw)` in the + protobuf dtype reader (`vortex-array/src/dtype/serde/proto.rs:36`). Part of N10. +2. N11: require `values_type >= canonical width` in `DecimalData::validate`. Make DBP assembly + widen a narrow MSP. Stop the btrblocks v1 path from writing narrowed `vortex.decimal` nodes. + Widen legacy narrow nodes on read. Add compat fixtures for narrow, wide, and Arrow-style + widths first. The `patch` panic, the fill-null upcast, and the "does not fit" constant tests + become unreachable for same-dtype inputs and can go. Update tests that build narrow arrays + on purpose, for example `decimal_narrow_storage` + (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). +3. Add `DecimalDType::storage_type()`, returning `DecimalType` until N9 lands, and route the 25 + `smallest_decimal_value_type` calls through it so that the width function has one owner. +4. N1: the extension opacity fixes. + +Wave 1, foundations: + +5. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. +6. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for + the Timestamp unit casts of N4. +7. N4: cast hooks (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. +8. N2, then N3: settle the rewrite design, then add pairwise derives. This is phase 2 of the + extension notes. Use the decimal rewrites of section 3.4 as acceptance tests. +9. N23: plugin registries for DuckDB and DataFusion type mapping. + +Wave 2, new types: + +10. N9 in the order of section 5.3, without user-visible wide columns. +11. N10: the core dtype edition kind and the protobuf oneof. +12. N17: FixedSizeBinary, after the wire representation decision. +13. N5: `validate_array`, starting with decimal precision at import boundaries. + +Wave 3, decimal restructure: + +14. N12: the wrapper and the legacy plugin. +15. N13: rewrites. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, `decimal/utils.rs`, + the decimal aggregates, and the per-width builder. +16. N14: DBP under the wrapper, optional. +17. N18 and N27. + +Wave 4, API unification: + +18. N16, then N15. Revisit N19 and N20 with the results of N22. + +## 9. Open questions + +1. Are `Primitive(I128)` and `Primitive(I256)` user-visible columns from the start, or storage + only until an engine needs them? DuckDB HUGEINT is the only current consumer. +2. Does `is_int()` mean any integer or a lane integer? Section 5.3 proposes to remove it, so that + each of the 53 call sites chooses. +3. Are `widen` and `checked_narrow` lazy `Cast` scalar-function arrays, or a dedicated in-memory + encoding? +4. Which crate owns the decimal and DBP serialized ids once DBP moves under the wrapper: + `vortex-array` or `encodings/decimal-byte-parts`? +5. Does FixedSizeBinary use a new core wire tag, or an extension descriptor over + `FixedSizeList` that old readers can still read? +6. After N13, is the move of `DType::Decimal` into a Rust extension still worth its API cost? +7. Does Vortex implement decimal arithmetic and comparison across different `(p, s)` pairs? The + alternative keeps the rule of equal dtypes, and engines coerce first. +8. Do file reads check decimal precision, or only external imports? Zone-map minimum and maximum + statistics can make the file check cheap. +9. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist + (`vortex-array/src/dtype/decimal/mod.rs:47`). Tests use -128. DuckDB rejects negative scales, + and Arrow allows them. +10. Does the storage width function keep the I8 minimum, or align with an engine? Arrow device + export uses an I32 minimum, and DuckDB an I16 minimum. The choice is in-memory only, so it + can change later. +11. PRs #10467 and #10468 leave two decisions open. One is the order of intervals: field-wise as + in Arrow, or normalized as in DuckDB. The other is the edition membership of + `vortex.duration` and `vortex.interval`. +12. When do `U128` and `U256` become worth adding? +13. Is the wrapper always on top of generic encodings (`Constant`, `Dict`, `Chunked`, `Sparse`)? + Section 3.3 says yes, like part 3 of the extension notes. The alternative keeps generic + encodings above the wrapper, as `ExtensionConstantRule` does today. + +## 10. Evidence log and limits + +Code read at commit `5fc5cb8`: the files cited in sections 2 to 6, plus the supporting +inventories. Every count in this report comes from `grep` over the workspace, excluding `target`. + +RFC drafts read from the `vortex-data/rfcs` repository: + +- Branch `ngates/types`: `rfcs/0062-value-types-and-representations.md`, read in full. +- Branch `ct/types`: `rfcs/0055-type-system.md`, read in full. +- Branch `ngates/rfc-0029-types`: `rfcs/0029-types.md`, read in full. It is an earlier draft of + RFC 0055 with the same decision framework and the same open FixedSizeBinary question. It frames + `DType` as a quotient type and canonicalization as a section, where RFC 0055 talks about + canonical execution. +- Branch `ngates/rfc-0005-extension`: `rfcs/0005-extension.md`, read in full. +- Branch `ngates/scalar-values`: `rfcs/0056-scalar-values.md`, read in part: the design sections + and every mention of decimals, extensions, and binary values. +- Branch `adamg/map-type-rfc`: `rfcs/0063-map-type.md` is an empty template. +- Branch `ngates/execution`: `rfcs/0019-array-execution.md`, skimmed. It is not relevant here. + +GitHub, read only: + +| Reference | Topic, and the depth of reading where it was recorded | +| --- | --- | +| [Issue #5820](https://github.com/vortex-data/vortex/issues/5820) | Decimal logical and physical mismatch. The body and all five human comments in full, and the 21 automated crash reports at the level of their summaries and errors | +| [Epic #7706](https://github.com/vortex-data/vortex/issues/7706) | Type system | +| [Issue #8570](https://github.com/vortex-data/vortex/issues/8570) | Cast hooks for extension types | +| [Issue #9902](https://github.com/vortex-data/vortex/issues/9902) | FixedSizeBinary, with its five comments | +| [Epic #10171](https://github.com/vortex-data/vortex/issues/10171) | Time series | +| [PR #8864](https://github.com/vortex-data/vortex/pull/8864) | Fixed-width sharing, merged | +| [PR #8865](https://github.com/vortex-data/vortex/pull/8865) | FixedSizeBinary prototype, closed. The description, the comments, and the first 100 of its 118 changed files | +| [PR #9886](https://github.com/vortex-data/vortex/pull/9886) | DataFusion literal for a UUID column, closed | +| [PR #10467](https://github.com/vortex-data/vortex/pull/10467) | Duration extension type. The files and the comments | +| [PR #10468](https://github.com/vortex-data/vortex/pull/10468) | Interval extension type. The file list and the new `interval.rs` | + +Verified outside the repository: + +- prost-derive 0.14.4 enum getters fall back to the first variant on unknown values + (`src/field/scalar.rs:286-300`, `src/lib.rs:308` in the cargo registry). +- arrow-buffer 59.3.0 `i256` layout and order (`src/bigint/mod.rs:59-66`, `:171-183`). +- Byte order against integer order, with the script in appendix A. + +A script checked that every `file:line` reference in this report points at an existing range, and +the first line of each range was reviewed. The external crate references above were checked by +hand. A second review checked the main claims of sections 1 and 2 by hand again. + +Not verified, because building or running Rust code was out of scope: + +- Any performance statement, including the cost of 128-bit and 256-bit kernels and the effect of + rewrites on fused arithmetic. +- The exact set of compile errors from the PType migration. The counts are a `grep` proxy. +- The array cast failure for decimal to `f32`. It comes from reading the dispatch code, not from + a run. +- The alignment of `i128` on targets other than x86_64 and aarch64, for example wasm32. +- Whether the fuzzer reports in issue #5820 that end in a null decimal sum come from narrow + storage. This report does not rely on that. + +## Appendix A: byte order script + +Run it with `python3 -I byte_order.py`. In this run, little-endian memcmp agreed with integer +order for 10,060 of 19,999 random pairs. Big-endian memcmp with the sign bit flipped agreed for all +19,999 pairs. + +```python +"""Compare memcmp order of 128-bit two's complement bytes with integer order.""" + +import random + + +def le(v: int) -> bytes: + return v.to_bytes(16, "little", signed=True) + + +def be_flip(v: int) -> bytes: + b = bytearray(v.to_bytes(16, "big", signed=True)) + b[0] ^= 0x80 + return bytes(b) + + +random.seed(7) +values = [random.randint(-(2**127), 2**127 - 1) for _ in range(20000)] +pairs = list(zip(values, values[1:])) +ok_le = sum((le(a) < le(b)) == (a < b) for a, b in pairs) +ok_flip = sum((be_flip(a) < be_flip(b)) == (a < b) for a, b in pairs) +print(f"little-endian: {ok_le} / {len(pairs)}") +print(f"big-endian, sign bit flipped: {ok_flip} / {len(pairs)}") +for a, b in [(1, 256), (-1, 0)]: + print(f"{a} < {b}: little-endian memcmp says {le(a) < le(b)}") +``` diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md index e7331d3511e..c7f61f3cdf3 100644 --- a/research/extension-types/03-design.md +++ b/research/extension-types/03-design.md @@ -112,6 +112,12 @@ type-specific encodings (`DateTimeParts`) live under the wrapper and see the sto removes the two shapes of EXT-S1 and EXT-S2. The compressor chooses a type-specific encoding from the extension metadata. +Then no generic encoding needs code for extension dtypes, and `ext_storage` only reads the child. +The cost is that code which builds a constant, a dictionary or chunks of an extension dtype must +put the wrapper on top. The [Decimal notes](../decimal/02-design.md) use the same rule for the +built-in `Decimal` wrapper. A writer can still produce an older wire shape with the wrapper lower +down when an older format needs it. + ### Statistics Zone maps and file statistics store storage statistics for extension columns. The rewrites in the @@ -155,6 +161,10 @@ plugins like the Arrow registry. Then a new extension type does not need code in mandatory rewrite pass that also covers the row path. - Open question: are rewrites owned only by the extension type? Or can a function owner register a rewrite for a type that it does not own? +- Open question: is the wrapper always on top of generic encodings? The alternative keeps generic + encodings above the wrapper, as `ExtensionConstantRule` does for constants today. That matches + the current code, but each generic encoding then needs a pushdown rule for `ext_storage`, or a + fallback that executes it. This proposal puts the wrapper on top. - Open question: are operations on values with different metadata rejected, or coerced to a common type? Rust and the built-in `Decimal` reject them today. SQL coerces them. - Open question: are `EQ` and `ORD` enough? A refinement over a storage dtype can need a third flag diff --git a/research/extension-types/05-implementation.md b/research/extension-types/05-implementation.md index f9bbcc0ec6e..28556401df5 100644 --- a/research/extension-types/05-implementation.md +++ b/research/extension-types/05-implementation.md @@ -112,5 +112,16 @@ build on the binder. ## Relation to the Decimal and type roadmap -The Decimal work in [`../decimal/`](../decimal/README.md) depends on phases 2 and 3. An opaque -Decimal wrapper uses the same derives, casts and normal form. +The [Decimal notes](../decimal/README.md) place these phases in a larger graph of type work. Three +points affect the order: + +- The array-level fix for `Decimal` does not wait for this plan. The built-in `Decimal` wrapper + owns its rewrites, so it does not need the `ExtVTable` hooks. +- A move of `Decimal` into an extension type needs phases 0, 2 and 3 first. Without them, a cast + from a decimal to an integer returns the unscaled values. +- Phase 3 and the Decimal work share one integer kernel family, `rescale` and `check_range`. + `Timestamp` unit casts and decimal rescales both use it. + +The Decimal rewrites are also good acceptance tests for the binder of phase 2. The decimal-like +reference type of phase 1 can run them against the built-in `Decimal`. Phase 4 and the Decimal +wrapper share one normal form. diff --git a/research/extension-types/README.md b/research/extension-types/README.md index cead7f700fa..c3e18dd6b74 100644 --- a/research/extension-types/README.md +++ b/research/extension-types/README.md @@ -38,3 +38,8 @@ That causes wrong results, panics, and code paths that disagree with each other. The findings come from reading `develop` at `5fc5cb8`, and from a throwaway test against `vortex-array` in a debug build. Findings marked "lead" in the agent catalog are not confirmed. + +## Related + +The [Decimal notes](../decimal/README.md) use `Decimal` as a test of this design. They also place +this plan in a larger graph of type work. diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md index 96dba437ba8..0572de92d5a 100644 --- a/research/extension-types/agents/design-notes.md +++ b/research/extension-types/agents/design-notes.md @@ -242,6 +242,15 @@ Consequences: the encoding from the extension metadata, and compresses the storage child. - The same applies to any future encoding for one extension type. +Open: the direction of the normal form. The alternative is "wrapper below generic encodings": +`Constant(extension scalar)`, `Dict(codes, Extension(d, values))` and `Chunked[Extension(d, a_i)]` +stay as they are, and only value-reading encodings must sit under the wrapper. That matches +`ExtensionConstantRule` and the legacy decimal wire form. It has two costs. `ext_storage` needs a +pushdown rule for each generic encoding, or a fallback that executes the encoding to the canonical +wrapper. Also, each generic encoding keeps an extension arm in its canonicalization. The wrapper on +top removes those arms. The Decimal report (`../../decimal/agents/report.md`, sections 3.3 and +3.5) uses the wrapper on top in memory, and its wire plugin writes the older shapes. + ## 7. Statistics - **S1.** Zone maps and file statistics for an extension column store statistics of the storage, From 033debe25b3718001599899d7eee4e593b41b800 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Mon, 19 Oct 2026 11:30:08 +0100 Subject: [PATCH 04/12] Refine extension type and Decimal research after review Derives are now a property of one extension dtype. A derive applies only when every operand in a derived position has that dtype. Operations across dtypes or metadata need explicit compatibility, coercion or rewrite rules, and the notes do not assume them. The design notes now define these parts: - A Rust contract for Scalar equality and order that keeps PartialEq and PartialOrd consistent. - Exact rules for the removal of ext_storage and ext_wrap pairs, for checked and unchecked wraps. - A validation model with validated and unvalidated arrays, and a default validate_array that checks each value. - Function declarations that authorize type-aware behavior, and a precise storage-only invariant. - The position of the wrapper relative to generic encodings, as an open decision with a labeled recommendation. A follow-up probe ran in vortex-btrblocks. It confirms the DateTimeParts compare bug (EXT-Q3), the Decimal to f32 cast disagreement, and a new finding about scalar storage checks (EXT-V7). The notes also correct the U5 and U6 node mapping, a claim about float equality, and a claim about the decimal wire form. Signed-off-by: Connor Tsui --- research/decimal/01-current-state.md | 6 +- research/decimal/02-design.md | 34 +- research/decimal/03-roadmap.md | 10 +- research/decimal/README.md | 7 +- research/decimal/agents/report.md | 130 ++-- .../extension-types/02-current-problems.md | 21 +- research/extension-types/03-design.md | 173 +++-- research/extension-types/04-urgent-changes.md | 18 +- research/extension-types/05-implementation.md | 57 +- research/extension-types/README.md | 21 +- .../extension-types/agents/design-notes.md | 658 +++++++++++++----- research/extension-types/agents/findings.md | 116 ++- .../extension-types/agents/probe_followup.rs | 522 ++++++++++++++ 13 files changed, 1426 insertions(+), 347 deletions(-) create mode 100644 research/extension-types/agents/probe_followup.rs diff --git a/research/decimal/01-current-state.md b/research/decimal/01-current-state.md index fe11dda73f9..cfd45819e29 100644 --- a/research/decimal/01-current-state.md +++ b/research/decimal/01-current-state.md @@ -29,7 +29,7 @@ All statements are for `develop` at `5fc5cb8`. Sections 2 and 3 of | One dtype has up to six physical forms | Kernels must handle mixed widths, widen on the fly, or test whether a constant fits the storage. `DecimalArray::patch` panics when the width cannot hold the dtype. Issue [#5820](https://github.com/vortex-data/vortex/issues/5820) collects the fuzzer crashes | | The canonical width is a convention, not a contract | 25 call sites compute it. The row codec uses it so that keys "from differently compressed chunks stay memcmp-comparable" | | Arrays never check the precision | Scalars check it, but arrays do not. So out-of-range values can reach kernels, and a comment in the arithmetic kernel says so | -| The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel" | +| The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel". A throwaway test confirmed the `f32` case | | `Decimal` duplicates `Primitive` | It has its own canonical variant (51 dispatch sites), builder, compare, arithmetic and aggregates. Decimal-specific files hold about 7,600 non-test lines | | DecimalByteParts carries the decimal dtype | It splits wide integers into 64-bit parts, but only decimals can use it | | Unknown enum values decode silently | An unknown `values_type` decodes as `I8`. An unknown protobuf `PType` decodes as `U8`, and Variant metadata stores protobuf dtypes in files | @@ -56,6 +56,10 @@ design: operations that derive from the storage, and operations that the type mu | `sum` | An integer sum with a wider result precision | Not applicable | | Casts | Owned by the type | Owned by the type | +One precision and scale is one dtype. In the terms of the extension design, `=`, `<`, hash, `min`, +`max` and sort within one precision and scale are derives for one dtype. The other operations are +rewrites that the type owns. + Equality and hashing need care. On `develop`, one value can have several physical widths, so the storage bytes do not identify a value. For this reason, `DecimalValue` converts to `i256` before it compares or hashes. With one storage width for each precision, a value has exactly one storage form diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index a65435c3d93..76d549f9f69 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -91,14 +91,16 @@ Generic rules then cover both types: - Slice, filter, take and mask push into the child and wrap the result again. - `scalar_at` wraps the scalar of the child. - Compression compresses the child with the ordinary schemes. -- Sparse, run-end, dictionary and chunked arrays work over the storage dtype. Today a sparse array - of an extension type cannot canonicalize at all, so this rule also closes a gap for extension - types. -The wrapper stays on top of generic encodings, as in the -[extension design](../extension-types/03-design.md#one-normal-form). Type-specific encodings, such -as DecimalByteParts and `DateTimeParts`, live under the wrapper and see only the storage dtype. The -position of the wrapper is still an open question (see below). +The position of the wrapper relative to generic encodings, such as `Constant`, `Dict` and +`Sparse`, is an open question (see below). The +[extension design](../extension-types/03-design.md#physical-form-open) compares both placements. +It recommends the wrapper on top, but no maintainer decision exists. With the wrapper on top, +sparse, run-end, dictionary and chunked arrays work over the storage dtype. That also closes a gap +for extension types, because a sparse array of an extension type cannot canonicalize today. + +Type-specific encodings, such as DecimalByteParts and `DateTimeParts`, live under the wrapper and +see only the storage dtype. This choice does not depend on the position of the wrapper. ## Integer child or FixedSizeBinary child @@ -212,8 +214,8 @@ FixedSizeBinary is a core type for opaque bytes of a fixed length, as RFC 0062 p equality and hashing, but UUID order needs a separate decision. `Decimal`, wide integers and FixedSizeBinary share only the fixed-width structural layer. That -layer is the right place for shared code. PR [#8864](https://github.com/vortex-data/vortex/pull/8864) -started it. +layer is the right place for shared code. PR +[#8864](https://github.com/vortex-data/vortex/pull/8864) started it. ## The file format does not change @@ -229,9 +231,10 @@ The wrapper reads every existing decimal wire form: Writes produce forms that old readers understand. A flat child becomes `vortex.decimal`. A compressed child with a lane-integer type becomes DecimalByteParts v1, which the first core edition -already contains. A dictionary child becomes a dictionary of `vortex.decimal` values. So wide -`PType` values never reach the wire for decimal columns, and decimal statistics keep the `Decimal` -dtype. +already contains. A constant child becomes a `vortex.constant` node with a decimal scalar, which is +the form that the compressor writes today. A dictionary child becomes a dictionary of +`vortex.decimal` values. So wide `PType` values never reach the wire for decimal columns, and +decimal statistics keep the `Decimal` dtype. Before the reader changes, the compatibility fixtures need narrow, wide and Arrow-style widths. Today they cover only `i32` and `i64`. @@ -255,9 +258,10 @@ acceptance test for the extension hooks. The decision on the move comes after th - Open question: are `I128` and `I256` user-visible columns from the start, or storage only until an engine needs them? DuckDB `HUGEINT` is the only current consumer. -- Open question: is the wrapper always on top of generic encodings such as `Constant` and `Dict`? - This design and the extension design say yes. Today `ExtensionConstantRule` puts `Constant` - above the extension wrapper, and the old decimal wire form puts `Dict` above `vortex.decimal`. +- Open question: is the wrapper on top of generic encodings such as `Constant` and `Dict`? The + extension notes recommend it, but no maintainer decision exists. Today `ExtensionConstantRule` + puts `Constant` above the extension wrapper, and the compressor writes a constant decimal column + as a `vortex.constant` node with a decimal scalar. - Open question: does Vortex compare and compute across different precisions and scales? The alternative keeps the rule of equal dtypes, and engines coerce first. - Open question: does FixedSizeBinary get a new core wire tag, or an extension descriptor over diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 4062e9ac6d1..5aa8c003489 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -14,7 +14,7 @@ of [`agents/report.md`](agents/report.md), which gives a reason for each edge. T | --- | --- | --- | | N1 | Extension opacity fixes ([part 4](../extension-types/04-urgent-changes.md) of the extension notes) | Urgent | | N2 | Bind-time rewrites for extension types | Research, then implementation | -| N3 | Pairwise derives | Implementation | +| N3 | Derives for one dtype | Implementation | | N4 | Cast hooks for extension types ([#8570](https://github.com/vortex-data/vortex/issues/8570)) | Implementation | | N5 | Array validation where data enters | Implementation | | N6 | Nullability on the extension dtype | Research, optional | @@ -50,7 +50,7 @@ flowchart TD subgraph EXT["Extension types"] N1["N1 Opacity fixes"] N2["N2 Bind-time rewrites"] - N3["N3 Pairwise derives"] + N3["N3 Derives for one dtype"] N4["N4 Cast hooks"] end subgraph DEC["Decimal"] @@ -146,9 +146,9 @@ the built-in wrapper owns its rewrites. Only the move to an extension type (N15) extension notes adds a decimal-like extension type over `i64`, and that type can run the same tests against the built-in `Decimal`. - The fixed-width structural kernels (N7) serve wide integers and FixedSizeBinary. -- The wrapper normal form and the `StorageWrapper` rules serve `Decimal` and every extension type. - `DateTimeParts` under the `Timestamp` wrapper is the temporal twin of DecimalByteParts under the - `Decimal` wrapper. +- The `StorageWrapper` rules serve `Decimal` and every extension type. So does the open decision + on the position of the wrapper relative to generic encodings. `DateTimeParts` under the + `Timestamp` wrapper is the temporal twin of DecimalByteParts under the `Decimal` wrapper. ## Risks diff --git a/research/decimal/README.md b/research/decimal/README.md index 93910b0e75f..b2865b931c3 100644 --- a/research/decimal/README.md +++ b/research/decimal/README.md @@ -39,6 +39,7 @@ physical forms, and `Decimal` duplicates much of `Primitive`. These notes propos ## Evidence -The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. No Rust -code was built or run for this part. One short Python script checked byte order. Section 10 of the -report lists the claims that are not verified, which include every performance claim. +The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. One +throwaway Rust test confirmed the `f32` cast disagreement (probe R1 in the extension notes). One +short Python script checked byte order. Section 10 of the report lists the claims that are not +verified, which include every performance claim. diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index 50f53f70d01..4eb90cfacea 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -28,9 +28,11 @@ Conventions: - "Lane integer" means U8 to U64 and I8 to I64. "Wide integer" means I128 and I256. - Issue and pull request numbers refer to `vortex-data/vortex` on GitHub. Section 10 links each one. -- No claim in this report comes from running Rust code. The claims come from reading code, RFC - drafts, issues, and pull requests. Appendix A holds the one script that was run. Section 10 - lists what is not verified. +- Two claims come from running Rust code, with probes R1 and R6 in + `../../extension-types/agents/probe_followup.rs`. They are the cast in section 1.1 and the + pushdown rules in section 3.3. The other claims come from reading code, RFC drafts, issues, and + pull requests. Appendix A holds the one Python script that ran. Section 10 lists what is not + verified. ## 1. Summary @@ -53,7 +55,8 @@ dtype therefore has up to six physical forms. That single fact explains most of - The scalar path and the array path disagree. A decimal scalar casts to `f32` and `f16` (`vortex-array/src/scalar/typed_view/decimal/scalar.rs:111-115`). The array kernel only handles `f64` (`vortex-array/src/arrays/decimal/compute/cast.rs:104`), so the array cast fails with - "No CastKernel" (`vortex-array/src/scalar_fn/fns/cast/mod.rs:143`). + "No CastKernel" (`vortex-array/src/scalar_fn/fns/cast/mod.rs:143`). Probe R1 confirms it: the + array path fails, and the row path returns `[1f32, 2.5f32]` for `Decimal(10, 2) [1.00, 2.50]`. ### 1.2 Recommendation in short @@ -73,9 +76,9 @@ dtype therefore has up to six physical forms. That single fact explains most of 5. Treat FixedSizeBinary as an independent type for opaque bytes (UUID, hashes). It shares the fixed-width structural layer with Primitive, but it never carries integer semantics. Decimal does not depend on it, and wide integers do not depend on it. -6. Move Decimal to a well-known extension in the Rust API (RFC 0062) only after the extension - opacity fixes, cast hooks, and pairwise derives land. The array-level wrapper design is the - same either way, so the early work carries over. +6. Move Decimal to a well-known extension in the Rust API (RFC 0062) only after the extension work + lands. That work is the opacity fixes, the cast hooks, and derives for one dtype. The + array-level wrapper design is the same either way, so the early work carries over. Section 7 gives the dependency graph. Section 8 gives the recommended order of work. @@ -475,26 +478,33 @@ Generic rules then cover both: | Behavior | Extension today | Decimal today | Shared rule | | --- | --- | --- | --- | | slice, filter, take, mask | Pushdown rules (`arrays/extension/compute/rules.rs:48-79`) | Buffer kernels through `FixedWidthArray` | Push into the child and rewrap | -| Constant child | `ExtensionConstantRule` (`rules.rs:28-46`) turns `Ext(Constant(s))` into `Constant(ext s)` | `Constant(decimal scalar)` | The wrapper stays on top: `Constant(wrapper s)` becomes `Wrapper(Constant(storage s))` | +| Constant child | `ExtensionConstantRule` (`rules.rs:28-46`) turns `Ext(Constant(s))` into `Constant(ext s)` | `Constant(decimal scalar)` | Open. With the wrapper on top, `Constant(wrapper s)` becomes `Wrapper(Constant(storage s))` | | scalar_at | Wraps the storage scalar | Reads the buffer at the stored width | `wrap_scalar(child.scalar_at(i))` | | Validity | From the child | Validity slot | From the child | | Compression | Compress storage and rewrap (`vortex-compressor/src/compressor/cascade.rs:175-205`) | `DecimalScheme` | Compress the child with ordinary schemes and rewrap | -| Sparse, run-end, dictionary, chunked | Sparse bails (`encodings/sparse/src/canonical.rs:361`) | Dedicated decimal paths | The wrapper stays on top: `Dict(codes, Wrapper(v))` becomes `Wrapper(Dict(codes, v))`, so the structural encoding sees only storage | +| Sparse, run-end, dictionary, chunked | Sparse bails (`encodings/sparse/src/canonical.rs:361`) | Dedicated decimal paths | Open. With the wrapper on top, `Dict(codes, Wrapper(v))` becomes `Wrapper(Dict(codes, v))`, so the structural encoding sees only storage | -The last row fixes a real gap for extensions as a side effect: a sparse array of an extension -type currently cannot canonicalize at all. +With the wrapper on top, the last row also closes a gap for extensions: a sparse array of an +extension type cannot canonicalize today. -This normal form is the one in part 3 of the extension notes, and it reverses -`ExtensionConstantRule`. It is still an open question (section 9). The alternative keeps generic -encodings above the wrapper, as the current code does for constants. Then each generic encoding -needs a pushdown rule for `ext_storage`, or a fallback that executes it. Old wire forms that put -the wrapper lower, such as `Dict(codes, vortex.decimal)`, come from the plugin at write time -(section 3.5). +The position of the wrapper is open (section 9, question 13). Section 6 of +`../../extension-types/agents/design-notes.md` compares both placements. It recommends the wrapper +on top, and no maintainer decision exists. The wrapper on top reverses `ExtensionConstantRule`. +The alternative keeps generic encodings above the wrapper, as the current code does for constants. +Then `Masked` and `Sparse` need a pushdown rule for `ext_storage`. Existing rules already push it +into dictionary values and chunks (probe R6 in the extension notes). + +On the wire, `vortex.decimal` is a leaf with one buffer. So a generic encoding of decimal data sits +above it. For example, the compressor writes a constant decimal column as a `vortex.constant` node +with a decimal scalar (`vortex-compressor/src/compressor/cascade.rs:272-273`, +`vortex-compressor/src/compressor/constant/mod.rs:80-86`). With the wrapper on top in memory, the +plugin writes these older shapes (section 3.5). Type-specific encodings move under the wrapper and see only the storage dtype. For decimals that means DBP stops carrying `DType::Decimal`. It becomes an encoding of `Primitive(I128)` or -`Primitive(I256)` (node N14). The same normal form applies to DateTimeParts under temporal -wrappers, which today carries the extension dtype (`encodings/datetime-parts/src/array.rs:294-305`). +`Primitive(I256)` (node N14). The same rule applies to DateTimeParts under temporal wrappers, +which today carries the extension dtype (`encodings/datetime-parts/src/array.rs:294-305`). This +choice does not depend on the position of the wrapper relative to generic encodings. ### 3.4 Decimal semantics as storage rewrites @@ -503,8 +513,8 @@ dtype, the rewrites below are exact. | Operation on `Decimal(p, s)` | Rewrite onto the unscaled child | Notes | | --- | --- | --- | -| Compare, same `(p, s)` | `compare(unscaled(a), unscaled(b))` | Valid because the unscaled value is unique within one `(p, s)` | -| Compare across `(p1, s1)` and `(p2, s2)` | Rescale both to `s = max(s1, s2)` at a width that holds `max(p1 - s1, p2 - s2) + s` digits, then compare | Pairwise derive. Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | +| Compare, same `(p, s)` | `compare(unscaled(a), unscaled(b))` | A derive for one dtype. Valid because the unscaled value is unique within one `(p, s)` | +| Compare across `(p1, s1)` and `(p2, s2)` | Rescale both to `s = max(s1, s2)` at a width that holds `max(p1 - s1, p2 - s2) + s` digits, then compare | A rewrite that the type owns, not a derive. Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | | Compare with a constant that has more fractional digits than `s` | Fold the constant and adjust the operator. `x < 1.005` at `s = 2` becomes `unscaled <= 100` | Needs floor and ceiling rounding | | Add, subtract | Widen to the result width, rescale to the larger scale, checked operation, precision check | Arrow rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`) | | Multiply | Widen, checked multiply, precision check | Result scale `s1 + s2` | @@ -557,8 +567,8 @@ Two points matter for the design: - **Temporal types need the same kernels.** Timestamp, Time, and Duration units are powers of ten apart, except Days. A Timestamp cast from milliseconds to nanoseconds (issue #8570) is - `rescale(delta = 6)`. A Duration comparison across units (PR #10467) is the same pairwise - rescale as a decimal comparison across scales. One kernel family serves both. + `rescale(delta = 6)`. A Duration comparison across units (PR #10467) needs the same rescale as + a decimal comparison across scales. One kernel family serves both. - **Fusion keeps performance.** The current decimal arithmetic kernel checks the operation and the precision bound in one pass (`numeric/decimal.rs`, `DecimalOpConstants`). A chain of separate array operations makes several passes. So generalize the fused kernel from @@ -590,9 +600,12 @@ Writing: | `Decimal { widen(flat Primitive(I_v)) }` | `vortex.decimal`, `values_type = v` | Yes, same bytes as today | | `Decimal { compressed child with a lane-integer ptype }` | `vortex.decimal_byte_parts` v1 with the child as MSP | Yes, core 2025.05 | | `Decimal { WideIntParts }` | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2 | -| `Decimal { Dict, RunEnd, Chunked, ... }` | Push the wrapper below the structural encoding, for example `Dict(codes, Decimal { values })` | Yes | +| `Decimal { Constant, Dict, RunEnd, Chunked, ... }` | Push the wrapper below the generic encoding, for example `vortex.constant` with a decimal scalar, or `Dict(codes, Decimal { values })` | Yes | | Any other child | Execute the child to a flat array and write `vortex.decimal` | Yes | +The fifth row applies only if the wrapper sits above generic encodings in memory. With generic +encodings above the wrapper, the memory form already has the wire shape. + Consequences: - Old readers read every file that a new writer produces, with the same edition rules as today. @@ -660,7 +673,7 @@ question is only where the Rust type lives: | --- | --- | --- | | Array | `DecimalArray` wrapper | `ExtensionArray` with the decimal ext dtype | | Shared machinery | Through `StorageWrapper` | Directly | -| Semantics | Decimal-owned rewrites | `ExtVTable` hooks: cast (#8570), pairwise derives, validation | +| Semantics | Decimal-owned rewrites | `ExtVTable` hooks: cast (#8570), derives for one dtype, rewrites, validation | | API change | Small | 269 lines in 97 files mention `DType::Decimal` | | Wire | Unchanged | Tag translation in both directions, recursively | | Risk | Low | High until extension semantics are opaque by default | @@ -1032,11 +1045,11 @@ must come first. **urgent** means a correctness bug that exists today. | Id | Node | Kind | Notes | | --- | --- | --- | --- | -| N1 | Extension opacity fixes | impl, urgent | Stop storage casts, storage-and-wrap scalar casts, lone-side storage compares, storage zero values | -| N2 | Bind-time rewrite infrastructure | research, then impl | `ext_storage` and `ext_wrap` style nodes, rewrites keyed by function and type ids | -| N3 | Pairwise derives | impl | Eq, Ord, Hash as rewrites that see both operand dtypes | +| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes: storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values, debug output | +| N2 | Bind-time rewrite infrastructure | research, then impl | `ext_storage` and `ext_wrap` nodes, extension declarations on functions, type-owned rewrites, and rules across dtypes | +| N3 | Derives for one dtype | impl | `EQ` and `ORD` declared on one dtype. The binder applies them when every operand in a derived position has that dtype. Operations across dtypes use rewrites (N2) | | N4 | Cast hooks (#8570) | impl | `cast_to` and `cast_from` on `ExtVTable`, bound `CastFn` | -| N5 | Array validation at trust boundaries | impl | `validate_array`: decimal precision, UTF-8, FixedSizeBinary length | +| N5 | Array validation at trust boundaries | impl | `validate_array` and validated or unvalidated arrays: decimal precision, UTF-8, FixedSizeBinary length | | N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes | | N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width | | N8 | Scaled-integer kernels | impl | `rescale`, `check_range`, fused scaled arithmetic and compare | @@ -1077,7 +1090,7 @@ flowchart TD N1["N1 Extension opacity fixes"]:::urgent N11["N11 Decimal storage at least canonical width"]:::urgent N2["N2 Bind-time rewrite infrastructure"]:::research - N3["N3 Pairwise derives"]:::impl + N3["N3 Derives for one dtype"]:::impl N4["N4 Cast hooks #8570"]:::impl N5["N5 Array validation"]:::impl N6["N6 Nullability restructure"]:::research @@ -1150,17 +1163,21 @@ but it is cheaper, safer, or faster with it. 1. N1 to N15: as an extension on `develop`, Decimal inherits storage casts and lone-side storage compares (`arrays/extension/compute/cast.rs:17-22`, `scalar_fn/fns/binary/compare/mod.rs:177-194`). -2. N1 to N21: without N1, `Duration(ms)` and `Duration(ns)` compare and cast raw `i64` storage. - New types must not ship with a known leak that users can come to depend on. +2. N1 to N21: without N1, a `Duration` cast across units reinterprets the storage on the row path + (EXT-C1). A comparison with a raw `i64` literal compares storage (EXT-Q1). New types must not + ship with a known leak that users can come to depend on. 3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The engine literal paths then change once, before the storage changes. PR #9886 shows the current workaround. -4. N2 to N3: pairwise derives are rewrites over both operands. +4. N2 to N3: the binder applies a derive as a rewrite onto `ext_storage`. 5. N2 to N13 (soft): expression-level rewrites let zone maps and compressed integer kernels evaluate decimal predicates on the integer child. -6. N3 to N15: decimal comparison across scales is a pairwise rescale. -7. N3 to N21 (soft): Duration comparison across units is the same pairwise rescale. Until N3 - lands, a comparison across units is rejected, which is correct but limited. +6. N3 to N15: after N1 and N2, a decimal extension type without derives loses compare, hash, + `min`, `max` and sort within one `(p, s)`. A comparison across scales is a rewrite that the + type owns (N2), not a derive. +7. N3 to N21 (soft): `Duration` declares `EQ` and `ORD` for one unit. Before N3, the shared storage + compare kernel gives the same answer for one unit. A comparison across units needs a + type-owned rewrite in both cases. 8. N4 to N15: rescaling casts and casts to and from integers and floats must be type-owned. 9. N4 to N21: unit conversions are casts. 10. N8 to N4 (soft): Timestamp, Time, and Duration unit casts are powers of ten, except Days. @@ -1229,15 +1246,12 @@ U8) and phases (0 to 6). This table maps them onto the graph. | Extension research item | Node here | | --- | --- | -| U1 one cast rule, U2 one compare rule, U7 zero values | N1 | -| U3 errors from reads | Not a separate node. It belongs with N1 | -| U4 infallible nullability changes | Not a separate node. It removes the panic in the rebuild that edge 34 describes | -| U5 temporal validation, U6 `Extension::validate` checks the storage | N5 | -| U8 debug output | Not in this graph | +| U1 to U8, the urgent changes | N1, in wave 0. U5 and U6 are dtype and storage checks, not the value validation of N5 | +| U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 34 describes | | Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 | | Phase 2, binder and derives | N2 and N3 | | Phase 3, casts | N4 | -| Phase 4, normal form and storage statistics | The shared rules of section 3.3, used by N12. DateTimeParts under the wrapper is the temporal twin of N14 | +| Phase 4, physical form and storage statistics | The shared rules of section 3.3, used by N12. The position of the wrapper is open (section 9). DateTimeParts under the wrapper is the temporal twin of N14 | | Phase 5, validation and nullability | N5 and N6 | | Phase 6, authoring helpers and engine export plugins | N23 | @@ -1293,7 +1307,7 @@ Wave 0, small and independent: (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). 3. Add `DecimalDType::storage_type()`, returning `DecimalType` until N9 lands, and route the 25 `smallest_decimal_value_type` calls through it so that the width function has one owner. -4. N1: the extension opacity fixes. +4. N1: the urgent extension changes U1 to U8. Wave 1, foundations: @@ -1301,8 +1315,10 @@ Wave 1, foundations: 6. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for the Timestamp unit casts of N4. 7. N4: cast hooks (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. -8. N2, then N3: settle the rewrite design, then add pairwise derives. This is phase 2 of the - extension notes. Use the decimal rewrites of section 3.4 as acceptance tests. +8. N2, then N3: settle the rewrite design, then add derives for one dtype. This is phase 2 of the + extension notes. Use the decimal rewrites of section 3.4 as acceptance tests. A checked + `ext_wrap` in N2 calls the default `validate_array`, which phase 1 of the extension notes adds. + N5 adds the rest later. 9. N23: plugin registries for DuckDB and DataFusion type mapping. Wave 2, new types: @@ -1340,7 +1356,8 @@ Wave 4, API unification: 7. Does Vortex implement decimal arithmetic and comparison across different `(p, s)` pairs? The alternative keeps the rule of equal dtypes, and engines coerce first. 8. Do file reads check decimal precision, or only external imports? Zone-map minimum and maximum - statistics can make the file check cheap. + statistics can make the file check cheap. The extension notes leave the same question open for + value checks on file reads. 9. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist (`vortex-array/src/dtype/decimal/mod.rs:47`). Tests use -128. DuckDB rejects negative scales, and Arrow allows them. @@ -1351,9 +1368,10 @@ Wave 4, API unification: in Arrow, or normalized as in DuckDB. The other is the edition membership of `vortex.duration` and `vortex.interval`. 12. When do `U128` and `U256` become worth adding? -13. Is the wrapper always on top of generic encodings (`Constant`, `Dict`, `Chunked`, `Sparse`)? - Section 3.3 says yes, like part 3 of the extension notes. The alternative keeps generic - encodings above the wrapper, as `ExtensionConstantRule` does today. +13. Is the wrapper on top of generic encodings (`Constant`, `Dict`, `Chunked`, `Masked`, + `Sparse`)? The extension notes recommend it (`design-notes.md`, section 6), but no maintainer + decision exists. The alternative keeps generic encodings above the wrapper, as + `ExtensionConstantRule` does today. ## 10. Evidence log and limits @@ -1396,17 +1414,25 @@ Verified outside the repository: - arrow-buffer 59.3.0 `i256` layout and order (`src/bigint/mod.rs:59-66`, `:171-183`). - Byte order against integer order, with the script in appendix A. +Verified with a run: `../../extension-types/agents/probe_followup.rs` ran in a debug build of +`vortex-btrblocks` on `5fc5cb8`. + +- Probe R1 checks the decimal to `f32` cast of section 1.1. The array path fails with `No + CastKernel to cast canonical array vortex.decimal from decimal(10,2) to f32`. The row path + returns `[1f32, 2.5f32]`, and the scalar cast returns `1f32`. +- Probe R6 checks the pushdown rules of section 3.3. Existing rules push `ext_storage` into + dictionary values and chunks. `ext_storage` decodes a `Masked` input and fails on a `Sparse` + input with extension values. + A script checked that every `file:line` reference in this report points at an existing range, and the first line of each range was reviewed. The external crate references above were checked by hand. A second review checked the main claims of sections 1 and 2 by hand again. -Not verified, because building or running Rust code was out of scope: +Not verified, because no run covered them: - Any performance statement, including the cost of 128-bit and 256-bit kernels and the effect of rewrites on fused arithmetic. - The exact set of compile errors from the PType migration. The counts are a `grep` proxy. -- The array cast failure for decimal to `f32`. It comes from reading the dispatch code, not from - a run. - The alignment of `i128` on targets other than x86_64 and aarch64, for example wasm32. - Whether the fuzzer reports in issue #5820 that end in a null decimal sum come from narrow storage. This report does not rely on that. diff --git a/research/extension-types/02-current-problems.md b/research/extension-types/02-current-problems.md index 3dc1df83f4a..1d74ad0b1c9 100644 --- a/research/extension-types/02-current-problems.md +++ b/research/extension-types/02-current-problems.md @@ -7,8 +7,8 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors All findings are for `develop` at `5fc5cb8`. Each ID refers to an entry in [`agents/findings.md`](agents/findings.md), which has file and line references, reproductions and -fix sketches. "Confirmed" means that a throwaway test ([`agents/probe.rs`](agents/probe.rs)) -reproduced the behavior. +fix sketches. "Confirmed" means that a throwaway test ([`agents/probe.rs`](agents/probe.rs) or +[`agents/probe_followup.rs`](agents/probe_followup.rs)) reproduced the behavior. ## Results depend on the encoding and the read path (EXT-P1) @@ -42,6 +42,10 @@ implementation. compare by storage. This happens in the compare kernel, the nested comparator and `ExtScalar`. A test type with no ordering still supports `<`. The built-in kernel runs first, and the registry cannot override it, so an extension type has no way to replace it. +- **`DateTimeParts` compare (EXT-Q3).** The kernel calls `as_extension()` on its constant operand + before it checks the type, and it does not compare time units. Confirmed: against an `i64` + constant, the array path panics. Against a `Timestamp(ns)` constant, a `Timestamp(ms)` column + gets a wrong result. - **Engine boundaries (EXT-B1 to EXT-B5).** DataFusion converts non-temporal extension scalars to storage scalars. Arrow export to a storage-like target type exports storage through the cast fallback. Python `as_py()` returns the storage value. DuckDB decides aggregate pushdown from the @@ -51,6 +55,10 @@ implementation. - **Arrays are never value-checked (EXT-V1).** `ExtensionArray::new` and `ExtensionBuilder` accept values that the type rejects. Reading such a row panics. Confirmed with a type that rejects zero. + `ext_storage` over the same array returns the zero on the array path and panics on the row path. +- **Scalars skip the storage check (EXT-V7).** An extension scalar can hold a value of another + storage type, for example a `Timestamp` with an `f64` value. Then `==` panics in one direction + and returns `false` in the other. Confirmed. - **Nullability changes can panic (EXT-N1).** `ExtDType::with_nullability` validates the type again and panics if validation fails. A type that accepts only non-nullable storage panics in `take` with nullable indices. Confirmed. @@ -59,12 +67,9 @@ implementation. - **Built-in temporal types (EXT-V4, EXT-V5).** `Date` and `Time` panic on non-primitive storage, so a malformed file panics instead of returning an error. `Timestamp` accepts the `Days` unit and unknown time zones, and then rejects every value. -- **Array validation is a no-op (EXT-V2).** `Extension::validate` compares the dtype with itself. - It accepts `Utf8` storage under a `Timestamp` dtype. Confirmed. -- **Lead: `DateTimeParts` compare (EXT-Q3).** The kernel calls `as_extension()` on its constant - operand before it checks the type, and it does not compare time units. The compare adaptor does - not check operand types. A `DateTimeParts` column compared with an `i64` constant can panic. This - needs a direct test. +- **Arrays skip the storage dtype check (EXT-V2).** `Extension::validate` checks the length, but + it compares the dtype with itself. It accepts `Utf8` storage under a `Timestamp` dtype. + Confirmed. ## Disagreements diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md index c7f61f3cdf3..36f2a148a88 100644 --- a/research/extension-types/03-design.md +++ b/research/extension-types/03-design.md @@ -8,17 +8,20 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors ## Goal Extension types behave like Rust newtypes. Structural operations always work. A semantic operation -works only where the extension type opts in or provides its own rule. No code path applies storage -semantics to an extension value without that opt-in. +works only where the extension type opts in or provides its own rule. A function that declares that +it understands the type can also allow it. No code path applies storage semantics to an extension +value without that opt-in. ## Rules -1. A semantic operation on an extension type is rejected unless the type opts in. +1. A semantic operation on an extension type is rejected unless the type or a type-aware function + opts in. 2. Structural operations are always allowed. -3. One place decides semantics: the binder, which runs when an expression or a scalar function - array is built. -4. Below the binder, everything is storage. Kernels, encodings, statistics, pruning and compression - see storage dtypes only. +3. One place decides how generic operations treat extension values: the binder. It runs when an + expression, a scalar function array or an aggregate is bound. +4. Below the binder, the kernels of generic semantic operations, statistics and pruning see + storage dtypes only. A function that declares an extension type can see that type. Whether + generic encodings carry extension dtypes depends on the physical form (open, see below). 5. The extension type owns every conversion that changes meaning. There are no implicit casts. 6. An unknown extension type supports structural operations and `ext_storage` only. @@ -63,35 +66,66 @@ The scalar path and the array path then evaluate the same expression, so the res cannot differ. Pushdown, pruning and compression keep working, because the rewritten expression is an ordinary storage expression. -### Pairwise derives +### Derives for one dtype -A derive says that a semantic operation on two extension values is the same operation on their -storage values. It depends on the metadata of both operands: +A derive says that a semantic operation on values of one extension dtype is the same operation on +their storage values. It is a property of one dtype: ```rust -/// Storage behavior that values of `lhs` and `rhs` share, like `#[derive]` on a newtype. +/// Storage behavior that values of `dtype` inherit, like `#[derive]` on a newtype. /// -/// The default shares nothing, so every semantic operation on this type is rejected. -fn derives(lhs: &ExtDType, rhs: &ExtDType) -> Derives { - Derives::NONE +/// The default inherits nothing, so every semantic operation on this dtype is rejected. +fn derives(dtype: &ExtDType) -> Derives { + Derives::default() } ``` `Derives` has two flags. `EQ` means that storage equality is value equality, which also makes the storage hash valid. `ORD` means that storage order is value order. -| Type | Pair | Derives | -| --- | --- | --- | -| `Timestamp` | Same unit, and both with a time zone or both without | `EQ`, `ORD` | -| `Date`, `Time` | Same unit | `EQ`, `ORD` | -| `Decimal`, if it follows this design | Same scale and same storage width | `EQ`, `ORD` | -| `Uuid` | Any | `EQ`, plus `ORD` if byte order is the chosen order | -| `Vector`, `FixedShapeTensor` | Equal metadata | `EQ` | -| `Json`, WKB, native geometry | Any | None | -| Unknown type | Any | None | +A derive applies only when every operand in a derived position has the same extension dtype: + +- A comparison needs two such operands, and `between` needs three. +- A unary aggregate, such as `min` or `is_sorted`, reads the flags of its one input. +- A struct or list that holds extension values inherits a flag only if every nested extension + type declares it. + +So `ts < 5i64` stays rejected, because only one side is an extension. + +| Type | Derives | +| --- | --- | +| `Timestamp` | `EQ`, `ORD` | +| `Date`, `Time` | `EQ`, `ORD` | +| `Decimal`, if it follows this design | `EQ`, `ORD` | +| `Uuid` | `EQ`, plus `ORD` if byte order is the chosen order | +| `Vector`, `FixedShapeTensor` | `EQ` | +| `Json`, WKB, native geometry | None | +| Unknown type | None | + +### Operations across dtypes + +Two dtypes with different metadata are different dtypes, for example `Timestamp(ms)` and +`Timestamp(ns)`. A derive never applies across them, and the binder rejects such an operation by +default. A type can support one only through a rule that the type owns: + +- A compatibility rule: the two dtypes compare by storage as they are. An example is two zoned + timestamps with one unit. +- A coercion: a cast hook of the type converts one operand first. +- A rewrite of the whole call: for example, a comparison of two decimals with different scales + rescales one side first. + +### Function declarations + +Each function declares how it uses an operand that contains an extension type: + +- Structural: it moves or selects values, for example `fill_null` or `case_when`. +- Derive: it needs `EQ` or `ORD`, for example `=` or `<`. +- Type-aware: it implements the semantics of named extension types itself, for example + `replace_time_zone` for `Timestamp`. +- Nothing: the default. The binder must rewrite or reject the call. -A derive is a preset rewrite. A type that needs something else writes its own rewrite. An example is -a comparison of two decimals with different scales, which needs a rescale first. +Only the function owner declares. The binder never infers a declaration from the input types that a +function accepts, because `Binary` accepts extension operands today. ### Casts @@ -102,21 +136,40 @@ result can be pushed down and evaluated in the same way on both paths. This is t Two explicit expressions cover everything else: -- `ext_storage(x)` reads the storage. It exists today. -- `ext_wrap(x, dtype)` builds an extension value from storage and checks it. It is new. +- `ext_storage(x)` reads the storage. It exists today. It never checks values. +- `ext_wrap(x, dtype, mode)` builds an extension value from storage. It is new. A checked wrap + validates the values. An unchecked wrap skips the check, and its result stays unvalidated. -### One normal form +A simplification can remove a wrap only if it keeps every error. `ext_storage(ext_wrap(y))` +simplifies to `y` only for an unchecked wrap. With a checked wrap, `y = 0` under a type that rejects +zero fails before the simplification and succeeds after it. -Only the wrapper carries an extension dtype. Generic encodings (`Constant`, `Dict`, `Chunked`) and -type-specific encodings (`DateTimeParts`) live under the wrapper and see the storage dtype. This -removes the two shapes of EXT-S1 and EXT-S2. The compressor chooses a type-specific encoding from -the extension metadata. +### Scalar identity -Then no generic encoding needs code for extension dtypes, and `ext_storage` only reads the child. -The cost is that code which builds a constant, a dictionary or chunks of an extension dtype must -put the wrapper on top. The [Decimal notes](../decimal/02-design.md) use the same rule for the -built-in `Decimal` wrapper. A writer can still produce an older wire shape with the wrapper lower -down when an older format needs it. +`Scalar` implements `PartialEq`, `Eq` and `Hash` as identity: the same dtype and the same stored +value. These traits serve Rust code, such as maps and tests, not SQL semantics. `PartialOrd` +returns `Equal` for identical scalars, and an order only for ordered dtypes. An extension dtype is +ordered only if it declares `ORD`. This keeps the Rust rule that `==` and `partial_cmp` agree. +Expression comparisons follow the binder, not these traits. + +### Physical form (open) + +Opacity does not decide where the wrapper sits relative to generic encodings such as `Constant`, +`Dict` and `Chunked`. The binder works with either placement. + +| | Wrapper on top | Generic encodings on top | +| --- | --- | --- | +| Shapes of one column | One | Several, as today | +| Current code | The compressor uses it for non-constant columns, unless a type-specific encoding such as `DateTimeParts` wins | The compressor and `ExtensionConstantRule` use it for constants | +| Main cost | Reverse `ExtensionConstantRule`, and look through the wrapper wherever code looks for constants | Push `ext_storage` through `Masked` and `Sparse`, and keep extension code in each generic encoding | +| Wire | No new form for extension columns. `Decimal` translates on write | Memory and wire forms match | + +These notes recommend the wrapper on top, after the binder lands. No maintainer decision exists. +[`agents/design-notes.md`](agents/design-notes.md), section 6, has the full comparison. + +Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an +`i64` encoding, so its compare kernel works on storage. The compressor chooses it from the +extension metadata. ### Statistics @@ -126,10 +179,19 @@ binder produce predicates on `ext_storage(col)`, and pruning uses the storage st ### Validation -An extension type can define `validate_array`. The core calls it where data enters: `try_new`, -Arrow import, `ext_wrap`, and optionally file reads. Kernels that keep the invariants use an -unchecked constructor. Reads return errors and never panic. Validated arrays also remove the -per-row check that each read does today. +Three kinds of validation exist: + +- Dtype validation checks the metadata and the storage dtype, once for each dtype. +- Storage-shape validation checks that storage has the storage dtype, for arrays and scalars. +- Value validation (`validate_array`) checks each value, at trust boundaries. + +Each extension array is validated or unvalidated. Only a successful value check makes an array +validated, and structural operations keep that state. File reads without a value check give +unvalidated arrays. A read of a row from an unvalidated array checks the value, and returns an +error for an invalid value. Reads never panic. + +The default `validate_array` checks each value with the existing scalar check of the type. It +skips rows under a null parent, because builders pad those rows. ### Nullability @@ -147,37 +209,34 @@ plugins like the Arrow registry. Then a new extension type does not need code in ## Costs and risks - Binding must be mandatory. The array API builds scalar function arrays directly, so the rewrite - must happen when such an array is built, not in an optional optimizer pass. + must happen when such an array is built, not in an optional optimizer pass. Aggregates that code + binds directly, such as zone-map statistics, need the binder too. - `DateTimeParts` changes from a `Timestamp` encoding to an `i64` encoding. That is a file format change for the encoding, and it needs an edition. - Engines must send extension literals for extension columns. Today the lone-extension compare rule can hide literals that have the storage type. - After binding, plans show storage operations, so plans are harder to read. +- The new scalar order must land with or after the change that rejects `min` and `max` without + `ORD`. Otherwise `min` and `max` can panic, for example in zone maps of `Json` columns. This + risk comes from reading the code, not from a run. ## Open questions - Open question: where does binding happen for the array API? One option is a rewrite when the array is built, with hooks on `ExtVTable`, so no session is necessary. The other option is a mandatory rewrite pass that also covers the row path. -- Open question: are rewrites owned only by the extension type? Or can a function owner register a - rewrite for a type that it does not own? -- Open question: is the wrapper always on top of generic encodings? The alternative keeps generic - encodings above the wrapper, as `ExtensionConstantRule` does for constants today. That matches - the current code, but each generic encoding then needs a pushdown rule for `ext_storage`, or a - fallback that executes it. This proposal puts the wrapper on top. -- Open question: are operations on values with different metadata rejected, or coerced to a common - type? Rust and the built-in `Decimal` reject them today. SQL coerces them. +- Open question: can a crate register a rewrite or a declaration for a function and a type that it + does not own? +- Open question: is the wrapper on top of generic encodings? These notes recommend it. The + alternative keeps generic encodings above the wrapper, as `ExtensionConstantRule` does for + constants today. +- Open question: does Vortex coerce operations across dtypes or metadata, or do engines coerce + first? Rust and the built-in `Decimal` reject such operations today. SQL coerces them. - Open question: are `EQ` and `ORD` enough? A refinement over a storage dtype can need a third flag that allows comparisons with plain storage values. -- Open question: builders pad non-nullable children under null parents with a zero. Does the - padding have to be a valid extension value? +- Open question: do file reads check values by default? - Open question: existing files store min and max for extension columns as extension scalars. Readers can convert them to storage values, but the switch needs a compatibility decision. -- Open question: is `validate_array` a no-op by default, or a check of each value through - `validate_scalar_value`? - Open question: is the nullability contract enough, or does nullability move onto the extension dtype? - Open question: does `Uuid` order by bytes, or have no order? -- Open question: `Scalar` implements `PartialEq` and `Hash` for use in Rust code. Can these stay - structural (same dtype and same storage value) if they are documented as identity, not SQL - equality? `PartialOrd` must follow the derives. diff --git a/research/extension-types/04-urgent-changes.md b/research/extension-types/04-urgent-changes.md index 68549625ef4..dacfec36cad 100644 --- a/research/extension-types/04-urgent-changes.md +++ b/research/extension-types/04-urgent-changes.md @@ -16,7 +16,7 @@ fits in a small PR with a regression test. Finding IDs refer to | U3 | Return errors from reads | EXT-V1 | No panic when a row is read | | U4 | Make nullability changes infallible | EXT-N1 | No panic in `take`, `mask` or `min_max` | | U5 | Fix the built-in temporal validation | EXT-V4, EXT-V5 | A malformed dtype returns an error | -| U6 | Make `Extension::validate` check the storage | EXT-V2 | An array with the wrong storage is rejected | +| U6 | Check the storage dtype of arrays and scalars | EXT-V2, EXT-V7 | An array or a scalar with the wrong storage is rejected | | U7 | Stop invalid zero values | EXT-V3 | No invalid scalars | | U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output | @@ -46,7 +46,11 @@ Visible changes: `cast(Timestamp, i64)` returns an error, and `ext_storage` repl - Remove the lone-extension case in `compare_arrays` (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`). - `CompareExecuteAdaptor` checks that the operand dtypes match before it calls an encoding kernel - (`compare/mod.rs:73-121`). This also closes the `DateTimeParts` lead (EXT-Q3). + (`compare/mod.rs:73-121`). With the stricter `return_dtype`, this also fixes the `DateTimeParts` + compare (EXT-Q3). +- The `DateTimeParts` compare kernel checks that the constant has the dtype of the column, + including the time unit, before it splits the constant + (`encodings/datetime-parts/src/compute/compare.rs:31-52`). Coupling: an engine converter that sends a storage-typed literal for an extension column must send an extension literal. DuckDB and DataFusion already convert temporal literals to extension scalars. @@ -76,10 +80,16 @@ when the dtype is built is the better place. - `Timestamp::validate_dtype` rejects the `Days` unit and unknown time zones (`vortex-array/src/extension/datetime/timestamp.rs:198-204`). -## U6. `Extension::validate` +## U6. Storage checks for arrays and scalars -- Compare the dtype of the storage child with `ext_dtype.storage_dtype()` +- `Extension::validate` compares the dtype of the storage child with `ext_dtype.storage_dtype()` (`vortex-array/src/arrays/extension/vtable/mod.rs:96-123`). +- Scalar validation checks that an extension value matches the storage dtype, before it calls the + check of the type (`vortex-array/src/scalar/validate.rs:100`, `:294`). This fixes EXT-V7. +- `Scalar::try_new` converts the `U64` form of an `F16` value to `F16` + (`vortex-array/src/scalar/scalar_impl.rs:73`, `vortex-array/src/scalar/validate.rs:147-157`). + Then `==` stays symmetric for `F16` scalars (probe R10). This case is not specific to extension + types. ## U7. Zero values diff --git a/research/extension-types/05-implementation.md b/research/extension-types/05-implementation.md index 28556401df5..7f8d75c8ac4 100644 --- a/research/extension-types/05-implementation.md +++ b/research/extension-types/05-implementation.md @@ -10,11 +10,11 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors | Phase | Content | Depends on | | --- | --- | --- | | 0 | The urgent changes in part 4, and the path-consistency test helper | Nothing | -| 1 | The `ExtVTable` contract, a conformance harness, and three reference test types | Nothing | +| 1 | The `ExtVTable` contract with the default `validate_array`, a conformance harness, and three reference test types | Nothing | | 2 | The binder and derives: `Derives`, `ext_wrap`, and rewrites for compare, min, max and sort | 0, 1 | | 3 | Casts through type-owned hooks, and an audit of engine literals | 2 | -| 4 | The normal form, and storage statistics for extension columns | 2 | -| 5 | Array validation, and nullability on the extension dtype | 0 | +| 4 | The physical form, and storage statistics for extension columns | 2 | +| 5 | Array validation, and nullability on the extension dtype | 0, 1 | | 6 | Authoring helpers, engine export plugins, and documentation | 2, 3 | ```mermaid @@ -22,14 +22,15 @@ flowchart LR P0["0. Urgent changes"] --> P2["2. Binder and derives"] P1["1. Contract and harness"] --> P2 P2 --> P3["3. Casts"] - P2 --> P4["4. Normal form and statistics"] + P2 --> P4["4. Physical form and statistics"] P0 --> P5["5. Validation and nullability"] + P1 --> P5 P2 --> P6["6. Authoring and boundaries"] P3 --> P6 ``` -Phases 0, 1 and 5 can run in parallel. Phase 2 is the critical step, because phases 3, 4 and 6 -build on the binder. +Phases 0 and 1 can run in parallel. Phase 5 can run in parallel with phase 2. Phase 2 is the +critical step, because phases 3, 4 and 6 build on the binder. ## Phase 0: urgent changes @@ -43,23 +44,31 @@ build on the binder. - Document the contract on `ExtVTable`: structural and semantic operations, the derive rules, validation, and nullability independence. -- Add a conformance harness for `ExtVTable` implementations. It checks the metadata round trip, that - `validate_dtype` ignores nullability, that `ORD` agrees with `EQ`, cast round trips, and path - consistency. +- Add `validate_array` to `ExtVTable`, with a default that checks each value through + `validate_scalar_value`. Phase 2 needs it for a checked `ext_wrap`, and phase 5 builds on it. +- Add a conformance harness for `ExtVTable` implementations. It checks the metadata round trip, + that `validate_dtype` ignores nullability, the derive rules, cast round trips, and path + consistency. It also checks that a `validate_array` override agrees with the default. - Add three reference test types: - An opaque type that derives nothing. Every semantic operation on it must fail at bind time. - - A decimal-like type over `i64` that derives `EQ` and `ORD` for the same scale. A differential - test compares its results with the built-in `Decimal`. + - A decimal-like type over `i64` that derives `EQ` and `ORD` for each dtype, with a rescale + rewrite across scales. A differential test compares its results with the built-in `Decimal`. - A refinement type that derives everything and adds a value check. ## Phase 2: binder and derives - Add `Derives`, `ExtVTable::derives`, and the erased forwarder on `ExtDTypeRef`. -- Add the `ext_wrap` expression. +- Add extension declarations to scalar and aggregate functions. Bind aggregates wherever code builds + them, including zone-map statistics. +- Add the `ext_wrap` expression. A checked `ext_wrap` calls `validate_array` from phase 1. - Rewrite compare, `between`, min, max, `is_sorted` and sort keys at bind time. +- Make the row path of `ext_storage` read storage without a value check. - Remove the code that applies storage semantics. That code is the extension compare kernel, the extension branch of the nested comparator, the extension path of `min_max`, and storage order in `ExtScalar`. +- Change `PartialOrd` for `Scalar` and the operators of `scalar_cmp` in the same step as `min` and + `max`. An earlier change of `PartialOrd` can make zone maps of `Json` columns panic. This risk + comes from reading the code. - Give each type in the repository its derives (the table in part 3). - Move the open PRs onto derives. #10408 becomes a derive check. #10246 and #10566 become aggregate rewrites. @@ -71,9 +80,14 @@ build on the binder. - Audit the DuckDB and DataFusion converters: every literal for an extension column must be an extension scalar. -## Phase 4: normal form and statistics +## Phase 4: physical form and statistics -- Add rewrite rules that move the wrapper above `Constant`, `Dict` and `Chunked`. +- Implement the physical form that maintainers choose (part 3): + - With the wrapper on top, reverse `ExtensionConstantRule` and add rules that lift the wrapper + above `Constant`, `Dict`, `Chunked`, `Masked` and `Sparse`. Constant detection then looks + through the wrapper. + - With generic encodings on top, push `ext_storage` through `Masked` and `Sparse`. `Dict` and + `Chunked` already push it down. - Store storage statistics for extension columns. Readers convert existing extension-typed statistics to storage values. - Move `DateTimeParts` under the wrapper as an `i64` encoding. An edition gates the new form, and @@ -81,8 +95,13 @@ build on the binder. ## Phase 5: validation and nullability -- Add `validate_array`, a checked `try_new`, and an unchecked constructor for kernels. -- Trust validated arrays when a row is read, and remove the per-row check. +- Record whether each array is validated. Only a successful check, or a structural operation on + validated input, makes an array validated. +- Run value validation in checked constructors and imports. File reads validate when a read + option asks for it. +- Add faster overrides of `validate_array` where a type has one, for example `Timestamp`. +- Row reads from unvalidated arrays check each value. Row reads from validated arrays skip the + check. - Optionally move nullability onto the extension dtype. ## Phase 6: authoring and boundaries @@ -95,8 +114,10 @@ build on the binder. ## Test strategy - **Path consistency.** Every semantic operation gives the same answer on the whole array, on one - row, and with constant inputs. + row, and with constant inputs. This also holds for unvalidated arrays with invalid values. - **Opaque rejection.** The opaque reference type fails at bind time for every semantic operation. +- **Scalar contract.** For generated scalars, `a == b` holds exactly when `partial_cmp(a, b)` + returns `Some(Equal)`, and equal scalars have equal hashes. - **Differential parity.** The decimal-like reference type matches the built-in `Decimal` for every operation that both support, including errors. - **Engines.** DuckDB and DataFusion pushdown tests use extension literals for temporal, `Uuid` and @@ -124,4 +145,4 @@ points affect the order: The Decimal rewrites are also good acceptance tests for the binder of phase 2. The decimal-like reference type of phase 1 can run them against the built-in `Decimal`. Phase 4 and the Decimal -wrapper share one normal form. +wrapper share the decision on the physical form. diff --git a/research/extension-types/README.md b/research/extension-types/README.md index c3e18dd6b74..7d3cabe1daf 100644 --- a/research/extension-types/README.md +++ b/research/extension-types/README.md @@ -21,23 +21,30 @@ That causes wrong results, panics, and code paths that disagree with each other. The outcome depends on the encoding of the input, and on whether one row or the whole array is read. - The proposal: extension types are opaque by default. Semantics are decided once, when an operation - is bound, by a rewrite into storage operations. Kernels, encodings, statistics and compression see - only storage. + is bound, by a rewrite into storage operations. The kernels of generic operations, statistics + and pruning see only storage. +- Each extension dtype declares the storage behavior that it inherits, such as equality and order. + Operations across dtypes need explicit rules, and the design does not assume that they exist. +- The position of the wrapper relative to generic encodings is a separate decision, and it is + open. - Some fixes are urgent, because they cause wrong results or panics. They cover implicit casts - through storage, comparisons against raw storage, and panics on reads and on nullability changes. + through storage, comparisons against raw storage, storage checks, and panics on reads and on + nullability changes. ## Agent documents - [`agents/findings.md`](agents/findings.md): every finding, with file and line references, reproductions, observed and expected behavior, and fix sketches. - [`agents/design-notes.md`](agents/design-notes.md): API sketches, invariants, rewrite rules, the - derive table for each type, and file lists for each phase. -- [`agents/probe.rs`](agents/probe.rs): the throwaway test that reproduced the confirmed findings. + derive table for each type, the validation model, and file lists for each phase. +- [`agents/probe.rs`](agents/probe.rs) and [`agents/probe_followup.rs`](agents/probe_followup.rs): + the throwaway tests that reproduced the confirmed findings. ## Evidence -The findings come from reading `develop` at `5fc5cb8`, and from a throwaway test against -`vortex-array` in a debug build. Findings marked "lead" in the agent catalog are not confirmed. +The findings come from reading `develop` at `5fc5cb8`, and from two throwaway tests in debug +builds. Findings marked "lead" in the agent catalog are not confirmed. No performance claim is +measured. ## Related diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md index 0572de92d5a..39ac9021a76 100644 --- a/research/extension-types/agents/design-notes.md +++ b/research/extension-types/agents/design-notes.md @@ -7,32 +7,44 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors This file is the detailed design behind `../03-design.md` and `../05-implementation.md`. It is a proposal, not an accepted design. "Open" marks a decision that is not made. Finding IDs (`EXT-*`) -refer to `findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. +refer to `findings.md`, which covers current code only. Probe IDs (R1 to R10) refer to appendix A +of `findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. ## 1. Invariants The design holds these invariants. Every change in the plan must keep them. -- **I1.** A semantic operation with an extension operand is rejected, unless the extension type - declares a derive for it or provides a rewrite for it. +- **I1.** A semantic operation on an operand that contains an extension dtype is rejected by + default. A declared derive, a type-owned rewrite, or a declaration of the function can allow it + (sections 3 and 4). - **I2.** A structural operation on an extension value is always allowed and keeps the extension dtype. -- **I3.** The binder is the only code that decides extension semantics. It runs in - `Expression::bind` (`vortex-array/src/expr/bound_expression.rs:304`) and when a scalar function - array is built (`vortex-array/src/arrays/scalar_fn/array.rs:79`). -- **I4.** After binding, no semantic operation has an extension operand. Kernels, encodings, - statistics, pruning and compression see storage dtypes only. +- **I3.** The binder is the only code that decides how a generic operation treats extension + values. It runs at three kinds of sites: + - `Expression::bind` (`vortex-array/src/expr/bound_expression.rs:304`). + - The construction of a scalar function array (`vortex-array/src/arrays/scalar_fn/array.rs:79`). + - Every site that binds an aggregate to an input dtype, for example the zone-map writer + (`vortex-layout/src/layouts/zoned/writer.rs:217-247`). +- **I4.** After binding, a generic operation applies storage semantics to an extension value only + where the binder checked a declared derive (section 4.3). The kernels of generic semantic + operations, statistics and pruning see storage dtypes only. Whether generic encodings and + compression carry extension dtypes depends on the physical form (section 6). A type-aware + function sees the extension types that it declares. `DateTimeParts` stays an exception until + phase 4 (EXT-S2). - **I5.** The only ways to move between an extension dtype and another dtype are `ext_storage`, `ext_wrap`, and cast hooks that the extension type owns. - **I6.** The row path and the array path evaluate the same bound expression, so they return the - same result (EXT-P1). -- **I7.** Reading a value never panics. A value that the type rejects produces an error. + same result (EXT-P1). This also applies to an unvalidated array with invalid values. +- **I7.** Reading a value never panics. A read from an unvalidated array checks the value. A value + that the type rejects produces an error (section 8). - **I8.** A nullability change never fails and never validates the type again. -- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and no rewrites. +- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and no rewrites, and + no function declaration matches it. ## 2. Operation classes -Structural operations do not read values. Semantic operations depend on what values mean. +Structural operations do not read values. Semantic operations depend on what values mean. A +type-aware function implements the semantics of the extension types that it declares. | Operation | Class | Note | | --- | --- | --- | @@ -43,7 +55,8 @@ Structural operations do not read values. Semantic operations depend on what val | `first`, `last` | Structural | Selects an existing value | | `uncompressed_size_in_bytes` | Structural | Size of the storage | | `pack`, `merge`, `select`, `get_item` on a struct that holds extension fields | Structural | The extension field stays opaque | -| `scalar_at`, serialization, compression of storage | Structural | | +| `cast` to the same dtype with another nullability | Structural | | +| `scalar_at`, serialization, compression of storage | Structural | Constant detection for compression compares storage | | `ext_storage` | Structural | Explicit unwrap | | `=`, `!=` | Semantic | Needs `EQ` or a rewrite | | `<`, `<=`, `>`, `>=`, `between` | Semantic | Needs `ORD` or a rewrite | @@ -54,81 +67,135 @@ Structural operations do not read values. Semantic operations depend on what val | `is_nan`, `nan_count`, `like`, `byte_length` | Semantic | Needs a rewrite | | Zero and default values | Semantic | Needs a `zero_value` hook | | Export to Arrow, DuckDB, DataFusion, Python, JNI | Semantic | Needs an export plugin | +| `replace_time_zone`, tensor functions, spatial functions | Type-aware | The function owner declares the types (section 4.2) | ## 3. Derives -### Definition +### 3.1 Definition + +A derive is a declaration on one extension dtype. It says that a semantic operation on values of +that dtype is the same operation on their storage values. ```rust -/// Storage behavior that two values of one extension type share. +/// Storage behavior that values of one extension dtype inherit, like `#[derive]` on a newtype. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub struct Derives { - /// Storage equality is value equality, in both directions. The storage hash is then a valid - /// hash for value equality. + /// Storage equality is value equality for values of this dtype. The storage hash is then a + /// valid hash for value equality. pub eq: bool, - /// Storage order is value order. `ord` requires `eq`. + /// Storage order is value order for values of this dtype. `ord` requires `eq`. pub ord: bool, } pub trait ExtVTable { // The existing methods stay. - /// Returns the storage behavior that values of `lhs` and `rhs` share. + /// Returns the storage behavior that values of `ext` inherit. /// - /// Both dtypes have this extension type. The default shares nothing. - fn derives(lhs: &ExtDType, rhs: &ExtDType) -> Derives { + /// The default inherits nothing, so every semantic operation on this dtype is rejected. + fn derives(ext: &ExtDType) -> Derives { Derives::default() } } ``` -`ExtDTypeRef` gets an erased forwarder: `fn derives(&self, other: &ExtDTypeRef) -> Derives`. It -returns `Derives::default()` when the two extension IDs differ. - -### Rules - -- **D1.** A derive applies only when the two storage dtypes are equal, ignoring nullability. For - other pairs, the type must provide a rewrite, for example a cast of one storage to the other. -- **D2.** `derives(a, b)` must equal `derives(b, a)`. The conformance harness checks this. -- **D3.** `eq` requires canonical storage: each value has exactly one storage form under both - metadata values. -- **D4.** `ord` requires `eq`, and the storage order must be a total order on values. -- **D5.** A type with `eq` must not let two different metadata values give different meanings to - the same storage value. If they do, `derives` returns no `eq` for that pair. - -### Table for the types in the repository - -| Type | Pair | `eq` | `ord` | Reason | -| --- | --- | --- | --- | --- | -| `Timestamp` | Same unit, both with a time zone | Yes | Yes | Both store UTC instants | -| `Timestamp` | Same unit, both without a time zone | Yes | Yes | Both store wall-clock counts | -| `Timestamp` | One with and one without a time zone | No | No | Different meanings | -| `Timestamp` | Different units | No | No | Needs a cast first | -| `Date`, `Time` | Same unit | Yes | Yes | | -| `Uuid` | Any version metadata | Yes | Open | Byte order is the Postgres order | -| `Json` | Any | No | No | Text is not canonical | -| `Vector` | Same dimension and element type | Yes | No | No natural order | -| `FixedShapeTensor` | Equal metadata | Yes | No | Shape and permutation change the meaning | -| WKB and native geometry types | Any | No | No | Two encodings can hold one geometry | -| `ForeignExtDType` | Any | No | No | I9 | - -The float element types of `Vector` give storage equality IEEE semantics (`NaN != NaN`). That is -correct for a derive, because the derive inherits the storage semantics on purpose. +`ExtDTypeRef` gets an erased forwarder: `fn derives(&self) -> Derives`. The binder reads the flags +from the non-nullable form of the dtype. So a type that breaks D4 still gives one answer. -## 4. Binder +### 3.2 Rules + +- **D1.** A derive applies only when every operand in a derived position has the same extension + dtype, ignoring nullability. A lone extension operand never gets a derive, so `ts < 5i64` stays + rejected. Operations across dtypes need an explicit rule (section 3.5). +- **D2.** `eq` requires canonical storage: each value of the dtype has exactly one storage form. +- **D3.** `ord` requires `eq`, and storage order must equal value order. +- **D4.** The flags do not depend on nullability. + +The conformance harness checks D4, and that `ord` implies `eq`. It checks D2 and D3 only when a +type supplies an optional reference order on native values for tests. `NativeValue` requires only +`Display` (`vortex-array/src/dtype/extension/vtable.rs:29-32`), so the harness has no other oracle. -### Resolution order +### 3.3 Use by operation -For a scalar function call that has at least one extension operand: +| Operation | Operands in derived positions | Flag | +| --- | --- | --- | +| `=`, `!=` | Both operands | `eq` | +| `<`, `<=`, `>`, `>=` | Both operands | `ord` | +| `between` | All three operands. `Between::return_dtype` already requires one dtype (`vortex-array/src/scalar_fn/fns/between/mod.rs:282-305`) | `ord` | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | The one input | `ord` | +| `is_constant`, `all_non_distinct`, distinct, group by | The one input | `eq` | +| `list_contains` | The list element and the needle | `eq` | +| Join keys, hash partitioning | The key columns on both sides | `eq` | + +An aggregate binds to one input dtype (`AggregateFnVTable::return_dtype`, +`vortex-array/src/aggregate_fn/vtable.rs:170`). So a unary operation reads the flags of its one +input, and it never needs a second dtype. A function outside this table gets no derive. It uses a +declaration or a type-owned rewrite (section 4). + +### 3.4 Containers + +A struct, list, fixed-size list or map dtype that holds extension values has `eq` or `ord` only if +every nested extension dtype declares it. An order comparison of two `struct{ts: Timestamp(ms)}` +values needs `ord` on `Timestamp(ms)`. Any comparison of two `struct{doc: Json}` values is +rejected. The nested comparator reads nested extension values by storage +(`vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`). It does so only after the +binder checked these flags. + +### 3.5 Operations across dtypes + +Derives never apply across dtypes. Two dtypes with different metadata are different dtypes, for +example `Timestamp(ms)` and `Timestamp(ns)`, or two time zones. The binder rejects such an +operation by default. A type can support one only through an explicit rule that the type owns: + +- **Compatibility.** The type states that two of its dtypes compare by storage as they are. An + example is two zoned timestamps with one unit, because both store UTC instants. +- **Coercion.** The binder casts one operand to the dtype of the other with a type-owned cast + hook. Then the derive of that dtype applies. +- **Rewrite.** The type rewrites the whole call into a storage expression, for example a rescale + before a decimal comparison (section 4.4). + +Open: whether Vortex coerces at all, or engines coerce before they send an expression. Rust and the +built-in `Decimal` reject such operations today. SQL engines coerce them. + +### 3.6 Table for the types in the repository + +| Type | `eq` | `ord` | Reason | +| --- | --- | --- | --- | +| `Timestamp` | Yes | Yes | One dtype fixes the unit and the time zone, so storage counts one unit from one epoch | +| `Date`, `Time` | Yes | Yes | | +| `Uuid` | Yes | Open | Byte order is the Postgres order | +| `Json` | No | No | Text is not canonical | +| `Vector` | Yes | No | No natural order | +| `FixedShapeTensor` | Yes | No | No natural order | +| WKB and native geometry types | No | No | Two encodings can hold one geometry | +| `ForeignExtDType` | No | No | I9 | + +Vortex compares floats with a total order. Equality is bitwise, so `NaN` equals a `NaN` with the +same bits, and `-0.0` differs from `+0.0` +(`vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs:31-34`, `nested.rs:258`). Probe R3 +shows this for primitive and fixed-size list compares. A `Vector` derive inherits these storage +semantics on purpose. + +## 4. Binder -1. If the function is structural (section 2), bind it without a change. -2. The function can be a comparison, `between`, or an order-based aggregate. If the extension type - declares the matching derive for the operand pair, rewrite the call to the same operation on - `ext_storage` of each operand. -3. If the extension type provides a rewrite (`ExtVTable::rewrite`), use it. -4. If the function itself handles the extension type (for example `replace_time_zone` for - `Timestamp`), bind it without a change. The function owner then owns the semantics. -5. Otherwise, return an error that names the function, the extension ID, and the missing derive. +### 4.1 Resolution order + +The binder handles every call that has at least one operand whose dtype contains an extension +dtype. It tries these steps in order: + +1. The function declares the call structural for these operands. Bind it without a change. +2. The function declares itself type-aware for every extension type in the operands. Bind it + without a change. The function owner owns the semantics. +3. The function declares a derive, and the operands meet D1 and section 3.4. If every operand in + a derived position is a top-level extension operand, rewrite each one to `ext_storage`. A + scalar result with the extension dtype, such as the result of `min`, gets a checked + `ext_wrap`. Otherwise, bind the call without a change. An example is `list_contains`, where the + list holds the extension values. Its kernel then compares nested values by storage, which the + derive check authorizes. +4. An extension type of an operand provides a rewrite for the call. Use it. The binder asks the + types in argument order. +5. Otherwise, return an error that names the function, the extension ID, and the missing + declaration. For a cast, the binder tries these steps in order: @@ -137,7 +204,70 @@ For a cast, the binder tries these steps in order: 3. `cast_from` of the target type. 4. An error. -### Proposed hooks +### 4.2 Function declarations + +```rust +/// How a function uses an operand whose dtype contains an extension dtype. +pub enum ExtensionUse { + /// The function moves or selects values and never reads them. + Structural, + /// The function applies storage equality or order. The binder checks these flags (D1). + Derive(Derives), + /// The function implements the semantics of the matched extension type itself. + TypeAware, + /// The function has no semantics for extension values. The binder rewrites or rejects it. + None, +} + +pub trait ScalarFnVTable { + /// Declares how this function uses the operand at `arg`, which contains `ext`. + /// + /// The default is `ExtensionUse::None`. + fn extension_use( + &self, + options: &Self::Options, + arg: usize, + ext: &ExtDTypeRef, + ) -> ExtensionUse { + ExtensionUse::None + } +} +``` + +`AggregateFnVTable` gets the same method without `arg`, because an aggregate has one input. + +- Only the function owner declares, in the definition of the function. The core grants no + type-aware declaration by default. +- A declaration can depend on the options. `Binary` declares `Derive` with `eq` or `ord` by + operator. `cast` is structural only for a nullability change. `stat` depends on its aggregate + (`vortex-array/src/scalar_fn/fns/stat.rs:112-114`). +- A `TypeAware` declaration matches an extension type by its vtable, for example + `ext.is::()`, never by ID string. A `ForeignExtDType` keeps the real ID + (`vortex-array/src/dtype/extension/foreign.rs:28-56`). So a match by ID also covers a + placeholder, which breaks I9. +- `replace_time_zone` declares `TypeAware` for `Timestamp`. Today its `return_dtype` accepts only + timestamps (`vortex-array/src/scalar_fn/fns/replace_time_zone.rs:131-148`). The tensor and + spatial functions declare their own types in the same way. +- The binder never infers a declaration from `return_dtype`. `Binary::return_dtype` accepts any + extension operand in a comparison today (`vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`, + EXT-Q1). +- A value that a function keeps in its options counts as an operand. `DynamicComparison` keeps its + right operand in its options and calls `Binary` directly + (`vortex-array/src/scalar_fn/fns/dynamic.rs:80-111`). Its declaration covers that value, and a + derive rewrite converts the value to a storage scalar. + +### 4.3 Nested extension values + +An operand counts as an extension operand if its dtype contains an extension dtype at any depth. +Examples are `list_contains` on `List` +(`vortex-array/src/scalar_fn/fns/list_contains/mod.rs:178-194`) and the spatial `collect` on +`List` (`vortex-spatial/src/scalar_fn/collect.rs:60-75`). +`ext_storage` unwraps only the top level (`vortex-array/src/scalar_fn/fns/ext_storage.rs:58-63`). +So a derive rewrite happens only when every derived operand is a top-level extension operand +(section 4.1). Otherwise the call binds unchanged. The kernel then reads nested members by storage, +which the container rule of section 3.4 authorizes. + +### 4.4 Proposed hooks on the extension type ```rust pub trait ExtVTable { @@ -160,23 +290,26 @@ pub trait ExtVTable { } /// Rewrites a scalar function call that has an operand of this type, or returns `None`. + /// + /// The call holds every argument, so a rewrite handles any arity and operands of other dtypes. fn rewrite(ext: &ExtDType, call: &ScalarFnCall) -> VortexResult> { Ok(None) } } ``` -`ScalarFnCall` is a new view of the function, its options and its bound arguments. Aggregates -need the same pair of hooks (`rewrite_aggregate`). Open: whether a session registry keyed by -`(function ID, extension ID)` also exists, so that a function owner can support a type that it does -not own. +`ScalarFnCall` is a new view of the function, its options and its bound arguments. Aggregates need +the same pair of hooks (`rewrite_aggregate`). Open: whether a session registry keyed by function +and extension type also exists. It lets a crate add support for a function and a type that it +does not own. -### Examples +### 4.5 Examples ```text Timestamp(ms) a < Timestamp(ms) b => ext_storage(a) < ext_storage(b) Timestamp(ms) a < lit(Timestamp(ms) 5) => ext_storage(a) < lit(5i64) -min(Timestamp(ms) a) => ext_wrap(min(ext_storage(a)), Timestamp(ms), unchecked) +Timestamp(ms) a < lit(5i64) => rejected (D1) +min(Timestamp(ms) a) => ext_wrap(min(ext_storage(a)), Timestamp(ms), checked) cast(Timestamp(ms) a, Timestamp(ns)) => ext_wrap(checked_mul(ext_storage(a), 1_000_000), Timestamp(ns), unchecked) cast(Timestamp(ns) a, Timestamp(ms)) => rejected unless the type defines rounding @@ -186,70 +319,153 @@ DecimalLike(10, 1) a < DecimalLike(10, 2) b Json a = Json b => rejected ``` -`ext_wrap(..., unchecked)` is correct in these examples, because `min` returns an existing value and -the unit cast keeps the range check in `checked_mul`. +`min` uses a checked wrap, because its result is a scalar and every scalar is valid (section 8). +The unit cast uses an unchecked wrap. Its result stays unvalidated, so a read of a row still +checks the value. -### Array API +### 4.6 Array API The array API builds `ScalarFnArray` values directly. The binder must run in `ScalarFnArray::try_new` (or in the typed constructors such as `Binary::try_new`). It returns the -rewritten tree, so the new array holds `ext_storage` children and an `ext_wrap` parent where needed. -The row path (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42`) then evaluates the -rewritten tree, and I6 holds without other changes. +rewritten tree, so the new array holds `ext_storage` children and an `ext_wrap` parent where +needed. The row path (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42`) then +evaluates the rewritten tree. Open: some rewrites can need session state. The hooks above use only the `ExtDType`, which travels with the dtype, so the binder does not need a session. -### Literals +### 4.7 Simplification of wraps -- `ext_storage(lit(s))` simplifies to `lit(s.to_storage_scalar())`. -- `ext_wrap(ext_storage(x), dtype(x))` simplifies to `x`. -- `ext_storage(ext_wrap(y, d))` simplifies to `y`. +`ext_storage` is infallible today (`vortex-array/src/scalar_fn/fns/ext_storage.rs:87-93`). A +checked `ext_wrap` is fallible in the sense of `ScalarFnVTable::is_infallible` +(`vortex-array/src/scalar_fn/vtable.rs:262-279`). It fails when a row that it evaluates holds a +value that the type rejects. A simplification must not remove such a failure. It must also not add +a failure for a row that the original expression does not evaluate. + +| Rule | Valid when | Reason | +| --- | --- | --- | +| `ext_storage(lit(s))` to `lit(s.to_storage_scalar())` | Always | Both read the storage of one valid scalar, and the dtypes match | +| `ext_storage(ext_wrap(y, d, unchecked))` to `y` | `y` has the storage dtype of `d`, ignoring nullability | Both steps are infallible and keep the values. The result dtype is the dtype of `y` | +| `ext_storage(ext_wrap(y, d, checked))` to `y` | Never | The wrap fails on a value that `d` rejects, and `y` alone does not fail | +| `ext_wrap(ext_storage(x), d, unchecked)` to `x` | `d` equals the dtype of `x`, ignoring nullability | Same values and dtype. The result keeps the validation state of `x`, which came from a real check | +| `ext_wrap(ext_storage(x), d, checked)` to `x` | Never at bind time | `x` can be unvalidated, and then the check can fail | +| `ext_wrap(lit(v), d, mode)` to a literal | `Scalar::try_new(d.with_nullability(n), v)` succeeds, where `n` is the nullability of `v` | A literal is a scalar, and every scalar is valid | + +A counterexample for the third rule uses `NonZero` over `u64` and `y = lit(0u64)`. The original +expression fails, because `NonZero` rejects zero. The simplified expression returns `0`. Probe R4 +shows that the check fails: `Scalar::try_new` with `NonZero` and storage `0` returns an error. +`ext_wrap` itself does not exist yet. + +Binding checks a literal argument of `ext_wrap` in both modes (section 5). A literal can also +appear after binding. `Expression::bind` folds nothing +(`vortex-array/src/expr/bound_expression.rs:309-323`), but simplification turns `cast(lit)` into a +literal (`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`). So +`ext_wrap(cast(lit(0i64), u64), NonZero, checked)` binds, and a later fold meets an invalid literal. +When the fold check fails, the wrap stays in the tree, as a failing cast does. Then a checked wrap +fails only for rows that it evaluates, and an unchecked wrap leaves the check to reads. The binder +emits an unchecked wrap only where the values are valid by construction. ## 5. `ext_wrap` -- Options: the target `ExtDTypeRef`, and a flag that is `checked` or `unchecked`. +- Options: the target `ExtDTypeRef`, and a mode that is `checked` or `unchecked`. - `return_dtype`: the target dtype, with the nullability of the input. -- Argument: must have the storage dtype of the target, ignoring nullability. -- Checked execution: calls `validate_array` (section 8), then builds the wrapper. -- Unchecked execution: builds the wrapper. Only the binder and kernels that keep the invariants of - the type use it. -- Row path: `Scalar::try_new` for checked, `Scalar::new_unchecked` for unchecked. - -## 6. Normal form - -Only the wrapper (`vortex.ext`) carries an extension dtype. These reduce rules move the wrapper to -the top: - -| From | To | -| --- | --- | -| `Constant(extension scalar s)` | `Extension(Constant(s.to_storage_scalar()))` | -| `Chunked[Extension(d, a_i)]` | `Extension(d, Chunked[a_i])` | -| `Dict(codes, Extension(d, values))` | `Extension(d, Dict(codes, values))` | -| `Masked(Extension(d, x), m)` | `Extension(d with the result nullability, Masked(x, m))` | -| `Sparse` with extension values | `Extension(Sparse` over storage`)` | - -Consequences: - -- `ExtensionConstantRule` (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`) reverses - direction. -- `ArrayRef::as_constant()` must look through the wrapper, or a new helper does it. Many kernels - call `as_constant()` on an operand. -- `encodings/sparse/src/canonical.rs:360-362` no longer needs to reject extension arrays. -- `DateTimeParts` becomes an `i64` encoding with the time unit in its metadata. Today its dtype is - `Timestamp` (`encodings/datetime-parts/src/canonical.rs:28`). Readers convert the old form on - read. An edition gates writes of the new form. `vortex-btrblocks/src/schemes/temporal.rs` chooses - the encoding from the extension metadata, and compresses the storage child. -- The same applies to any future encoding for one extension type. - -Open: the direction of the normal form. The alternative is "wrapper below generic encodings": -`Constant(extension scalar)`, `Dict(codes, Extension(d, values))` and `Chunked[Extension(d, a_i)]` -stay as they are, and only value-reading encodings must sit under the wrapper. That matches -`ExtensionConstantRule` and the legacy decimal wire form. It has two costs. `ext_storage` needs a -pushdown rule for each generic encoding, or a fallback that executes the encoding to the canonical -wrapper. Also, each generic encoding keeps an extension arm in its canonicalization. The wrapper on -top removes those arms. The Decimal report (`../../decimal/agents/report.md`, sections 3.3 and -3.5) uses the wrapper on top in memory, and its wire plugin writes the older shapes. +- Argument: must have the storage dtype of the target, ignoring nullability. Otherwise binding + fails. +- Literal argument: binding checks it with `Scalar::try_new`, in both modes, and an invalid + literal is a bind error. A literal that simplification makes later follows the fold rule of + section 4.7. +- Checked execution: calls `validate_array` (section 8), then builds a validated wrapper. It is + fallible. +- Unchecked execution: builds the wrapper without a check. The result is unvalidated. It is + infallible. +- Row path: both modes build the scalar with `Scalar::try_new`. `Scalar::new_unchecked` checks + values only in debug builds, and it panics there on an invalid value + (`vortex-array/src/scalar/scalar_impl.rs:97-104`). + +## 6. Physical form (open) + +Opacity is a semantic rule. It does not decide where the wrapper sits relative to generic +encodings such as `Constant`, `Dict`, `Chunked`, `Masked` and `Sparse`. The binder works with +either placement, because `ext_storage` works with either. This section compares the two. No +maintainer decision exists. + +### 6.1 Current code + +- The compressor writes `Constant(extension scalar)` for a constant extension column, and + `Masked(Constant(extension scalar))` when some rows are null + (`vortex-compressor/src/compressor/cascade.rs:272-273`, + `vortex-compressor/src/compressor/constant/mod.rs:175-193`). For other columns, it compares two + results and keeps the smaller one (`cascade.rs:175-205`). The first comes from a scheme that + matches the extension type. For example, `TemporalScheme` writes `DateTimeParts` for a + `Timestamp` (`vortex-btrblocks/src/schemes/temporal.rs:45-56`, `:124`). The second is + `vortex.ext` over the compressed storage. Probe R8 shows the constant, masked and `vortex.ext` + shapes. Its compressor had no temporal scheme. +- `ExtensionConstantRule` rewrites `Extension(Constant(storage))` to `Constant(extension scalar)` + (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`). Probe R5 shows the rewrite. +- `ArrayRef::as_constant()` matches only the `Constant` encoding + (`vortex-array/src/array/erased.rs:485-487`). Probe R5 shows that it misses + `Extension(Constant)`. +- `ext_storage` reads a constant input directly. For other inputs, it executes the input to a + canonical `ExtensionArray` (`vortex-array/src/scalar_fn/fns/ext_storage.rs:78-84`). +- Generic rules push infallible scalar functions into dictionary values and into chunks + (`vortex-array/src/arrays/dict/compute/rules.rs:96-138`, + `vortex-array/src/arrays/chunked/compute/rules.rs:35-36`). Probe R6 shows that + `ext_storage` keeps both encodings. Over `Masked`, it decodes the input. Over `Sparse` with + extension values, it fails, because sparse canonicalization rejects extension arrays + (`encodings/sparse/src/canonical.rs:360-362`). +- Constant canonicalization and masked execution have extension arms + (`vortex-array/src/arrays/constant/vtable/canonical.rs:204-219`, + `vortex-array/src/arrays/masked/execute.rs:178-193`). So does dictionary execution + (`vortex-array/src/arrays/dict/execute.rs:182-193`). +- `vortex.ext`, `vortex.constant`, `vortex.dict` and `vortex.chunked` are in core edition 2025.05 + (`vortex-edition/src/declarations/core/v2025_05.rs:27-33`). `vortex.masked` is in 2025.10 + (`v2025_10.rs:24`). + +### 6.2 Comparison + +| Aspect | Wrapper above generic encodings | Generic encodings above the wrapper | +| --- | --- | --- | +| Shapes of an extension column | One after reduction: `Extension(child)` | Several: `Extension(x)`, `Constant(ext s)`, `Masked(Constant(ext s))`, `Dict(codes, Extension(v))`, `Chunked[...]` | +| Constructors | Code that builds a `Constant`, `Dict` or `Chunked` with an extension dtype makes a form that reduce rules lift. A constructor can also build the lifted form directly. `ExtensionConstantRule` reverses | No change. `Constant` already sits on top | +| Constant detection | `as_constant()` misses `Extension(Constant)`, so callers need a helper that looks through the wrapper. The compressor writes `Extension(Constant(storage))` | `as_constant()` works as today | +| Storage extraction, array path | `ext_storage` reads the child | `ext_storage` keeps `Dict` and `Chunked` through existing rules. `Masked` decodes, and `Sparse` fails (R6) | +| Storage extraction, row path | One reduce rule, `ext_storage(Extension(s))` to `s`, makes the row path read storage. No such rule fired in probe R9 | The same rule, plus a pushdown for `Masked` and `Sparse`. `Dict` and `Chunked` already push `ext_storage` down (R6) | +| Extension arms in generic encodings | None. Sparse works over storage | `Constant`, `Masked` and `Dict` keep theirs. `Sparse` needs one | +| Statistics | Child statistics are storage statistics | `Constant(ext s)` holds an extension scalar. A reader converts it | +| Wire compatibility | Readers read every old form, and reduce rules normalize it. Old readers read `vortex.ext` over `vortex.constant`. `Decimal` needs a translation on write, because `vortex.decimal` is a leaf | The memory form and the wire form match for both types | + +### 6.3 Recommendation + +These notes recommend the wrapper above generic encodings. This is a recommendation, not a +decision. The reasons: + +- One shape removes EXT-S1, and generic encodings lose their extension arms. +- The compressor already writes this shape for non-constant extension columns, unless a + type-specific scheme such as `DateTimeParts` wins (R8). +- Extension columns need no new wire form, because old readers read `vortex.ext` over + `vortex.constant`. + +The costs: + +- `ExtensionConstantRule` reverses. Constant detection in the compressor and in kernels must look + through the wrapper. +- The change lands best after the binder. Before the binder, a kernel that reads a constant + extension operand with `as_constant()` misses `Extension(Constant)`. The `DateTimeParts` compare + kernel then declines (`encodings/datetime-parts/src/compute/compare.rs:31-33`). The extension + compare kernel still compares the storage + (`vortex-array/src/arrays/extension/compute/compare.rs:31-51`). +- The `Decimal` plugin translates the wrapper form to the legacy decimal forms on write + (`../../decimal/agents/report.md`, section 3.5). + +### 6.4 Type-specific encodings + +Type-specific encodings are a separate question. `DateTimeParts` carries the `Timestamp` dtype +today (`encodings/datetime-parts/src/canonical.rs:28`, EXT-S2). Under the binder, a compare +reaches it as `ext_storage(column) < lit(i64)`. To keep its compare kernel, `DateTimeParts` +becomes an `i64` encoding with the time unit in its metadata. Readers convert the old form on +read. An edition gates writes of the new form. `vortex-btrblocks/src/schemes/temporal.rs` chooses +the encoding from the extension metadata, and compresses the storage child. The same applies to +any future encoding for one extension type. ## 7. Statistics @@ -266,33 +482,86 @@ top removes those arms. The Decimal report (`../../decimal/agents/report.md`, se - **S5.** Bounded min and max (truncation) apply to the storage dtype, so they also apply to extensions over `Utf8` and `Binary`. -## 8. Validation +## 8. Validation and trust + +### 8.1 Three kinds of validation + +| Kind | What it checks | When | Cost | +| --- | --- | --- | --- | +| Dtype validation | `validate_dtype`: the metadata and the storage dtype | When a dtype is built or read, for both nullabilities (section 9) | Once for each dtype | +| Storage-shape validation | The storage child has the length of the array, which `Extension::validate` checks today, and the storage dtype (U6). A scalar value has the storage dtype (EXT-V7) | When an array or a scalar is built or read | Constant | +| Value validation | `validate_array`: each checked value is a valid value of the type | At the trust boundaries in section 8.3 | One pass over the values | + +### 8.2 Validation state + +Each extension array is validated or unvalidated. + +- An array is validated only after a value check succeeds for all its checked values, for example + `validate_array`. The result of a structural operation is validated only if all its extension + inputs are validated. +- In this design, a `Scalar` is always valid, because `Scalar::try_new` runs the value check. + Today two gaps exist. A scalar can hold a value of the wrong storage type (EXT-V7, fixed by U6). + `Scalar::zero_value` skips the check (EXT-V3, fixed by U7). +- A constant array checks its scalar when it is decoded + (`vortex-array/src/arrays/constant/vtable/mod.rs:160-161`). So a constant extension array is + validated. +- Every other array is unvalidated. This includes file reads without a value check, unchecked + wraps, and constructors that skip the value check. +- No constructor marks an array validated without a check. A skipped check is never a proof of + validity. + +### 8.3 Where value validation runs + +- Checked constructors and checked `ext_wrap`. The result is validated. +- Imports from Arrow, DuckDB, FFI and Python. The result is validated. +- `Scalar::try_new`, for every extension scalar, as today. +- File reads, when a read option asks for it. The result is validated. Open: the default of this + option. +- Row reads from an unvalidated array check each read value, and return an error for an invalid + value (U3). Row reads from a validated array skip the check. +- A consumer that needs every value to be valid, such as an export, validates an unvalidated array + first. + +`ext_storage` never checks values, on either path. Today the row path checks them. It reads each +child as an extension scalar (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`), and +that read runs the per-row check (`vortex-array/src/arrays/extension/vtable/operations.rs:19-30`). +The array path reads the storage without a check +(`vortex-array/src/scalar_fn/fns/ext_storage.rs:83-84`). Probe R9 shows the difference: the array +path returns `[0, 1]` for `NonZero [0, 1]`, and the row path panics. The row path must read the +storage child directly. + +### 8.4 The array validator and the scalar validator ```rust pub trait ExtVTable { - /// Checks that every non-null value of `storage` is valid for this type. + /// Checks every value of `storage` that is valid in `storage` and in `ancestors`. /// - /// The default accepts every value. A type with value-level rules overrides it. + /// The default applies `validate_scalar_value` to each of those values. An override must + /// accept exactly the arrays that the default accepts. fn validate_array( ext: &ExtDType, storage: &ArrayRef, + ancestors: &Mask, ctx: &mut ExecutionCtx, ) -> VortexResult<()> { - Ok(()) + validate_each_value::(ext, storage, ancestors, ctx) } } ``` -- Callers: `ExtensionArray::try_new`, `ext_wrap` (checked), Arrow import, and file reads when a read - option enables it. -- Kernels that keep the invariants use `ExtensionArray::new_unchecked`. -- A row read from an array uses `Scalar::new_unchecked`. The per-row validation in the current read - path (`vortex-array/src/arrays/extension/vtable/operations.rs:29`) then goes away. -- Examples: - - `Timestamp` computes the storage min and max, then checks only those two values with `jiff`. - - `Uuid` checks the version nibble of each value against the metadata. - - `NormalizedVector` checks the norm of each row within a tolerance. -- Open: is the default a no-op, or a check of each value through `validate_scalar_value`? +- `validate_scalar_value` stays the rule for one value. Its default calls `unpack_native` + (`vortex-array/src/dtype/extension/vtable.rs:53-66`). +- The default `validate_array` is not a no-op. It applies `validate_scalar_value` to each value + that is valid in the array and in every ancestor. +- The caller passes the ancestor mask. Builders pad the rows under a null parent with default + values (`vortex-array/src/builders/fixed_size_list.rs:292-305`). `vortex-spatial` treats + storage behind a null row as arbitrary (`vortex-spatial/src/scalar_fn/row.rs:77-78`). So a + padding value never has to be valid. +- A type can override `validate_array` with a faster check. The conformance harness compares the + override with the default on generated inputs. +- Examples of overrides: `Timestamp` checks the storage minimum and maximum against its valid + range. `Uuid` checks the version nibble of each value against the metadata. A hypothetical + `NormalizedVector` type checks the norm of each row within a tolerance. ## 9. Nullability @@ -317,18 +586,84 @@ pub trait ExtVTable { ``` - `Scalar::zero_value` and `ScalarValue::try_zero_value` use this hook for extension dtypes. -- Open: builders pad a non-nullable child under a null parent - (`vortex-array/src/builders/fixed_size_list.rs:288` and `:303`). A type with no zero has no valid - padding value. One option is that `validate_array` checks only rows that are valid in every - ancestor. Another option is that such a child must be nullable. - -## 11. Scalar identity - -`Scalar` implements `PartialEq` and `Hash` for Rust code: tests, maps of constants, and -deduplication. Keep these structural: two scalars are equal when the dtypes are equal (ignoring -nullability) and the storage values are equal. Document this as identity, not as SQL equality. -`PartialOrd` for extension scalars must follow `derives`, because ordering is semantic. The -compare paths (`scalar_cmp`) must go through the binder rules, not through `PartialOrd`. +- Builders pad a non-nullable child under a null parent + (`vortex-array/src/builders/fixed_size_list.rs:303`, `vortex-array/src/builders/mod.rs:156-166`). + Value validation skips those rows (section 8.4), so a type with no zero still works as such a + child. The padding needs only a storage value. + +## 11. Scalar identity and order + +### 11.1 Current code + +- `Scalar` derives `Eq` (`vortex-array/src/scalar/mod.rs:41`). `PartialEq` and `Hash` compare the + dtype without nullability and the value (`vortex-array/src/scalar/scalar_impl.rs:336-356`). +- `PartialOrd` returns `None` for different dtypes. Otherwise it compares the stored values + (`scalar_impl.rs:358-401`), so extension values compare by storage for every type (EXT-Q2). + Probe R2 shows `Some(Less)` for two `Opaque` values. +- Floats use bitwise equality and `total_cmp` (`vortex-array/src/dtype/ptype.rs:328-336`). +- `scalar_cmp` computes every comparison operator from `partial_cmp` + (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:241-267`). Constant inputs and the row + path use it. +- A `Variant` value has no order, so `partial_cmp` returns `None` even when `==` holds + (`scalar_impl.rs:449-451`). +- An extension scalar can hold a value of the wrong storage type. Then `==` can panic (EXT-V7, + probe R2). +- A valid `F16` scalar can hold a `U64` value, for compatibility with an old serialized form + (`vortex-array/src/scalar/validate.rs:147-157`). Then `==` panics in one direction and returns + `false` in the other (probe R10). + +### 11.2 The Rust rule + +`PartialOrd` must agree with `PartialEq`: `a == b` holds exactly when `partial_cmp(a, b)` returns +`Some(Equal)`. Suppose `PartialEq` stays structural and `PartialOrd` follows the derives. Then the +rule breaks. For an `Opaque` scalar `x` without `ord`, `x == x` is true, but `partial_cmp(x, x)` +returns `None`. + +### 11.3 Contract + +- `PartialEq`, `Eq` and `Hash` describe identity, as today: the same dtype without nullability, + and equal stored values. Integer and decimal values of different widths compare by value + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-176`, + `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:255-281`). +- `PartialOrd` follows three steps: + 1. If `a == b`, return `Some(Equal)`. + 2. If both have one ordered dtype, ignoring nullability, return the order of the stored values. + Nulls sort first. + 3. Otherwise, return `None`. +- A built-in dtype keeps its current order. An extension dtype is ordered only if it declares + `ord`. A container is ordered only if all its members are ordered. The flags come from the + non-nullable dtype. +- `Ord` is not implemented, as today. +- A valid `Eq` needs values whose variant matches the dtype. So U6 checks the storage dtype of + extension scalars (EXT-V7). Scalar construction also converts the `U64` form of an `F16` value + to `F16` (probe R10). + +These steps keep the Rust rule. Step 1 returns `Some(Equal)` exactly when `==` holds. Under the +stored-value order, two values that are not equal never compare as `Equal`, because float +equality is bitwise and float order is `total_cmp`. The steps also fix the `Variant` case. + +### 11.4 Relation to expression semantics + +- Identity serves Rust code: maps of constants, deduplication, tests and caches. It never decides + an expression result for extension values. +- Identity implies semantic equality, because one dtype and one storage value give one value. + The reverse does not hold. Without `eq`, or with a rule across dtypes, two values that are not + identical can be equal. +- Expression comparisons follow the binder. After binding, `scalar_cmp` sees storage scalars at + the top level. It must compute `=` and `!=` with `==`, and the order operators with + `partial_cmp`. Then `=` works on a container with an `eq`-only member, such as + `struct{v: Vector}`. +- The nested comparator follows the container rule of section 3.4. + +### 11.5 Order of work + +Change `PartialOrd` only together with, or after, the phase 2 change that rejects `min` and `max` +for extension types without `ord`. Today `min` and `max` merge partial results with +`partial_min(..).vortex_expect` (`vortex-array/src/aggregate_fn/fns/min/mod.rs:59`, +`vortex-array/src/aggregate_fn/fns/min_max/mod.rs:247`, `:252`). The zone-map writer binds them +for every extension column (`vortex-layout/src/layouts/zoned/writer.rs:217-231`). With the new +order, a `Json` zone that merges two different maxima panics. The merge of several partial results +in one zone is an inference from the code, not a run. ## 12. Files to change, by phase @@ -339,28 +674,33 @@ compare paths (`scalar_cmp`) must go through the binder rules, not through `Part | 0, U3 | `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/scalar/constructor.rs` | | 0, U4 | `vortex-array/src/dtype/extension/typed.rs`, `vortex-array/src/dtype/extension/vtable.rs` | | 0, U5 | `vortex-array/src/extension/datetime/{date,time,timestamp}.rs` | -| 0, U6 | `vortex-array/src/arrays/extension/vtable/mod.rs` | +| 0, U6 | `vortex-array/src/arrays/extension/vtable/mod.rs`, `vortex-array/src/scalar/validate.rs`, `vortex-array/src/scalar/scalar_impl.rs` | | 0, U7 | `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/scalar_value.rs` | | 0, U8 | `vortex-array/src/dtype/extension/erased.rs` | -| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/scalar_fn/fns/{binary,between}/`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | +| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/scalar_fn/vtable.rs`, `vortex-array/src/aggregate_fn/vtable.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/scalar_fn/fns/{binary,between,dynamic.rs,ext_storage.rs}`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-layout/src/layouts/zoned/writer.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | | 3 | `vortex-array/src/scalar_fn/fns/cast/mod.rs`, `vortex-array/src/extension/datetime/`, `vortex-duckdb/src/convert/expr.rs`, `vortex-datafusion/src/convert/exprs/mod.rs` | -| 4 | `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/`, `encodings/sparse/src/canonical.rs`, `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/`, `vortex-edition/` | -| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/builders/extension.rs`, `vortex-array/src/dtype/extension/` | +| 4 | The files depend on the chosen physical form (section 6). Both forms change `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/` and `vortex-edition/`. The wrapper on top also changes `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/` and `vortex-compressor/src/compressor/constant/`. Generic encodings on top also change `encodings/sparse/src/canonical.rs` | +| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/builders/extension.rs`, `vortex-array/src/dtype/extension/` | | 6 | `vortex-arrow/src/session.rs`, `vortex-duckdb/src/convert/`, `vortex-duckdb/src/exporter/`, `vortex-datafusion/src/convert/`, `vortex-python/src/scalar/`, `docs/developer-guide/extending/extension-dtypes.md` | ## 13. Tests - **Path consistency.** A helper in `vortex-array/src/compute/conformance/` takes an expression builder and inputs. It evaluates the expression on the array path, on the row path, and with - constant-encoded inputs. It asserts equal values or an error on all three. + constant-encoded inputs. It asserts equal values or an error on all three. The inputs include + unvalidated arrays with invalid values. - **Reference types.** `vortex-array/src/extension/tests/` already has `DivisibleInt`. Add: - `Opaque` over `i64`, with no derives. Every semantic operation fails when it is bound. - - `DecimalLike` over `i64`, with `eq` and `ord` for the same scale and a rescale rewrite. A - differential test runs each operation on `DecimalLike` and on the built-in `Decimal`, and - compares values or errors. + - `DecimalLike` over `i64`, with `eq` and `ord` for each dtype and a rescale rewrite across + scales. A differential test runs each operation on `DecimalLike` and on the built-in + `Decimal`, and compares values or errors. - `Refined` over `i64` (for example, positive values only), with every derive and a - `validate_array` check. -- **Conformance harness for `ExtVTable`.** Checks the metadata round trip, that `validate_dtype` - ignores nullability, rules D1 to D4, cast round trips, and path consistency. + `validate_array` override. +- **Conformance harness for `ExtVTable`.** It checks the metadata round trip, that + `validate_dtype` ignores nullability, rules D1 to D4, cast round trips, and path consistency. D2 + and D3 need the optional reference order. The harness also compares a `validate_array` override + with the default. +- **Scalar contract.** For generated scalars, `a == b` holds exactly when `partial_cmp(a, b)` + returns `Some(Equal)`, and equal scalars have equal hashes. - **Engines.** DuckDB and DataFusion pushdown tests with extension literals for `Timestamp`, `Date`, `Uuid` and `Json` columns. A `Json` comparison must not be pushed down. diff --git a/research/extension-types/agents/findings.md b/research/extension-types/agents/findings.md index af9e5d22ca8..73fa8c6fa9b 100644 --- a/research/extension-types/agents/findings.md +++ b/research/extension-types/agents/findings.md @@ -9,6 +9,9 @@ This file is the detailed catalog behind `../02-current-problems.md`. Each findi the human documents use. All paths are relative to the repository root. All line numbers are for `develop` at commit `5fc5cb8` (2026-10-17). +This catalog covers current code only. Problems in the proposed design appear as design rationale +in `design-notes.md`. + ## Conventions Severity: @@ -22,8 +25,8 @@ Severity: Status: -- **probe** means that the throwaway test in `probe.rs` reproduced the behavior. The probe ran - against `vortex-array` in a debug build. +- **probe** means that a throwaway test reproduced the behavior. `probe.rs` ran against + `vortex-array`, and `probe_followup.rs` ran in `vortex-btrblocks`. Both ran in debug builds. - **code** means that the behavior follows directly from the cited code. Nothing ran. - **lead** means that the code suggests a problem, but the problem is not confirmed. A lead names the check that confirms or rejects it. @@ -46,7 +49,7 @@ Terms: | EXT-C3 | Same-type extension cast hides the storage error | S3 | code | | EXT-Q1 | Compare skips the type check when one side is an extension | S1 | probe | | EXT-Q2 | Every extension type gets storage equality and order, with no override | S1 | probe | -| EXT-Q3 | `DateTimeParts` compare kernel trusts its operand types | S2 | lead | +| EXT-Q3 | `DateTimeParts` compare kernel trusts its operand types | S1 | probe | | EXT-A1 | Ordering-based operations disagree about extension types | S3 | probe | | EXT-A2 | Zone maps fall back to storage statistics | S3 | code | | EXT-V1 | Arrays are never value-checked, and reads panic on bad values | S2 | probe | @@ -55,6 +58,7 @@ Terms: | EXT-V4 | `Date` and `Time` panic on non-primitive storage | S2 | code | | EXT-V5 | `Timestamp` accepts dtypes that reject every value | S2 | code | | EXT-V6 | Eight extension types never check values | S3 | code | +| EXT-V7 | Extension scalars skip the storage dtype check | S2 | probe | | EXT-N1 | A nullability change can panic | S2 | probe | | EXT-S1 | A constant extension column has two shapes | S3 | code | | EXT-S2 | Some encodings carry the extension dtype themselves | S3 | code | @@ -189,9 +193,9 @@ Terms: - **Impact:** The result is wrong for any type whose storage order or storage equality differs from its value order or value equality. Examples in the repository: JSON (key order and whitespace), WKB geometry (two encodings of one geometry), vectors and tensors (no natural order). -- **Fix:** See `design-notes.md` (pairwise derives that the binder reads). +- **Fix:** See `design-notes.md` (derives for one dtype that the binder reads). -### EXT-Q3: `DateTimeParts` compare kernel trusts its operand types (lead) +### EXT-Q3: `DateTimeParts` compare kernel trusts its operand types - **Locations:** - `encodings/datetime-parts/src/compute/compare.rs:24-61`: the kernel calls @@ -201,16 +205,18 @@ Terms: the kernel of the encoding without checking that the operand dtypes match. - `vortex-array/src/scalar/downcast.rs:161-164`: `Scalar::as_extension` panics if the scalar is not an extension. -- **Suspected behavior:** A `DateTimeParts`-encoded timestamp column compared with an `i64` - constant panics in `as_extension()`. EXT-Q1 lets that expression through. A comparison with a - timestamp constant in a different unit compares sub-second parts in different units. -- **Counter-evidence:** `vortex-file/src/tests.rs:2369` and `:2410` assert that a unit mismatch - returns an error in a file scan. The tests do not show which layer raises the error. Zone-map - pruning can raise it before the kernel runs. -- **Check:** Build a `DateTimePartsArray` directly, compare it with an `i64` constant, and execute - it on the array path. Then repeat with a timestamp constant in another unit. +- **Observed** [R7]: the probe encodes a `Timestamp(ms)` column `[1.6 s, 1.4 s]` as + `DateTimeParts`. The correct answer to `< 1.5 s` is `[false, true]`. + - Against the `i64` constant `1500`, the array path panics with `Failed to convert scalar to + extension`. The row path returns `Cannot compare scalars with incompatible types`. + - Against a `Timestamp(ns)` constant of 1.5 s, the array path returns `[true, true]`, which is + wrong. The row path returns an error. + - Against a `Timestamp(ms)` constant of 1.5 s, the array path returns `[false, true]`. +- **Earlier counter-evidence:** `vortex-file/src/tests.rs:2369` and `:2410` assert that a unit + mismatch returns an error in a file scan. The probe shows that the kernel itself raises no + error. So another layer raises it in those tests. This catalog does not identify that layer. - **Fix:** Make `CompareExecuteAdaptor` require equal dtypes (ignoring nullability) before it calls - any kernel. Make the kernel check the unit. + any kernel. Make the kernel check the unit. U2 covers both. ### EXT-A1: Ordering-based operations disagree about extension types @@ -256,9 +262,12 @@ Terms: `vortex_expect` if the value is invalid. `Scalar::extension::` (line 243) also panics. - **Observed:** With a `NonZero` test type, `ExtensionArray::new(non_zero, [0])` succeeds [L3]. `append_zeros(1)` succeeds [L1]. Reading the row panics with `unable to construct an extension - Scalar` [L2]. -- **Fix:** Return an error from these read paths. Add value validation at trust boundaries - (`design-notes.md`, "Validation"). + Scalar` [L2]. `ext_storage` over `NonZero [0, 1]` returns `[0, 1]` on the array path and panics + on the row path [R9]. The row path reads the child as an extension scalar + (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`). +- **Fix:** Return an error from these read paths. Make the row path of `ext_storage` read the + storage without a value check. Add value validation at trust boundaries (`design-notes.md`, + "Validation and trust"). ### EXT-V2: `Extension::validate` never checks the storage dtype @@ -316,6 +325,32 @@ Terms: - **Impact:** These types check nothing at the value level, and `Display` shows raw storage. This is allowed by the trait, but it means that "scalars are checked" is not true for these types. +### EXT-V7: Extension scalars skip the storage dtype check + +- **Locations:** + - `vortex-array/src/scalar/validate.rs:100` and `:294`: for an extension dtype, scalar validation + calls only `validate_storage_value`. + - `vortex-array/src/dtype/extension/erased.rs:107-109`: `validate_storage_value` calls only the + `validate_scalar_value` of the type. + - `vortex-array/src/scalar/validate.rs:46-50`: a primitive dtype requires a value of the same + `PType`. Extension storage never gets this check. + - `vortex-array/src/extension/datetime/timestamp.rs:211`: `Timestamp` casts any primitive value + to `i64` before it checks the range. + - `vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-160`: `PValue` equality and order + upcast the other value with `vortex_expect`. The upcast of a float to an integer fails + (`pvalue.rs:197-203`, `:397-418`). +- **Observed** [R2]: `Scalar::try_new(Timestamp(ms), F64(5.0))` succeeds and displays as + `1970-01-01T00:00:00.005Z`. Call that scalar `f`, and call the `Timestamp(ms)` scalar with + `I64(5)` `i`. Then `i == f` panics with `upcast`, and `f == i` returns `false`. So `==` is not + symmetric. `i.partial_cmp(&f)` also panics. +- **Related primitive case** [R10]: a valid `F16` scalar can hold a `U64` value, for compatibility + with an old serialized form (`vortex-array/src/scalar/validate.rs:147-157`). `==` between the + `U64` form and the `F16` form of one value panics in one direction and returns `false` in the + other. +- **Reach:** Not measured. The probe builds the scalars with `Scalar::try_new` directly. +- **Fix:** Check that an extension value matches the storage dtype before the check of the type. + U6 covers it. Convert the `U64` form of an `F16` value to `F16` when the scalar is built. + ### EXT-N1: A nullability change can panic - **Locations:** @@ -341,7 +376,8 @@ Terms: `TODO(connor): This is maybe incorrect unless this is a refinement type?`. - **Impact:** Kernels and extension crates must handle both shapes. `vortex-tensor` handles both by hand. The same choice exists for `Dict` and `Chunked`. -- **Fix:** Choose one normal form (see `design-notes.md`, "Normal form"). +- **Fix:** Choose one physical form. `design-notes.md`, section 6, compares the two options. The + choice is open. ### EXT-S2: Some encodings carry the extension dtype themselves @@ -487,6 +523,50 @@ build of `vortex-array`. "Row" means `execute_scalar` per row on the lazy result | P | `Timestamp(ms) + Timestamp(ms)` | build | `incompatible types for arithmetic operation` | | Q | `Scalar::is_zero(Timestamp(ms)=0)` | call | `Some(true)` | +### Follow-up probes + +`probe_followup.rs` ran in a debug build of `vortex-btrblocks`, on `5fc5cb8`. It reuses `NonZero` +and `Opaque`. Its session registers the `DateTimeParts` and `Sparse` encodings. R8 uses a cascading +compressor with the integer dictionary scheme only. With the default schemes, `TemporalScheme` can +also write `DateTimeParts` at the top of a `Timestamp` column +(`vortex-btrblocks/src/schemes/temporal.rs:45-56`). + +| Probe | Expression | Path | Result | +| --- | --- | --- | --- | +| R1 | `cast(Decimal(10, 2) [1.00, 2.50], f32)` | array | `No CastKernel to cast canonical array vortex.decimal from decimal(10,2) to f32` | +| R1 | same | row | `[1f32, 2.5f32]` | +| R1 | `cast(Decimal(10, 2) 1.00, f32)` on a scalar | scalar | `1f32` | +| R2 | `Opaque(1).partial_cmp(Opaque(2))` | call | `Some(Less)` | +| R2 | `Opaque(1) == Opaque(1)`, then `partial_cmp` | call | `true`, `Some(Equal)` | +| R2 | `Scalar::try_new(Timestamp(ms), F64(5.0))` | call | succeeds: `1970-01-01T00:00:00.005Z` | +| R2 | `Timestamp(ms) I64(5) == Timestamp(ms) F64(5.0)` | call | panic: `upcast` | +| R2 | `Timestamp(ms) F64(5.0) == Timestamp(ms) I64(5)` | call | `false` | +| R2 | `Timestamp(ms) I64(5)` `partial_cmp` `F64(5.0)` | call | panic: `upcast` | +| R3 | `NaN == NaN`, then `-0.0 == +0.0`, on scalars | call | `true`, `false` | +| R3 | `[NaN, -0.0] = [NaN, +0.0]` | array | `[true, false]` | +| R3 | `FixedSizeList`: `[[NaN, -0.0], [NaN, 1.0]] = [[NaN, +0.0], [NaN, 1.0]]` | array | `[false, true]` | +| R4 | `Scalar::try_new(NonZero, 0u64)` | call | `zero is not a valid NonZero value` | +| R5 | `Extension(Constant(1000i64))`: tree, then `as_constant()` | call | `vortex.ext` over `vortex.constant`, `None` | +| R5 | same, after `optimize()` | call | `vortex.constant(vortex.timestamp[ms](i64))`, `Some(1970-01-01T00:00:01Z)` | +| R6 | `ext_storage(Extension(Dict))`, one step | call | `vortex.dict(i64)` | +| R6 | `ext_storage(Dict(codes, Extension(values)))` | call | The lazy tree pushes `ext_storage` into the values. One step gives `vortex.dict(i64)` | +| R6 | `ext_storage(Chunked[Extension, Extension])` | call | The lazy tree pushes `ext_storage` into each chunk | +| R6 | `ext_storage(Masked(Extension))`, one step | call | No pushdown. One step decodes to `vortex.primitive(i64?)` | +| R6 | `ext_storage(Sparse)` with extension values, one step | call | `Sparse canonicalization does not support Extension arrays yet` | +| R7 | `DateTimeParts` of `Timestamp(ms) [1.6 s, 1.4 s]` `< 1500i64` | array | panic: `Failed to convert scalar to extension` | +| R7 | same | row | `Cannot compare scalars with incompatible types` | +| R7 | same column `< Timestamp(ns)` 1.5 s | array | `[true, true]`, which is wrong. Expected: `[false, true]` | +| R7 | same | row | `Cannot compare scalars with incompatible types` | +| R7 | same column `< Timestamp(ms)` 1.5 s, the control | array | `[false, true]` | +| R8 | Compress a constant `Timestamp(ms)` column | call | `vortex.constant(vortex.timestamp[ms](i64))` | +| R8 | Compress a low-cardinality `Opaque` column | call | `vortex.ext` over `vortex.dict(i64)` | +| R8 | Compress a constant `Timestamp(ms)` column with some nulls | call | `vortex.masked` over `vortex.constant(vortex.timestamp[ms](i64?))` | +| R9 | `ext_storage(NonZero [0, 1])` | array | `[0u64, 1u64]` | +| R9 | same | row | panic: ``unable to construct an extension `Scalar` `` | +| R10 | `Scalar::try_new(F16, U64 bits of 1.0)` | call | succeeds: `1f16` | +| R10 | `U64` form `==` `F16` form of 1.0 | call | panic: `upcast` | +| R10 | `F16` form `==` `U64` form of 1.0 | call | `false` | + ## Appendix B: where storage semantics are applied This table lists every place where code outside an extension crate applies storage semantics to an diff --git a/research/extension-types/agents/probe_followup.rs b/research/extension-types/agents/probe_followup.rs new file mode 100644 index 00000000000..c4d154e836c --- /dev/null +++ b/research/extension-types/agents/probe_followup.rs @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Throwaway follow-up probes for the type research on `develop` at `5fc5cb8`. +//! +//! This file is not part of any crate, so nothing compiles it. To run it, copy it to +//! `vortex-btrblocks/tests/research_probe.rs` and run: +//! +//! ```text +//! cargo test -p vortex-btrblocks --test research_probe -- --nocapture --test-threads=1 +//! ``` +//! +//! Delete the copy afterwards. Appendix A of `findings.md` records the output of the last run. +//! "Array path" means `execute::` on the lazy array, then a read of each row. +//! "Row path" means `execute_scalar` on each row of the lazy array. + +#![allow(clippy::all, clippy::disallowed_methods, unused)] + +use std::panic::AssertUnwindSafe; +use std::panic::catch_unwind; +use std::sync::LazyLock; + +use vortex_array::ArrayRef; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::RecursiveCanonical; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::ChunkedArray; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::DecimalArray; +use vortex_array::arrays::DictArray; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::TemporalArray; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::DecimalDType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::dtype::extension::ExtDTypeRef; +use vortex_array::dtype::extension::ExtId; +use vortex_array::dtype::extension::ExtVTable; +use vortex_array::expr::ext_storage; +use vortex_array::expr::root; +use vortex_array::extension::datetime::TimeUnit; +use vortex_array::extension::datetime::Timestamp; +use vortex_array::optimizer::ArrayOptimizer; +use vortex_array::scalar::DecimalValue; +use vortex_array::scalar::PValue; +use vortex_array::scalar::Scalar; +use vortex_array::scalar::ScalarValue; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_array::validity::Validity; +use vortex_btrblocks::CascadingCompressor; +use vortex_btrblocks::Scheme; +use vortex_btrblocks::schemes::integer::IntDictScheme; +use vortex_buffer::buffer; +use vortex_datetime_parts::DateTimeParts; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_session::VortexSession; +use vortex_sparse::Sparse; + +/// The array session with the `DateTimeParts` and `Sparse` encodings and their kernels registered. +static SESSION: LazyLock = LazyLock::new(|| { + let session = vortex_array::array_session(); + vortex_datetime_parts::initialize(&session); + vortex_sparse::initialize(&session); + session +}); + +/// u64 values where zero is not a valid value. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonZero; + +impl ExtVTable for NonZero { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_zero") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::U64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + let v = storage_value.as_primitive().cast::()?; + if v == 0 { + vortex_bail!("zero is not a valid NonZero value"); + } + Ok(v) + } +} + +/// An extension type with no meaningful order (think "opaque handle"). +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct Opaque; + +impl ExtVTable for Opaque { + type Metadata = EmptyMetadata; + type NativeValue<'a> = i64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.opaque") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::I64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +fn report(name: &str, f: impl FnOnce() -> VortexResult) { + match catch_unwind(AssertUnwindSafe(f)) { + Ok(Ok(v)) => println!("[{name}] OK: {v:?}"), + Ok(Err(e)) => println!( + "[{name}] ERR: {}", + e.to_string().lines().next().unwrap_or_default() + ), + Err(p) => { + let msg = p + .downcast_ref::() + .cloned() + .or_else(|| p.downcast_ref::<&str>().map(|s| s.to_string())) + .unwrap_or_default(); + println!( + "[{name}] PANIC: {}", + msg.lines().next().unwrap_or_default() + ) + } + } +} + +fn ts(unit: TimeUnit, nullability: Nullability) -> ExtDTypeRef { + Timestamp::new(unit, nullability).erased() +} + +fn opaque() -> ExtDTypeRef { + ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + ) + .unwrap() + .erased() +} + +fn non_zero() -> ExtDTypeRef { + ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased() +} + +/// Reads every row with `execute_scalar` (the row path). +fn rows(a: &ArrayRef) -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let mut out = vec![]; + for i in 0..a.len() { + out.push(a.execute_scalar(i, &mut ctx)?.to_string()); + } + Ok(format!("{} [{}]", a.dtype(), out.join(", "))) +} + +/// Executes the whole array through the array kernels, then reads the stored values. +fn array(a: &ArrayRef) -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let canonical = a + .clone() + .execute::(&mut ctx)? + .0 + .into_array(); + rows(&canonical) +} + +/// The encoding tree of an array, on one line. +fn tree(a: &ArrayRef) -> String { + a.display_tree_encodings_only() + .to_string() + .lines() + .map(str::trim) + .collect::>() + .join(" | ") +} + +#[test] +fn r1_decimal_to_f32() { + let dec = || { + DecimalArray::new( + buffer![100i64, 250], + DecimalDType::new(10, 2), + Validity::NonNullable, + ) + .into_array() + }; + let f32_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); + + report("R1 array path: Decimal(10, 2) [1.00, 2.50] -> f32", || { + array(&dec().cast(f32_dtype.clone())?) + }); + report("R1 row path: Decimal(10, 2) [1.00, 2.50] -> f32", || { + rows(&dec().cast(f32_dtype.clone())?) + }); + report("R1 scalar path: Decimal(10, 2) 1.00 -> f32", || { + let s = Scalar::decimal( + DecimalValue::I64(100), + DecimalDType::new(10, 2), + Nullability::NonNullable, + ); + Ok(s.cast(&f32_dtype)?.to_string()) + }); +} + +#[test] +fn r2_scalar_traits() { + let ms = ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let a = Scalar::extension_ref(opaque(), Scalar::from(1i64)); + let b = Scalar::extension_ref(opaque(), Scalar::from(2i64)); + + report("R2 Opaque(1).partial_cmp(Opaque(2))", || Ok(a.partial_cmp(&b))); + report("R2 Opaque(1) == Opaque(1), and partial_cmp", || { + Ok((a == a.clone(), a.partial_cmp(&a))) + }); + + let f64_ts = || { + Scalar::try_new( + DType::Extension(ms.clone()), + Some(ScalarValue::Primitive(PValue::F64(5.0))), + ) + }; + let i64_ts = Scalar::extension_ref(ms.clone(), Scalar::from(5i64)); + + report("R2 try_new(Timestamp(ms), F64 5.0)", || { + Ok(f64_ts()?.to_string()) + }); + report("R2 Timestamp(ms) I64 5 == Timestamp(ms) F64 5.0", || { + Ok(i64_ts == f64_ts()?) + }); + report("R2 Timestamp(ms) F64 5.0 == Timestamp(ms) I64 5", || { + Ok(f64_ts()? == i64_ts) + }); + report("R2 Timestamp(ms) I64 5 partial_cmp F64 5.0", || { + Ok(i64_ts.partial_cmp(&f64_ts()?)) + }); +} + +#[test] +fn r3_float_identity() { + report("R3 Scalar NaN == NaN", || { + Ok(Scalar::from(f64::NAN) == Scalar::from(f64::NAN)) + }); + report("R3 Scalar -0.0 == +0.0", || { + Ok(Scalar::from(-0.0f64) == Scalar::from(0.0f64)) + }); + report("R3 array path: [NaN, -0.0] = [NaN, +0.0]", || { + let l = buffer![f64::NAN, -0.0].into_array(); + let r = buffer![f64::NAN, 0.0].into_array(); + array(&l.binary(r, Operator::Eq)?) + }); + report( + "R3 array path: FSL [[NaN, -0.0], [NaN, 1.0]] = [[NaN, +0.0], [NaN, 1.0]]", + || { + let l = FixedSizeListArray::try_new( + buffer![f32::NAN, -0.0, f32::NAN, 1.0].into_array(), + 2, + Validity::NonNullable, + 2, + )? + .into_array(); + let r = FixedSizeListArray::try_new( + buffer![f32::NAN, 0.0, f32::NAN, 1.0].into_array(), + 2, + Validity::NonNullable, + 2, + )? + .into_array(); + array(&l.binary(r, Operator::Eq)?) + }, + ); +} + +#[test] +fn r4_non_zero_check() { + report("R4 Scalar::try_new(NonZero, 0u64)", || { + Ok(Scalar::try_new( + DType::Extension(non_zero()), + Some(ScalarValue::Primitive(PValue::U64(0))), + )? + .to_string()) + }); +} + +#[test] +fn r5_constant_detection() { + let ms = ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let ext_const = ExtensionArray::new( + ms, + ConstantArray::new(Scalar::from(1000i64), 3).into_array(), + ) + .into_array(); + + report("R5 Extension(Constant): tree and as_constant()", || { + Ok((tree(&ext_const), ext_const.as_constant().map(|s| s.to_string()))) + }); + report("R5 after optimize(): tree and as_constant()", || { + let optimized = ext_const.optimize()?; + Ok(( + tree(&optimized), + optimized.as_constant().map(|s| s.to_string()), + )) + }); +} + +#[test] +fn r6_storage_extraction() { + let codes = || buffer![0u8, 1, 0, 1].into_array(); + let values = || buffer![10i64, 20].into_array(); + + let one_step = |input: ArrayRef| -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let lazy = input.apply(&ext_storage(root()))?; + let stepped = lazy.clone().execute::(&mut ctx)?; + Ok(format!("lazy: {} || one step: {}", tree(&lazy), tree(&stepped))) + }; + + report("R6 ext_storage(Extension(Dict))", || { + let input = ExtensionArray::new( + opaque(), + DictArray::try_new(codes(), values())?.into_array(), + ) + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Dict(codes, Extension(values)))", || { + let input = DictArray::try_new( + codes(), + ExtensionArray::new(opaque(), values()).into_array(), + )? + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Chunked[Extension, Extension])", || { + let chunk = || ExtensionArray::new(opaque(), values()).into_array(); + let input = ChunkedArray::try_new([chunk(), chunk()], DType::Extension(opaque()))? + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Masked(Extension))", || { + let child = ExtensionArray::new(opaque(), buffer![10i64, 20, 10, 20].into_array()) + .into_array(); + let validity = + Validity::Array(BoolArray::from_iter([true, false, true, false]).into_array()); + one_step(MaskedArray::try_new(child, validity)?.into_array()) + }); + report("R6 ext_storage(Sparse with extension values)", || { + let input = Sparse::try_new( + buffer![1u64, 3].into_array(), + ExtensionArray::new(opaque(), values()).into_array(), + 4, + Scalar::extension_ref(opaque(), Scalar::from(0i64)), + )? + .into_array(); + one_step(input) + }); +} + +#[test] +fn r7_datetime_parts_compare() { + let ms = || ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let ns = ts(TimeUnit::Nanoseconds, Nullability::NonNullable); + + // 1.6 s and 1.4 s, so `< 1.5 s` is [false, true]. + let column = || -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let temporal = TemporalArray::new_timestamp( + buffer![1600i64, 1400].into_array(), + TimeUnit::Milliseconds, + None, + ); + Ok(DateTimeParts::try_from_temporal(temporal, &mut ctx)?.into_array()) + }; + let i64_const = || ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + let ns_const = || { + ConstantArray::new( + Scalar::extension_ref(ns.clone(), Scalar::from(1_500_000_000i64)), + 2, + ) + .into_array() + }; + let ms_const = || { + ConstantArray::new(Scalar::extension_ref(ms(), Scalar::from(1500i64)), 2).into_array() + }; + + report("R7 column tree", || Ok(tree(&column()?))); + report("R7 array path: DateTimeParts(ms) < i64 1500", || { + array(&column()?.binary(i64_const(), Operator::Lt)?) + }); + report("R7 row path: DateTimeParts(ms) < i64 1500", || { + rows(&column()?.binary(i64_const(), Operator::Lt)?) + }); + report("R7 array path: DateTimeParts(ms) < Timestamp(ns) 1.5 s", || { + array(&column()?.binary(ns_const(), Operator::Lt)?) + }); + report("R7 row path: DateTimeParts(ms) < Timestamp(ns) 1.5 s", || { + rows(&column()?.binary(ns_const(), Operator::Lt)?) + }); + report("R7 array path (control): DateTimeParts(ms) < Timestamp(ms) 1500", || { + array(&column()?.binary(ms_const(), Operator::Lt)?) + }); +} + +#[test] +fn r8_compressor_shapes() { + let compressor = CascadingCompressor::new(vec![&IntDictScheme as &'static dyn Scheme]); + let compress = |input: ArrayRef| -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + Ok(tree(&compressor.compress(&input, &mut ctx)?)) + }; + + report("R8 constant Timestamp(ms) column", || { + compress( + ExtensionArray::new( + ts(TimeUnit::Milliseconds, Nullability::NonNullable), + buffer![1000i64; 64].into_array(), + ) + .into_array(), + ) + }); + report("R8 low-cardinality Opaque column", || { + let values: Vec = (0..300).map(|i| [10i64, 20, 30][i % 3]).collect(); + compress( + ExtensionArray::new(opaque(), PrimitiveArray::from_iter(values).into_array()) + .into_array(), + ) + }); + report("R8 partly null constant Timestamp(ms) column", || { + let values = (0..64).map(|i| (i % 4 != 0).then_some(1000i64)); + compress( + ExtensionArray::new( + ts(TimeUnit::Milliseconds, Nullability::Nullable), + PrimitiveArray::from_option_iter(values).into_array(), + ) + .into_array(), + ) + }); +} + +#[test] +fn r10_f16_backcompat() { + let f16_dtype = DType::Primitive(PType::F16, Nullability::NonNullable); + let one = vortex_array::dtype::half::f16::from_f32(1.0); + let bits_form = || { + Scalar::try_new( + f16_dtype.clone(), + Some(ScalarValue::Primitive(PValue::U64(u64::from(one.to_bits())))), + ) + }; + let native_form = Scalar::primitive(one, Nullability::NonNullable); + + report("R10 try_new(F16, U64 bits of 1.0)", || { + Ok(bits_form()?.to_string()) + }); + report("R10 U64 form == F16 form", || Ok(bits_form()? == native_form)); + report("R10 F16 form == U64 form", || Ok(native_form == bits_form()?)); +} + +#[test] +fn r9_ext_storage_paths() { + let input = + ExtensionArray::new(non_zero(), buffer![0u64, 1].into_array()).into_array(); + + report("R9 lazy ext_storage(NonZero [0, 1]) tree", || { + Ok(tree(&input.clone().apply(&ext_storage(root()))?)) + }); + report("R9 array path: ext_storage(NonZero [0, 1])", || { + array(&input.clone().apply(&ext_storage(root()))?) + }); + report("R9 row path: ext_storage(NonZero [0, 1])", || { + rows(&input.clone().apply(&ext_storage(root()))?) + }); +} From fc1b5cc39015ce622a32769879918da45d1ab71b Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Mon, 19 Oct 2026 11:47:17 +0100 Subject: [PATCH 05/12] Correct cast and aggregate rules in the type research U1 now also removes the storage target case in ExtScalar::cast. Without that change, constants and the row path still cast an extension value to its storage dtype after the array path stops. The Decimal rewrite of a cast to an integer keeps the current rule: a value outside the target range fails before truncation. Only min and max wrap their result again, because is_sorted and is_constant return Bool. Signed-off-by: Connor Tsui --- research/decimal/02-design.md | 5 +++-- research/decimal/agents/report.md | 7 ++++--- research/extension-types/02-current-problems.md | 7 ++++--- research/extension-types/04-urgent-changes.md | 4 ++++ research/extension-types/agents/design-notes.md | 2 +- research/extension-types/agents/findings.md | 12 +++++++++--- 6 files changed, 25 insertions(+), 12 deletions(-) diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index 76d549f9f69..9de59b3f172 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -60,10 +60,11 @@ Every decimal operation becomes an expression over the child: | `x < 1.005` with scale 2 | `unscaled(x) <= 100` | | `+`, `-`, `*`, `/` | Widen, do a checked integer operation, then check the precision | | Cast to `Decimal(p2, s2)` | Rescale with an exactness check, then check the precision | -| Cast to an integer | Divide by `10^s` toward zero, then do a range-checked integer cast | +| Cast to an integer | Check the target range before truncation, then divide by `10^s` toward zero. So `127.1` fails for `i8`, as today | | Cast to a float | Convert to `f64`, divide by `10^s`, then narrow. Scalars and arrays use one path | | `sum` | Sum the integers at a wider width, then check the precision | -| `min`, `max`, `is_sorted` | Run the storage aggregate, then wrap the result | +| `min`, `max` | Run the storage aggregate, then wrap the result | +| `is_sorted`, `is_constant` | Run the storage aggregate. The `Bool` result stays as it is | These rewrites need two integer kernels that do not exist yet. `rescale` multiplies or divides by a power of ten with a rounding mode, and `check_range` fails on values outside a range. The unit diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index 4eb90cfacea..95a1e841ad2 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -187,7 +187,7 @@ factory picks the canonical width (`vortex-array/src/builders/mod.rs:412-416`). | Operation | Location | Behavior | | --- | --- | --- | | Cast, nullability only | `arrays/decimal/compute/cast.rs:45-80` | Reduce rule, same decimal dtype | -| Cast to integer | `cast.rs:97-103`, `:172-196` | Truncate toward zero, range checked | +| Cast to integer | `cast.rs:97-103`, `:172-196` | Truncate toward zero. A value outside the target range fails before truncation | | Cast to float | `cast.rs:104-107`, `:198-245` | `f64` only, one rounding step | | Cast decimal to decimal | `cast.rs:108-170`, plan at `:348` | Zero-copy when the scale is equal, the precision widens, and the width already fits. Otherwise rescale, exact scale-down only | | Integer to decimal | `arrays/primitive/compute/cast.rs:87-89`, `:127-173` | Floats rejected. Zero-copy when the width equals the canonical width and the scale is 0 | @@ -520,11 +520,12 @@ dtype, the rewrites below are exact. | Multiply | Widen, checked multiply, precision check | Result scale `s1 + s2` | | Divide | Multiply the dividend by `10^k`, checked divide toward zero, precision check | As today | | Cast to `Decimal(p2, s2)` | Rescale with exactness, integer width cast, precision check | Replaces `DecimalCastPlan` | -| Cast to an integer | Divide by `10^s` toward zero, then range-checked integer cast | Replaces `DecimalToIntegerCast` | +| Cast to an integer | For a positive scale, `check_range` on the unscaled values with the bounds `MIN · 10^s` and `MAX · 10^s` of the target, then divide by `10^s` toward zero. For a negative scale, a checked multiply. Then narrow | Replaces `DecimalToIntegerCast` and keeps its rule: a value outside the range fails before truncation, so `127.1` fails for `i8` (`scalar/typed_view/decimal/integer_cast.rs:14-53`, test `cast_decimal_to_integer_bounds`) | | Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays | | Cast from an integer | Widen, multiply by `10^s`, precision check | Replaces `cast_to_decimal` | | Sum | Integer sum with accumulator `storage_ptype(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | -| Min, max, is_sorted, is_constant | Storage aggregate, then wrap | Matches `min_max/extension.rs:17-35` | +| Min, max | Storage aggregate, then wrap the result | Matches `min_max/extension.rs:17-35` | +| is_sorted, is_constant | Storage aggregate. The `Bool` result stays as it is | Valid within one `(p, s)` | | Hash, group by, distinct, join keys | Storage hash and equality | Valid within one `(p, s)` | | Zone-map pruning | Rewrite the predicate onto storage statistics | Old files keep decimal-typed statistics, which hold the same integers | | IN list | Storage set over rescaled needles | Replaces `haystack/mod.rs:118-125` | diff --git a/research/extension-types/02-current-problems.md b/research/extension-types/02-current-problems.md index 1d74ad0b1c9..ebda0758587 100644 --- a/research/extension-types/02-current-problems.md +++ b/research/extension-types/02-current-problems.md @@ -31,9 +31,10 @@ implementation. - **Casts reinterpret storage (EXT-C1, EXT-C2).** A scalar cast into an extension type casts to the storage dtype and wraps the result. The source type plays no part. An array cast out of an - extension type returns the storage cast, so `Timestamp(ms)` casts to `i64`, `f64` and `u8`. Issue - [#7504](https://github.com/vortex-data/vortex/issues/7504) decided to remove the second behavior, - but it is still on `develop`. + extension type returns the storage cast, so `Timestamp(ms)` casts to `i64`, `f64` and `u8`. A + scalar cast out of an extension type accepts the exact storage dtype as a target. Issue + [#7504](https://github.com/vortex-data/vortex/issues/7504) decided to remove the array cast out of + an extension type, but it is still on `develop`. - **Comparisons against raw storage (EXT-Q1).** The compare type check is skipped when either side is an extension. A lone extension side then compares its storage, so `ts < 1500` compares raw `i64` values. `between` checks types strictly, so `ts BETWEEN 0 AND 1500` fails while diff --git a/research/extension-types/04-urgent-changes.md b/research/extension-types/04-urgent-changes.md index dacfec36cad..a6f2842d180 100644 --- a/research/extension-types/04-urgent-changes.md +++ b/research/extension-types/04-urgent-changes.md @@ -32,6 +32,10 @@ must be errors. The table in EXT-P1 lists the first cases. Keep only casts between one extension dtype and the same dtype with another nullability. - Remove the storage fallback in the extension cast kernel (`vortex-array/src/arrays/extension/compute/cast.rs:18-22`). +- Remove the storage target case in `ExtScalar::cast` + (`vortex-array/src/scalar/typed_view/extension/mod.rs:87-96`). `Scalar::cast` sends extension + sources there (`vortex-array/src/scalar/cast.rs:84`). Without this change, constants and the row + path still cast `Timestamp(ms)` to `i64` after the array path stops. - Return the storage error instead of "no `CastKernel`" (EXT-C3). - Change the Arrow exporters that call `array.cast(target)` (EXT-B2). An exporter that exports storage on purpose calls `ext_storage`. Every other exporter returns an error. diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md index 39ac9021a76..76b2551df11 100644 --- a/research/extension-types/agents/design-notes.md +++ b/research/extension-types/agents/design-notes.md @@ -669,7 +669,7 @@ in one zone is an inference from the code, not a run. | Phase | Files | | --- | --- | -| 0, U1 | `vortex-array/src/scalar/cast.rs`, `vortex-array/src/arrays/extension/compute/cast.rs`, `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | +| 0, U1 | `vortex-array/src/scalar/cast.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-array/src/arrays/extension/compute/cast.rs`, `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | | 0, U2 | `vortex-array/src/scalar_fn/fns/binary/mod.rs`, `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs`, `encodings/datetime-parts/src/compute/compare.rs` | | 0, U3 | `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/scalar/constructor.rs` | | 0, U4 | `vortex-array/src/dtype/extension/typed.rs`, `vortex-array/src/dtype/extension/vtable.rs` | diff --git a/research/extension-types/agents/findings.md b/research/extension-types/agents/findings.md index 73fa8c6fa9b..9373d444f87 100644 --- a/research/extension-types/agents/findings.md +++ b/research/extension-types/agents/findings.md @@ -133,6 +133,10 @@ Terms: - **Location:** `vortex-array/src/arrays/extension/compute/cast.rs:18-22`. - **Mechanism:** If the target is not the same extension dtype (ignoring nullability), the kernel returns `storage_array().cast(target)`. +- **Scalar path:** `ExtScalar::cast` accepts the exact storage dtype as a target, ignoring + nullability (`vortex-array/src/scalar/typed_view/extension/mod.rs:87-96`). `Scalar::cast` sends + extension sources there (`vortex-array/src/scalar/cast.rs:84`). So a constant `Timestamp(ms)` + casts to `i64`, even if the kernel stops doing it. - **Observed:** `Timestamp(ms)` casts to `i64`, `f64` and `u8` [E1, E2, E3, E1']. - **History:** PR #7469 added this behavior. Issue #7504 decided to remove it, and was closed as a duplicate of #8570, which is still open. The conformance suite says the opposite of the kernel: @@ -143,9 +147,10 @@ Terms: they receive an extension array, they export its storage through this fallback (EXT-B2). Removing the fallback without changing these callers turns a silent export into an error. - **Expected:** An error, unless the extension type defines the cast. -- **Fix:** Remove the fallback. Callers that intend to read storage must call `ext_storage`. -- **Regression test:** Casting `Timestamp(ms)` to `i64` returns an error on both paths. - `ext_storage` on the same input returns the `i64` storage. +- **Fix:** Remove the fallback in the kernel and the storage target case in `ExtScalar::cast`. + Callers that intend to read storage must call `ext_storage`. +- **Regression test:** Casting `Timestamp(ms)` to `i64` returns an error on the row path, the + array path, and with constant inputs. `ext_storage` on the same input returns the `i64` storage. ### EXT-C3: Same-type extension cast hides the storage error @@ -587,6 +592,7 @@ any extension type. | `vortex-array/src/scalar/typed_view/extension/mod.rs:107-137` | Storage eq, order, hash | Semantic (EXT-Q2) | | `vortex-array/src/aggregate_fn/fns/min_max/extension.rs` | Storage min and max | Semantic (EXT-A1) | | `vortex-array/src/arrays/extension/compute/cast.rs:18-22` | Storage cast | Semantic (EXT-C2) | +| `vortex-array/src/scalar/typed_view/extension/mod.rs:87-96` | Storage cast of a scalar, exact storage target only | Semantic (EXT-C2) | | `vortex-array/src/scalar/cast.rs:56-61` | Storage cast and wrap | Semantic (EXT-C1) | | `vortex-array/src/scalar/scalar_value.rs:90-95`, `builders/extension.rs:110` | Storage zero | Semantic (EXT-V3) | | `vortex-layout/src/layouts/zoned/schema.rs:112-121` | Storage stats dtype | Semantic (EXT-A2) | From 380fb4356b7e96943ef5368b68ede54eb6add3ce Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Mon, 19 Oct 2026 21:52:30 +0100 Subject: [PATCH 06/12] Record type research decisions and update the notes to them The new research/decisions.md records the decisions from the review on 2026-10-19 and the questions that stay open. The other notes follow it. Extension types now implement their behavior as rewrite rules on arrays. The rules are registered with the type, and there is no ext_wrap expression. A lift rule pulls generic structural arrays, such as Constant and Dict, out from under the extension wrapper. ExtensionArray::try_new checks values, and a new new_unchecked serves kernels. Decimal stays a leaf with one buffer and no data child. The precision fixes the buffer width, so each decimal dtype has one canonical encoding. A Narrow encoding stores a column at a lower precision with the same scale. A reader wraps the narrow most significant part of an old DecimalByteParts array in an integer Narrow, without a copy. The notes also list two costs of the one-width rule: an old I128 leaf at precision 39 to 76 needs a widening copy, and an Arrow import at another width copies its buffer. Signed-off-by: Connor Tsui --- research/README.md | 3 + research/decimal/01-current-state.md | 12 +- research/decimal/02-design.md | 365 +++--- research/decimal/03-roadmap.md | 134 +- research/decimal/README.md | 41 +- research/decimal/agents/decimal-call-sites.md | 30 +- research/decimal/agents/report.md | 1166 ++++++++++------- research/decisions.md | 104 ++ research/extension-types/03-design.md | 228 ++-- research/extension-types/04-urgent-changes.md | 26 +- research/extension-types/05-implementation.md | 146 ++- research/extension-types/README.md | 15 +- .../extension-types/agents/design-notes.md | 762 ++++++----- research/extension-types/agents/findings.md | 51 +- 14 files changed, 1828 insertions(+), 1255 deletions(-) create mode 100644 research/decisions.md diff --git a/research/README.md b/research/README.md index 32fca7750f5..265619da89f 100644 --- a/research/README.md +++ b/research/README.md @@ -11,3 +11,6 @@ merged. Each summary document links to the agent documents. The agent documents hold file and line references, reproductions, fix sketches and file lists. + +[`decisions.md`](decisions.md) records the decisions that the maintainer made while reviewing these +notes, and the questions that stay open. The other documents follow it. diff --git a/research/decimal/01-current-state.md b/research/decimal/01-current-state.md index cfd45819e29..6e438389ff8 100644 --- a/research/decimal/01-current-state.md +++ b/research/decimal/01-current-state.md @@ -5,8 +5,8 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # 1. Current state of `Decimal` -All statements are for `develop` at `5fc5cb8`. Sections 2 and 3 of -[`agents/report.md`](agents/report.md) give file and line references for each one. +All statements are for `develop` at `5fc5cb8`. Section 2 of [`agents/report.md`](agents/report.md) +gives file and line references for each one. ## How it works @@ -66,6 +66,8 @@ compares or hashes. With one storage width for each precision, a value has exact within one precision and scale. Then storage equality and storage hashing are value equality and value hashing. -Other pairs need a rewrite. `1.0` at scale 1 is stored as `10`, and `1.00` at scale 2 is stored as -`100`. So a comparison across scales rescales one side first. Two precisions with the same scale -can have different storage widths, so their comparison widens one side first. +Other pairs are different dtypes. `1.0` at scale 1 is stored as `10`, and `1.00` at scale 2 is +stored as `100`. So a comparison across scales must rescale one side first. Two precisions with +the same scale can have different storage widths, so their comparison must widen one side first. +Under decision X3 in [`decisions.md`](../decisions.md), the engine does this step with a cast to a +common dtype. Vortex then compares only equal dtypes. diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index 9de59b3f172..4d4ae69a1b4 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -5,115 +5,168 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # 2. Design +This part follows [`decisions.md`](../decisions.md), which records the decisions of the maintainer +from 2026-10-19. "Decided" marks a settled choice. "Recommendation" marks a proposal of these notes +that is still open. + ## Goal -`Decimal` behaves like an extension type over integer storage, and it stays a built-in type. -Built-in means that every reader knows the type and that its wire tag never changes. It does not -fix the Rust representation. +`Decimal` stays a built-in type that behaves like a hardcoded extension type (decided, D1). Each +decimal dtype has exactly one canonical encoding (decided, D3). Decimal kernels reuse the integer +kernels. + +## The leaf -## The wrapper +A canonical decimal array is a leaf: one values buffer and validity, with no data child (decided, +D2). `DecimalArray` and the `vortex.decimal` wire node already have this shape. -A decimal array becomes a wrapper with no buffers and one child: the unscaled integers. The child -dtype is a function of the precision, with the current canonical widths: +The precision fixes the width of the buffer: ```rust impl DecimalDType { - /// The integer type that holds every unscaled value of this precision. - pub const fn storage_ptype(&self) -> PType { + /// The width of the values buffer for this precision. + pub const fn values_type(&self) -> DecimalType { match self.precision.get() { - 1..=2 => PType::I8, - 3..=4 => PType::I16, - 5..=9 => PType::I32, - 10..=18 => PType::I64, - 19..=38 => PType::I128, - _ => PType::I256, + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, } } } ``` -The wrapper keeps three invariants: +Recommendation (DQ1): use this mapping, which gives the smallest signed width that holds the +digits. It already exists as `DecimalType::smallest_decimal_value_type`. + +The leaf keeps three invariants: -1. **Type.** The child dtype is `Primitive(storage_ptype(p), nullability)`. Construction checks it, - and the check costs nothing for each row. +1. **Width.** The buffer has the width of the precision. Construction checks it, and the check + costs nothing for each row. 2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. Imports from Arrow, DuckDB, FFI and Python check it. Kernels that make new values check their own results. -3. **No data.** The wrapper owns no buffer. Validity comes from the child. - -A narrower physical width becomes an encoding of the child. For precision 1 to 18, the integer -compression schemes narrow the values better than a fixed smaller width does. Examples are frame of -reference, bit packing and dictionaries. For precision 19 and up, the child is a lazy widening of -an `i64` array when every value fits. Otherwise the child is a split into 64-bit parts, which is -DecimalByteParts in a general form. +3. **Leaf.** The array has one buffer and validity, and no data child. So a kernel never receives a width that it does not expect. The `patch` panic and the "does the constant fit" special cases go away. -## Decimal operations become integer operations +`DecimalType` already has `I128` and `I256`. So the leaf needs no new `PType`, and this fix can +land before the wide integers. + +## The integer view + +The integer view is a `PrimitiveArray` over the same buffer and validity. It needs no copy. +Decimal kernels delegate to integer kernels through it. + +For precision 1 to 18, the view uses `I8` to `I64`, which exist today. For precision 19 to 76, the +view needs `I128` and `I256` (decided, D5). Until those types exist, the kernels for wide decimals +stay decimal-specific. + +For extension types, `ext_storage` is the explicit way to reach the storage. For `Decimal`, the +integer view plays that role. + +## `Narrow` + +Without a narrow form, D3 forces the full width on every column. A `Narrow` encoding solves this +(decided, D4). It stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)`, with +`p' < p` and the same nullability. + +- `Narrow` has no buffers. Validity comes from the child. +- It decodes by widening the child buffer to the width of `p`. +- It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. + +Recommendation (DQ2): one `Narrow` encoding for integers and decimals. Integers narrow by width +(issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by +precision. The integer view of a decimal `Narrow` is an integer `Narrow` over the view of the +child. So the integer `Narrow` kernels give decimals their narrow-width speed. + +Precision and integer width do not line up. An `i32` holds values up to 2,147,483,647, which needs +precision 10. But 9 is the largest precision that maps to `i32`. So a writer needs value bounds to +choose `p'`: + +- The compressor has them from statistics. +- A cast that widens the precision has them from the source dtype. It returns a `Narrow` over the + source without a copy. +- An old narrow buffer, read without a scan, does not have them. DecimalByteParts holds it instead, + as the next section describes. + +Recommendation (DQ3): decimal `Narrow` gets its own wire ID, unless it lands before the edition of +PR [#10333](https://github.com/vortex-data/vortex/pull/10333) ships. That draft edition already +defines `vortex.narrow` for integers. + +## DecimalByteParts + +DecimalByteParts (DBP) stays the compressed decimal form. Its children are integers: a signed most +significant part (MSP), and 64-bit lower parts for wide values. The ordinary integer schemes +compress each part. + +Recommendation (DQ4): the dtype of a DBP array fixes the type of its MSP. Then narrowing happens in +only two places: `Narrow` for the precision, and integer encodings such as bit packing for the +bits. The current narrowing step of the compressor (`narrowed_decimal`) has no job left. + +Readers still accept the narrow MSPs of old files. A reader wraps an old narrow MSP in an integer +`Narrow` of the fixed type, without a copy. So the in-memory MSP always has the fixed type. An old +narrow `vortex.decimal` leaf of up to 64 bits becomes a DBP array in the same way. Its buffer +becomes the child of that integer `Narrow`. + +## Where structural encodings sit + +Generic structural encodings sit above the type layer (decided, X5). These notes recommend this +list for them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave` +(XQ1). Decimal storage encodings, DBP and `Narrow`, sit below them. -Every decimal operation becomes an expression over the child: +The decimal wire forms already follow this rule. A constant decimal column is a `vortex.constant` +node with a decimal scalar, or `vortex.masked` over that node when some rows are null. Run-end and +dictionary arrays already hold decimal values. So `Decimal` needs no plugin that translates an +in-memory form back to the legacy wire forms. -| Operation | Rewrite onto the unscaled integers | +## Decimal operations as rules over the view + +`Decimal` implements its semantics as rewrite rules on arrays that the type owns (decided, X4). +Each rule runs integer kernels on the view and builds a decimal result. There is no `ext_wrap` +expression. + +| Operation | Rule over the integer view | | --- | --- | -| `=`, `<`, hash, with the same precision and scale | The same operation | -| `<` across scales | Rescale one side to the larger scale, then compare | -| `x < 1.005` with scale 2 | `unscaled(x) <= 100` | -| `+`, `-`, `*`, `/` | Widen, do a checked integer operation, then check the precision | -| Cast to `Decimal(p2, s2)` | Rescale with an exactness check, then check the precision | +| `=`, `<`, hash, with the same precision and scale | The same integer operation | +| `=`, `<` across precisions or scales | None. The engine casts both sides to one dtype first (decided, X3) | +| `+`, `-`, `*`, `/` with the same dtype | Widen, do a checked integer operation, then check the precision | +| `+`, `-`, `*`, `/` across precisions or scales | Open (DQ6). Recommendation: accept them, with the Arrow rules for the result type | +| Cast to `Decimal(p2, s2)` | Rescale with an exactness check, then check the precision. A wider precision with the same scale needs no copy | | Cast to an integer | Check the target range before truncation, then divide by `10^s` toward zero. So `127.1` fails for `i8`, as today | | Cast to a float | Convert to `f64`, divide by `10^s`, then narrow. Scalars and arrays use one path | | `sum` | Sum the integers at a wider width, then check the precision | -| `min`, `max` | Run the storage aggregate, then wrap the result | -| `is_sorted`, `is_constant` | Run the storage aggregate. The `Bool` result stays as it is | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max` | Run the integer aggregate, then make a decimal of each result value | +| `is_sorted`, `is_constant` | Run the integer aggregate. The `Bool` result stays as it is | + +A cast exists only where a rule for the pair of dtypes exists (decided, C1). `Decimal` ships rules +for the common pairs above. Recommendation (CQ1): the default cast fails on a value that it cannot +represent exactly, and lossy steps such as `trunc` are separate expressions. Under CQ1, the integer +cast above becomes `trunc` and then a cast. The table keeps the current behavior until CQ1 is +settled. -These rewrites need two integer kernels that do not exist yet. `rescale` multiplies or divides by a +These rules need two integer kernels that do not exist yet. `rescale` multiplies or divides by a power of ten with a rounding mode, and `check_range` fails on values outside a range. The unit casts of `Timestamp`, `Time` and `Duration` need the same kernels, because their units are powers of ten apart (except days). The current decimal arithmetic kernel does the operation and the precision check in one pass. A fused integer kernel keeps that speed. -## One wrapper abstraction for `Decimal` and extension types - -`Decimal` and `Extension` need the same structural behavior, but each one implements it separately -today. One trait can hold the shared part: - -```rust -/// A logical wrapper with no data over one storage child. -pub trait StorageWrapper: VTable { - fn storage(array: ArrayView<'_, Self>) -> &ArrayRef; - fn storage_dtype(dtype: &DType) -> VortexResult; - fn rewrap(array: ArrayView<'_, Self>, storage: ArrayRef) -> VortexResult; - fn wrap_scalar(dtype: &DType, storage: Scalar) -> VortexResult; -} -``` - -Generic rules then cover both types: - -- Slice, filter, take and mask push into the child and wrap the result again. -- `scalar_at` wraps the scalar of the child. -- Compression compresses the child with the ordinary schemes. - -The position of the wrapper relative to generic encodings, such as `Constant`, `Dict` and -`Sparse`, is an open question (see below). The -[extension design](../extension-types/03-design.md#physical-form-open) compares both placements. -It recommends the wrapper on top, but no maintainer decision exists. With the wrapper on top, -sparse, run-end, dictionary and chunked arrays work over the storage dtype. That also closes a gap -for extension types, because a sparse array of an extension type cannot canonicalize today. +## Integer lanes or FixedSizeBinary -Type-specific encodings, such as DecimalByteParts and `DateTimeParts`, live under the wrapper and -see only the storage dtype. This choice does not depend on the position of the wrapper. - -## Integer child or FixedSizeBinary child +D2 and D3 fix the shape of the leaf. The remaining choice is the type of the values in the buffer, +and of the view over it. ### Options -| Option | Child | +| Option | Values in the buffer | | --- | --- | -| (a) Integer child | `Primitive(I8)` to `Primitive(I256)`, with two new primitive types | -| (b) FixedSizeBinary child | `FixedSizeBinary(n)`, a new core type | +| (a) Integer lanes | Little-endian signed integers, as today, with an integer view up to `I256` | +| (b) FixedSizeBinary | `FixedSizeBinary(n)` values, a new core type | | (c) Hybrid | Integers up to 64 bits, FixedSizeBinary above | | (d) Separate wide-integer family | A new dtype and canonical array for 128 and 256 bits | -| (e) Current layout, opaque API | The current buffer, with the width fixed to the canonical width | +| (e) Integer lanes without a view | The buffer of (a), with decimal-specific kernels | ### Byte order is not integer order @@ -128,27 +181,26 @@ integer order: Byte comparison gives integer order only for big-endian bytes with the sign bit flipped. The row codec already uses that form. But that layout breaks zero-copy exchange with Arrow and DuckDB -decimals, and it needs a conversion before each arithmetic operation. With either layout, a -FixedSizeBinary child keeps every decimal kernel and adds a reinterpretation step. +decimals, and it needs a conversion before each arithmetic operation. With either layout, +FixedSizeBinary values keep every decimal kernel and add a reinterpretation step. ### Comparison -| | (a) Integer child | (b) FixedSizeBinary child | (e) Current layout | +| | (a) Integer lanes | (b) FixedSizeBinary | (e) No view | | --- | --- | --- | --- | | Compare, arithmetic, aggregates | Generic integer kernels | Decimal-specific kernels | Decimal-specific kernels | -| Compression | All integer schemes up to 64 bits. Narrowing or a 64-bit split for wide values | Dictionaries and general-purpose schemes only | DecimalByteParts only | +| Compression | `Narrow`, DecimalByteParts, and all integer schemes on its parts | Dictionaries and general-purpose schemes only | As (a) | | Min and max statistics | Correct | Wrong for little-endian bytes | Correct | -| Arrow and DuckDB decimals | Zero-copy | Zero-copy | Zero-copy | | DuckDB `HUGEINT` | Possible | Not possible | Not possible | | Decimal file format | No change | Needs FixedSizeBinary first | No change | | Size of the change | Largest, because of the `PType` migration. The compiler finds most sites | A new core type | Smallest | | Duplicated logic | Lowest | High | High, about 7,600 lines stay | -### Verdict: an integer child +### Verdict: integer lanes -Option (a) is the proposal, built as a separate wide-integer tier. The tier keeps the useful part of -option (d). Wide integers opt into a small set of kernels, and they do not act like 64-bit -integers. +Option (a) is the proposal. The leaf keeps integer lanes, and the integer view reaches the integer +kernels. Option (e) is the state before the wide `PType` values land. It fixes the crashes, but +the kernels for precision 19 to 76 stay decimal-specific until then. Wide integers do not need FixedSizeBinary: @@ -157,53 +209,45 @@ Wide integers do not need FixedSizeBinary: take and filter. - Compression of wide integers is a split into 64-bit parts, which DecimalByteParts already does. - Arrow export of decimals uses the Arrow decimal types. Arrow has no 128-bit integer type, so only - a top-level `I128` or `I256` column needs an export rule. `FixedSizeBinary(16)` is one option for + a top-level wide-integer column needs an export rule. `FixedSizeBinary(16)` is one option for that rule, not a prerequisite. -Three findings can change the verdict: - -- If maintainers reject wide integers as a Vortex type, option (e) is the fallback for one release. - It fixes the crashes, but `Decimal` stays a separate world. -- If benchmarks show that generic 128-bit and 256-bit kernels are clearly slower, the decimal - kernels stay as specializations behind the rewrites. This result is unlikely, because the current - decimal kernels already run on `i128` and `i256` values. -- If Vortex needs more than 76 digits or arbitrary precision, no fixed-width child works. - -## The wide-integer tier +## Wide integers -- `PType` gets `I128 = 11` and `I256 = 12`. The values 13 and 14 stay reserved for `U128` and - `U256`, which wait until a consumer needs them. +- `PType` gets `I128`, `I256`, `U128` and `U256` (decided, D5), for example as the values 11 to 14. - Two typed sub-enums split the integers. `LaneIntPType` holds the eight integer types up to 64 - bits, and `WideIntPType` holds `I128` and `I256`. Offsets, codes, take indices and lane encodings - such as FastLanes accept only lane integers. + bits, and `WideIntPType` holds the four wide types. Offsets, codes, take indices and lane + encodings such as FastLanes accept only lane integers. - `is_int()` goes away, so each of its 53 call sites must choose between "lane integer" and "any - integer". Most of them guard offsets, codes or indices. -- The dispatch macros that end in `other => panic!` retire, because a new type turns them into - run-time panics. Typed macros replace them. -- The compiler drives the migration. The sub-enums and macros come first, and the removal of - `is_int()` comes next. Then the macros migrate one crate at a time. The two new variants come - last, so every exhaustive match fails to compile until it handles them. -- The flatbuffer `PType` enum gets the two values, and old readers fail loudly on them. The + integer". The dispatch macros that end in `other => panic!` retire, and typed macros replace + them. +- The compiler drives the migration. The four new variants come last, so every exhaustive match + fails to compile until it handles them. +- The flatbuffer `PType` enum gets the four values, and old readers fail loudly on them. The protobuf `PType` enum stays as it is, because old readers decode an unknown value as `U8` without an error. A new protobuf `oneof` variant carries wide integers instead. An edition gates top-level wide columns. -- Both wide types use 16-byte alignment. Two places assume that the alignment equals the byte - width, and they need a fix for `I256`. +- All wide types use 16-byte alignment. Two places assume that the alignment equals the byte + width, and they need a fix for the 256-bit types. +- Vortex has an `i256` type but no `u256` type. `U256` needs a new one. - Wide integers get no FastLanes encodings, never act as offsets or codes, and get no CUDA dynamic dispatch at first. +Recommendation (DQ5): wide integers are user-visible columns behind an edition. They get a small +kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations. + Top-level wide columns need a rule in each engine: -| Engine | `I128` | `I256` | -| --- | --- | --- | -| Arrow | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)` | -| DuckDB | `HUGEINT` | Error | -| DataFusion | Error | Error | -| Python | Python `int` scalars | Python `int` scalars | +| Engine | `I128` | `U128` | `I256`, `U256` | +| --- | --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)` | +| DuckDB | `HUGEINT` | `UHUGEINT` | Error | +| DataFusion | Error | Error | Error | +| Python | Python `int` scalars | Python `int` scalars | Python `int` scalars | -## FixedSizeBinary on its own track +## FixedSizeBinary -FixedSizeBinary is a core type for opaque bytes of a fixed length, as RFC 0062 proposes. It refines +FixedSizeBinary is a core type for opaque bytes of a fixed length (decided, D6). It refines `Binary` in the same way that `FixedSizeList` refines `List`. - **Layout.** One buffer of `len * n` bytes, validity for whole values, and an explicit length so @@ -212,68 +256,75 @@ FixedSizeBinary is a core type for opaque bytes of a fixed length, as RFC 0062 p casts to and from `Binary` and `FixedSizeList`. No arithmetic and no integer semantics. - **UUID.** `Uuid` storage becomes `FixedSizeBinary(16)` in memory. Files keep the current `FixedSizeList` storage until an edition allows the new form. A 128-bit lane can implement UUID - equality and hashing, but UUID order needs a separate decision. + equality and hashing. Whether `Uuid` orders by its bytes is open (XQ6), with no recommendation + yet. + +Recommendation (DQ7): FixedSizeBinary appears on the wire as a new core dtype tag behind an +edition. `Decimal`, wide integers and FixedSizeBinary share only the fixed-width structural layer. That layer is the right place for shared code. PR [#8864](https://github.com/vortex-data/vortex/pull/8864) started it. -## The file format does not change +## The file format -The wrapper reads every existing decimal wire form: +The leaf is the `vortex.decimal` wire node, so the decimal file format does not change. Readers +handle every existing decimal wire form: | Wire node | In memory | | --- | --- | -| `vortex.decimal` at the canonical width | A wrapper over the buffer, zero-copy | -| `vortex.decimal` at a narrower width | A wrapper over a lazy widening, zero-copy | -| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at low precision | A wrapper over a lazy checked narrowing | +| `vortex.decimal` at the canonical width | The leaf, zero-copy | +| `vortex.decimal` at a narrower width of up to 64 bits | DecimalByteParts with the buffer as its MSP, zero-copy | +| `vortex.decimal` at `I128` for precision 39 to 76 | The leaf, after a widening copy | +| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at low precision | The leaf, after a checked narrowing copy | | `vortex.decimal` with an unknown width | An error, not a silent `I8` | -| DecimalByteParts v1 and v2 | DecimalByteParts at first, later a split of the wide child | +| DecimalByteParts v1 and v2 | DecimalByteParts. An old narrow MSP becomes an integer `Narrow` | + +Writers produce forms that old readers understand. The leaf becomes `vortex.decimal` at the +canonical width. DBP becomes DecimalByteParts v1, which the first core edition already contains, +or v2 where an edition allows it. Decimal `Narrow` is written only where an edition allows its wire +ID. Constant, dictionary and run-end arrays of decimals keep their current wire forms. -Writes produce forms that old readers understand. A flat child becomes `vortex.decimal`. A -compressed child with a lane-integer type becomes DecimalByteParts v1, which the first core edition -already contains. A constant child becomes a `vortex.constant` node with a decimal scalar, which is -the form that the compressor writes today. A dictionary child becomes a dictionary of -`vortex.decimal` values. So wide `PType` values never reach the wire for decimal columns, and -decimal statistics keep the `Decimal` dtype. +So wide `PType` values never reach the wire for decimal columns, and decimal statistics keep the +`Decimal` dtype. Before the reader changes, the compatibility fixtures need narrow, wide and +Arrow-style widths. Today they cover only `i32` and `i64`. -Before the reader changes, the compatibility fixtures need narrow, wide and Arrow-style widths. -Today they cover only `i32` and `i64`. +## `Decimal` stays built in -## Does `Decimal` become an extension type? +`Decimal` stays a built-in type (decided, D1). It behaves like a hardcoded extension type: -RFC 0062 proposes `Decimal` as a well-known extension type over signed-integer storage, with the -old wire tag kept. The wrapper above is the array side of that proposal. Only the location of the -Rust type is still open. +- It is opaque by default. Integer semantics apply only where a rule of the type says so. +- It derives `eq` and `ord` for one precision and scale. With one width for each precision, the + integers on the view then give value equality and value order. +- Its other operations are rewrite rules that the type owns. -The move is not safe today, because extension types inherit storage semantics. If `Decimal` -becomes an extension type today, a cast from `Decimal(10, 2)` to `i64` returns the unscaled -integers, and `price < 5` compares `500` with `5`. So the move waits for the extension fixes, the -cast hooks and the derives. After the decimal rewrites land, the move changes the API, not the -behavior. About 270 lines in 97 files mention `DType::Decimal`. +A move to an extension type adds no capability. If the built-in type and an extension type become +identical, a later change can remove the built-in type. That step is optional. -The proposal builds the wrapper with `DType::Decimal` first, and it uses `Decimal` as an -acceptance test for the extension hooks. The decision on the move comes after the rewrites land. +RFC 0062 proposes `Decimal` as a well-known extension type. If the optional move happens, it waits +for the extension fixes. Today, as an extension type, a cast from `Decimal(10, 2)` to `i64` returns +the unscaled integers, and `price < 5` compares `500` with `5`. ## Open questions -- Open question: are `I128` and `I256` user-visible columns from the start, or storage only until - an engine needs them? DuckDB `HUGEINT` is the only current consumer. -- Open question: is the wrapper on top of generic encodings such as `Constant` and `Dict`? The - extension notes recommend it, but no maintainer decision exists. Today `ExtensionConstantRule` - puts `Constant` above the extension wrapper, and the compressor writes a constant decimal column - as a `vortex.constant` node with a decimal scalar. -- Open question: does Vortex compare and compute across different precisions and scales? The - alternative keeps the rule of equal dtypes, and engines coerce first. -- Open question: does FixedSizeBinary get a new core wire tag, or an extension descriptor over - `FixedSizeList` that old readers can read? -- Open question: are the lazy widening and narrowing nodes cast arrays, or a small dedicated - encoding? -- Open question: do file reads check the decimal precision, or only imports? Zone-map minimum and - maximum statistics can make the check on file reads cheap. -- Open question: what is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does - not exist, and tests use -128. DuckDB rejects negative scales, and Arrow allows them. -- Open question: does the canonical width keep its `I8` minimum, or match an engine? The choice - affects only memory, so it can change later. -- Open question: after the rewrites land, is the move of `Decimal` into a Rust extension type worth - its API cost? +The decisions file lists these questions as open. Each answer is a recommendation of these notes. + +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ1 | Which width does each precision get? | The smallest signed width, as above. A wider function, such as the `I128` minimum of the Arrow export, avoids copies at Arrow import but costs memory | +| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes | +| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless it lands before the edition of PR #10333 ships | +| DQ4 | Does the dtype of a DBP array fix the type of its MSP? | Yes | +| DQ5 | Are wide integers user-visible, and which kernels come first? | User-visible behind an edition, with a small kernel set first | +| DQ6 | Does decimal arithmetic accept different precisions and scales? | Yes, with the Arrow rules. Comparisons keep the rule of equal dtypes | +| DQ7 | How does FixedSizeBinary appear on the wire? | A new core dtype tag behind an edition | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error | + +These notes add three smaller open questions: + +- What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist, + and tests use -128. DuckDB rejects negative scales, and Arrow allows them. +- When DBP decodes an old narrow MSP, does it compute value bounds and return a decimal `Narrow`, + or does it always widen? +- Do file reads check the decimal precision, or trust the file? Recommendation XQ3 for extension + values suggests trust, with checks only for what memory safety needs. diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 5aa8c003489..8d138db7a7b 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -9,26 +9,28 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors Many planned type changes depend on each other. The IDs below match the full graph in section 7 of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. +The decision IDs (D1 to D6, X1 to X9, C1) and the open questions (DQ1 to DQ7, XQ1 to XQ6, CQ1) +come from [`decisions.md`](../decisions.md). | ID | Work | Kind | | --- | --- | --- | | N1 | Extension opacity fixes ([part 4](../extension-types/04-urgent-changes.md) of the extension notes) | Urgent | -| N2 | Bind-time rewrites for extension types | Research, then implementation | -| N3 | Derives for one dtype | Implementation | -| N4 | Cast hooks for extension types ([#8570](https://github.com/vortex-data/vortex/issues/8570)) | Implementation | -| N5 | Array validation where data enters | Implementation | +| N2 | Rewrite rules that the type owns, registered with the type (X4) | Research, then implementation | +| N3 | Derives for one dtype (X2) | Implementation | +| N4 | Cast rules for pairs of dtypes (C1), with cast hooks for extension types ([#8570](https://github.com/vortex-data/vortex/issues/8570)) | Implementation | +| N5 | Value checks in `try_new`, with `new_unchecked` for kernels (X7) | Implementation | | N6 | Nullability on the extension dtype | Research, optional | | N7 | Shared fixed-width structural kernels | Implementation | | N8 | Scaled-integer kernels: `rescale` and `check_range` | Implementation | -| N9 | Wide integers: `I128` and `I256` | Implementation | +| N9 | Wide integers: `I128`, `I256`, `U128` and `U256` (D5) | Implementation | | N10 | Wire safety: strict enum decoding, and edition gates for core dtypes | Implementation | -| N11 | Decimal storage at least as wide as the canonical width | Urgent | -| N12 | Decimal wrapper over an integer child | Implementation | -| N13 | Decimal operations as integer rewrites | Implementation | -| N14 | DecimalByteParts as a split of wide integers | Implementation, optional | -| N15 | `Decimal` as a well-known extension type in Rust | Implementation, later | +| N11 | One canonical width for each decimal dtype (D3) | Urgent | +| N12 | Zero-copy integer view of the decimal leaf | Implementation | +| N13 | Decimal operations as rules over the view | Implementation | +| N14 | Limb split for wide-integer columns | Implementation, optional | +| N15 | `Decimal` as an extension type (D1 keeps it built in) | Optional, later | | N16 | Wire translation for well-known types | Research, then implementation | -| N17 | FixedSizeBinary core type | Implementation | +| N17 | FixedSizeBinary core type (D6) | Implementation | | N18 | `Uuid` on FixedSizeBinary | Implementation | | N19 | `Utf8` as a well-known extension type | Research, low priority | | N20 | `FixedSizeList` and `Map` as well-known extension types | Research, low priority | @@ -37,30 +39,32 @@ of [`agents/report.md`](agents/report.md), which gives a reason for each edge. T | N23 | Engine type plugins for DuckDB and DataFusion | Implementation | | N24 | Refinement types (RFC 0055) | Research | | N26 | Scalar value model (RFC 0056) | Research, then implementation | -| N27 | DuckDB `HUGEINT` import and export | Implementation | +| N27 | DuckDB `HUGEINT` and `UHUGEINT` import and export | Implementation | +| N28 | One `Narrow` encoding for integers and decimals (D4) | In progress | ## Graph -A solid line is a hard dependency: without the earlier work, the later work is wrong or unsafe. A -dashed line is a soft dependency: the earlier work makes the later work cheaper. This graph shows -all hard dependencies and the three most useful soft ones. The report lists every edge. +A solid line is a hard dependency: the later work must not land before the earlier work. A dashed +line is a soft dependency: the earlier work makes the later work cheaper. This graph shows all hard +dependencies and the five most useful soft ones. The report lists every edge. ```mermaid flowchart TD subgraph EXT["Extension types"] N1["N1 Opacity fixes"] - N2["N2 Bind-time rewrites"] + N2["N2 Rules owned by the type"] N3["N3 Derives for one dtype"] - N4["N4 Cast hooks"] + N4["N4 Cast rules"] end subgraph DEC["Decimal"] - N11["N11 Storage at least canonical width"] + N28["N28 One Narrow encoding"] + N11["N11 One canonical width"] + N12["N12 Integer view"] N7["N7 Fixed-width kernels"] N8["N8 Scaled-integer kernels"] N9["N9 Wide integers"] - N12["N12 Wrapper over integer child"] - N13["N13 Operations as rewrites"] - N14["N14 Wide-integer split"] + N13["N13 Rules over the view"] + N14["N14 Limb split, optional"] end subgraph BYTES["Bytes"] N17["N17 FixedSizeBinary"] @@ -69,10 +73,10 @@ flowchart TD subgraph WIRE["Wire and engines"] N10["N10 Wire safety"] N16["N16 Wire translation"] - N27["N27 DuckDB HUGEINT"] + N27["N27 DuckDB HUGEINT and UHUGEINT"] end - subgraph LATER["Later"] - N15["N15 Decimal as extension type"] + subgraph LATER["Later or optional"] + N15["N15 Decimal as extension type, optional"] N19["N19 Utf8 as extension type"] N20["N20 FixedSizeList and Map as extension types"] N22["N22 Execution layouts"] @@ -82,12 +86,11 @@ flowchart TD N2 --> N3 N1 --> N21 N4 --> N21 - N11 --> N12 + N28 --> N11 N9 --> N12 N12 --> N13 N8 --> N13 N9 --> N14 - N12 --> N14 N9 --> N27 N10 --> N27 N17 --> N18 @@ -100,6 +103,8 @@ flowchart TD N16 --> N20 N22 --> N19 N22 --> N20 + N11 -.-> N12 + N28 -.-> N13 N7 -.-> N9 N7 -.-> N17 N8 -.-> N4 @@ -107,6 +112,10 @@ flowchart TD The nodes N5, N6, N23, N24 and N26 have only soft edges, so the graph leaves them out. +Two edges need a note. N9 to N12 applies only to precision 19 to 76. The view for precision 1 to +18 uses existing types, so it can land at any point. N11 does not depend on N9, because the leaf +uses `DecimalType`, which already has 128-bit and 256-bit widths. + ## Urgent changes These changes are small and independent. They are safe to make now: @@ -114,49 +123,76 @@ These changes are small and independent. They are safe to make now: 1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a silent `I8` or `U8`. Readers must have this change before any writer emits a new value. -2. **Decimal storage at least as wide as the canonical width (N11).** `DecimalData::validate` - rejects narrower storage. DecimalByteParts assembly widens a narrow part, the compressor stops - writing narrowed `vortex.decimal` nodes, and readers widen old narrow nodes. Then the `patch` - panic and the "does the constant fit" special cases are unreachable for inputs with the same - dtype. This is option 3 of issue [#5820](https://github.com/vortex-data/vortex/issues/5820). - The compatibility fixtures for narrow, wide and Arrow-style widths come first. -3. **One owner for the width function.** `DecimalDType::storage_type()` replaces the 25 direct - calls to `smallest_decimal_value_type`. +2. **The DecimalByteParts nullability fix.** The cast rule of DecimalByteParts declines a + nullability change that the validity cannot make trivially. This is the first reusable part of + PR [#10230](https://github.com/vortex-data/vortex/pull/10230). +3. **One owner for the width function.** `DecimalDType::values_type()` replaces the 25 direct + calls to `smallest_decimal_value_type`, and producers take the width from the precision. This + is the first step of N11. 4. **Extension opacity fixes (N1).** These are the urgent changes in [part 4 of the extension notes](../extension-types/04-urgent-changes.md). +5. **The integer `Narrow` PRs.** PRs [#10223](https://github.com/vortex-data/vortex/pull/10223) + and [#10332](https://github.com/vortex-data/vortex/pull/10332) can merge as they are. Their only + integer-only check is the private `validate_dtypes`, so decimal support stays additive. + +## One canonical form + +N11 fixes the crashes of issue [#5820](https://github.com/vortex-data/vortex/issues/5820). The +decisions file splits the reusable parts of PR #10230 into small PRs. After the urgent changes, +the rest of N11 follows that order: + +1. Decide DQ3, then merge PR [#10333](https://github.com/vortex-data/vortex/pull/10333). It adds + `vortex.narrow` to the draft edition `preview2026.10.0`. +2. Add `Narrow` over decimals (N28). When `validate_dtypes` opens for decimals, the compare kernel + and the sum pushdown need decimal code in the same change. +3. Make the compressor and DecimalByteParts decoding stop narrowing and return `Narrow` instead. +4. Make readers convert old buffers at other widths. Old narrow leaves of up to 64 bits become + DecimalByteParts arrays without a copy, with an integer `Narrow` as the most significant part. + An old I128 leaf at precision 39 to 76 needs a widening copy. The compatibility fixtures for + narrow, wide and Arrow-style widths come first. +5. Make `DecimalArray::try_new` accept only the canonical width. + +Then the `patch` panic and the "does the constant fit" special cases are unreachable. Decimal +kernels can move onto integer kernels through the view (N12) at any point. ## Order of work | Wave | Work | | --- | --- | -| 0, urgent | Strict enum decoding, N11, one owner for the width function, and N1 | -| 1, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast hooks, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rewrites as acceptance tests. N23 engine plugins | -| 2, new types | N9 wide integers, without user-visible wide columns. N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary. N5 array validation | -| 3, decimal restructure | N12 wrapper and wire plugin. N13 rewrites, which delete most decimal kernels. N14, which is optional. N18 and N27 | -| 4, API unification | N16, then N15. N19 and N20 after N22 | +| 0, urgent | Strict enum decoding, the DecimalByteParts nullability fix, one owner for the width function, N1, and PRs #10223 and #10332 | +| 1, one canonical form | DQ3 and PR #10333. N28 for decimals. The rest of N11. N12 for precision 1 to 18 | +| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins | +| 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation | +| 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 | +| 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 | -The array-level fix for `Decimal` (N11, N12 and N13) does not wait for the extension work, because -the built-in wrapper owns its rewrites. Only the move to an extension type (N15) waits for N1 to N4. +Waves 1 and 2 can run in parallel. The decimal fix (N28, N11, N12 and N13) does not wait for the +extension work, because the built-in type owns its rules. Only the optional move to an extension +type (N15) waits for N1 to N4. ## Shared work between the tracks - The scaled-integer kernels (N8) serve decimal rescales and the unit casts of `Timestamp`, `Time` and `Duration`. -- The decimal rewrites are acceptance tests for the extension binder (N2 and N3). Phase 1 of the - extension notes adds a decimal-like extension type over `i64`, and that type can run the same - tests against the built-in `Decimal`. +- The decimal rules are acceptance tests for the rules and derives of extension types (N2 and + N3). Phase 1 of the extension notes adds a decimal-like extension type over `i64`, and that type + can run the same tests against the built-in `Decimal`. - The fixed-width structural kernels (N7) serve wide integers and FixedSizeBinary. -- The `StorageWrapper` rules serve `Decimal` and every extension type. So does the open decision - on the position of the wrapper relative to generic encodings. `DateTimeParts` under the - `Timestamp` wrapper is the temporal twin of DecimalByteParts under the `Decimal` wrapper. +- One `Narrow` encoding (N28, recommendation DQ2) serves integers and decimals. +- The placement rule (X5, recommendation XQ1) serves `Decimal` and every extension type. + Structural encodings sit above, and storage encodings sit below. `DateTimeParts` under the + `Timestamp` type is the temporal twin of DecimalByteParts under `Decimal`. ## Risks | Risk | Mitigation | | --- | --- | | The `PType` migration touches 36 crates and several languages | Typed sub-enums, compiler-driven steps, and one crate in each PR | -| A leftover `other => panic!` arm panics on a wide type | Retire those macros, and fuzz decimals at every width | +| A leftover `other => panic!` arm panics on a wide type | Retire those macros, and fuzz decimals and wide integers at every width | | An old reader reads a protobuf wide `PType` as `U8` | A new protobuf `oneof` variant, and strict readers early | | Rewritten arithmetic makes several passes over the data | Fused scaled-integer kernels, checked with the existing decimal and compare benchmarks | -| The public API breaks: `DecimalArray::buffer::()`, `values_type()` | Deprecation shims. The in-memory ID stays `vortex.decimal`, which the Python bindings use | +| D3 without `Narrow` forces the full width on every column | `Narrow` for decimals lands before `try_new` rejects other widths | +| A narrow buffer does not give a precision | The compressor uses statistics. Old narrow leaves become DecimalByteParts arrays without a scan | +| Arrow import at another width copies the buffer | Accept the copy, or choose another width function in DQ1 | +| The public API breaks: `DecimalArray::new::` with a `T` other than the canonical type | Constructors that convert, and errors that name the canonical width. The in-memory ID stays `vortex.decimal`, which the Python bindings use | | Wide integers reach users before engines can export them | Edition gates, and an explicit export rule for each engine | diff --git a/research/decimal/README.md b/research/decimal/README.md index b2865b931c3..a69d56de7d6 100644 --- a/research/decimal/README.md +++ b/research/decimal/README.md @@ -2,29 +2,35 @@ `Decimal` is a built-in dtype with the shape of an extension type: an integer storage value plus a scale. On `develop`, the decimal array does not keep to that shape. One decimal dtype can have six -physical forms, and `Decimal` duplicates much of `Primitive`. These notes propose an opaque -`Decimal` wrapper over an integer child, and they put the type work that it needs in order. +physical forms, and `Decimal` duplicates much of `Primitive`. These notes describe the target +design that the maintainer settled on 2026-10-19, and they put the type work that it needs in +order. [`decisions.md`](../decisions.md) records the decisions and the open questions. | Part | Content | | --- | --- | | [1. Current state](01-current-state.md) | How `Decimal` works on `develop`, and what is wrong | -| [2. Design](02-design.md) | The wrapper, the child type, wide integers, FixedSizeBinary, and the open questions | +| [2. Design](02-design.md) | The leaf, the integer view, `Narrow`, wide integers, FixedSizeBinary, and the open questions | | [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the urgent changes, and the order of work | ## Summary - The width of the decimal buffer does not depend on the precision. So one dtype has up to six physical forms, every kernel must handle mixed widths, and some paths panic. -- Proposal: a decimal array is a wrapper with one child, the unscaled integers. The child dtype is - a function of the precision. A narrower physical width becomes an encoding of the child. -- The child is an integer array, not a FixedSizeBinary array. Each decimal operation is an integer - operation plus bookkeeping for the scale, and byte order is not integer order. -- Integer storage for precision 19 to 76 needs two new primitive types, `I128` and `I256`. They do - not need FixedSizeBinary. -- FixedSizeBinary is still a good addition, on its own track, for opaque bytes such as UUIDs. -- The decimal file format does not change. The wrapper reads and writes the existing wire forms. -- `Decimal` stays a built-in type. A move into a well-known extension type in Rust is possible - after the extension fixes land, and then it changes only the API. +- Decided: a decimal array stays a leaf with one buffer and validity. Each decimal dtype has + exactly one canonical encoding, because the precision fixes the width of the buffer. +- Decided: a `Narrow` encoding stores a column as a child with a lower precision and the same + scale. So narrow storage stays possible. These notes recommend one `Narrow` encoding for + integers and decimals. +- Decimal kernels delegate to integer kernels through a zero-copy integer view of the buffer. Each + decimal operation is an integer operation plus bookkeeping for the scale. +- The view needs the new primitive types `I128` and `I256` for precision 19 to 76. The leaf does + not need them, so the fix for one canonical form can come first. +- Decided: Vortex adds `I128`, `I256`, `U128` and `U256`. None of them needs FixedSizeBinary. +- Decided: FixedSizeBinary becomes a core type for opaque bytes, such as UUIDs and hashes. +- The decimal file format does not change. Only decimal `Narrow` needs a new wire ID, behind an + edition. +- Decided: `Decimal` stays a built-in type that behaves like a hardcoded extension type. A move to + an extension type is an optional later step. ## Agent documents @@ -39,7 +45,8 @@ physical forms, and `Decimal` duplicates much of `Primitive`. These notes propos ## Evidence -The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. One -throwaway Rust test confirmed the `f32` cast disagreement (probe R1 in the extension notes). One -short Python script checked byte order. Section 10 of the report lists the claims that are not -verified, which include every performance claim. +The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. On +2026-10-19 the review also read the draft `Narrow` PRs and PR #10230. One throwaway Rust test +confirmed the `f32` cast disagreement and the `ext_storage` pushdown rules (probes R1 and R6 in the +extension notes). One short Python script checked byte order. Section 10 of the report lists the +claims that are not verified, which include every performance claim. diff --git a/research/decimal/agents/decimal-call-sites.md b/research/decimal/agents/decimal-call-sites.md index fd1cf9a40ec..0fab6f128aa 100644 --- a/research/decimal/agents/decimal-call-sites.md +++ b/research/decimal/agents/decimal-call-sites.md @@ -11,10 +11,10 @@ and files named `tests.rs` count as tests. ## 1. Decimal-specific files -The first column is non-test lines, the second is test lines. "Fate under the wrapper design" uses -these labels: +The first column is non-test lines, the second is test lines. "Fate under the leaf design" uses +these labels. The leaf design is decision D2 and D3 in `../../decisions.md`. -- **delete**: the logic becomes a rewrite to integer kernels or a shared wrapper rule. +- **delete**: the logic becomes a call to an integer kernel through the integer view. - **shrink**: a smaller decimal-specific piece remains (dtype rules, rescaling, legacy wire). - **keep**: the logic stays, possibly moved. - **generalize**: the logic becomes a generic wide-integer component. @@ -85,29 +85,29 @@ these labels: 7637 5915 TOTAL (code, tests) ``` -| File or group | Role today | Fate under the wrapper design | +| File or group | Role today | Fate under the leaf design | | --- | --- | --- | -| `vortex-array/src/arrays/decimal/array.rs` | `DecimalData { decimal_dtype, values, values_type }`, constructors, `patch` | shrink: wrapper over one child | -| `vortex-array/src/arrays/decimal/vtable/*` | `vortex.decimal` vtable, `DecimalMetadata { values_type }` serde | shrink: wrapper vtable plus legacy plugin | +| `vortex-array/src/arrays/decimal/array.rs` | `DecimalData { decimal_dtype, values, values_type }`, constructors, `patch` | shrink: one buffer at the width that the precision fixes | +| `vortex-array/src/arrays/decimal/vtable/*` | `vortex.decimal` vtable, `DecimalMetadata { values_type }` serde | shrink: a width check, and readers that convert old widths | | `vortex-array/src/arrays/decimal/compute/cast.rs` | decimal to decimal, integer, and f64 casts, plus `upcast_decimal_values` | shrink: rescale rules over integer casts | -| `vortex-array/src/arrays/decimal/compute/{between,fill_null,mask,rules,fixed_width}.rs` | per-width kernels and storage-fit special cases | delete: wrapper rules plus integer kernels | +| `vortex-array/src/arrays/decimal/compute/{between,fill_null,mask,rules,fixed_width}.rs` | per-width kernels and storage-fit special cases | delete: integer kernels through the integer view | | `vortex-array/src/arrays/decimal/utils.rs` | `widened_buffer`, `converted_buffer`, `narrowed_decimal` | delete: no mixed widths in canonical form | | `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs` | same-dtype compare with width widening | delete: storage compare | | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs` | checked arithmetic at a working width | shrink: rescale and precision check over integer ops | | `vortex-array/src/aggregate_fn/fns/*/decimal.rs` | sum, min/max, is_sorted, is_constant, all_non_distinct, size | delete: storage aggregates plus result wrapping | -| `vortex-array/src/builders/decimal.rs` | `DecimalBuilder` with six buffer variants | shrink: wrapper over `PrimitiveBuilder` | +| `vortex-array/src/builders/decimal.rs` | `DecimalBuilder` with six buffer variants | shrink: one width for each dtype, over `PrimitiveBuilder` | | `vortex-array/src/dtype/decimal/*` | `DecimalDType`, `DecimalType`, `NativeDecimalType`, i256 bound tables | keep `DecimalDType`, and keep `DecimalType` only for legacy metadata | | `vortex-array/src/scalar/typed_view/decimal/*` | `DecimalValue`, `DecimalScalar`, Arrow arithmetic rules | shrink: scalar view over a storage integer | -| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | generalize: wide-integer limb split under the wrapper | -| `vortex-arrow/src/executor/decimal.rs` | width adaptation to Decimal32/64/128/256 | keep, reads the child | -| `vortex-duckdb/src/exporter/decimal.rs` | width adaptation to DuckDB widths | keep, reads the child | -| `vortex-btrblocks/src/schemes/decimal.rs` | narrow then split | generalize: wide-integer scheme for the child | +| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | keep as the compressed form. Its dtype fixes the type of its MSP (DQ4) | +| `vortex-arrow/src/executor/decimal.rs` | width adaptation to Decimal32/64/128/256 | keep, reads the one canonical width | +| `vortex-duckdb/src/exporter/decimal.rs` | width adaptation to DuckDB widths | keep, reads the one canonical width | +| `vortex-btrblocks/src/schemes/decimal.rs` | narrow then split | shrink: no narrowing, which moves to `Narrow` and integer encodings | | `vortex-cuda/src/kernel/*decimal*` | device filter and DBP | keep or generalize to wide-integer lanes | ## 2. Uses of `DecimalType::smallest_decimal_value_type` This function is the de facto canonical width. Every site below already treats the width as a -function of precision. The wrapper design makes that rule the contract instead of a convention. +function of precision. The leaf design makes that rule the contract instead of a convention. The first `types.rs` entry is the definition, so the list holds 25 call sites. ```text @@ -168,8 +168,8 @@ encodings/sparse/src/canonical.rs:316 ## 5. Canonical dispatch sites that name `Canonical::Decimal` (non-test) -Each site needs a decision when the decimal canonical form changes. With a wrapper, most of -them reduce to "push into the child" through a shared wrapper rule. +Each site needs a decision when the decimal canonical form changes. With one width for each +dtype, most of them reduce to a call on the integer view. ```text vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:38 diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index 95a1e841ad2..92d3d9c6fae 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -9,6 +9,10 @@ This file is the detailed report behind the summary in `../README.md`. It is res accepted design, and it changes no code. Paths and line numbers are for `develop` at `5fc5cb8` (2026-10-17), the same commit as the extension-type notes in `../../extension-types/`. +The maintainer reviewed these notes on 2026-10-19. [`decisions.md`](../../decisions.md) records the +decisions (D1 to D6, X1 to X9, C1) and the open questions (DQ1 to DQ7, XQ1 to XQ6, CQ1, CQ2). This +report follows that file. If the two disagree, the decisions file wins. + Audience: Vortex maintainers and coding agents who implement the type-system work. Companion files in this directory: @@ -21,11 +25,16 @@ Companion files in this directory: Conventions: - Paths are relative to the repository root. A `file:line` reference points at the commit above. -- "Storage" means the integer child of a decimal, or the storage child of an extension array. -- "Canonical width" means `DecimalType::smallest_decimal_value_type` - (`vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, +- "Settled" marks a decision of the maintainer in the decisions file. "Recommendation" marks a + proposal of these notes that the maintainer has not confirmed. +- "Canonical width" means the buffer width that the precision fixes (D3). These notes recommend + the existing function `DecimalType::smallest_decimal_value_type` for it (DQ1, + `vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, 5-9 to I32, 10-18 to I64, 19-38 to I128, and 39-76 to I256. -- "Lane integer" means U8 to U64 and I8 to I64. "Wide integer" means I128 and I256. +- "Integer view" means a `PrimitiveArray` over the buffer and the validity of a decimal array, + without a copy (section 3.2). +- "Storage" means the storage child of an extension array. +- "Lane integer" means U8 to U64 and I8 to I64. "Wide integer" means I128, I256, U128 and U256. - Issue and pull request numbers refer to `vortex-data/vortex` on GitHub. Section 10 links each one. - Two claims come from running Rust code, with probes R1 and R6 in @@ -60,25 +69,33 @@ dtype therefore has up to six physical forms. That single fact explains most of ### 1.2 Recommendation in short -1. Make the decimal array a zero-data wrapper with exactly one child, the unscaled integers. The - child dtype is a pure function of the precision: `Primitive(storage_ptype(p), nullability)`. - Narrow physical storage becomes an encoding of the child, never a property of the canonical - form. -2. Use an integer child, not a FixedSizeBinary child. Add `PType::I128` and `PType::I256` as a - distinct "wide integer" tier. The tier opts into a small kernel set: compare, arithmetic, cast, - sum, min and max, hash, and structural operations. Do not add `U128` or `U256` until a - consumer needs them. Section 4 gives the full comparison. -3. Keep every decimal wire format. The new in-memory wrapper reads the legacy `vortex.decimal` - and DecimalByteParts nodes and writes them back. No new array id and no new dtype tag is - needed for decimals. Wide PTypes never reach the wire for decimal columns. -4. Express decimal semantics as rewrites onto integer kernels plus one shared "scaled integer" - kernel family. The same family serves temporal unit conversions (Timestamp, Duration, Time). -5. Treat FixedSizeBinary as an independent type for opaque bytes (UUID, hashes). It shares the - fixed-width structural layer with Primitive, but it never carries integer semantics. Decimal - does not depend on it, and wide integers do not depend on it. -6. Move Decimal to a well-known extension in the Rust API (RFC 0062) only after the extension work - lands. That work is the opacity fixes, the cast hooks, and derives for one dtype. The - array-level wrapper design is the same either way, so the early work carries over. +Each point names the settled decision or the open question behind it. + +1. Keep `DecimalArray` a leaf: one values buffer and validity, and no data child (D2). Give each + decimal dtype exactly one canonical encoding, so the precision fixes the buffer width (D3). + Recommendation (DQ1): use the existing smallest-width mapping. +2. Give decimal kernels a zero-copy integer view: a `PrimitiveArray` over the same buffer and + validity. Decimal kernels delegate to integer kernels through the view. Rewrite rules that the + type owns add the bookkeeping for the scale (X4). +3. Store narrow columns with a `Narrow` encoding (D4). A `Decimal(p, s)` column holds one child of + dtype `Decimal(p', s)` with `p' < p`. Recommendation (DQ2): one `Narrow` encoding for integers + and decimals. +4. Keep DecimalByteParts (DBP) as the compressed decimal form. Recommendation (DQ4): the dtype of a + DBP array fixes the type of its most significant part. Then `Narrow` narrows the precision, and + integer encodings such as bit packing narrow the bits. +5. Keep every decimal wire format. Generic structural encodings sit above the decimal storage + encodings (X5, recommendation XQ1). The wire forms of today already have this shape. Only + decimal `Narrow` needs a wire ID, behind an edition (DQ3). +6. Add `I128`, `I256`, `U128` and `U256` as primitive integer types (D5). Build them as a separate + wide-integer tier with typed dispatch. Recommendation (DQ5): user-visible behind an edition, + with a small kernel set first. +7. The integer view needs `I128` and `I256` for precision 19 to 76. The leaf does not need them, + because `DecimalType` already has both widths. So the one-canonical-form fix can land before + the `PType` migration. +8. Add FixedSizeBinary as a core type for opaque bytes (D6). Decimal and wide integers do not + depend on it. Recommendation (DQ7): a new core dtype tag behind an edition. +9. Keep `Decimal` a built-in type that behaves like a hardcoded extension type (D1). A move to an + extension type is an optional later step. Section 7 gives the dependency graph. Section 8 gives the recommended order of work. @@ -219,8 +236,8 @@ Paths in the table are relative to `vortex-array/src/`. Shared code also has dec 64-bit lower parts (`encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:6-15`). The MSP holds validity (`decimal_byte_parts/array.rs:404-409`). Each part is a normal integer array that any integer scheme can compress. -- DBP carries `DType::Decimal` itself (`decimal_byte_parts/array.rs:102-104`). It mixes a - logical type with a physical layout. +- DBP carries `DType::Decimal` itself (`decimal_byte_parts/array.rs:102-104`). So it is a decimal + storage encoding, not an integer encoding. - Two wire ids exist. `vortex.decimal_byte_parts` (v1, MSP only) is in the first core edition. `vortex.decimal_byte_parts.v2` (with lower parts) is in no edition yet (`decimal_byte_parts/plugin/mod.rs:30-40`, @@ -341,197 +358,236 @@ Four different width functions exist: 11. Decimal scalars on the wire carry the width of whatever variant the writer used, and readers do not check precision. 12. Sparse canonicalization supports decimals with a dedicated path but rejects all extension - arrays (`encodings/sparse/src/canonical.rs:361`). The wrapper idea is not applied anywhere - as a shared rule. + arrays (`encodings/sparse/src/canonical.rs:361`). No shared rule decides where structural + encodings sit relative to a type. X5 and XQ1 address this (section 3.3). -## 3. Target: Decimal as an opaque wrapper over an integer child +## 3. Target: one canonical decimal leaf ### 3.1 Definition The logical dtype does not change. `DType::Decimal(DecimalDType, Nullability)` keeps its Rust -variant in this phase and keeps its wire tag forever (section 3.8 covers the later move to a -well-known extension). +variant and its wire tag (D1, section 3.8). -The storage dtype becomes a pure function of the precision. It reuses the current canonical -width: +`DecimalArray` stays a leaf (D2). It holds one values buffer and validity, and it has no data +child. This is also the shape of the `vortex.decimal` wire node: one buffer, an optional validity +child, and no data child (`vortex-array/src/arrays/decimal/vtable/mod.rs:155-170`). + +The precision fixes the width of the buffer (D3). The width function gets one owner: ```rust impl DecimalDType { - /// The integer type that holds every unscaled value of this precision. - pub const fn storage_ptype(&self) -> PType { + /// The width of the values buffer for this precision. + /// + /// This is the existing `DecimalType::smallest_decimal_value_type` (recommendation DQ1). + pub const fn values_type(&self) -> DecimalType { match self.precision.get() { - 1..=2 => PType::I8, - 3..=4 => PType::I16, - 5..=9 => PType::I32, - 10..=18 => PType::I64, - 19..=38 => PType::I128, - _ => PType::I256, + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, } } +} - /// The dtype of the unscaled child for this precision and the given nullability. - pub fn storage_dtype(&self, nullability: Nullability) -> DType { - DType::Primitive(self.storage_ptype(), nullability) - } +impl Array { + /// Fails unless `T::DECIMAL_TYPE` equals `decimal_dtype.values_type()`. + pub fn try_new( + buffer: Buffer, + decimal_dtype: DecimalDType, + validity: Validity, + ) -> VortexResult; + + /// The unscaled integers over the same buffer and validity, without a copy (section 3.2). + pub fn unscaled(&self) -> PrimitiveArray; } ``` -The array becomes a zero-data wrapper with one slot: +`try_new` exists today with this signature, but it accepts any width +(`vortex-array/src/arrays/decimal/array.rs:433-443`). `DecimalType` already has `I128` and `I256` +(`vortex-array/src/dtype/decimal/types.rs:25-42`). So the leaf needs no new `PType`, and the +one-canonical-form fix (node N11) does not wait for the wide integers (node N9). -```rust -#[array_slots(Decimal)] -pub struct DecimalSlots { - /// The unscaled integers. The dtype is `decimal_dtype.storage_dtype(nullability)`. - #[slot(0)] - pub unscaled: ArrayRef, -} +Invariants: -impl VTable for Decimal { - type TypedArrayData = EmptyArrayData; - type OperationsVTable = Self; // scalar_at wraps the child scalar - type ValidityVTable = ValidityVTableFromChild; +- **I1, width.** The buffer width equals `decimal_dtype.values_type()`. `DecimalData::validate` + checks it next to the length and alignment checks of today + (`vortex-array/src/arrays/decimal/array.rs:301-319`). The check costs nothing for each row. +- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. The places where + data enters establish it. These are the Arrow, DuckDB, FFI, and Python imports, and the value + check in `try_new` (node N5, X7). Kernels that produce new values (arithmetic, rescaling casts, + sums) check their own outputs, as `numeric/decimal.rs` does today with `DecimalValueBounds`. +- **I3, leaf.** The array has one buffer and one validity slot + (`vortex-array/src/arrays/decimal/array.rs:46-51`). It has no data child. - fn id(&self) -> ArrayId { - // The in-memory id stays `vortex.decimal`. Section 3.5 covers the wire. - static ID: CachedId = CachedId::new("vortex.decimal"); - *ID - } +The value of a row is `unscaled * 10^(-scale)`. Nothing else is stored. - fn nbuffers(_array: ArrayView<'_, Self>) -> usize { - 0 - } +Recommendation DQ1 keeps the I8 minimum. Section 2.11 lists three other width functions in the +engines. A function with an I128 minimum below precision 39 matches the Arrow export. It keeps +Arrow `Decimal128` imports zero-copy, but it costs up to 16 times the memory at small precisions. +Any of these functions fits the existing wire form, because `vortex.decimal` records its width. - // validate: check that the child dtype equals the storage dtype and that lengths match. - // execute: return the wrapper unchanged. The child stays as it is. -} +### 3.2 Canonical form and the integer view -impl Array { - /// Wraps unscaled integers whose dtype is the storage dtype of `decimal_dtype`. - pub fn try_new(decimal_dtype: DecimalDType, unscaled: ArrayRef) -> VortexResult; +`Canonical::Decimal(DecimalArray)` stays a variant. With I1, one decimal dtype has one canonical +form. So a kernel never receives a width that it does not expect. The `patch` panic and the "does +the constant fit" special cases go away (section 2.13, items 1 and 3). - /// The unscaled integers. This is the only way to see the storage. - pub fn unscaled(&self) -> &ArrayRef; -} +Issue #5820 also proposed a decimal array with an integer child "that is separately decompressed, +like list offsets". D2 rejects that shape for the canonical form. The narrow representation lives +in a separate `Narrow` encoding instead (section 3.3). + +The integer view is a `PrimitiveArray` over the same buffer and validity. It costs no copy. Its +dtype is `Primitive(I_w, nullability)`, where `I_w` is the signed integer type of the canonical +width: + +```rust +// Zero-copy. For `Decimal(9, 2)`, the dtype of the view is `Primitive(I32, nullability)`. +let unscaled: PrimitiveArray = decimal.unscaled(); ``` -Invariants: +Decimal kernels delegate to integer kernels through the view. A result of an integer kernel +becomes a decimal again without a copy, when its buffer has the canonical width of the result +dtype. + +The view for precision 1 to 18 uses I8 to I64, which exist today. The view for precision 19 to 76 +needs I128 and I256 (D5, node N9). Until N9 lands, the kernels for those precisions stay +decimal-specific. Kernels can move onto the view one at a time, at any point in the order of work. + +For extension types, `ext_storage` is the explicit unwrap to the storage, and it stays. For +`Decimal`, the integer view plays the same role. -- **I1, type.** `unscaled.dtype() == decimal_dtype.storage_dtype(nullability)`. `validate` - checks this. It costs nothing at run time. -- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. Trust boundaries - establish it (Arrow, DuckDB, FFI, and Python imports, and the array `validate_array` hook of - node N5). Kernels that produce new values (arithmetic, rescaling casts, sums) check their own - outputs, as `numeric/decimal.rs` does today with `DecimalValueBounds`. -- **I3, no data.** The wrapper owns no buffer. Validity comes from the child, as for - `ExtensionArray` (`vortex-array/src/arrays/extension/vtable/mod.rs:85-89`) and DBP - (`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:404-409`). +### 3.3 Storage encodings and the placement rule -The value of a row is `unscaled * 10^(-scale)`. Nothing else is stored. +#### `Narrow` -### 3.2 Canonical form +A `Narrow` encoding stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)` with +`p' < p` (D4). The child has the same scale and the same nullability. `Narrow` has three +properties: -`Canonical::Decimal(DecimalArray)` remains a variant in this phase, but the wrapped child does -not have to be canonical. That matches the existing rule that canonical form is not recursive -(`vortex-array/src/canonical.rs:85-88`) and matches `ExtensionArray`, whose `execute` returns -itself without touching the storage (`vortex-array/src/arrays/extension/vtable/mod.rs:189-191`). -A kernel that needs flat integers executes the child explicitly: +- It has no buffers. Validity comes from the child. +- It decodes by widening the child buffer to the canonical width of `p`. +- It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. -```rust -let unscaled = decimal.unscaled().clone().execute::(ctx)?; -``` +Recommendation (DQ2): one `Narrow` encoding for integers and decimals. Integers narrow by width +(issue #10020), and decimals narrow by precision. The integer view of a decimal `Narrow` is then +an integer `Narrow` over the integer view of the child. So the integer `Narrow` kernels give +decimals their narrow-width speed. -This is the parent and child relation proposed in issue #5820 ("an integer array as a child that -is separately decompressed, like list offsets"). It also answers the objection in the same thread -that the two designs are equivalent. They are not equivalent. With a fixed child dtype, a kernel -never receives a width that it does not expect. Also, the decimal-specific kernels disappear -instead of being duplicated (section 3.4). +Precision and integer width do not line up. An `i32` holds values up to 2,147,483,647, which needs +precision 10. But 9 is the largest precision that maps to `i32`. So a narrow buffer alone does not +give `p'`, and a writer needs value bounds to choose it: -### 3.3 Shared wrapper machinery with extensions +- The compressor has them from statistics. +- A cast that widens the precision has them from the source dtype. It returns a `Narrow` over the + source without a copy. +- An old narrow buffer, read without a scan, does not have them. DBP holds it instead, as the next + part describes. -Decimal and extensions need the same structural behavior. Today each type implements it -separately, and some cases exist for only one of them. One trait can carry the shared part: +The integer `Narrow` is a draft stack today: PRs #10223, #10332 and #10333. Their only integer-only +check is the private `validate_dtypes`, so decimal support stays additive. When that check opens +for decimals, the compare kernel and the sum pushdown need decimal code in the same change. -```rust -/// A zero-data logical wrapper over one storage child. -/// -/// Structural operations push into the storage child and rewrap the result. Semantic -/// operations never read the storage unless a type-owned rewrite says so. -pub trait StorageWrapper: VTable { - /// The storage child. - fn storage(array: ArrayView<'_, Self>) -> &ArrayRef; - - /// The storage dtype implied by a wrapper dtype. It must be a pure function of the dtype. - fn storage_dtype(dtype: &DType) -> VortexResult; - - /// Rebuilds a wrapper with the same logical dtype around a new storage child. - fn rewrap(array: ArrayView<'_, Self>, storage: ArrayRef) -> VortexResult; - - /// Wraps a storage scalar into a logical scalar of `dtype`. - fn wrap_scalar(dtype: &DType, storage: Scalar) -> VortexResult; -} -``` +PR #10333 adds `vortex.narrow` to the draft edition `preview2026.10.0`, with one metadata byte that +holds the child `PType`. Recommendation (DQ3): decimal `Narrow` gets a second wire ID, unless it +lands before that edition ships in a release. A second wire ID needs no change to the open PRs. +The editions spec says that a change to what readers must understand needs a new wire ID +(`docs/specs/versioning/editions.md:118-121`). -`Extension` implements it with `ExtensionArray::new` and `Scalar::extension_ref`. `Decimal` -implements it with `DecimalArray::try_new` and a decimal scalar built from the storage value. -Generic rules then cover both: +#### DecimalByteParts -| Behavior | Extension today | Decimal today | Shared rule | -| --- | --- | --- | --- | -| slice, filter, take, mask | Pushdown rules (`arrays/extension/compute/rules.rs:48-79`) | Buffer kernels through `FixedWidthArray` | Push into the child and rewrap | -| Constant child | `ExtensionConstantRule` (`rules.rs:28-46`) turns `Ext(Constant(s))` into `Constant(ext s)` | `Constant(decimal scalar)` | Open. With the wrapper on top, `Constant(wrapper s)` becomes `Wrapper(Constant(storage s))` | -| scalar_at | Wraps the storage scalar | Reads the buffer at the stored width | `wrap_scalar(child.scalar_at(i))` | -| Validity | From the child | Validity slot | From the child | -| Compression | Compress storage and rewrap (`vortex-compressor/src/compressor/cascade.rs:175-205`) | `DecimalScheme` | Compress the child with ordinary schemes and rewrap | -| Sparse, run-end, dictionary, chunked | Sparse bails (`encodings/sparse/src/canonical.rs:361`) | Dedicated decimal paths | Open. With the wrapper on top, `Dict(codes, Wrapper(v))` becomes `Wrapper(Dict(codes, v))`, so the structural encoding sees only storage | - -With the wrapper on top, the last row also closes a gap for extensions: a sparse array of an -extension type cannot canonicalize today. - -The position of the wrapper is open (section 9, question 13). Section 6 of -`../../extension-types/agents/design-notes.md` compares both placements. It recommends the wrapper -on top, and no maintainer decision exists. The wrapper on top reverses `ExtensionConstantRule`. -The alternative keeps generic encodings above the wrapper, as the current code does for constants. -Then `Masked` and `Sparse` need a pushdown rule for `ext_storage`. Existing rules already push it -into dictionary values and chunks (probe R6 in the extension notes). - -On the wire, `vortex.decimal` is a leaf with one buffer. So a generic encoding of decimal data sits -above it. For example, the compressor writes a constant decimal column as a `vortex.constant` node -with a decimal scalar (`vortex-compressor/src/compressor/cascade.rs:272-273`, -`vortex-compressor/src/compressor/constant/mod.rs:80-86`). With the wrapper on top in memory, the -plugin writes these older shapes (section 3.5). - -Type-specific encodings move under the wrapper and see only the storage dtype. For decimals that -means DBP stops carrying `DType::Decimal`. It becomes an encoding of `Primitive(I128)` or -`Primitive(I256)` (node N14). The same rule applies to DateTimeParts under temporal wrappers, -which today carries the extension dtype (`encodings/datetime-parts/src/array.rs:294-305`). This -choice does not depend on the position of the wrapper relative to generic encodings. - -### 3.4 Decimal semantics as storage rewrites - -Every decimal operation is an integer operation plus scale bookkeeping. With a fixed storage -dtype, the rewrites below are exact. - -| Operation on `Decimal(p, s)` | Rewrite onto the unscaled child | Notes | +DBP stays the compressed decimal form. Its children are integers: a signed most significant part +(MSP), and unsigned 64-bit lower parts for wide values. The ordinary integer schemes compress each +part (section 2.8). + +Recommendation (DQ4): the dtype of a DBP array fixes the type of its MSP. These notes propose the +types that `split_decimal` produces for canonical input +(`encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:84-121`). For precision 1 to 18, +the MSP has the canonical width. For precision 19 to 76, the MSP is `i64`, with one lower part for +`i128` values and three for `i256` values. + +Narrowing then happens in only two places. `Narrow` narrows the precision, and integer encodings +such as bit packing narrow the bits of the MSP. So `narrowed_decimal` +(`vortex-array/src/arrays/decimal/utils.rs:124-143`) has no job left. Today the btrblocks +`DecimalScheme` calls it first (`vortex-btrblocks/src/schemes/decimal.rs:133`). + +Readers still accept the narrow MSPs of old files. The DBP read plugin wraps an old narrow MSP in +an integer `Narrow` of the fixed type, without a copy. So the in-memory MSP always has the type +that the dtype fixes. The integer `Narrow` of PR #10223 carries the old width. + +A reader reads an old narrow `vortex.decimal` leaf as DBP in the same way. This applies when the +leaf is narrower than the canonical width and has 64 bits or less. Its buffer becomes the child of +that integer `Narrow`. +Today `DecimalParts::from_narrow` wraps such a buffer directly as the MSP +(`encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:60-65`). + +Today DBP decoding with no lower parts builds a leaf at the MSP width +(`encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-93`). Under D3 it must decode +to the canonical width. Under DQ4 this needs no special case for old files, because the integer +`Narrow` widens the old MSP when it decodes. When the decoder knows value bounds, it returns a +decimal `Narrow` instead (step 4 in section 8.3). + +#### Placement of structural encodings + +Rule X5 pulls generic structural arrays out from under the extension wrapper. Recommendation XQ1 +names them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. So +generic structural encodings sit above the type layer. The decimal storage encodings, DBP and +`Narrow`, sit below them. + +For `Decimal`, the wire forms of today already follow this rule: + +- The compressor writes a constant decimal column as `vortex.constant` with a decimal scalar + (`vortex-compressor/src/compressor/cascade.rs:272-273`). When some rows are null, it writes + `vortex.masked` over that node (`vortex-compressor/src/compressor/constant/mod.rs:175-193`). +- Run-end arrays already decode decimal values (`encodings/runend/src/array.rs:488-494`). +- A dictionary over decimal values works (`vortex-array/src/arrays/dict/execute.rs:102-111`). +- Sparse and masked arrays have decimal paths (`encodings/sparse/src/canonical.rs:314-327`, + `vortex-array/src/arrays/masked/execute.rs:89-95`). + +So `Decimal` needs no write plugin that translates an in-memory form back to the legacy forms. An +earlier version of these notes needed one for a wrapper design. + +`DateTimeParts` is the temporal twin of DBP. It is a storage encoding below the structural +encodings, and it carries the extension dtype today +(`encodings/datetime-parts/src/array.rs:294-305`). + +Probe R6 in the extension notes checks the related `ext_storage` pushdown rules. Existing rules +push `ext_storage` into dictionary values and chunks. `ext_storage` decodes a `Masked` input and +fails on a `Sparse` input with extension values. + +### 3.4 Decimal semantics as rewrite rules + +`Decimal` implements its semantics as rewrite rules on arrays that the type owns (X4). Each rule +runs integer kernels on the integer view and builds a decimal result. There is no `ext_wrap` +expression, because a rule builds its result array directly. + +Three settled rules frame the table. Vortex runs an operation only when the operand dtypes already +match, and engines convert operands to a common dtype (X3). A cast exists only where a rule for the +pair of dtypes exists (C1). `Decimal` ships built-in rules for the common pairs below. + +| Operation on `Decimal(p, s)` | Rule over the integer view | Notes | | --- | --- | --- | -| Compare, same `(p, s)` | `compare(unscaled(a), unscaled(b))` | A derive for one dtype. Valid because the unscaled value is unique within one `(p, s)` | -| Compare across `(p1, s1)` and `(p2, s2)` | Rescale both to `s = max(s1, s2)` at a width that holds `max(p1 - s1, p2 - s2) + s` digits, then compare | A rewrite that the type owns, not a derive. Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | -| Compare with a constant that has more fractional digits than `s` | Fold the constant and adjust the operator. `x < 1.005` at `s = 2` becomes `unscaled <= 100` | Needs floor and ceiling rounding | -| Add, subtract | Widen to the result width, rescale to the larger scale, checked operation, precision check | Arrow rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`) | +| Compare, same `(p, s)` | `compare(view(a), view(b))` | A derive for one dtype (X2). Valid because the unscaled value is unique within one `(p, s)` after D3 | +| Compare across `(p1, s1)` and `(p2, s2)` | None in Vortex | The engine casts both operands to one dtype first (X3). Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | +| Compare a column cast to a larger scale with a constant | Fold the cast into the constant and adjust the operator. `cast(x, s = 3) < 1.005` at `s = 2` becomes `view(x) <= 100` | An optimization for engine casts. Needs floor and ceiling rounding | +| Add, subtract, same dtype | Widen to the result width, checked operation, precision check | Arrow rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`). Equal dtypes only today (`scalar_fn/fns/binary/mod.rs:146-153`) | +| Arithmetic across `(p, s)` | Open (DQ6). Recommendation: accept it, rescale to the larger scale, and use the Arrow rules for the result type | Comparisons keep the rule of equal dtypes | | Multiply | Widen, checked multiply, precision check | Result scale `s1 + s2` | | Divide | Multiply the dividend by `10^k`, checked divide toward zero, precision check | As today | -| Cast to `Decimal(p2, s2)` | Rescale with exactness, integer width cast, precision check | Replaces `DecimalCastPlan` | -| Cast to an integer | For a positive scale, `check_range` on the unscaled values with the bounds `MIN · 10^s` and `MAX · 10^s` of the target, then divide by `10^s` toward zero. For a negative scale, a checked multiply. Then narrow | Replaces `DecimalToIntegerCast` and keeps its rule: a value outside the range fails before truncation, so `127.1` fails for `i8` (`scalar/typed_view/decimal/integer_cast.rs:14-53`, test `cast_decimal_to_integer_bounds`) | -| Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays | +| Cast to `Decimal(p2, s2)` | Rescale with exactness, integer width cast, precision check | Replaces `DecimalCastPlan`. A wider precision with the same scale needs no copy. It relabels the buffer, or returns a `Narrow` over the source when the width grows | +| Cast to an integer | For a positive scale, `check_range` on the unscaled values with the inclusive bounds `MIN · 10^s` and `MAX · 10^s` of the target, then divide by `10^s` toward zero. A bound that does not fit the storage width saturates. For a negative scale, multiply by `10^-s` with a checked multiply at a width that holds the result. Then do a checked narrowing to the target. A zero value needs no factor, so it succeeds even when `10^-s` fits no integer width | Replaces `DecimalToIntegerCast` and keeps its rule: a value outside the range fails before truncation, so `127.1` fails for `i8` (`scalar/typed_view/decimal/integer_cast.rs:14-53`, test `cast_decimal_to_integer_bounds` in `arrays/decimal/compute/cast.rs:651`). The current code computes in `i256` and rejects a factor `10^s` that does not fit (`integer_cast.rs:37-39`). Under recommendation CQ1, truncation becomes a separate `trunc` step | +| Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays. CQ1 also decides whether this rounding cast stays a default cast | | Cast from an integer | Widen, multiply by `10^s`, precision check | Replaces `cast_to_decimal` | -| Sum | Integer sum with accumulator `storage_ptype(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | -| Min, max | Storage aggregate, then wrap the result | Matches `min_max/extension.rs:17-35` | -| is_sorted, is_constant | Storage aggregate. The `Bool` result stays as it is | Valid within one `(p, s)` | -| Hash, group by, distinct, join keys | Storage hash and equality | Valid within one `(p, s)` | -| Zone-map pruning | Rewrite the predicate onto storage statistics | Old files keep decimal-typed statistics, which hold the same integers | -| IN list | Storage set over rescaled needles | Replaces `haystack/mod.rs:118-125` | -| Row encoding | Storage at the canonical width, big-endian with the sign bit flipped | Already done (`vortex-row/src/codec.rs:19`, `:1466-1480`) | - -These rewrites need a small set of integer kernels that do not exist as reusable pieces today. +| Sum | Integer sum with accumulator `values_type(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | +| Min, max, min_max, bounded_min, bounded_max | Integer aggregate on the view, then a decimal of the input dtype for each result value | Matches `min_max/extension.rs:17-35`. `min_max` returns a struct of two values. Truncation applies only to `Utf8` and `Binary`, so the bounded variants return exact values for decimals | +| is_sorted, is_constant | Integer aggregate on the view. The `Bool` result stays as it is | Valid within one `(p, s)` | +| Hash, group by, distinct, join keys | Integer hash and equality on the view | Valid within one `(p, s)` | +| Zone-map pruning | Rewrite the predicate onto the integer statistics | Decimal statistics hold the same integers. Recommendation XQ2 writes rules over `ReduceNode`, so they also rewrite bound expressions for pruning | +| IN list | Integer set over the needles, which have the column dtype (X3) | Replaces `haystack/mod.rs:118-125` | +| Row encoding | The canonical width, big-endian with the sign bit flipped | Already done (`vortex-row/src/codec.rs:19`, `:1466-1480`) | + +These rules need a small set of integer kernels that do not exist as reusable pieces today. They are the scaled-integer family (node N8): ```rust @@ -568,8 +624,8 @@ Two points matter for the design: - **Temporal types need the same kernels.** Timestamp, Time, and Duration units are powers of ten apart, except Days. A Timestamp cast from milliseconds to nanoseconds (issue #8570) is - `rescale(delta = 6)`. A Duration comparison across units (PR #10467) needs the same rescale as - a decimal comparison across scales. One kernel family serves both. + `rescale(delta = 6)`. A Duration cast across units (PR #10467) needs the same rescale as a + decimal cast across scales. One kernel family serves both. - **Fusion keeps performance.** The current decimal arithmetic kernel checks the operation and the precision bound in one pass (`numeric/decimal.rs`, `DecimalOpConstants`). A chain of separate array operations makes several passes. So generalize the fused kernel from @@ -578,134 +634,136 @@ Two points matter for the design: ### 3.5 Wire compatibility -No new wire id and no new dtype tag is needed. The `ArrayPlugin` mechanism already lets one -in-memory array read and write several historical wire forms -(`vortex-array/src/array/plugin.rs:102-157`). +The leaf is the `vortex.decimal` wire node, so it needs no new wire ID and no new dtype tag. Only +decimal `Narrow` needs a wire ID (DQ3). The `ArrayPlugin` mechanism already lets one in-memory +array read several historical wire forms (`vortex-array/src/array/plugin.rs:102-157`). Reading: | Wire node | In-memory result | | --- | --- | -| `vortex.decimal`, `values_type` equals the canonical width | `Decimal { Primitive(I_w) }`, zero-copy | -| `vortex.decimal`, `values_type` narrower (compressor narrowing) | `Decimal { widen(Primitive(I_v)) }`, zero-copy and lazy | -| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | `Decimal { checked_narrow(Primitive(I_v)) }`, lazy. Fails at execution only if a value violates the precision | +| `vortex.decimal`, `values_type` equals the canonical width | The leaf, zero-copy | +| `vortex.decimal`, `values_type` narrower and at most 64 bits (compressor narrowing) | DBP with an integer `Narrow` over the buffer as its MSP, zero-copy (section 3.3) | +| `vortex.decimal`, `values_type` I128 at precision 39 to 76 (compressor narrowing) | The leaf after a widening copy. DBP has no 128-bit MSP | +| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | The leaf after a checked narrowing copy. A value that does not fit the canonical width is an error, not a panic | | `vortex.decimal`, unknown `values_type` | Error. Decode with `DecimalType::try_from(raw)`, not the prost getter | -| `vortex.decimal_byte_parts` (v1) | Phase 2: unchanged DBP that executes into the wrapper. Later: `Decimal { widen(msp) }` | -| `vortex.decimal_byte_parts.v2` | Phase 2: unchanged DBP. Later: `Decimal { WideIntParts { msp, lower } }` | +| `vortex.decimal_byte_parts` (v1) and `vortex.decimal_byte_parts.v2` | DBP. An old narrow MSP becomes an integer `Narrow` of the fixed type | +| Decimal `Narrow`, with the wire ID of DQ3 | `Narrow` over its child | Writing: | In-memory shape | Wire node | Readable by old readers | | --- | --- | --- | -| `Decimal { flat Primitive(I_w) }` | `vortex.decimal`, `values_type = w`, the buffer and validity of the child | Yes | -| `Decimal { widen(flat Primitive(I_v)) }` | `vortex.decimal`, `values_type = v` | Yes, same bytes as today | -| `Decimal { compressed child with a lane-integer ptype }` | `vortex.decimal_byte_parts` v1 with the child as MSP | Yes, core 2025.05 | -| `Decimal { WideIntParts }` | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2 | -| `Decimal { Constant, Dict, RunEnd, Chunked, ... }` | Push the wrapper below the generic encoding, for example `vortex.constant` with a decimal scalar, or `Dict(codes, Decimal { values })` | Yes | -| Any other child | Execute the child to a flat array and write `vortex.decimal` | Yes | - -The fifth row applies only if the wrapper sits above generic encodings in memory. With generic -encodings above the wrapper, the memory form already has the wire shape. +| The leaf | `vortex.decimal`, `values_type` equals the canonical width | Yes | +| DBP with an MSP only | `vortex.decimal_byte_parts` v1 | Yes, core 2025.05 | +| DBP with lower parts | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2 | +| Decimal `Narrow` | The wire ID of DQ3 | Only when an edition allows it. Otherwise the compressor does not produce it | +| `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked` or `Masked` over decimal values | The same nodes as today | Yes | Consequences: - Old readers read every file that a new writer produces, with the same edition rules as today. -- Wide PTypes never appear on the wire for decimal columns. The legacy metadata carries - `DecimalType`, and DBP carries lane-integer PTypes. Only top-level `Primitive(I128)` columns - (DuckDB HUGEINT, node N27) need new wire values and edition gating. +- Wide PTypes never appear on the wire for decimal columns. The leaf carries `DecimalType`, and + DBP carries lane-integer PTypes. Only top-level wide-integer columns (DuckDB HUGEINT and + UHUGEINT, node N27) need new wire values and edition gating. - Decimal statistics and zone-map entries keep the `Decimal` dtype tag. They stay readable. -- Writing DBP from the decimal plugin needs one owner for the decimal and DBP serialized ids. In - phase 2 the simplest path keeps DBP as its own encoding and lets the compressor produce it. - Moving DBP under the wrapper (node N14) is a later step. Section 9 lists the ownership choice. -- Extend the compat fixtures before the reader changes. Add narrowed widths (I8 at p 10), wide - widths (I128 at p 20, I256 at p 40), and Arrow-style wider widths (I128 at p 5). +- A writer without an edition for `Narrow` or DBP v2 loses one compact form. Today the v1 + compressor writes `vortex.decimal` at I128 for precision 39 to 76 when the values fit + (`vortex-btrblocks/src/schemes/decimal.rs:133-137`). Under D3, that writer stores I256 values. +- Extend the compat fixtures before the reader changes. Add narrowed widths (I8 at p 10) and wide + widths (I128 at p 20, I256 at p 40). Also add Arrow-style wider widths (I128 at p 5) and the + narrow 128-bit form (I128 at p 40). ### 3.6 What happens to `values_type` narrowing -Narrowing becomes an encoding of the child: - -- **Lane-integer children (p <= 18).** The existing integer schemes (FoR with BitPacking, - ZigZag, Delta, RLE, Dict) already narrow integers better than a fixed smaller width does. No - decimal-specific scheme is needed. -- **Wide children (p >= 19).** If every value fits `i64`, the child is `widen(i64 array)` and - the inner array uses the lane-integer schemes. Otherwise the child is a limb split, which is - DBP generalized (node N14). It has a signed MSP and unsigned 64-bit lower parts, and the - lane-integer schemes compress each part. -- **Legacy reads** produce `widen(...)` or `checked_narrow(...)` nodes as shown in section 3.5. - -`widen` has two possible representations. One is the existing lazy `Cast` scalar-function array, -which already has reduce and kernel pushdown machinery. The other is a small dedicated in-memory -encoding. Either way, the encoding needs one generic rule that already exists in DBP form -(`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). A comparison of -`widen(x)` with a constant becomes a comparison of `x` with the constant when the constant fits -the narrow type. Otherwise the result is a constant. The generalized rule helps every widened -integer column, not only decimals. +Narrowing leaves the canonical form. Two mechanisms replace it: + +- **Precision.** `Narrow` stores a column at a lower precision with the same scale (section 3.3). + The compressor picks `p'` from the value bounds in its statistics. +- **Bits.** The integer encodings narrow the MSP of DBP. FoR with BitPacking, ZigZag, Delta, RLE, + and Dict already narrow integers better than a fixed smaller width does. + +`narrowed_decimal` and the first step of the btrblocks `DecimalScheme` go away. The compressor +returns a `Narrow` over a leaf or over a DBP array instead (step 4 in section 8.3). Legacy reads +keep working: old narrow leaves become DBP arrays, and old narrow MSPs become integer `Narrow` +arrays (section 3.5). + +`Narrow` replaces the lazy `widen` and `checked_narrow` nodes of an earlier version of these notes. +It carries the generic comparison rule that DBP already has +(`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). A comparison of a +narrow column with a constant compares the child when the constant fits the child. Otherwise the +result is a constant. PR #10223 adds this rule for integers. The rule helps every narrow integer +column, not only decimals. ### 3.7 Builders, scalars, engines, and the public API -- **Builder.** `DecimalBuilder` becomes the decimal dtype plus a `PrimitiveBuilder` of the - storage type. `append_value::` converts with a checked cast and fails when the value is - outside the precision. The six-variant buffer enum - (`vortex-array/src/builders/decimal.rs:38-56`) goes away. -- **Scalars.** `DecimalScalar` keeps its API. `DecimalValue` is normalized to the storage width - at construction with the existing `normalize` (`dvalue.rs:208-216`). `scalar_at` wraps the - child scalar. -- **Deprecated accessors.** `DecimalArray::buffer::()` becomes - `unscaled().execute::(ctx)?.as_slice::()` with `T` equal to the storage - type. `values_type()` returns the storage width. `DecimalArray::new::` converts a buffer of - another width, or wraps it in a lazy `widen` or `checked_narrow` child. -- **Engines.** The Arrow, DuckDB, and CUDA exporters read the child. An Arrow `Decimal128` - export of a `checked_narrow` child can reuse the original 128-bit buffer without a copy. This - is a small case of the execution layouts of RFC 0062. +- **Builder.** `DecimalBuilder` takes its width from the dtype and never changes it. The + six-variant buffer enum (`vortex-array/src/builders/decimal.rs:38-56`) can stay, because the + width depends on the dtype. `append_value::` converts with a checked cast and fails when the + value is outside the precision. The generic builder factory already picks the canonical width + (`vortex-array/src/builders/mod.rs:412-416`). +- **Scalars.** `DecimalScalar` keeps its API. `DecimalValue` is normalized to the canonical width + at construction with the existing `normalize` + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:208-216`). `scalar_at` reads the buffer + at the canonical width. +- **Accessors.** `DecimalArray::buffer::()` keeps its signature, and `T` is always the type of + the canonical width. `values_type()` always returns the canonical width. `unscaled()` is new and + returns the integer view. `DecimalArray::new::` fails for any other `T` (step 6 in section + 8.3). +- **Engines.** The Arrow, DuckDB, and CUDA exporters read the leaf as today. Arrow export already + widens buffers below 128 bits for `Decimal128` (`vortex-arrow/src/executor/decimal.rs:152-165`). + Arrow import copies when the Arrow width differs from the canonical width, for example for + `Decimal128` data at precision 18 or below. Today that import is zero-copy + (`vortex-arrow/src/convert.rs:204-277`). DQ1 decides this cost. - **Python.** The binding keys its `DecimalArray` subclass on the in-memory encoding id - (`vortex-python/src/arrays/native.rs:174-175`). Keeping the id `vortex.decimal` keeps it working. + (`vortex-python/src/arrays/native.rs:174-175`). The id stays `vortex.decimal`. -### 3.8 Does Decimal become a well-known extension in Rust? +### 3.8 Decimal stays a built-in type -RFC 0062 (branch `ngates/types`, `rfcs/0062-value-types-and-representations.md`) proposes -`vortex.decimal` as a well-known extension with signed-integer storage, while the existing wire -tag keeps working. It also states the precondition directly: "Moving Decimal requires integer -storage capable of its currently supported precisions, including wide lanes." +D1 settles this question. `Decimal` stays a built-in type, and it behaves like a hardcoded +extension type: -The wrapper in sections 3.1 to 3.7 is exactly the array side of that proposal. The remaining -question is only where the Rust type lives: +- It is opaque by default. No code applies integer semantics to a decimal value unless a rule of + the type says so, as X1 requires of extension types. +- It derives `eq` and `ord` for one `(p, s)`, as X2 describes. After D3, the unscaled value is + unique within one dtype, so integer equality and order on the view are value equality and order. +- Its other operations are rewrite rules that the type owns (X4, section 3.4). -| Aspect | Keep `DType::Decimal` (phase 2) | Well-known extension (phase 4) | -| --- | --- | --- | -| Array | `DecimalArray` wrapper | `ExtensionArray` with the decimal ext dtype | -| Shared machinery | Through `StorageWrapper` | Directly | -| Semantics | Decimal-owned rewrites | `ExtVTable` hooks: cast (#8570), derives for one dtype, rewrites, validation | -| API change | Small | 269 lines in 97 files mention `DType::Decimal` | -| Wire | Unchanged | Tag translation in both directions, recursively | -| Risk | Low | High until extension semantics are opaque by default | - -Today the extension machinery leaks storage semantics. An extension array cast to any other -dtype casts its storage (`vortex-array/src/arrays/extension/compute/cast.rs:17-22`). A scalar cast -into an extension casts to the storage and rewraps (`vortex-array/src/scalar/cast.rs:56-61`). A -comparison with a lone extension side compares raw storage -(`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-194`). If Decimal moves now, a cast -from `decimal(10, 2)` to `i64` returns the unscaled integers, and `price < 5` compares `500` -with `5`. So the move must wait for nodes N1 to N4. Built-in registration is not an -obstacle: Date, Time, and Timestamp are already default extensions -(`vortex-array/src/dtype/session.rs:36-39`). - -Recommendation: build the wrapper with `DType::Decimal` first. Use Decimal as the acceptance test -for the extension hooks while they are designed. Decide on the Rust move after node N13, when it -has become a mechanical API change. - -## 4. The child representation question +A move to an extension type adds no capability. If the built-in type and an extension type become +identical, a later change can remove the built-in type. That step is optional (node N15). + +RFC 0062 (branch `ngates/types`, `rfcs/0062-value-types-and-representations.md`) proposes +`vortex.decimal` as a well-known extension with signed-integer storage. It states the precondition +directly: "Moving Decimal requires integer storage capable of its currently supported precisions, +including wide lanes." D1 does not adopt that move. + +If the optional move happens, it waits for the extension fixes. Today an extension array cast to +any other dtype casts its storage (`vortex-array/src/arrays/extension/compute/cast.rs:17-22`). A +scalar cast into an extension casts to the storage and rewraps +(`vortex-array/src/scalar/cast.rs:56-61`). A comparison with a lone extension side compares raw +storage (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-194`). If Decimal moves now, a +cast from `decimal(10, 2)` to `i64` returns the unscaled integers, and `price < 5` compares `500` +with `5`. So the move waits for nodes N1 to N4. The API cost is large: 269 lines in 97 files +mention `DType::Decimal`. Built-in registration is not an obstacle, because Date, Time, and +Timestamp are already default extensions (`vortex-array/src/dtype/session.rs:36-39`). + +## 4. The buffer type of the leaf + +D2 and D3 fix the shape of the leaf: one buffer, with one width for each precision. This section +keeps the earlier comparison of integer and FixedSizeBinary representations as background. The +verdict now concerns the type of the values in that buffer, and the type of the view over it. ### 4.1 The options -- **(a) Integer child with new PTypes.** The child is `Primitive(I8 .. I256)`. Add `I128` and - `I256` to `PType`. `U128` and `U256` are a separate decision (section 5.7). -- **(b) FixedSizeBinary child.** The child is `FixedSizeBinary(n)` with `n` in 1, 2, 4, 8, 16, 32, - or only 16 and 32 for the wide widths. FixedSizeBinary does not exist yet. -- **(c) Hybrid.** `Primitive` child up to 64 bits, `FixedSizeBinary(16)` or `(32)` above. +- **(a) Integer lanes with an integer view.** The buffer holds little-endian signed integers, as + today. The view is `Primitive(I8 .. I256)`, which needs `I128` and `I256` in `PType`. +- **(b) FixedSizeBinary values.** The buffer holds `FixedSizeBinary(n)` values with `n` in 1, 2, + 4, 8, 16, 32, or only 16 and 32 for the wide widths. +- **(c) Hybrid.** Integer lanes up to 64 bits, `FixedSizeBinary(16)` or `(32)` above. - **(d) Separate wide-integer family.** A new canonical family for 128 and 256 bits, outside `Primitive`. An example is a new dtype variant with a new canonical array. -- **(e) Status quo layout, opaque API.** Keep the typed buffer inside `DecimalArray`, pin the - width to the canonical width, and hide `values_type`. +- **(e) Integer lanes without a view.** The buffer of option (a), with the width fixed to the + canonical width, but with decimal-specific kernels instead of an integer view. ### 4.2 Byte order is not integer order @@ -723,38 +781,38 @@ Vortex already uses the sign-flip transform where it needs byte order. The row c (`vortex-row/src/codec.rs:19`). The DuckDB UUID exporter flips the top bit of a big-endian `u128` to get the ordered HUGEINT form of DuckDB (`vortex-duckdb/src/exporter/uuid.rs`, export loop). -So a FixedSizeBinary child can reuse memcmp order only if it stores big-endian sign-flipped -bytes. That layout breaks zero-copy exchange with Arrow `Decimal128` and `Decimal256` and with -DuckDB `DECIMAL`. It also forces a conversion before every arithmetic operation. With the native -little-endian layout, every ordered operation on the child needs a decimal-specific +So FixedSizeBinary values can reuse memcmp order only if they hold big-endian sign-flipped bytes. +That layout breaks zero-copy exchange with Arrow `Decimal128` and `Decimal256` and with DuckDB +`DECIMAL`. It also forces a conversion before every arithmetic operation. With the native +little-endian layout, every ordered operation on the values needs a decimal-specific reinterpretation. Either way, the decimal logic stays duplicated. ### 4.3 Comparison Compute: -| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | | --- | --- | --- | --- | --- | --- | | Compare | Generic primitive kernel at 128 and 256 bits | Decimal reinterpretation | Integer to 64 bits, reinterpretation above | New kernels, or shared generic code as in (a) | Existing decimal kernel | | Checked arithmetic | `i128` native. `i256` through num-traits `Checked*` | Reinterpretation | Mixed | As (a) | Existing | | Sum, min, max, sorted | Integer aggregates with a wider accumulator | Decimal-specific | Mixed | As (a) | Existing | | Casts | Integer width casts plus rescale | Decimal-specific | Mixed | As (a) | Existing | -| Hash | Native | Byte hash, correct at a fixed width | Both | Native | Upcast to i256 | +| Hash | Native | Byte hash, correct at a fixed width | Both | Native | Decimal-specific | | Lane layout | Array of little-endian lanes. A limb split gives a SIMD-friendly layout | Same bytes, no typed lanes | Mixed | As (a) | As today | Data: -| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | | --- | --- | --- | --- | --- | --- | -| Compression | All lane schemes to 64 bits. Wide: narrow if it fits, limb split, Dict, RLE, run-end, constant, sparse, zstd | Dict, zstd, byte split. No FoR or BitPacking without reinterpretation | Mixed | As (a) | DBP stays decimal-only | +| Compression | `Narrow`, DBP, and all lane schemes on the MSP. Dict, RLE, run-end, constant, sparse, zstd | Dict, zstd, byte split. No FoR or BitPacking without reinterpretation | Mixed | As (a) | As (a) | | Statistics | Integer min, max, and sum are meaningful | Memcmp min and max are wrong for numbers | Mixed | As (a) | Existing | | Ordering | Native | Wrong for little-endian bytes | Wrong above 64 bits | Native | Native | Interop: -| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | | --- | --- | --- | --- | --- | --- | -| Arrow | Decimal export zero-copy at 128 and 256 bits. A top-level `Primitive(I128)` needs a policy, because Arrow has no Int128 | Decimal export zero-copy | Zero-copy | As (a) | As today | +| Arrow | Decimal export as today. A top-level `Primitive(I128)` needs a policy, because Arrow has no Int128 | Decimal export as today | As today | As (a) | As today | | DuckDB | DECIMAL as today. HUGEINT becomes possible | DECIMAL needs reinterpretation. No HUGEINT | No HUGEINT | HUGEINT possible | No HUGEINT | | DataFusion | Decimals as today. No Int128 scalar exists | As today | As today | As today | As today | | Python, NumPy | Python `int` scalars. NumPy has no 128-bit integer | As today | As today | As (a) | As today | @@ -762,21 +820,25 @@ Interop: Cost: -| Dimension | (a) Integer child | (b) FSB child | (c) Hybrid | (d) Wide family | (e) Status quo | +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | | --- | --- | --- | --- | --- | --- | | Decimal wire | No change | Needs FSB first | Needs FSB first | No change | No change | | New wire values | Flatbuffer enum values, a new protobuf oneof, edition gating. Only for top-level wide columns | FSB tag or extension descriptor | As (b) | New dtype tag if user-visible | None | | Blast radius | Largest: about 380 macro uses, about 45 exhaustive matches, 53 `is_int()` sites, 12 mirrors. The compiler finds most of them | An FSB type: the closed prototype PR #8865 changed 118 files | As (b), plus two decimal paths | A dtype variant and a canonical variant: 51 canonical sites plus dtype matches | Smallest | -| Alignment | 16 bytes for both wide widths. Fix two `byte_width == alignment` assumptions | 1 byte by default. Typed access needs 16 | Mixed | As (a) | Unchanged | -| Duplicated logic | Lowest. Decimal kernels become rewrites | High | Medium | Medium | High, about 7,600 lines stay | +| Alignment | 16 bytes for all wide widths. Fix two `byte_width == alignment` assumptions | 1 byte by default. Typed access needs 16 | Mixed | As (a) | Unchanged | +| Duplicated logic | Lowest. Decimal kernels become rules over the view | High | Medium | Medium | High, about 7,600 lines stay | ### 4.4 Verdict -Choose (a), built as a distinct wide-integer tier (section 5). The tier keeps the useful part of -(d): wide integers opt into a small, explicit kernel set instead of pretending to be lane -integers. +Choose (a). The leaf keeps little-endian integer lanes, and the integer view gives decimal kernels +the integer kernels. D5 adds the wide `PType` values that the view needs. The wide-integer tier +(section 5) keeps the useful part of (d). Wide integers opt into a small, explicit kernel set +instead of pretending to be lane integers. -The suspected dependency in the request does not hold. Wide PTypes do not need FixedSizeBinary: +Option (e) is the state between nodes N11 and N9. It fixes the crash class of issue #5820. But the +kernels for precision 19 to 76 stay decimal-specific until the wide `PType` values land. + +Wide PTypes do not need FixedSizeBinary: - Rust stores `i128` natively, and Vortex already has `i256` with checked arithmetic. - The structural layer that both need already exists. `FixedWidthArray` moves records of 1, 2, @@ -791,19 +853,18 @@ limb split for comparisons. Neither type carries the semantics of the other. Evidence that changes the decision: -- Maintainers can decide that wide integers never become a Vortex type, or that the PType - migration is too large. Then fall back to (e) for one release. Require storage at least as wide - as the precision (node N11), and share generic lane kernels at the function level. This fixes - the crash class of issue #5820, but Decimal stays a separate world. - Benchmarks can show that the generic integer kernels at 128 and 256 bits are clearly slower than the current decimal kernels. Then keep the decimal kernels as specializations behind the - rewrites. This result is unlikely, because the current decimal kernels already run on `i128` + rules. This result is unlikely, because the current decimal kernels already run on `i128` and `i256` lanes. - If a requirement appears for precisions above 76 or arbitrary precision (for example PostgreSQL - NUMERIC), no fixed-width child works, and a variable-width representation is needed. + NUMERIC), no fixed-width buffer works, and a variable-width representation is needed. ## 5. Design of the wide-integer tier +D5 settles that Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types. This +section describes how. + ### 5.1 Types ```rust @@ -812,17 +873,16 @@ pub enum PType { U8 = 0, U16 = 1, U32 = 2, U64 = 3, I8 = 4, I16 = 5, I32 = 6, I64 = 7, F16 = 8, F32 = 9, F64 = 10, - I128 = 11, - I256 = 12, - // 13 and 14 stay reserved for U128 and U256. + I128 = 11, I256 = 12, + U128 = 13, U256 = 14, } /// Integers that fit a 64-bit lane. Only these can be offsets, codes, sizes, or take indices, /// and only these reach FastLanes and other lane kernels. pub enum LaneIntPType { U8, U16, U32, U64, I8, I16, I32, I64 } -/// Signed integers wider than 64 bits. -pub enum WideIntPType { I128, I256 } +/// Integers wider than 64 bits. +pub enum WideIntPType { I128, I256, U128, U256 } impl PType { pub const fn as_lane_int(self) -> Option; @@ -836,26 +896,31 @@ impl PType { Trait plan: -- Implement `NativePType` for `i128` and `i256`. `i128` already has every supertrait. `i256` - needs `Num`, `NumCast`, `FromPrimitive`, `Bounded`, and the byte conversion traits, about two - hundred lines over the Arrow type. Most `match_each_native_ptype!` bodies are generic over - `NativePType`, so they then compile and run correctly at the new widths. +- Implement `NativePType` for `i128`, `u128`, `i256` and a new `u256`. `i128` and `u128` already + have every supertrait. `i256` needs `Num`, `NumCast`, `FromPrimitive`, `Bounded`, and the byte + conversion traits, about two hundred lines over the Arrow type. Most + `match_each_native_ptype!` bodies are generic over `NativePType`, so they then compile and run + correctly at the new widths. +- Vortex has no native `u256` type. `primitive_types::U256` appears only in the fuzzing support for + `i256` (`vortex-array/src/dtype/arbitrary/decimal.rs:14-20`). So `U256` needs a new type, built + like `i256`. - Keep `IntegerPType` (which requires `PrimInt`) for lane integers only. - Add one trait for any-width integer kernels, for example `NativeIntPType: NativePType + Ord + Hash + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv + BigCast`, implemented for the eight - lane integers and the two wide ones. Compare, arithmetic, sum, min and max, casts, and the + lane integers and the four wide ones. Compare, arithmetic, sum, min and max, casts, and the scaled-integer family bound on it. -- Fold `NativeDecimalType` into it: the precision bound tables become a `Pow10` trait on the - signed integers, and `DecimalType` survives only to decode legacy metadata. +- `DecimalType` stays the width type of the leaf metadata. After N9, each of its values maps onto + one signed `PType`. `NativeDecimalType` can later fold into the new trait, with the precision + bound tables as a `Pow10` trait on the signed integers. ### 5.2 Dispatch macros | Macro | Arms | Use | | --- | --- | --- | -| `match_each_native_ptype!` | All 13 | Generic code over `NativePType` | +| `match_each_native_ptype!` | All 15 | Generic code over `NativePType` | | `match_each_lane_int_ptype!` | 8, takes `LaneIntPType` | Offsets, codes, indices, lane encodings. Exhaustive, no panic arm | -| `match_each_int_ptype!` | 10 | Any-width integer kernels | -| `match_each_wide_int_ptype!` | 2 | Wide-only code, such as the limb split | +| `match_each_int_ptype!` | 12 | Any-width integer kernels | +| `match_each_wide_int_ptype!` | 4 | Wide-only code, such as the limb split | Retire `match_each_integer_ptype!` and its siblings after the migration, because their `other => panic!` arm turns a type mistake into a run-time panic @@ -872,13 +937,14 @@ Let the compiler enumerate the sites: encodings and choose `is_lane_int`. 3. Migrate `match_each_integer_ptype!` uses (169) and `match_each_unsigned_integer_ptype!` uses (118) to the typed macros, crate by crate. -4. Add the two variants last. Every exhaustive match (about 45 outside `ptype.rs`) and every +4. Add the four variants last. Every exhaustive match (about 45 outside `ptype.rs`) and every mirror then fails to compile until it handles them. -5. Teach the fuzzer to generate decimal children at every width before exposing wide columns. +5. Teach the fuzzer to generate decimal views and wide-integer columns at every width before + exposing wide columns. ### 5.4 Wire encoding and gating -- **Flatbuffers.** Append `I128` and `I256` to `enum PType: uint8` +- **Flatbuffers.** Append `I128`, `I256`, `U128` and `U256` to `enum PType: uint8` (`vortex-array/flatbuffers/vortex-dtype/dtype.fbs:4-16`). Old readers fail loudly with "Unknown PType variant" (`vortex-array/src/dtype/serde/flatbuffers.rs:559`). - **Protobuf.** Do not append to `enum PType` in `dtype.proto`. Old readers decode an unknown @@ -889,44 +955,51 @@ Let the compiler enumerate the sites: "Unrecognized DType" (`proto.rs:32`). Also change new readers to decode `PType` with `TryFrom` and fail on unknown values. - **Editions.** Add a component kind for core dtype features, or treat `vortex.primitive.i128` - and `vortex.primitive.i256` as dtype members, and check them in `validate_dtype_editions` + and the other wide types as dtype members, and check them in `validate_dtype_editions` (`vortex-file/src/writer.rs:429-475`). Use the same mechanism for any later core dtype tag, - including FixedSizeBinary if it gets a tag. Decimal columns do not need it, because their - storage never reaches the wire (section 3.5). + including the FixedSizeBinary tag that DQ7 recommends. Decimal columns do not need it, because + the leaf carries `DecimalType` (section 3.5). ### 5.5 Scalars -- Add `PValue::I128(i128)` and `PValue::I256(i256)`. `PValue` equality and hashing upcast within - each signedness family today (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-178`). - Extend the upcast target to `i256` for signed values. +- Add `PValue::I128`, `PValue::I256`, `PValue::U128` and `PValue::U256`. `PValue` equality and + hashing upcast within each signedness family today + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-178`). Extend the upcast targets to + `i256` for signed values and to `u256` for unsigned values. - Serialize them as protobuf `bytes_value` holding 16 or 32 little-endian bytes, decoded by the dtype, the same way decimals work (`vortex-array/src/scalar/proto.rs:402-445`). Old readers reject `bytes_value` for a primitive dtype (`proto.rs:442-444`), which is a loud failure. -- Decide later whether `DecimalValue` becomes a storage `PValue` plus the dtype. In phase 2 it - can stay, normalized to the canonical width at construction. +- Decide later whether `DecimalValue` becomes a storage `PValue` plus the dtype. Until then it + stays, normalized to the canonical width at construction. ### 5.6 Engine policy for top-level wide columns -| Engine | `Primitive(I128)` | `Primitive(I256)` | -| --- | --- | --- | -| Arrow | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `i128` needs 39 digits) or `FixedSizeBinary(16)` through an export plugin | Error by default. Opt-in `FixedSizeBinary(32)`. `Decimal256` holds at most 76 digits, and `i256` needs 77 | -| DuckDB | HUGEINT, zero-copy little-endian | No type. Error | -| DataFusion | Error | Error | -| Python | Python `int` scalars. Arrays follow the Arrow policy | Same | -| FFI, C++, Java | New enum values in `vx_ptype` and mirrors | Same | -| CUDA | Not supported in dynamic dispatch at first | Same | +| Engine | `Primitive(I128)` | `Primitive(U128)` | `Primitive(I256)`, `Primitive(U256)` | +| --- | --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `i128` needs 39 digits) or `FixedSizeBinary(16)` through an export plugin | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `u128` needs 39 digits) or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)`. `Decimal256` holds at most 76 digits, `i256` needs 77, and `u256` needs 78 | +| DuckDB | HUGEINT, zero-copy little-endian | UHUGEINT | No type. Error | +| DataFusion | Error | Error | Error | +| Python | Python `int` scalars. Arrays follow the Arrow policy | Same | Same | +| FFI, C++, Java | New enum values in `vx_ptype` and mirrors | Same | Same | +| CUDA | Not supported in dynamic dispatch at first | Same | Same | ### 5.7 U128 and U256 -Defer them. Decimal needs only signed widths. The only current consumer is DuckDB UHUGEINT -(`vortex-duckdb/src/convert/dtype.rs:101`). Reserve the wire values. Until they exist, -`to_unsigned()` and `to_signed()` must not be called on wide types. Audit their 77 call sites -during step 3 of section 5.3, because they reinterpret lane integers for indices and codes. +D5 adds them, which overrides the earlier recommendation of these notes to defer them. Decimal +needs only the signed widths. DuckDB UHUGEINT is the first consumer of `U128` +(`vortex-duckdb/src/convert/dtype.rs:101`). `U256` needs a new native type (section 5.1). + +With all four types, `to_unsigned()` and `to_signed()` have a target for each wide type. Audit +their 77 call sites during step 3 of section 5.3 anyway, because many of them reinterpret lane +integers for indices and codes. + +Recommendation (DQ5): wide integers are user-visible columns behind an edition. They get a small +kernel set first: compare, arithmetic, cast, sum, min, max, hash, and structural operations. ### 5.8 What wide integers deliberately do not get - No FastLanes BitPacking, FoR, or Delta on 128-bit lanes. The `fastlanes` lanes stop at 64 - bits. Compression goes through narrowing or the limb split. + bits. Compression goes through `Narrow` or the limb split. - Never offsets, sizes, codes, or take indices. - No CUDA dynamic dispatch at first. The decimal device kernels keep their own width handling. @@ -972,7 +1045,8 @@ Apply the RFC 0055 test: nested values, including extensions over nested storage, to arrays. Under that rule a UUID over `FixedSizeList` has no cheap scalar, while a UUID over `FixedSizeBinary(16)` does. -Conclusion: make it core, as RFC 0062 proposes. +D6 settles the question: FixedSizeBinary is a core type, as RFC 0062 proposes. The test above is +the background for that decision. ### 6.3 Canonical layout @@ -1015,10 +1089,10 @@ depend on byte order. It must not implement ordering without the big-endian sign | New core tag (the next free flatbuffer union value and protobuf oneof field) and a new `vortex.fixed_size_binary` array id | Fail loudly on the schema | Needs the core dtype feature gate of section 5.4 | | Extension descriptor `vortex.fixed_size_binary` over `FixedSizeList` storage, and the array written in that storage form | Read it as a foreign extension over `FixedSizeList`, so the data stays readable (`vortex-array/src/dtype/extension/foreign.rs:26-75`) | The existing dtype edition kind | -The second option is the inverse of the decimal plan: the Rust API has a core variant, and the -wire uses an extension descriptor. The rule behind both is the same: choose the wire form for -reader compatibility and the Rust form for kernels and API. Translate at the serialization -boundary. Section 9 keeps the choice open. +Recommendation (DQ7): a new core dtype tag behind an edition. Then the wire form matches the Rust +form, and no translation is needed. The extension descriptor keeps old readers working. But then +the wire form differs from the Rust form, and node N16 must translate at the serialization +boundary. The question stays open (section 9). ### 6.7 UUID on FixedSizeBinary @@ -1029,6 +1103,7 @@ boundary. Section 9 keeps the choice open. new storage. The conversion is zero-copy for flat data. The existing Arrow plugin already does the same conversion (`encodings/uuid/src/arrow.rs`, `try_fsl_to_fsb`). - The DuckDB exporter keeps its big-endian sign-flip transform. +- XQ6 asks whether `Uuid` orders by its bytes. It is open, and there is no recommendation yet. ### 6.8 Relation to decimals and wide integers @@ -1041,29 +1116,30 @@ values"). ### 7.1 Nodes -Kind: **impl** means implementation work with a known shape. **research** means a design decision -must come first. **urgent** means a correctness bug that exists today. +Kind: **impl** means implementation work with a known shape. A **research** node needs a design +decision first. An **urgent** node fixes a correctness bug that exists today. No settled decision +needs an **optional** node. | Id | Node | Kind | Notes | | --- | --- | --- | --- | -| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes: storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values, debug output | -| N2 | Bind-time rewrite infrastructure | research, then impl | `ext_storage` and `ext_wrap` nodes, extension declarations on functions, type-owned rewrites, and rules across dtypes | -| N3 | Derives for one dtype | impl | `EQ` and `ORD` declared on one dtype. The binder applies them when every operand in a derived position has that dtype. Operations across dtypes use rewrites (N2) | -| N4 | Cast hooks (#8570) | impl | `cast_to` and `cast_from` on `ExtVTable`, bound `CastFn` | -| N5 | Array validation at trust boundaries | impl | `validate_array` and validated or unvalidated arrays: decimal precision, UTF-8, FixedSizeBinary length | +| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values and debug output | +| N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode` | +| N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) | +| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types get `cast_to` and `cast_from` hooks. CQ1, the default for inexact values, is open | +| N5 | Value checks in `try_new` | impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. The decimal leaf runs its own precision check. Checks: decimal precision, UTF-8, FixedSizeBinary length | | N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes | | N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width | | N8 | Scaled-integer kernels | impl | `rescale`, `check_range`, fused scaled arithmetic and compare | -| N9 | Wide-integer tier (I128, I256) | impl | Section 5 | +| N9 | Wide-integer tier (I128, I256, U128, U256) | impl | Section 5 and D5. Recommendation (DQ5): user-visible behind an edition, with a small kernel set first | | N10 | Wire safety and core dtype gating | impl | Strict enum decoding, protobuf oneof for wide PTypes, edition kind for core dtype features | -| N11 | Decimal storage at least as wide as the precision | impl, urgent | Option 3 of issue #5820. Removes the crash class | -| N12 | Decimal wrapper over an integer child | impl | Section 3, with the legacy wire plugin | -| N13 | Decimal semantics as storage rewrites | impl | Delete the decimal-specific kernels | -| N14 | DBP generalized as a wide-integer limb split | impl, optional | DBP moves under the wrapper and loses its decimal dtype | -| N15 | Decimal as a well-known extension in Rust | impl, later | RFC 0062. Wire tag unchanged | +| N11 | One canonical width for each decimal dtype | impl, urgent | D3. Producers take the width from the precision, readers convert old buffers, and `DecimalArray::try_new` accepts only the canonical width. Removes the crash class of issue #5820 | +| N12 | Zero-copy integer view | impl | Section 3.2. Precision 1 to 18 needs no new `PType`, so that part can land at any point. Precision 19 to 76 needs N9 | +| N13 | Decimal semantics as rewrite rules | impl | Section 3.4. Decimal kernels delegate to integer kernels through the view. Most decimal-specific kernels go away | +| N14 | Limb split for wide-integer columns | impl, optional | The split of DBP, offered as an encoding of wide-integer columns. DBP itself keeps its decimal dtype (DQ4) | +| N15 | Decimal as an extension type | impl, optional, later | D1 keeps the built-in type. A later change can remove it if the two become identical. Wire tag unchanged | | N16 | Wire translation layer for well-known types | research, then impl | Rust representation and wire representation chosen separately | -| N17 | FixedSizeBinary core type | impl | Section 6 | -| N18 | UUID on FixedSizeBinary | impl | Accept frozen storage, write frozen storage | +| N17 | FixedSizeBinary core type | impl | D6 and section 6. Recommendation (DQ7): a new core dtype tag behind an edition | +| N18 | UUID on FixedSizeBinary | impl | Accept frozen storage, write frozen storage. XQ6, the byte order of `Uuid`, is open | | N19 | Utf8 as a well-known extension | research, low priority | 776 lines in 212 files mention `DType::Utf8` | | N20 | FixedSizeList and Map as well-known extensions | research, low priority | Needs requested layouts for the offset-free FixedSizeList form | | N21 | Duration and Interval extensions (PRs #10467, #10468) | impl, in flight | Both PRs leave edition membership open | @@ -1071,13 +1147,13 @@ must come first. **urgent** means a correctness bug that exists today. | N23 | Engine type plugins for DuckDB and DataFusion | impl | Arrow already has plugin registries (`vortex-arrow/src/session.rs`) | | N24 | Refinement types (RFC 0055) | research only | UUID within FixedSizeBinary(16), Utf8 within Binary | | N26 | Scalar value model (RFC 0056) | research, then impl | Decides which types keep a host `Scalar` | -| N27 | DuckDB HUGEINT import and export | impl | First user-visible wide-integer column | +| N27 | DuckDB HUGEINT and UHUGEINT import and export | impl | First user-visible wide-integer columns | +| N28 | One `Narrow` encoding | impl, in flight | D4. Integers first (PRs #10223, #10332, #10333, issue #10020), then decimals. Recommendation (DQ2): one encoding. DQ3 decides the decimal wire ID | (There is no N25. The Variant interaction is an edge on N10.) Not in the graph: Union completion. Union casts and several aggregates are still TODOs (`vortex-array/src/scalar_fn/fns/cast/mod.rs:223-228`). That work is independent of this graph. -It gains from the shared structural rules of section 3.3. ### 7.2 Graph @@ -1087,22 +1163,24 @@ flowchart TD classDef critical fill:#fff3cd,stroke:#997404,color:#000 classDef impl fill:#d1e7dd,stroke:#0f5132,color:#000 classDef research fill:#e2e3e5,stroke:#41464b,color:#000,stroke-dasharray: 4 3 + classDef optional fill:#ffffff,stroke:#41464b,color:#000,stroke-dasharray: 2 2 N1["N1 Extension opacity fixes"]:::urgent - N11["N11 Decimal storage at least canonical width"]:::urgent - N2["N2 Bind-time rewrite infrastructure"]:::research + N11["N11 One canonical decimal width"]:::urgent + N28["N28 One Narrow encoding"]:::critical + N2["N2 Rewrite rules owned by the type"]:::research N3["N3 Derives for one dtype"]:::impl - N4["N4 Cast hooks #8570"]:::impl - N5["N5 Array validation"]:::impl + N4["N4 Cast rules #8570"]:::impl + N5["N5 Value checks in try_new"]:::impl N6["N6 Nullability restructure"]:::research - N7["N7 Fixed-width structural layer"]:::critical - N8["N8 Scaled-integer kernels"]:::impl - N9["N9 Wide-integer tier I128 I256"]:::critical + N7["N7 Fixed-width structural layer"]:::impl + N8["N8 Scaled-integer kernels"]:::critical + N9["N9 Wide integers I128 I256 U128 U256"]:::critical N10["N10 Wire safety and core dtype gating"]:::impl - N12["N12 Decimal wrapper over integer child"]:::critical - N13["N13 Decimal semantics as rewrites"]:::critical - N14["N14 DBP as wide-integer limb split"]:::impl - N15["N15 Decimal as well-known extension"]:::critical + N12["N12 Zero-copy integer view"]:::critical + N13["N13 Decimal semantics as rewrite rules"]:::critical + N14["N14 Limb split for wide integers"]:::optional + N15["N15 Decimal as extension type"]:::optional N16["N16 Wire translation layer"]:::research N17["N17 FixedSizeBinary core type"]:::impl N18["N18 UUID on FixedSizeBinary"]:::impl @@ -1113,7 +1191,7 @@ flowchart TD N23["N23 Engine type plugins"]:::impl N24["N24 Refinement types"]:::research N26["N26 Scalar value model"]:::research - N27["N27 DuckDB HUGEINT"]:::impl + N27["N27 DuckDB HUGEINT and UHUGEINT"]:::impl N1 --> N15 N1 --> N21 @@ -1128,7 +1206,8 @@ flowchart TD N8 --> N13 N7 -.-> N9 N7 -.-> N17 - N11 --> N12 + N28 --> N11 + N11 -.-> N12 N9 --> N12 N9 --> N14 N9 --> N27 @@ -1136,7 +1215,7 @@ flowchart TD N10 -.-> N9 N10 -.-> N17 N12 --> N13 - N12 --> N14 + N28 -.-> N13 N13 --> N15 N16 --> N15 N16 -.-> N18 @@ -1148,7 +1227,7 @@ flowchart TD N23 -.-> N21 N23 -.-> N27 N23 -.-> N18 - N5 -.-> N12 + N5 -.-> N13 N5 -.-> N17 N26 -.-> N17 N24 -.-> N18 @@ -1156,8 +1235,8 @@ flowchart TD N6 -.-> N15 ``` -Solid arrows are hard dependencies. Dashed arrows are soft: the target works without the source, -but it is cheaper, safer, or faster with it. +Solid arrows are hard dependencies: the target must not land before the source. Dashed arrows are +soft: the target works without the source, but it is cheaper, safer, or faster with it. ### 7.3 Edges with reasons @@ -1165,75 +1244,85 @@ but it is cheaper, safer, or faster with it. compares (`arrays/extension/compute/cast.rs:17-22`, `scalar_fn/fns/binary/compare/mod.rs:177-194`). 2. N1 to N21: without N1, a `Duration` cast across units reinterprets the storage on the row path - (EXT-C1). A comparison with a raw `i64` literal compares storage (EXT-Q1). New types must not - ship with a known leak that users can come to depend on. + (EXT-C1). On the array path, a comparison with a raw `i64` literal compares storage (EXT-Q1). + New types must not ship with a known leak that users can come to depend on. 3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The engine literal paths then change once, before the storage changes. PR #9886 shows the current workaround. -4. N2 to N3: the binder applies a derive as a rewrite onto `ext_storage`. -5. N2 to N13 (soft): expression-level rewrites let zone maps and compressed integer kernels - evaluate decimal predicates on the integer child. +4. N2 to N3: a derive is one kind of rule that the type owns. It runs the storage operation for + operands of one dtype. +5. N2 to N13 (soft): the decimal rules can reuse the rule mechanism of extension types. With + recommendation XQ2, rules over `ReduceNode` also rewrite bound expressions for pruning. The + built-in type can own its rules without N2. 6. N3 to N15: after N1 and N2, a decimal extension type without derives loses compare, hash, - `min`, `max` and sort within one `(p, s)`. A comparison across scales is a rewrite that the - type owns (N2), not a derive. -7. N3 to N21 (soft): `Duration` declares `EQ` and `ORD` for one unit. Before N3, the shared storage - compare kernel gives the same answer for one unit. A comparison across units needs a - type-owned rewrite in both cases. -8. N4 to N15: rescaling casts and casts to and from integers and floats must be type-owned. + `min`, `max` and sort within one `(p, s)`. Comparisons across scales stay with the engine (X3). +7. N3 to N21 (soft): `Duration` declares `eq` and `ord` for one unit. Before N3, the shared storage + compare kernel gives the same answer for one unit. Across units, the engine casts first (X3). +8. N4 to N15: rescaling casts and casts to and from integers and floats must be rules of the type + (C1). 9. N4 to N21: unit conversions are casts. 10. N8 to N4 (soft): Timestamp, Time, and Duration unit casts are powers of ten, except Days. -11. N8 to N13: decimal rewrites need `rescale` and `check_range`. +11. N8 to N13: decimal rules need `rescale` and `check_range`. 12. N7 to N9 (soft): width-keyed structural kernels give wide integers take, filter, run-end, sparse, dict, and constant support without new per-type code. 13. N7 to N17 (soft): width-keyed structural kernels give FixedSizeBinary run-end, sparse, dict, and constant support without new per-type code. Take and filter already come from `FixedWidthArray`. -14. N11 to N12: N11 forbids narrow storage, which removes the crash class and the "does not fit" - special cases. The wrapper refactor then starts from a consistent base. -15. N9 to N12: the wrapper child for precision 19 to 76 is `Primitive(I128)` or `Primitive(I256)`. -16. N9 to N14: the limb split is an encoding of wide integers. -17. N9 to N27: HUGEINT maps to `Primitive(I128)`. -18. N10 to N27: a top-level wide column writes new dtype values. An edition must gate them, and +14. N28 to N11: the last step of N11 makes `DecimalArray::try_new` accept only the canonical + width. Before that step, the compressor and DBP decoding must return `Narrow` instead of + narrow leaves. Without `Narrow`, D3 forces the full width on every column (D4). +15. N11 to N12 (soft): after N11, all arrays of one decimal dtype have one view dtype. Before N11, + a delegated binary kernel must widen one side first. +16. N9 to N12: the view for precision 19 to 76 is `Primitive(I128)` or `Primitive(I256)`. The + view for precision 1 to 18 does not wait for N9. +17. N9 to N14: the limb split is an encoding of wide integers. +18. N9 to N27: HUGEINT maps to `Primitive(I128)`, and UHUGEINT maps to `Primitive(U128)`. +19. N10 to N27: a top-level wide column writes new dtype values. An edition must gate them, and the protobuf encoding must fail loudly in old readers. -19. N10 to N9 (soft): ship the strict enum decoding in readers before any writer can emit wide +20. N10 to N9 (soft): ship the strict enum decoding in readers before any writer can emit wide PTypes. Variant shredding stores protobuf dtypes on disk (`vortex-array/src/arrays/variant/vtable/mod.rs:54`). -20. N10 to N17 (soft): needed only if FixedSizeBinary gets a new core wire tag. -21. N12 to N13: the rewrites rely on the fixed storage dtype. -22. N12 to N14: the limb split lives under the wrapper. -23. N13 to N15: once decimal semantics are storage rewrites, the move to an extension changes - the API, not the behavior. -24. N16 to N15: the `Decimal` wire tag must translate to and from the extension, recursively. -25. N16 to N18 (soft): UUID storage is `FixedSizeBinary(16)` in memory and `FixedSizeList` on +21. N10 to N17 (soft): needed only if FixedSizeBinary gets a new core wire tag. DQ7 recommends + one, and then this edge becomes hard. +22. N12 to N13: the rules run over the integer view. +23. N28 to N13 (soft): the view of a decimal `Narrow` is an integer `Narrow`. So the decimal rules + get the narrow-width speed of the integer `Narrow` kernels. +24. N13 to N15: once decimal semantics are rules over the view, the optional move to an extension + changes the API, not the behavior. +25. N16 to N15: the `Decimal` wire tag must translate to and from the extension, recursively. +26. N16 to N18 (soft): UUID storage is `FixedSizeBinary(16)` in memory and `FixedSizeList` on the wire. -26. N16 to N19 and N16 to N20: the `Utf8`, `FixedSizeList`, and `Map` tags need the same +27. N16 to N19 and N16 to N20: the `Utf8`, `FixedSizeList`, and `Map` tags need the same translation. -27. N17 to N18: UUID storage needs FixedSizeBinary. -28. N22 to N19 and N22 to N20: Utf8 views and the offset-free FixedSizeList layout must stay +28. N17 to N18: UUID storage needs FixedSizeBinary. +29. N22 to N19 and N22 to N20: Utf8 views and the offset-free FixedSizeList layout must stay available when those types become extensions. -29. N23 to N21, N27, and N18 (soft): with plugins, new types get DuckDB and DataFusion mappings +30. N23 to N21, N27, and N18 (soft): with plugins, new types get DuckDB and DataFusion mappings without edits to central `match` statements (`vortex-duckdb/src/convert/dtype.rs`, `vortex-datafusion/src/convert/scalars.rs`). Edits to the central statements also work. -30. N5 to N12 (soft): import boundaries establish invariant I2 (values within the precision). -31. N5 to N17 (soft): a cast from `Binary` must check every length. FixedSizeBinary can check +31. N5 to N13 (soft): the places where data enters establish invariant I2 (values within the + precision). The rules can then trust their inputs. +32. N5 to N17 (soft): a cast from `Binary` must check every length. FixedSizeBinary can check lengths in its own constructors and casts, so the shared hook only avoids a second mechanism. -32. N26 to N17 (soft): a FixedSizeBinary scalar is a binary payload, which RFC 0056 treats as +33. N26 to N17 (soft): a FixedSizeBinary scalar is a binary payload, which RFC 0056 treats as scalar-compatible. -33. N24 to N18 and N24 to N19 (soft): a refinement framework can express these types, but +34. N24 to N18 and N24 to N19 (soft): a refinement framework can express these types, but neither one depends on it. -34. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every +35. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every nullability change rebuilds the extension dtype (`vortex-array/src/dtype/extension/typed.rs:91-99`). ### 7.4 Critical path and parallel work -- **Critical path for fixing Decimal:** N11 runs first and alone. The hard chain is N9, N12, - N13. N8 must land before N13. N7 is soft, but it shrinks N9, so do it first. -- **Critical path for the RFC 0062 end state:** N9, N12, N13, N15, with N1, N2 then N3, N4, and - N16 joining at N15. +- **Critical path for one canonical form:** the integer `Narrow` PRs, then N28 for decimals, then + the rest of N11. The DBP nullability fix and the first step of N11 come before them. N11 does + not wait for N9. +- **Critical path for delegated decimal kernels:** N9, N12, N13. N8 must land before N13. N7 is + soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at any point. +- **The optional move (N15):** after N13, with N1, N2 then N3, N4, and N16 joining at N15. - **Parallel tracks:** - Extension correctness: N1, N2, N3, N4, N5. - - Decimal: N11, N7, N8, N9, N12, N13, N14. + - Decimal: N28, N11, N12, N7, N8, N9, N13, N14. - Bytes: N17, N18. - Temporal: N21, after N1 and N4. - Wire and engines: N10, N16, N23, N27. @@ -1243,142 +1332,191 @@ but it is cheaper, safer, or faster with it. ### 7.5 Mapping to the extension-type research The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to -U8) and phases (0 to 6). This table maps them onto the graph. +U8) and phases (0 to 6). The decisions file adds ids X1 to X9 and XQ1 to XQ6. This table maps them +onto the graph. | Extension research item | Node here | | --- | --- | | U1 to U8, the urgent changes | N1, in wave 0. U5 and U6 are dtype and storage checks, not the value validation of N5 | -| U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 34 describes | +| U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 35 describes | | Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 | -| Phase 2, binder and derives | N2 and N3 | -| Phase 3, casts | N4 | -| Phase 4, physical form and storage statistics | The shared rules of section 3.3, used by N12. The position of the wrapper is open (section 9). DateTimeParts under the wrapper is the temporal twin of N14 | -| Phase 5, validation and nullability | N5 and N6 | -| Phase 6, authoring helpers and engine export plugins | N23 | +| Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show | +| Phase 3, cast rules that the types own (C1) | N4 | +| Phase 4, the lift rule for structural arrays, and storage statistics (X5, XQ1) | The placement rule of section 3.3. DateTimeParts below the structural encodings is the temporal twin of DBP | +| Phase 5, value checks in `ExtensionArray::try_new`, and nullability (X7) | N5 and N6 | +| Phase 6, authoring and boundaries | N23 | -The array-level decimal fix (N11, N12, N13) does not wait for the extension phases, because the -built-in Decimal wrapper owns its rewrites (section 3.4). The RFC 0062 end state (N15) needs -phases 0, 2 and 3 first (N1, N3 and N4). Part 5 of the extension notes gives the same order. +The decimal fix (N28, N11, N12, N13) does not wait for the extension phases. The built-in Decimal +owns its rules (D1, section 3.4). The optional move (N15) needs phases 0, 2 and 3 first (N1, N3 +and N4). ## 8. Recommendations ### 8.1 Decision -Give Decimal an integer child. Add `I128` and `I256` as a wide-integer tier with typed dispatch. -Keep the decimal wire format and the `DType::Decimal` variant for now. Build FixedSizeBinary on its -own track for opaque bytes. Revisit the Rust-level move to a well-known extension after the -decimal rewrites land. +The maintainer settled the main choices on 2026-10-19: -Reasons, in order of weight: +- `DecimalArray` stays a leaf with one buffer and validity (D2). +- Each decimal dtype has one canonical encoding, and the precision fixes the width (D3). +- A `Narrow` encoding stores a column at a lower precision with the same scale (D4). +- Vortex adds `I128`, `I256`, `U128` and `U256` (D5). +- FixedSizeBinary is a core type for opaque bytes (D6). +- `Decimal` stays a built-in type (D1). -1. Every decimal operation is an integer operation plus scale bookkeeping. An integer child turns - the decimal kernels into rewrites over shared integer kernels. A FixedSizeBinary child keeps - every decimal kernel and adds a reinterpretation step. +These notes add the integer view of section 3.2, so that decimal kernels delegate to integer +kernels. Section 9 lists the open questions with the recommendations of these notes. + +Reasons for integer lanes and the integer view, in order of weight: + +1. Every decimal operation is an integer operation plus scale bookkeeping. The view turns the + decimal kernels into rules over shared integer kernels. FixedSizeBinary values keep every + decimal kernel and add a reinterpretation step. 2. Byte order is not integer order for the layout that Arrow and DuckDB use (section 4.2). -3. Integer statistics, integer compression, and zero-copy Arrow and DuckDB decimal exchange all - work unchanged with an integer child. -4. The decimal wire format needs no change, because the wrapper reads and writes the legacy forms. -5. The cost is a large but mechanical PType migration that the compiler can drive. +3. Integer statistics and integer compression work unchanged. Exchange with Arrow and DuckDB + stays zero-copy where the widths match. +4. The decimal wire format needs no change, because the leaf is the `vortex.decimal` node. +5. The cost is a large but mechanical PType migration that the compiler can drive. It does not + block the one-canonical-form fix. ### 8.2 Risks and mitigations | Risk | Mitigation | | --- | --- | | The PType migration touches 36 crates (counting each encoding crate) and several languages | Typed sub-enums and macros, compiler-driven steps (section 5.3), one crate per PR | -| A leftover `other => panic!` arm panics on a wide type | Retire the panic macros. Fuzz decimals at every width | +| A leftover `other => panic!` arm panics on a wide type | Retire the panic macros. Fuzz decimals and wide integers at every width | | Old readers silently read a protobuf wide PType as `U8` | Use a new protobuf oneof. Ship strict readers early (N10) | | Rewritten arithmetic makes several passes | Fused scaled-integer kernels (N8). Compare before and after with the existing `vortex-array/benches/binary_ops.rs`, `cast_decimal.rs`, and `compare.rs` | -| Arrow import of `Decimal128` at low precision loses zero-copy | Lazy `checked_narrow` child. The Arrow exporter can reuse the original buffer | -| Public API breaks: `DecimalArray::buffer::()`, `values_type()`, `DecimalBuilder::new::` | Deprecation shims that convert. Keep the in-memory id `vortex.decimal`, which the Python bindings key on (`vortex-python/src/arrays/native.rs:174-175`) | +| D3 without `Narrow` forces the full width on every column | Land `Narrow` for decimals before `try_new` rejects other widths (N28 before the end of N11) | +| A narrow buffer does not give a precision, because precision and integer width do not line up | The compressor uses statistics. Old narrow leaves become DBP arrays without a scan | +| Arrow import at a width other than the canonical width copies the buffer | Accept the copy, or choose another width function in DQ1. Arrow export already copies for widths below 128 bits | +| A writer without an edition for `Narrow` or DBP v2 stores precision 39 to 76 at 32 bytes, even when values fit 16 bytes | Allow `Narrow` or DBP v2 in an edition | +| Public API breaks: `DecimalArray::new::` with a `T` other than the canonical type, and `DecimalBuilder::new::` | Constructors that convert, and errors that name the canonical width. Keep the in-memory id `vortex.decimal`, which the Python bindings key on (`vortex-python/src/arrays/native.rs:174-175`) | | Wide integers become visible to users before engines can export them | Writer gating (N10) and an explicit engine policy (section 5.6) | ### 8.3 Sequence +PR #10230 gives decimals an integer child, which D2 replaces. The decisions file splits its +reusable parts into six smaller PRs, in this order. This section calls them steps 1 to 6: + +1. The nullability fix in DBP. +2. Producers take the width from the precision. +3. `Narrow` over decimals. +4. The compressor and DBP decoding stop narrowing and return `Narrow` instead. +5. Readers convert old buffers at other widths. +6. `DecimalArray::try_new` accepts only the canonical width. + +Decimal kernels can move onto integer kernels through the zero-copy view at any point. + Wave 0, small and independent: 1. Strict decoding: `DecimalType::try_from(raw)` in the decimal deserializer (`vortex-array/src/arrays/decimal/vtable/mod.rs:170`) and `PType::try_from(raw)` in the protobuf dtype reader (`vortex-array/src/dtype/serde/proto.rs:36`). Part of N10. -2. N11: require `values_type >= canonical width` in `DecimalData::validate`. Make DBP assembly - widen a narrow MSP. Stop the btrblocks v1 path from writing narrowed `vortex.decimal` nodes. - Widen legacy narrow nodes on read. Add compat fixtures for narrow, wide, and Arrow-style - widths first. The `patch` panic, the fill-null upcast, and the "does not fit" constant tests - become unreachable for same-dtype inputs and can go. Update tests that build narrow arrays - on purpose, for example `decimal_narrow_storage` - (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). -3. Add `DecimalDType::storage_type()`, returning `DecimalType` until N9 lands, and route the 25 - `smallest_decimal_value_type` calls through it so that the width function has one owner. +2. Step 1: the nullability fix in the DBP cast rule. Today the rule casts the MSP for every + nullability change + (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`). PR #10230 makes + it decline a change that the validity cannot make trivially. +3. Step 2: producers take the width from the precision, which is the first part of N11. Add + `DecimalDType::values_type()` and route the 25 `smallest_decimal_value_type` calls through + it, so that the width function has one owner. 4. N1: the urgent extension changes U1 to U8. +5. Merge the integer `Narrow` PRs #10223 and #10332 as they are. Their only integer-only check is + the private `validate_dtypes`, so decimal support stays additive. + +Wave 1, one canonical form: + +6. Decide DQ3, then merge PR #10333. It adds `vortex.narrow` to the draft edition + `preview2026.10.0`, with one metadata byte that holds the child `PType`. +7. Add compat fixtures for narrow, wide, and Arrow-style widths, and for the narrow 128-bit form + (section 3.5). +8. Step 3: `Narrow` over decimals (N28). When `validate_dtypes` opens for decimals, the compare + kernel and the sum pushdown get decimal code in the same change. +9. Step 4: the compressor and DBP decoding return `Narrow`. `narrowed_decimal` goes away. +10. Step 5: readers convert old buffers. Old narrow MSPs, and old narrow leaves of 64 bits or + less, become DBP arrays with an integer `Narrow` as the MSP, without a copy. An old I128 leaf + at precision 39 to 76 needs a widening copy. +11. Step 6: `DecimalArray::try_new` accepts only the canonical width, which completes N11. The + `patch` panic, the fill-null upcast, and the "does not fit" constant tests become unreachable + and can go. Update tests that build narrow arrays on purpose, for example + `decimal_narrow_storage` (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). +12. N12 for precision 1 to 18, at any point in this wave. + +Wave 2, foundations, in parallel with wave 1: + +13. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. +14. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for + the Timestamp unit casts of N4. +15. N4: cast rules (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. +16. N2, then N3: the rewrite rules that types own, then derives for one dtype. Use the decimal + rules of section 3.4 as acceptance tests. +17. N23: plugin registries for DuckDB and DataFusion type mapping. + +Wave 3, new types: + +18. N9 in the order of section 5.3, with all four wide types. Recommendation (DQ5): user-visible + columns only behind an edition. +19. N10: the core dtype edition kind and the protobuf oneof. +20. N17: FixedSizeBinary, after DQ7. +21. N5: the value check in `ExtensionArray::try_new`, and the precision check in + `DecimalArray::try_new`, starting at import boundaries. + +Wave 4, decimal kernels: + +22. N12 for precision 19 to 76. +23. N13: rules over the view. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, + `decimal/utils.rs`, and the decimal aggregates. +24. N14, which is optional. +25. N18 and N27. + +Wave 5, optional: + +26. N16, then N15 if the built-in type and an extension type become identical. Revisit N19 and + N20 with the results of N22. -Wave 1, foundations: - -5. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. -6. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for - the Timestamp unit casts of N4. -7. N4: cast hooks (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. -8. N2, then N3: settle the rewrite design, then add derives for one dtype. This is phase 2 of the - extension notes. Use the decimal rewrites of section 3.4 as acceptance tests. A checked - `ext_wrap` in N2 calls the default `validate_array`, which phase 1 of the extension notes adds. - N5 adds the rest later. -9. N23: plugin registries for DuckDB and DataFusion type mapping. - -Wave 2, new types: - -10. N9 in the order of section 5.3, without user-visible wide columns. -11. N10: the core dtype edition kind and the protobuf oneof. -12. N17: FixedSizeBinary, after the wire representation decision. -13. N5: `validate_array`, starting with decimal precision at import boundaries. - -Wave 3, decimal restructure: - -14. N12: the wrapper and the legacy plugin. -15. N13: rewrites. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, `decimal/utils.rs`, - the decimal aggregates, and the per-width builder. -16. N14: DBP under the wrapper, optional. -17. N18 and N27. - -Wave 4, API unification: +## 9. Open questions -18. N16, then N15. Revisit N19 and N20 with the results of N22. +### 9.1 Questions in the decisions file -## 9. Open questions +The decisions file lists these questions as open. Each answer below is a recommendation of these +notes, which the maintainer has not confirmed. -1. Are `Primitive(I128)` and `Primitive(I256)` user-visible columns from the start, or storage - only until an engine needs them? DuckDB HUGEINT is the only current consumer. -2. Does `is_int()` mean any integer or a lane integer? Section 5.3 proposes to remove it, so that - each of the 53 call sites chooses. -3. Are `widen` and `checked_narrow` lazy `Cast` scalar-function arrays, or a dedicated in-memory - encoding? -4. Which crate owns the decimal and DBP serialized ids once DBP moves under the wrapper: - `vortex-array` or `encodings/decimal-byte-parts`? -5. Does FixedSizeBinary use a new core wire tag, or an extension descriptor over - `FixedSizeList` that old readers can still read? -6. After N13, is the move of `DType::Decimal` into a Rust extension still worth its API cost? -7. Does Vortex implement decimal arithmetic and comparison across different `(p, s)` pairs? The - alternative keeps the rule of equal dtypes, and engines coerce first. -8. Do file reads check decimal precision, or only external imports? Zone-map minimum and maximum - statistics can make the file check cheap. The extension notes leave the same question open for - value checks on file reads. -9. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ1 | Which width does each precision get? | The smallest signed width that holds the digits. This is the existing mapping (`vortex-array/src/dtype/decimal/types.rs:46-56`), with an I8 minimum. Section 3.1 lists the cost of other functions | +| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes. Integers narrow by width (issue #10020). Decimals narrow by precision | +| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless decimal `Narrow` lands before the edition of PR #10333 ships in a release | +| DQ4 | Does the dtype of a DBP array fix the type of its MSP? | Yes. A reader wraps an old narrow MSP in an integer `Narrow` of the fixed type | +| DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. First compare, arithmetic, cast, sum, min, max, hash and structural operations | +| DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes (X3) | +| DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | +| XQ1 | Which arrays does the rule of X5 pull out, and what stays under the wrapper? | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. For `Decimal`, the wire forms already have this shape (section 3.3) | +| XQ2 | Where do the rules of X4 live? | On `ExtVTable`, written over `ReduceNode`. The built-in `Decimal` owns the same kind of rules | +| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, and never panic. For decimals, these notes read this as a width check on conversion, with trust for the precision | +| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. The answer affects N18 | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error. Lossy steps such as `trunc` and `round` are separate expressions. The decimal-to-integer rule of section 3.4 keeps the current truncation until CQ1 is settled | + +### 9.2 Other open questions + +1. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist (`vortex-array/src/dtype/decimal/mod.rs:47`). Tests use -128. DuckDB rejects negative scales, and Arrow allows them. -10. Does the storage width function keep the I8 minimum, or align with an engine? Arrow device - export uses an I32 minimum, and DuckDB an I16 minimum. The choice is in-memory only, so it - can change later. -11. PRs #10467 and #10468 leave two decisions open. One is the order of intervals: field-wise as - in Arrow, or normalized as in DuckDB. The other is the edition membership of - `vortex.duration` and `vortex.interval`. -12. When do `U128` and `U256` become worth adding? -13. Is the wrapper on top of generic encodings (`Constant`, `Dict`, `Chunked`, `Masked`, - `Sparse`)? The extension notes recommend it (`design-notes.md`, section 6), but no maintainer - decision exists. The alternative keeps generic encodings above the wrapper, as - `ExtensionConstantRule` does today. +2. PRs #10467 and #10468 leave two decisions open. One is the order of intervals: field-wise as + in Arrow, or normalized as in DuckDB. The other is the edition membership of + `vortex.duration` and `vortex.interval`. +3. Does `is_int()` mean any integer or a lane integer? Section 5.3 proposes to remove it, so that + each of the 53 call sites chooses. +4. When DBP decodes an old narrow MSP, does it compute value bounds and return a decimal `Narrow`, + or does it always widen? Statistics of the MSP can make the bounds cheap. ## 10. Evidence log and limits Code read at commit `5fc5cb8`: the files cited in sections 2 to 6, plus the supporting inventories. Every count in this report comes from `grep` over the workspace, excluding `target`. +The maintainer decisions of 2026-10-19 are recorded in [`decisions.md`](../../decisions.md). +Sections 1 and 3 to 9 follow them. Section 3 replaced a wrapper design with one integer child, +which D2 rejects. + RFC drafts read from the `vortex-data/rfcs` repository: - Branch `ngates/types`: `rfcs/0062-value-types-and-representations.md`, read in full. @@ -1407,6 +1545,11 @@ GitHub, read only: | [PR #9886](https://github.com/vortex-data/vortex/pull/9886) | DataFusion literal for a UUID column, closed | | [PR #10467](https://github.com/vortex-data/vortex/pull/10467) | Duration extension type. The files and the comments | | [PR #10468](https://github.com/vortex-data/vortex/pull/10468) | Interval extension type. The file list and the new `interval.rs` | +| [PR #10223](https://github.com/vortex-data/vortex/pull/10223) | `Narrow` integer encoding, draft. The description, the diff, and the comments, read on 2026-10-19 | +| [PR #10332](https://github.com/vortex-data/vortex/pull/10332) | Arithmetic and selection through `Narrow`, draft, stacked on #10223. Same depth, same date | +| [PR #10333](https://github.com/vortex-data/vortex/pull/10333) | `Narrow` through compression, and `vortex.narrow` in `preview2026.10.0`, draft, stacked on #10332. Same depth, same date | +| [PR #10230](https://github.com/vortex-data/vortex/pull/10230) | Canonical decimals with one integer child, draft, stacked on #10333. The description, the diff, and the comments, read on 2026-10-19 | +| [Epic #10020](https://github.com/vortex-data/vortex/issues/10020) | Narrow integer encoding. The title and the first part of the body, read on 2026-10-19 | Verified outside the repository: @@ -1421,9 +1564,9 @@ Verified with a run: `../../extension-types/agents/probe_followup.rs` ran in a d - Probe R1 checks the decimal to `f32` cast of section 1.1. The array path fails with `No CastKernel to cast canonical array vortex.decimal from decimal(10,2) to f32`. The row path returns `[1f32, 2.5f32]`, and the scalar cast returns `1f32`. -- Probe R6 checks the pushdown rules of section 3.3. Existing rules push `ext_storage` into - dictionary values and chunks. `ext_storage` decodes a `Masked` input and fails on a `Sparse` - input with extension values. +- Probe R6 checks the `ext_storage` pushdown rules that section 3.3 cites. Existing rules push + `ext_storage` into dictionary values and chunks. `ext_storage` decodes a `Masked` input and + fails on a `Sparse` input with extension values. A script checked that every `file:line` reference in this report points at an existing range, and the first line of each range was reviewed. The external crate references above were checked by @@ -1437,6 +1580,9 @@ Not verified, because no run covered them: - The alignment of `i128` on targets other than x86_64 and aarch64, for example wasm32. - Whether the fuzzer reports in issue #5820 that end in a null decimal sum come from narrow storage. This report does not rely on that. +- Whether a decimal `Narrow` reaches the speed of the integer `Narrow` kernels. No benchmark + covers it. The benchmark comment on PR #10230 measures a different design. +- The cost of the copies at Arrow import and in readers that convert old buffers. ## Appendix A: byte order script diff --git a/research/decisions.md b/research/decisions.md new file mode 100644 index 00000000000..5dae6193b0c --- /dev/null +++ b/research/decisions.md @@ -0,0 +1,104 @@ + + +# Decisions + +This file records the decisions that the maintainer made while reviewing these notes on +2026-10-19. It also lists the questions that stay open. The other documents follow this file. If a +document disagrees with it, this file wins. + +"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that +the maintainer has not confirmed yet. + +## Extension types + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| X1 | Extension types are opaque by default. No code applies storage semantics to an extension value unless the type opts in | Storage semantics give wrong results for most types (EXT-Q2, EXT-C2) | +| X2 | A derive (`eq` or `ord`) is a property of one extension dtype. It never applies across dtypes | Two dtypes with different metadata are different types | +| X3 | Engines convert operands to a common dtype. Vortex runs only operations whose operand dtypes already match | Engines already do this in their planners, and one owner avoids two rule sets that disagree | +| X4 | A type implements its behavior as rewrite rules on arrays. The rules are registered with the type, together with its compression scheme and the other parts of the type. There is no `ext_wrap` expression | A rule builds its result array directly, so a separate wrap expression adds nothing | +| X5 | A rewrite rule pulls generic structural arrays, such as `Constant` and `Dict`, out from under the extension wrapper | Code that looks for a constant or a dictionary then finds it at the top, without any knowledge of extension types | +| X6 | Extension values inside a struct, list or map are handled by removing extension layers at every depth. Nested kernels get no extension code | Kernels then see storage dtypes only | +| X7 | `ExtensionArray::try_new` checks every non-null value. A new `ExtensionArray::new_unchecked` skips the check for kernels that keep legal values | Today arrays are never checked (EXT-V1). This is a bug | +| X8 | API changes are acceptable where correctness needs them, for example fallible constructors and private enum variants | Correctness and clear abstractions matter more than the current API | +| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types | + +### Open + +| ID | Question | Recommendation | +| --- | --- | --- | +| XQ1 | Which arrays does the rule of X5 pull out, and what stays under the wrapper? | Pull out the persistent structural encodings: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. Slices, filters and scalar functions stay under the wrapper. A canonical extension array is a wrapper over canonical storage | +| XQ2 | Where do the rules of X4 live? | On the `ExtVTable`, so that they travel with the dtype and need no session lookup. Write them over `ReduceNode`, so that they also rewrite bound expressions for pruning | +| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, trust semantic constraints, and never panic on an illegal value. Offer a strict read mode that checks values | +| XQ4 | Which values can sit in null slots? | Any value. Accessors that take a child out of its parent mask it with the validity of the parent. This also allows a storage encoding to store nulls as a reserved value | +| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again | +| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet | + +## Decimal and new physical types + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| D1 | `Decimal` stays a built-in type. It behaves like a hardcoded extension type: opaque by default, with derives for one dtype and rules that the type owns. If the two become identical, a later change can remove the built-in type | A move to an extension type adds no capability, and the built-in type can follow the same rules | +| D2 | A canonical decimal array is a leaf with one buffer and validity. It has no data child | A leaf is simpler than a wrapper over an integer child, and it is the same as the `vortex.decimal` wire node | +| D3 | Each decimal dtype has exactly one canonical encoding. The precision fixes the width of the buffer | Today one dtype has six physical forms, which causes the crashes in issue [#5820](https://github.com/vortex-data/vortex/issues/5820) | +| D4 | A `Narrow` encoding stores a decimal column as a child with a lower precision and the same scale | Without it, D3 forces the full width on every column, which costs performance | +| D5 | Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types | Wide decimals need an integer view, and wide integers are useful as columns | +| D6 | FixedSizeBinary becomes a core type for opaque bytes of a fixed length, such as UUIDs and hashes | Its defining operations (equality, hashing, byte order, cheap scalars) are core operations. Arrow and Parquet have it natively | + +### Open + +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ1 | Which width does each precision get? | The smallest signed width that holds the digits, which is the existing mapping in `vortex-array/src/dtype/decimal/types.rs:46-56` | +| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)). Decimals narrow by precision | +| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless decimal `Narrow` lands before the edition of PR #10333 ships in a release. A second wire ID needs no change to the open PRs | +| DQ4 | Does the dtype of a DecimalByteParts array fix the type of its most significant part? | Yes. Narrowing then happens only through `Narrow` and integer encodings. A reader wraps the narrow part of an old file in an integer `Narrow` of the fixed type, without a copy | +| DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. A small kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations | +| DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes | +| DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | + +## Casts + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| C1 | A cast between two dtypes exists only if a rule for that pair exists. Vortex ships rules for common pairs of built-in types, as the Rust standard library ships `From` implementations. Reading storage as another type is never a cast | Opt-in conversions with good defaults. Today casts reinterpret storage and disagree between paths | + +### Open + +| ID | Question | Recommendation | +| --- | --- | --- | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error, like `TryFrom`. Lossy steps such as `trunc` and `round` are separate, named expressions | +| CQ2 | How do struct casts match fields? | By name, with unique field names (issue [#6900](https://github.com/vortex-data/vortex/issues/6900)). Extra source fields return an error | + +## The open `Narrow` pull requests + +This review read the draft stack on 2026-10-19. + +- PR [#10223](https://github.com/vortex-data/vortex/pull/10223) and PR + [#10332](https://github.com/vortex-data/vortex/pull/10332) can merge as they are. Their only + integer-only check is the private `validate_dtypes`, so decimal support stays additive. When + that check opens for decimals, the compare kernel and the sum pushdown need decimal code in the + same change. +- PR [#10333](https://github.com/vortex-data/vortex/pull/10333) adds `vortex.narrow` to the draft + edition `preview2026.10.0`, with one metadata byte that holds the child `PType`. Question DQ3 + decides it before the merge. The editions spec says that a change to what readers must + understand needs a new wire ID (`docs/specs/versioning/editions.md:118-121`). +- PR [#10230](https://github.com/vortex-data/vortex/pull/10230) gives decimals an integer child, + which D2 replaces. Split its reusable parts into smaller PRs, in this order: + 1. The nullability fix in DecimalByteParts. + 2. Producers take the width from the precision. + 3. `Narrow` over decimals. + 4. The compressor and DecimalByteParts decoding stop narrowing and return `Narrow` instead. + 5. Readers convert old buffers at other widths. + 6. `DecimalArray::try_new` accepts only the canonical width. + + Decimal kernels can move onto integer kernels through a zero-copy view at any point. diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md index 36f2a148a88..ddd5bfdc041 100644 --- a/research/extension-types/03-design.md +++ b/research/extension-types/03-design.md @@ -5,6 +5,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # 3. Design +This part follows [`../decisions.md`](../decisions.md), which records the decisions of the +maintainer. IDs such as X4 and XQ1 refer to that file. + ## Goal Extension types behave like Rust newtypes. Structural operations always work. A semantic operation @@ -15,13 +18,13 @@ value without that opt-in. ## Rules 1. A semantic operation on an extension type is rejected unless the type or a type-aware function - opts in. + opts in (X1). 2. Structural operations are always allowed. -3. One place decides how generic operations treat extension values: the binder. It runs when an - expression, a scalar function array or an aggregate is bound. -4. Below the binder, the kernels of generic semantic operations, statistics and pruning see - storage dtypes only. A function that declares an extension type can see that type. Whether - generic encodings carry extension dtypes depends on the physical form (open, see below). +3. Two kinds of rules decide how generic operations treat extension values: the derive rules of the + core and the rules that each type owns. Every node meets a typing check when code builds it, and + the rules run on every path. +4. After the rules run, the kernels of generic semantic operations, statistics and pruning see + storage dtypes only. A function that declares an extension type can see that type. 5. The extension type owns every conversion that changes meaning. There are no implicit casts. 6. An unknown extension type supports structural operations and `ext_storage` only. @@ -35,7 +38,7 @@ value without that opt-in. | Opaque, kernels per extension ID | Look up `(function, extension ID)` before the generic kernel | Computes results, but statistics and pruning learn nothing | | Type-owned encodings | Each extension type gets its own array encoding | Mixes logical type and encoding, and generic encodings still carry the type | | Full dtype plugin | The type defines its own arrays, scalars and compute | The most power and the most work, and storage becomes optional | -| **Opaque, bind-time rewrite** | The binder rewrites semantic operations into storage expressions | **Proposed** | +| **Opaque, rewrite rules** | Derive rules and rules that the type owns rewrite semantic operations into storage operations | **Chosen (X1, X4)** | The deciding property is the number of places that must understand extension semantics. With flags, every consumer must read the flag correctly. Today, three groups of consumers give three @@ -45,31 +48,37 @@ different answers to "is this type ordered?" (EXT-A1, EXT-B4): - `is_sorted` and `vortex-row` treat no extension type as ordered. - DuckDB pushdown treats every extension type except `Uuid` as ordered. -With bind-time rewrites, only the binder needs to understand extension semantics. +With rewrite rules, only the rules need to understand extension semantics. Unlike kernels, rules +also rewrite predicates, so statistics and pruning keep working on storage. ## Proposal -### Bind-time rewrite +### Typing and rewrite rules + +A call with an extension operand passes two steps: -When an operation is bound and an operand has an extension type, the binder asks the extension type -how to answer. The answer is a rewrite into an expression over storage, or a rejection: +1. Typing checks the call when code builds the node. It accepts four kinds of calls. They are + structural calls, calls of a type-aware function, calls that a derive covers, and calls that a + rule of the type covers. Everything else is an error. +2. Rewrite rules replace an accepted semantic call with operations on storage. ```text ts_ms < lit(Timestamp(ms) 1500) => ext_storage(ts_ms) < 1500i64 -min(ts_ms) => ext_wrap(min(ext_storage(ts_ms)), Timestamp(ms)) -cast(ts_ms, Timestamp(ns)) => ext_wrap(checked_mul(ext_storage(ts_ms), 1_000_000), - Timestamp(ns)) +min(ts_ms) => min(ext_storage(ts_ms)), and the rule builds a Timestamp(ms) + scalar from the result +cast(ts_ms, Timestamp(ns)) => Extension(Timestamp(ns), + checked_mul(ext_storage(ts_ms), 1_000_000)) json_a = json_b => rejected, because Json derives no equality ``` -The scalar path and the array path then evaluate the same expression, so the results of EXT-P1 -cannot differ. Pushdown, pruning and compression keep working, because the rewritten expression is -an ordinary storage expression. +The rules run in the optimizer and in the executor, so the row path and the array path apply the +same rewrite. A missing rule gives an error, never a wrong result. Pushdown, pruning and +compression keep working, because the rewritten operations are ordinary storage operations. ### Derives for one dtype A derive says that a semantic operation on values of one extension dtype is the same operation on -their storage values. It is a property of one dtype: +their storage values. It is a property of one dtype (X2): ```rust /// Storage behavior that values of `dtype` inherit, like `#[derive]` on a newtype. @@ -80,101 +89,116 @@ fn derives(dtype: &ExtDType) -> Derives { } ``` -`Derives` has two flags. `EQ` means that storage equality is value equality, which also makes the -storage hash valid. `ORD` means that storage order is value order. +`Derives` has two flags. `eq` means that storage equality is value equality, which also makes the +storage hash valid. `ord` means that storage order is value order. A derive applies only when every operand in a derived position has the same extension dtype: - A comparison needs two such operands, and `between` needs three. - A unary aggregate, such as `min` or `is_sorted`, reads the flags of its one input. - A struct or list that holds extension values inherits a flag only if every nested extension - type declares it. + type declares it. The derive rule then removes every extension layer at every depth, so the + kernel sees only storage (X6). So `ts < 5i64` stays rejected, because only one side is an extension. | Type | Derives | | --- | --- | -| `Timestamp` | `EQ`, `ORD` | -| `Date`, `Time` | `EQ`, `ORD` | -| `Decimal`, if it follows this design | `EQ`, `ORD` | -| `Uuid` | `EQ`, plus `ORD` if byte order is the chosen order | -| `Vector`, `FixedShapeTensor` | `EQ` | +| `Timestamp` | `eq`, `ord` | +| `Date`, `Time` | `eq`, `ord` | +| `Decimal`, which stays built-in but follows the same rules (D1) | `eq`, `ord` | +| `Uuid` | `eq`, plus `ord` if byte order is the chosen order | +| `Vector`, `FixedShapeTensor` | `eq` | | `Json`, WKB, native geometry | None | | Unknown type | None | ### Operations across dtypes Two dtypes with different metadata are different dtypes, for example `Timestamp(ms)` and -`Timestamp(ns)`. A derive never applies across them, and the binder rejects such an operation by -default. A type can support one only through a rule that the type owns: - -- A compatibility rule: the two dtypes compare by storage as they are. An example is two zoned - timestamps with one unit. -- A coercion: a cast hook of the type converts one operand first. -- A rewrite of the whole call: for example, a comparison of two decimals with different scales - rescales one side first. +`Timestamp(ns)`. A derive never applies across them, and Vortex rejects such an operation (X3). +An engine converts the operands to one dtype first, as its planner already does for SQL. A user of +the Rust or Python API writes an explicit cast. ### Function declarations Each function declares how it uses an operand that contains an extension type: - Structural: it moves or selects values, for example `fill_null` or `case_when`. -- Derive: it needs `EQ` or `ORD`, for example `=` or `<`. +- Derive: it needs `eq` or `ord`, for example `=` or `<`. - Type-aware: it implements the semantics of named extension types itself, for example `replace_time_zone` for `Timestamp`. -- Nothing: the default. The binder must rewrite or reject the call. +- Nothing: the default. A rule of the type covers the call, or typing rejects it. -Only the function owner declares. The binder never infers a declaration from the input types that a +Only the function owner declares. Typing never infers a declaration from the input types that a function accepts, because `Binary` accepts extension operands today. -### Casts +An aggregate with a derive also declares whether its result is a value of its input, a bound on its +input, or neither (XQ5, open). Only a value is wrapped again. For example, `min` returns a value, +and `bounded_max` over `Utf8` storage returns a truncated bound, which stays a storage value. + +### Rules that the type owns -The extension type owns its casts. The binder tries the cast hook of the source type, then the cast -hook of the target type, then rejects the cast. The hooks return expressions, not closures, so the -result can be pushed down and evaluated in the same way on both paths. This is the design in issue -[#8570](https://github.com/vortex-data/vortex/issues/8570), with expressions in place of closures. +A type implements its own semantics as rewrite rules on arrays (X4). The rules are registered with +the type, together with its compression scheme and the other parts of the type. Cast rules convert +the type to another dtype, or another dtype into the type. Call rules cover other functions and +aggregates, such as `sum` over a decimal-like type. -Two explicit expressions cover everything else: +A cast exists only if a rule covers it (C1). This is the design of issue +[#8570](https://github.com/vortex-data/vortex/issues/8570), with rules in place of closures. + +No `ext_wrap` expression exists. A rule that produces extension values builds the extension array +itself. It uses `ExtensionArray::new_unchecked` when legal input gives legal output, and +`ExtensionArray::try_new` otherwise. So no user expression can build a wrapper without a check. + +Two explicit expressions remain: - `ext_storage(x)` reads the storage. It exists today. It never checks values. -- `ext_wrap(x, dtype, mode)` builds an extension value from storage. It is new. A checked wrap - validates the values. An unchecked wrap skips the check, and its result stays unvalidated. +- `ext_storage_deep(x)` removes every extension layer at every depth. It is new. The derive rules + use it for operands that hold nested extension values, for example in a comparison of + `struct{ts: Timestamp}` values. -A simplification can remove a wrap only if it keeps every error. `ext_storage(ext_wrap(y))` -simplifies to `y` only for an unchecked wrap. With a checked wrap, `y = 0` under a type that rejects -zero fails before the simplification and succeeds after it. +Where the rules live is open (XQ2). These notes recommend a method on `ExtVTable`, so that the +rules travel with the dtype and need no session lookup. The existing session registry for array +kernels is keyed by encoding, and every extension array has the same encoding id. Also, 83 plain +`.optimize()` calls outside tests never consult that registry. ### Scalar identity `Scalar` implements `PartialEq`, `Eq` and `Hash` as identity: the same dtype and the same stored value. These traits serve Rust code, such as maps and tests, not SQL semantics. `PartialOrd` returns `Equal` for identical scalars, and an order only for ordered dtypes. An extension dtype is -ordered only if it declares `ORD`. This keeps the Rust rule that `==` and `partial_cmp` agree. -Expression comparisons follow the binder, not these traits. +ordered only if it declares `ord`. This keeps the Rust rule that `==` and `partial_cmp` agree. +Expression comparisons follow the derive rules, not these traits. -### Physical form (open) +### Physical form -Opacity does not decide where the wrapper sits relative to generic encodings such as `Constant`, -`Dict` and `Chunked`. The binder works with either placement. +A rewrite rule pulls generic structural arrays, such as `Constant` and `Dict`, out from under the +extension wrapper (X5). Code that looks for a constant or a dictionary then finds it at the top, +without any knowledge of extension types. -| | Wrapper on top | Generic encodings on top | -| --- | --- | --- | -| Shapes of one column | One | Several, as today | -| Current code | The compressor uses it for non-constant columns, unless a type-specific encoding such as `DateTimeParts` wins | The compressor and `ExtensionConstantRule` use it for constants | -| Main cost | Reverse `ExtensionConstantRule`, and look through the wrapper wherever code looks for constants | Push `ext_storage` through `Masked` and `Sparse`, and keep extension code in each generic encoding | -| Wire | No new form for extension columns. `Decimal` translates on write | Memory and wire forms match | +Which arrays the rule pulls out is open (XQ1). These notes recommend this split: -These notes recommend the wrapper on top, after the binder lands. No maintainer decision exists. -[`agents/design-notes.md`](agents/design-notes.md), section 6, has the full comparison. +- Above the wrapper: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. +- Below the wrapper: the canonical storage, storage encodings such as BitPacked or FSST, and slices, + filters and scalar functions that wait to run. +- A canonical extension array is a wrapper over canonical storage. So a canonical array never hides + a pending computation or a compressed child. + +Each structural encoding gets one method that rebuilds it with its value children passed through a +function. That method moves the wrapper, pushes `ext_storage` down, and decodes a structural +encoding that holds extension values. `RunEnd` and `Sparse`, which fail on extension values today, +get support from the same method. Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an `i64` encoding, so its compare kernel works on storage. The compressor chooses it from the extension metadata. +[`agents/design-notes.md`](agents/design-notes.md), section 6, has the details. + ### Statistics -Zone maps and file statistics store storage statistics for extension columns. The rewrites in the -binder produce predicates on `ext_storage(col)`, and pruning uses the storage statistics for them. +Zone maps and file statistics store storage statistics for extension columns. The derive rules +produce predicates on `ext_storage(col)`, and pruning uses the storage statistics for them. `vortex-spatial` already prunes in this way, by hand. ### Validation @@ -183,15 +207,27 @@ Three kinds of validation exist: - Dtype validation checks the metadata and the storage dtype, once for each dtype. - Storage-shape validation checks that storage has the storage dtype, for arrays and scalars. -- Value validation (`validate_array`) checks each value, at trust boundaries. +- Value validation (`validate_array`) checks that each non-null value is a legal value of the + type. + +Every extension array that Vortex builds holds only legal values (X7). `ExtensionArray::try_new` +checks the values in one batch with `validate_array`. Today it checks only the storage dtype, which +is a bug (EXT-V1). Kernels that keep legal values use a new `ExtensionArray::new_unchecked`, so they +pay nothing. + +The default `validate_array` applies the existing scalar check of the type to each value. A type +can override it with a faster check, for example a range check on the minimum and maximum of a +`Timestamp` column. -Each extension array is validated or unvalidated. Only a successful value check makes an array -validated, and structural operations keep that state. File reads without a value check give -unvalidated arrays. A read of a row from an unvalidated array checks the value, and returns an -error for an invalid value. Reads never panic. +Two questions stay open: -The default `validate_array` checks each value with the existing scalar check of the type. It -skips rows under a null parent, because builders pad those rows. +- Do file reads check values (XQ3)? These notes recommend that decoding checks only what memory + safety needs, as it already does for UTF-8, and trusts the rest. A strict read mode checks values. + Because files are trusted, building a scalar from an array always checks the value and returns an + error for an illegal one. No read panics. +- Which values can sit in null slots (XQ4)? These notes recommend any value. A child that leaves + its parent is masked with the validity of the parent. This keeps Arrow imports free of copies, + and it also allows a storage encoding to store nulls as a reserved value. ### Nullability @@ -208,35 +244,39 @@ plugins like the Arrow registry. Then a new extension type does not need code in ## Costs and risks -- Binding must be mandatory. The array API builds scalar function arrays directly, so the rewrite - must happen when such an array is built, not in an optional optimizer pass. Aggregates that code - binds directly, such as zone-map statistics, need the binder too. +- Typing must cover every path that builds a node. The types enforce it: `BoundExpression` loses + its public variants, and `Cast::new` becomes fallible in the same step as typing. The design + accepts these API changes for correctness (X8). +- The rules must run on every path: the static optimizer, the session-aware optimizer, and the + executor. A test checks that all paths give the same result. - `DateTimeParts` changes from a `Timestamp` encoding to an `i64` encoding. That is a file format change for the encoding, and it needs an edition. +- Old readers cannot decode `RunEnd` and `Sparse` over extension values. So writers keep the old + shape of those two until an edition allows the new one. - Engines must send extension literals for extension columns. Today the lone-extension compare rule can hide literals that have the storage type. -- After binding, plans show storage operations, so plans are harder to read. +- After the rules run, plans show storage operations, so plans are harder to read. - The new scalar order must land with or after the change that rejects `min` and `max` without - `ORD`. Otherwise `min` and `max` can panic, for example in zone maps of `Json` columns. This - risk comes from reading the code, not from a run. + `ord`. Otherwise `min` and `max` can panic, for example on a chunked `Json` column. This risk + comes from reading the code, not from a run. ## Open questions -- Open question: where does binding happen for the array API? One option is a rewrite when the - array is built, with hooks on `ExtVTable`, so no session is necessary. The other option is a - mandatory rewrite pass that also covers the row path. -- Open question: can a crate register a rewrite or a declaration for a function and a type that it - does not own? -- Open question: is the wrapper on top of generic encodings? These notes recommend it. The - alternative keeps generic encodings above the wrapper, as `ExtensionConstantRule` does for - constants today. -- Open question: does Vortex coerce operations across dtypes or metadata, or do engines coerce - first? Rust and the built-in `Decimal` reject such operations today. SQL coerces them. -- Open question: are `EQ` and `ORD` enough? A refinement over a storage dtype can need a third flag - that allows comparisons with plain storage values. -- Open question: do file reads check values by default? -- Open question: existing files store min and max for extension columns as extension scalars. - Readers can convert them to storage values, but the switch needs a compatibility decision. -- Open question: is the nullability contract enough, or does nullability move onto the extension - dtype? -- Open question: does `Uuid` order by bytes, or have no order? +The full list with recommendations is in [`../decisions.md`](../decisions.md). The questions for +extension types are: + +- XQ1: which arrays the rule of X5 pulls out, and whether canonical form requires canonical storage. +- XQ2: where the rules of a type live. A related question is whether a crate can register a rule + for a function and a type that it does not own. +- XQ3: whether file reads check values. +- XQ4: which values can sit in null slots. +- XQ5: how an aggregate declares its result. +- XQ6: whether `Uuid` orders by bytes, or has no order. + +Older questions also stay open: + +- Are `eq` and `ord` enough? A refinement over a storage dtype can need a third flag that allows + comparisons with plain storage values. +- Existing files store min and max for extension columns as extension scalars. Readers can convert + them to storage values, but the switch needs a compatibility decision. +- Is the nullability contract enough, or does nullability move onto the extension dtype? diff --git a/research/extension-types/04-urgent-changes.md b/research/extension-types/04-urgent-changes.md index a6f2842d180..492c6a05702 100644 --- a/research/extension-types/04-urgent-changes.md +++ b/research/extension-types/04-urgent-changes.md @@ -15,7 +15,7 @@ fits in a small PR with a regression test. Finding IDs refer to | U2 | Give compare one type rule | EXT-P1, EXT-Q1, EXT-Q3 | `ts < 1500` fails when it is built | | U3 | Return errors from reads | EXT-V1 | No panic when a row is read | | U4 | Make nullability changes infallible | EXT-N1 | No panic in `take`, `mask` or `min_max` | -| U5 | Fix the built-in temporal validation | EXT-V4, EXT-V5 | A malformed dtype returns an error | +| U5 | Fix the built-in temporal validation | EXT-V4, EXT-V5, EXT-V8 | A malformed dtype or an out-of-range value returns an error | | U6 | Check the storage dtype of arrays and scalars | EXT-V2, EXT-V7 | An array or a scalar with the wrong storage is rejected | | U7 | Stop invalid zero values | EXT-V3 | No invalid scalars | | U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output | @@ -62,15 +62,22 @@ A test for each engine confirms the other cases. ## U3. Errors from reads -- Row reads of an extension array and `ExtensionConstantRule` build the extension scalar with - `try_new` and return its error (`vortex-array/src/arrays/extension/vtable/operations.rs:29`, - `vortex-array/src/arrays/extension/compute/rules.rs:39`). +- Row reads of an extension array build the extension scalar with `try_new` and return its error + (`vortex-array/src/arrays/extension/vtable/operations.rs:29`). +- `ExtensionConstantRule` builds the extension scalar with `try_new` + (`vortex-array/src/arrays/extension/compute/rules.rs:39`). If the check fails, the rule returns + `Ok(None)`. The array stays `Extension(Constant)`, and a read of a row returns the error. A rule + that returns the error makes optimization fail, so `ext_storage` and structural operations on + the array also fail. - `Scalar::extension_ref` gets a fallible variant, and read paths use it. ## U4. Nullability changes cannot fail - `ExtDType::try_with_vtable` validates the dtype with both nullabilities. After that, `with_nullability` cannot fail and does not validate again. +- `with_nullability` builds the new dtype directly. Today it calls `try_with_vtable` + (`vortex-array/src/dtype/extension/typed.rs:91-98`). With the first change, that call validates + both nullabilities on every nullability change. - The documentation of `ExtVTable::validate_dtype` states that its result must not depend on nullability. @@ -83,6 +90,9 @@ when the dtype is built is the better place. (`vortex-array/src/extension/datetime/date.rs:104`, `time.rs:105`). - `Timestamp::validate_dtype` rejects the `Days` unit and unknown time zones (`vortex-array/src/extension/datetime/timestamp.rs:198-204`). +- `Timestamp::unpack_native` uses the `try_` setters of `jiff::Span` and returns their error + (`vortex-array/src/extension/datetime/timestamp.rs:214-229`). Today an out-of-range value + panics (EXT-V8). ## U6. Storage checks for arrays and scalars @@ -91,9 +101,11 @@ when the dtype is built is the better place. - Scalar validation checks that an extension value matches the storage dtype, before it calls the check of the type (`vortex-array/src/scalar/validate.rs:100`, `:294`). This fixes EXT-V7. - `Scalar::try_new` converts the `U64` form of an `F16` value to `F16` - (`vortex-array/src/scalar/scalar_impl.rs:73`, `vortex-array/src/scalar/validate.rs:147-157`). - Then `==` stays symmetric for `F16` scalars (probe R10). This case is not specific to extension - types. + (`vortex-array/src/scalar/scalar_impl.rs:73`). Validation then rejects the `U64` form. Today the + fast path accepts it (`vortex-array/src/scalar/validate.rs:46-50`), and so does the detailed + check (`:147-157`). The proto decoder already converts the old form + (`vortex-array/src/scalar/proto.rs:343`), so only direct construction makes the `U64` form. Then + `==` stays symmetric for `F16` scalars (probe R10). This case is not specific to extension types. ## U7. Zero values diff --git a/research/extension-types/05-implementation.md b/research/extension-types/05-implementation.md index 7f8d75c8ac4..2e34640277d 100644 --- a/research/extension-types/05-implementation.md +++ b/research/extension-types/05-implementation.md @@ -5,32 +5,34 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # 5. Implementation +This plan follows [`../decisions.md`](../decisions.md). IDs such as X4 and XQ1 refer to that file. + ## Phases | Phase | Content | Depends on | | --- | --- | --- | | 0 | The urgent changes in part 4, and the path-consistency test helper | Nothing | | 1 | The `ExtVTable` contract with the default `validate_array`, a conformance harness, and three reference test types | Nothing | -| 2 | The binder and derives: `Derives`, `ext_wrap`, and rewrites for compare, min, max and sort | 0, 1 | -| 3 | Casts through type-owned hooks, and an audit of engine literals | 2 | -| 4 | The physical form, and storage statistics for extension columns | 2 | -| 5 | Array validation, and nullability on the extension dtype | 0, 1 | +| 2 | Typing, derives and rewrite rules: `Derives`, the derive rules for compare, min, max and sort, `ext_storage_deep`, and rules that the type owns | 0, 1 | +| 3 | Cast rules that the types own, and an audit of engine literals | 2 | +| 4 | The lift rule for structural arrays, and storage statistics for extension columns | 2 | +| 5 | Value checks in `ExtensionArray::try_new`, and nullability on the extension dtype | 0, 1 | | 6 | Authoring helpers, engine export plugins, and documentation | 2, 3 | ```mermaid flowchart LR - P0["0. Urgent changes"] --> P2["2. Binder and derives"] + P0["0. Urgent changes"] --> P2["2. Typing and rules"] P1["1. Contract and harness"] --> P2 - P2 --> P3["3. Casts"] - P2 --> P4["4. Physical form and statistics"] - P0 --> P5["5. Validation and nullability"] + P2 --> P3["3. Cast rules"] + P2 --> P4["4. Lift rule and statistics"] + P0 --> P5["5. Value checks and nullability"] P1 --> P5 P2 --> P6["6. Authoring and boundaries"] P3 --> P6 ``` Phases 0 and 1 can run in parallel. Phase 5 can run in parallel with phase 2. Phase 2 is the -critical step, because phases 3, 4 and 6 build on the binder. +critical step, because phases 3, 4 and 6 build on the rules. ## Phase 0: urgent changes @@ -45,63 +47,86 @@ critical step, because phases 3, 4 and 6 build on the binder. - Document the contract on `ExtVTable`: structural and semantic operations, the derive rules, validation, and nullability independence. - Add `validate_array` to `ExtVTable`, with a default that checks each value through - `validate_scalar_value`. Phase 2 needs it for a checked `ext_wrap`, and phase 5 builds on it. + `validate_scalar_value`. Phase 5 builds on it. - Add a conformance harness for `ExtVTable` implementations. It checks the metadata round trip, - that `validate_dtype` ignores nullability, the derive rules, cast round trips, and path - consistency. It also checks that a `validate_array` override agrees with the default. + that `validate_dtype` ignores nullability, and that a `validate_array` override agrees with the + default. - Add three reference test types: - - An opaque type that derives nothing. Every semantic operation on it must fail at bind time. - - A decimal-like type over `i64` that derives `EQ` and `ORD` for each dtype, with a rescale - rewrite across scales. A differential test compares its results with the built-in `Decimal`. + - An opaque type that derives nothing. Typing must reject every semantic operation on it. + - A decimal-like type over `i64` that derives `eq` and `ord` for each dtype. A differential test + compares its results with the built-in `Decimal`. - A refinement type that derives everything and adds a value check. +- The harness and the reference types grow with later phases. Phase 2 adds the derive rules and + the derives. Phase 3 adds cast round trips. -## Phase 2: binder and derives +## Phase 2: typing, derives and rules - Add `Derives`, `ExtVTable::derives`, and the erased forwarder on `ExtDTypeRef`. -- Add extension declarations to scalar and aggregate functions. Bind aggregates wherever code builds - them, including zone-map statistics. -- Add the `ext_wrap` expression. A checked `ext_wrap` calls `validate_array` from phase 1. -- Rewrite compare, `between`, min, max, `is_sorted` and sort keys at bind time. -- Make the row path of `ext_storage` read storage without a value check. -- Remove the code that applies storage semantics. That code is the extension compare kernel, the - extension branch of the nested comparator, the extension path of `min_max`, and storage order in - `ExtScalar`. -- Change `PartialOrd` for `Scalar` and the operators of `scalar_cmp` in the same step as `min` and - `max`. An earlier change of `PartialOrd` can make zone maps of `Json` columns panic. This risk - comes from reading the code. +- Add extension declarations to scalar and aggregate functions. An aggregate also declares whether + its result is a value of its input, a bound, or neither (XQ5). +- Extend the typing check in its four places: `BoundExpression::try_new_vec`, + `ScalarFnArray::try_new_from_slots`, `AggregateDTypes::try_new` and `ScalarFn::validate`. Make + the variants of `BoundExpression` private. Make `Cast::new` fallible, because typing rejects + casts that no rule covers. +- Add the derive rules for compare, `between`, min, max, `is_sorted` and sort keys. They rewrite + derived operands to `ext_storage`, or to the new `ext_storage_deep` for nested values. A rule + that returns a value of the input builds the extension scalar itself. There is no `ext_wrap` + expression (X4). +- Add the mechanism for rules that a type owns, and register them with the type. These notes + recommend a method on `ExtVTable`, so the static optimizer, the session-aware optimizer and the + executor all find the rules through the dtype (XQ2). Write the rules over `ReduceNode`, so that + they also rewrite bound expressions for pruning. +- Add the rule `ext_storage(Extension(s))` to `s`, so the row path of `ext_storage` reads storage + without a value check. +- Remove the code that applies storage semantics: the extension compare kernel, the extension + branch of the nested comparator and the extension path of `min_max`. `ExtScalar` keeps the + storage order only for types with `ord`. +- Change `PartialOrd` for `Scalar` in the same step as `min` and `max`. An earlier change of + `PartialOrd` can make `max` panic on a chunked `Json` column, or in DuckDB aggregate pushdown. + This risk comes from reading the code. - Give each type in the repository its derives (the table in part 3). -- Move the open PRs onto derives. #10408 becomes a derive check. #10246 and #10566 become aggregate - rewrites. +- Move the open PRs onto derives and rules. #10408 becomes a derive check. #10246 and #10566 + become rules that the types own. -## Phase 3: casts +## Phase 3: cast rules -- Implement the hooks from #8570 as rewrites. -- Add `Timestamp` unit casts with overflow checks, and unit casts for `Date` and `Time`. +- Implement the casts of #8570 as rules that the types own (C1). A cast exists only if a rule + covers it. +- Add `Timestamp` unit casts with overflow checks, and unit casts for `Date` and `Time`. A unit + cast builds its result with `ExtensionArray::new_unchecked`, because legal input gives legal + output. - Audit the DuckDB and DataFusion converters: every literal for an extension column must be an extension scalar. -## Phase 4: physical form and statistics - -- Implement the physical form that maintainers choose (part 3): - - With the wrapper on top, reverse `ExtensionConstantRule` and add rules that lift the wrapper - above `Constant`, `Dict`, `Chunked`, `Masked` and `Sparse`. Constant detection then looks - through the wrapper. - - With generic encodings on top, push `ext_storage` through `Masked` and `Sparse`. `Dict` and - `Chunked` already push it down. +## Phase 4: lift rule and statistics + +- Add the static rule that pulls structural arrays out from under the extension wrapper (X5). The + set of arrays is open (XQ1). These notes recommend `Constant`, `Dict`, `RunEnd`, `Sparse`, + `Chunked`, `Masked` and `Interleave`. +- Give each structural encoding one method that rebuilds it with its value children passed through + a function. Use it for the lift rule, for the pushdown of `ext_storage`, and to decode structural + encodings that hold extension values. `RunEnd` and `Sparse` get extension support this way. +- Remove the extension take rule, which builds `Extension(Dict(indices, storage))`. +- Make the canonical form of an extension array a wrapper over canonical storage, if maintainers + accept that recommendation (XQ1). +- Keep the old wire shape for `RunEnd` and `Sparse` over extension values until an edition allows + the new one. Readers convert old shapes with the lift rule. - Store storage statistics for extension columns. Readers convert existing extension-typed statistics to storage values. - Move `DateTimeParts` under the wrapper as an `i64` encoding. An edition gates the new form, and readers keep the old form. -## Phase 5: validation and nullability +## Phase 5: value checks and nullability -- Record whether each array is validated. Only a successful check, or a structural operation on - validated input, makes an array validated. -- Run value validation in checked constructors and imports. File reads validate when a read - option asks for it. +- Make `ExtensionArray::try_new` check values with `validate_array`, and add + `ExtensionArray::new_unchecked`. Move kernels, builders and rules that keep legal values onto + `new_unchecked` ([`agents/design-notes.md`](agents/design-notes.md), section 8). +- Imports from Arrow, DuckDB, FFI and Python call `try_new`. File reads follow the answer to XQ3. - Add faster overrides of `validate_array` where a type has one, for example `Timestamp`. -- Row reads from unvalidated arrays check each value. Row reads from validated arrays skip the - check. +- Building a scalar from an array keeps the check of U3, because file data can hold illegal values + under the recommendation of XQ3. +- Apply the answer to XQ4 to the rows under a null parent. These notes recommend that any value is + allowed there, and that accessors that take a child out of its parent mask it. - Optionally move nullability onto the extension dtype. ## Phase 6: authoring and boundaries @@ -114,8 +139,10 @@ critical step, because phases 3, 4 and 6 build on the binder. ## Test strategy - **Path consistency.** Every semantic operation gives the same answer on the whole array, on one - row, and with constant inputs. This also holds for unvalidated arrays with invalid values. -- **Opaque rejection.** The opaque reference type fails at bind time for every semantic operation. + row, and with constant inputs. +- **Rule coverage.** The same expression gives the same result through `optimize`, `optimize_ctx`, + execution without optimization, and the row path. +- **Opaque rejection.** Typing rejects every semantic operation on the opaque reference type. - **Scalar contract.** For generated scalars, `a == b` holds exactly when `partial_cmp(a, b)` returns `Some(Equal)`, and equal scalars have equal hashes. - **Differential parity.** The decimal-like reference type matches the built-in `Decimal` for every @@ -127,22 +154,23 @@ critical step, because phases 3, 4 and 6 build on the binder. - Phases 0 to 3 change behavior, not the file format. The release notes must list the behavior changes from part 4. -- Phase 4 changes the `DateTimeParts` format and the statistics dtype of extension columns. Both - need editions and read compatibility. -- Phase 5 adds an optional check when a file is read. It does not change the format. +- Phase 4 changes the `DateTimeParts` format, the statistics dtype of extension columns, and the + shape of `RunEnd` and `Sparse` over extension values. All three need editions and read + compatibility. +- Phase 5 can add a value check when a file is read, depending on XQ3. It does not change the + format. ## Relation to the Decimal and type roadmap The [Decimal notes](../decimal/README.md) place these phases in a larger graph of type work. Three points affect the order: -- The array-level fix for `Decimal` does not wait for this plan. The built-in `Decimal` wrapper - owns its rewrites, so it does not need the `ExtVTable` hooks. -- A move of `Decimal` into an extension type needs phases 0, 2 and 3 first. Without them, a cast - from a decimal to an integer returns the unscaled values. +- `Decimal` stays a built-in type (D1). Its fixes do not wait for this plan, but it follows the same + rules: derives for one dtype and rules that the type owns. +- `Decimal` and extension types share the lift rule of phase 4. A constant decimal column is + already `vortex.constant` with a decimal scalar, so the old decimal wire forms fit it. - Phase 3 and the Decimal work share one integer kernel family, `rescale` and `check_range`. `Timestamp` unit casts and decimal rescales both use it. -The Decimal rewrites are also good acceptance tests for the binder of phase 2. The decimal-like -reference type of phase 1 can run them against the built-in `Decimal`. Phase 4 and the Decimal -wrapper share the decision on the physical form. +The decimal-like reference type of phase 1 compares the rules of phase 2 with the built-in +`Decimal`. diff --git a/research/extension-types/README.md b/research/extension-types/README.md index 7d3cabe1daf..e0f5ce2d18c 100644 --- a/research/extension-types/README.md +++ b/research/extension-types/README.md @@ -20,13 +20,16 @@ That causes wrong results, panics, and code paths that disagree with each other. - On `develop`, the same cast or comparison can return a correct value, a wrong value, or an error. The outcome depends on the encoding of the input, and on whether one row or the whole array is read. -- The proposal: extension types are opaque by default. Semantics are decided once, when an operation - is bound, by a rewrite into storage operations. The kernels of generic operations, statistics - and pruning see only storage. +- The proposal: extension types are opaque by default. A typing check rejects every semantic + operation that no declaration covers. Rewrite rules turn the allowed ones into storage + operations. The kernels of generic operations, statistics and pruning see only storage. - Each extension dtype declares the storage behavior that it inherits, such as equality and order. - Operations across dtypes need explicit rules, and the design does not assume that they exist. -- The position of the wrapper relative to generic encodings is a separate decision, and it is - open. + Each type also owns rewrite rules for its own operations, such as casts. Engines convert + operands of different dtypes before Vortex sees them. +- A rewrite rule keeps generic structural arrays such as `Constant` and `Dict` above the extension + wrapper. So code finds them without any knowledge of extension types. +- [`../decisions.md`](../decisions.md) records which of these points are decided and which are + still open. - Some fixes are urgent, because they cause wrong results or panics. They cover implicit casts through storage, comparisons against raw storage, storage checks, and panics on reads and on nullability changes. diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md index 76b2551df11..cb62149ca3f 100644 --- a/research/extension-types/agents/design-notes.md +++ b/research/extension-types/agents/design-notes.md @@ -5,41 +5,43 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # Extension type design notes (agent reference) -This file is the detailed design behind `../03-design.md` and `../05-implementation.md`. It is a -proposal, not an accepted design. "Open" marks a decision that is not made. Finding IDs (`EXT-*`) -refer to `findings.md`, which covers current code only. Probe IDs (R1 to R10) refer to appendix A -of `findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. +This file is the detailed design behind `../03-design.md` and `../05-implementation.md`. It +follows `../../decisions.md`, which records the decisions of the maintainer. IDs such as X4 and XQ1 +refer to that file. "Open" marks a decision that is not made. Finding IDs (`EXT-*`) refer to +`findings.md`, which covers current code only. Probe IDs (R1 to R10) refer to appendix A of +`findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. ## 1. Invariants The design holds these invariants. Every change in the plan must keep them. - **I1.** A semantic operation on an operand that contains an extension dtype is rejected by - default. A declared derive, a type-owned rewrite, or a declaration of the function can allow it - (sections 3 and 4). + default. A declared derive, a rule that the type owns, or a declaration of the function can allow + it (sections 3 and 4). - **I2.** A structural operation on an extension value is always allowed and keeps the extension dtype. -- **I3.** The binder is the only code that decides how a generic operation treats extension - values. It runs at three kinds of sites: - - `Expression::bind` (`vortex-array/src/expr/bound_expression.rs:304`). - - The construction of a scalar function array (`vortex-array/src/arrays/scalar_fn/array.rs:79`). - - Every site that binds an aggregate to an input dtype, for example the zone-map writer - (`vortex-layout/src/layouts/zoned/writer.rs:217-247`). -- **I4.** After binding, a generic operation applies storage semantics to an extension value only - where the binder checked a declared derive (section 4.3). The kernels of generic semantic - operations, statistics and pruning see storage dtypes only. Whether generic encodings and - compression carry extension dtypes depends on the physical form (section 6). A type-aware - function sees the extension types that it declares. `DateTimeParts` stays an exception until - phase 4 (EXT-S2). +- **I3.** Two kinds of rules decide how a generic operation treats extension values. They are the + derive rules of the core and the rules that each extension type owns (section 4). No other code + applies storage semantics to an extension value. Every node meets a typing check when code + builds it (section 4.1). The rules run on every path that optimizes or executes arrays and bound + expressions (section 4.5). +- **I4.** After the rules run, a generic operation applies storage semantics to an extension value + only where a derive covers it (sections 3.4 and 4.3). The kernels of generic semantic + operations, statistics and pruning see storage dtypes only, at every depth. Generic structural + encodings sit above the extension wrapper, and storage encodings sit below it (section 6). A + type-aware function sees the extension types that it declares. `DateTimeParts` stays an + exception until phase 4 (EXT-S2). - **I5.** The only ways to move between an extension dtype and another dtype are `ext_storage`, - `ext_wrap`, and cast hooks that the extension type owns. -- **I6.** The row path and the array path evaluate the same bound expression, so they return the - same result (EXT-P1). This also applies to an unvalidated array with invalid values. -- **I7.** Reading a value never panics. A read from an unvalidated array checks the value. A value - that the type rejects produces an error (section 8). -- **I8.** A nullability change never fails and never validates the type again. -- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and no rewrites, and - no function declaration matches it. + `ext_storage_deep` and the rules that the extension type owns. A rule that produces extension + values builds the extension array itself (section 5). No `ext_wrap` expression exists (X4). +- **I6.** The row path and the array path apply the same rules, so they return the same result + (EXT-P1). Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns the separate + question of whether a validity query raises errors from values that it skips (X9). +- **I7.** Reading a value never panics. Building an extension scalar from an array checks the value + and returns an error for an illegal value (section 8). +- **I8.** A nullability change of a dtype never fails and never validates the type again. +- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and owns no rules, and + no `TypeAware` declaration matches it. ## 2. Operation classes @@ -57,14 +59,14 @@ type-aware function implements the semantics of the extension types that it decl | `pack`, `merge`, `select`, `get_item` on a struct that holds extension fields | Structural | The extension field stays opaque | | `cast` to the same dtype with another nullability | Structural | | | `scalar_at`, serialization, compression of storage | Structural | Constant detection for compression compares storage | -| `ext_storage` | Structural | Explicit unwrap | -| `=`, `!=` | Semantic | Needs `EQ` or a rewrite | -| `<`, `<=`, `>`, `>=`, `between` | Semantic | Needs `ORD` or a rewrite | -| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | Semantic | Needs `ORD` or a rewrite | -| `is_constant`, `all_non_distinct`, `list_contains`, distinct, group by, join keys | Semantic | Needs `EQ` or a rewrite | -| Arithmetic, `sum`, `sum_v2`, `mean` | Semantic | Needs a rewrite, because the result type usually changes | -| `cast` other than a nullability change | Semantic | Needs a cast hook | -| `is_nan`, `nan_count`, `like`, `byte_length` | Semantic | Needs a rewrite | +| `ext_storage`, `ext_storage_deep` | Structural | Explicit unwrap of the top level, or of every level | +| `=`, `!=` | Semantic | Needs `eq` or a rule of the type | +| `<`, `<=`, `>`, `>=`, `between` | Semantic | Needs `ord` or a rule of the type | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | Semantic | Needs `ord` or a rule of the type | +| `is_constant`, `all_non_distinct`, `list_contains`, distinct, group by, join keys | Semantic | Needs `eq` or a rule of the type | +| Arithmetic, `sum`, `sum_v2`, `mean` | Semantic | Needs a rule of the type, because the result type usually changes | +| `cast` other than a nullability change | Semantic | Needs a cast rule of the type (section 4.4) | +| `is_nan`, `nan_count`, `like`, `byte_length` | Semantic | Needs a rule of the type | | Zero and default values | Semantic | Needs a `zero_value` hook | | Export to Arrow, DuckDB, DataFusion, Python, JNI | Semantic | Needs an export plugin | | `replace_time_zone`, tensor functions, spatial functions | Type-aware | The function owner declares the types (section 4.2) | @@ -73,8 +75,8 @@ type-aware function implements the semantics of the extension types that it decl ### 3.1 Definition -A derive is a declaration on one extension dtype. It says that a semantic operation on values of -that dtype is the same operation on their storage values. +A derive is a declaration on one extension dtype (X2). It says that a semantic operation on values +of that dtype is the same operation on their storage values. ```rust /// Storage behavior that values of one extension dtype inherit, like `#[derive]` on a newtype. @@ -99,14 +101,15 @@ pub trait ExtVTable { } ``` -`ExtDTypeRef` gets an erased forwarder: `fn derives(&self) -> Derives`. The binder reads the flags -from the non-nullable form of the dtype. So a type that breaks D4 still gives one answer. +`ExtDTypeRef` gets an erased forwarder: `fn derives(&self) -> Derives`. Typing and the derive rules +read the flags from the non-nullable form of the dtype. So a type that breaks D4 still gives one +answer. ### 3.2 Rules - **D1.** A derive applies only when every operand in a derived position has the same extension dtype, ignoring nullability. A lone extension operand never gets a derive, so `ts < 5i64` stays - rejected. Operations across dtypes need an explicit rule (section 3.5). + rejected. Operations across dtypes are rejected (section 3.5). - **D2.** `eq` requires canonical storage: each value of the dtype has exactly one storage form. - **D3.** `ord` requires `eq`, and storage order must equal value order. - **D4.** The flags do not depend on nullability. @@ -130,32 +133,26 @@ type supplies an optional reference order on native values for tests. `NativeVal An aggregate binds to one input dtype (`AggregateFnVTable::return_dtype`, `vortex-array/src/aggregate_fn/vtable.rs:170`). So a unary operation reads the flags of its one input, and it never needs a second dtype. A function outside this table gets no derive. It uses a -declaration or a type-owned rewrite (section 4). +declaration or a rule that the type owns (section 4). ### 3.4 Containers A struct, list, fixed-size list or map dtype that holds extension values has `eq` or `ord` only if -every nested extension dtype declares it. An order comparison of two `struct{ts: Timestamp(ms)}` -values needs `ord` on `Timestamp(ms)`. Any comparison of two `struct{doc: Json}` values is -rejected. The nested comparator reads nested extension values by storage -(`vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`). It does so only after the -binder checked these flags. +every nested extension dtype declares it. An extension dtype whose storage holds another extension +dtype follows the same rule. An order comparison of two `struct{ts: Timestamp(ms)}` values needs +`ord` on `Timestamp(ms)`. Any comparison of two `struct{doc: Json}` values is rejected. + +After the check, the derive rule removes every extension layer at every depth (section 4.3, X6). +The kernel then sees only storage dtypes. So the extension branch of the nested comparator +(`vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`) goes away in phase 2. ### 3.5 Operations across dtypes Derives never apply across dtypes. Two dtypes with different metadata are different dtypes, for -example `Timestamp(ms)` and `Timestamp(ns)`, or two time zones. The binder rejects such an -operation by default. A type can support one only through an explicit rule that the type owns: - -- **Compatibility.** The type states that two of its dtypes compare by storage as they are. An - example is two zoned timestamps with one unit, because both store UTC instants. -- **Coercion.** The binder casts one operand to the dtype of the other with a type-owned cast - hook. Then the derive of that dtype applies. -- **Rewrite.** The type rewrites the whole call into a storage expression, for example a rescale - before a decimal comparison (section 4.4). - -Open: whether Vortex coerces at all, or engines coerce before they send an expression. Rust and the -built-in `Decimal` reject such operations today. SQL engines coerce them. +example `Timestamp(ms)` and `Timestamp(ns)`, or two time zones. Vortex rejects such an operation +(X3). An engine converts the operands to one dtype first, as its planner already does for SQL. A +user of the Rust or Python API writes an explicit cast. Rust and the built-in `Decimal` reject such +operations today as well. ### 3.6 Table for the types in the repository @@ -163,7 +160,7 @@ built-in `Decimal` reject such operations today. SQL engines coerce them. | --- | --- | --- | --- | | `Timestamp` | Yes | Yes | One dtype fixes the unit and the time zone, so storage counts one unit from one epoch | | `Date`, `Time` | Yes | Yes | | -| `Uuid` | Yes | Open | Byte order is the Postgres order | +| `Uuid` | Yes | Open | Byte order is the Postgres order (XQ6) | | `Json` | No | No | Text is not canonical | | `Vector` | Yes | No | No natural order | | `FixedShapeTensor` | Yes | No | No natural order | @@ -176,33 +173,43 @@ same bits, and `-0.0` differs from `+0.0` shows this for primitive and fixed-size list compares. A `Vector` derive inherits these storage semantics on purpose. -## 4. Binder +## 4. Typing and rewrite rules -### 4.1 Resolution order +A call with an extension operand passes two steps. Typing checks the call when code builds the +node. Rewrite rules then replace an accepted semantic call with operations on storage. Typing never +rewrites, and the rules never accept a call that typing rejected. -The binder handles every call that has at least one operand whose dtype contains an extension -dtype. It tries these steps in order: +### 4.1 Typing -1. The function declares the call structural for these operands. Bind it without a change. -2. The function declares itself type-aware for every extension type in the operands. Bind it - without a change. The function owner owns the semantics. -3. The function declares a derive, and the operands meet D1 and section 3.4. If every operand in - a derived position is a top-level extension operand, rewrite each one to `ext_storage`. A - scalar result with the extension dtype, such as the result of `min`, gets a checked - `ext_wrap`. Otherwise, bind the call without a change. An example is `list_contains`, where the - list holds the extension values. Its kernel then compares nested values by storage, which the - derive check authorizes. -4. An extension type of an operand provides a rewrite for the call. Use it. The binder asks the - types in argument order. -5. Otherwise, return an error that names the function, the extension ID, and the missing - declaration. +Every node of a compute graph meets a typing check when code builds it. Today that check is +`return_dtype`, and it runs in four places: -For a cast, the binder tries these steps in order: +- `BoundExpression::try_new_vec` (`vortex-array/src/expr/bound_expression.rs:129-148`). + `Expression::bind`, `with_children` in the optimizer and the reduce rules of bound expressions + all reach it. +- `ScalarFnArray::try_new_from_slots` (`vortex-array/src/arrays/scalar_fn/array.rs:102-114`). + `try_new`, `try_new_with_len`, `ArrayRef::apply_bound` and the reduce rules of arrays all reach + it. +- `AggregateDTypes::try_new` (`vortex-array/src/aggregate_fn/vtable.rs:55-75`), where an aggregate + meets an input dtype. `Accumulator::try_new` reaches it, and DuckDB aggregate pushdown calls + `Accumulator::try_new` directly (`vortex-duckdb/src/convert/expr.rs:543-552`). + `Accumulator::from_dtypes` takes dtypes that are already resolved, so it accepts only checked + dtypes. `return_dtype` and `state_dtype` of `AggregateFnRef` answer dtype questions without an + accumulator (`vortex-array/src/aggregate_fn/erased.rs:86-95`), so they apply the same check. +- `ScalarFn::validate` (`vortex-array/src/arrays/scalar_fn/vtable/mod.rs:79-121`) checks a scalar + function array that code builds from parts. -1. A nullability change of the same dtype. -2. `cast_to` of the source type. -3. `cast_from` of the target type. -4. An error. +Typing accepts a call with an extension operand only in these cases, tried in this order: + +1. The function declares the call structural for these operands. +2. The function declares itself type-aware for every extension type in the operands. +3. The function declares a derive, and the operands meet D1 and section 3.4. +4. A rule of an extension type in the operands covers the call. Typing asks the types in argument + order. + +Otherwise typing returns an error that names the function, the extension ID and the missing +declaration. For a cast, typing tries three cases in order. The first is a nullability change of +the same dtype. The second is a rule of the source type, and the third is a rule of the target type. ### 4.2 Function declarations @@ -211,11 +218,12 @@ For a cast, the binder tries these steps in order: pub enum ExtensionUse { /// The function moves or selects values and never reads them. Structural, - /// The function applies storage equality or order. The binder checks these flags (D1). + /// The function applies storage equality or order. Typing checks these flags (D1). Derive(Derives), /// The function implements the semantics of the matched extension type itself. TypeAware, - /// The function has no semantics for extension values. The binder rewrites or rejects it. + /// The function has no semantics for extension values. A rule of the type covers the call, + /// or typing rejects it. None, } @@ -236,6 +244,22 @@ pub trait ScalarFnVTable { `AggregateFnVTable` gets the same method without `arg`, because an aggregate has one input. +An aggregate with a `Derive` declaration also declares how its result relates to its input (XQ5, +open). These notes recommend three kinds: + +- A value of the input is one of its stored values. The derive rule builds an extension scalar of + the input dtype from the storage result. Examples are the results of `min` and `max`, and each + field of the struct that `min_max` returns. +- A bound on the input stays a storage value (S5). `bounded_min` and `bounded_max` declare a bound + when they truncate the storage, which happens for `Utf8` and `Binary`. For other storage dtypes, + they declare a value. +- Any other result stays as it is, for example the `Bool` of `is_sorted`. + +A value result can only get its top-level layer back. So if a value result holds extension values +inside a container, for example `min` over `List`, typing rejects the call. + +More rules for declarations: + - Only the function owner declares, in the definition of the function. The core grants no type-aware declaration by default. - A declaration can depend on the options. `Binary` declares `Derive` with `eq` or `ord` by @@ -248,148 +272,196 @@ pub trait ScalarFnVTable { - `replace_time_zone` declares `TypeAware` for `Timestamp`. Today its `return_dtype` accepts only timestamps (`vortex-array/src/scalar_fn/fns/replace_time_zone.rs:131-148`). The tensor and spatial functions declare their own types in the same way. -- The binder never infers a declaration from `return_dtype`. `Binary::return_dtype` accepts any +- Typing never infers a declaration from `return_dtype`. `Binary::return_dtype` accepts any extension operand in a comparison today (`vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`, EXT-Q1). - A value that a function keeps in its options counts as an operand. `DynamicComparison` keeps its right operand in its options and calls `Binary` directly - (`vortex-array/src/scalar_fn/fns/dynamic.rs:80-111`). Its declaration covers that value, and a - derive rewrite converts the value to a storage scalar. + (`vortex-array/src/scalar_fn/fns/dynamic.rs:80-111`). Its declaration covers that value. The + value is a function that returns a new `ScalarValue` at each execution (`dynamic.rs:181-186`). + So the derive rule changes the dtype of the value to the storage dtype, and the values stay as + they are. After that change, nothing checks a new value against the extension type. So the rule + keeps one check: each new value goes through `validate_scalar_value`, and an illegal value is an + error. Today `Rhs::scalar` panics on such a value (`dynamic.rs:189-193`). ### 4.3 Nested extension values An operand counts as an extension operand if its dtype contains an extension dtype at any depth. Examples are `list_contains` on `List` (`vortex-array/src/scalar_fn/fns/list_contains/mod.rs:178-194`) and the spatial `collect` on -`List` (`vortex-spatial/src/scalar_fn/collect.rs:60-75`). -`ext_storage` unwraps only the top level (`vortex-array/src/scalar_fn/fns/ext_storage.rs:58-63`). -So a derive rewrite happens only when every derived operand is a top-level extension operand -(section 4.1). Otherwise the call binds unchanged. The kernel then reads nested members by storage, -which the container rule of section 3.4 authorizes. +`List` (`vortex-spatial/src/scalar_fn/collect.rs:60-96`). `ext_storage` unwraps only the +top level, and it rejects an operand whose top-level dtype is not an extension +(`vortex-array/src/scalar_fn/fns/ext_storage.rs:58-63`). + +So a derive rule uses a new structural expression, `ext_storage_deep` (X6). It replaces every +extension dtype in the operand with its storage dtype, at every depth. It repeats this while a +storage dtype holds another extension dtype. Typing has already checked section 3.4, so every layer +that it removes declares the flag. + +- For an array, it rebuilds each container around the storage of its children. It copies no + values. +- For a scalar, it changes only the dtype, because an extension scalar already holds its storage + value. +- For a top-level extension whose storage holds no extension dtype, it equals `ext_storage`. The + derive rule then emits `ext_storage` itself. So the statistics rule S2, the simplification rules + of section 4.8 and the pushdown rules of section 6 apply unchanged to the common case. Those + rules gain a matching case for `ext_storage_deep`, which appears only for nested extension + values. + +For example, `list_contains` over `List` and a `Timestamp(ms)` needle becomes +`list_contains` over `List` and an `i64` needle. Its kernel sees only storage. The spatial +`collect` declares `TypeAware` for `Point`, so no layer is removed there. + +### 4.4 Rules that the type owns + +A type implements its own semantics as rewrite rules (X4). The rules are registered with the type, +together with its compression scheme and the other parts of the type. A rule matches a call that +has an operand of the type. It replaces the call with operations on storage, or it declines. + +- **Cast rules.** A rule converts the type to another dtype, or another dtype into the type. Cast + rules implement the design of issue [#8570](https://github.com/vortex-data/vortex/issues/8570), + with rules in place of closures. A cast exists only if a rule covers it (C1). +- **Call rules.** A rule covers a scalar function or an aggregate with an operand of the type. An + example is `sum` over a decimal-like type, which needs a wider result dtype. + +A rule reaches storage through `ext_storage` or `ext_storage_deep`. When its result holds values of +an extension type, it builds the extension array or scalar itself (section 5). + +Where the rules live is open (XQ2). These notes recommend a method on `ExtVTable` that returns the +rules of the type. `ExtDTypeRef` holds the vtable, so the rules travel with the dtype and need no +session lookup. Registering the type in a session then registers its rules too. + +The existing session registry, `ArrayKernels` (`vortex-array/src/optimizer/kernels.rs`), is the +alternative. It has two problems for extension types: + +- It is keyed by encoding ids, and every extension array has the id `vortex.ext`. So each rule + checks the dtype itself, and the rules of every registered type run on every extension column. +- Only the session-aware optimizer consults it. A grep counts 83 plain `.optimize()` calls outside + tests, and they use static rules only. Two of them are `ArrayRef::cast` + (`vortex-array/src/builtins.rs:176`) and `ArrayRef::apply_bound` + (`vortex-array/src/expression.rs:41`). + +Rules written over `ReduceNode` serve both arrays and bound expressions, as `reduce_null` already +does (`vortex-array/src/scalar_fn/fns/is_not_null.rs:50`). Pruning works on bound expressions +(`vortex-file/src/pruning.rs:35-49`). So a rule that serves only arrays cannot rewrite a predicate +onto storage statistics. + +Open: whether a crate can register a rule for a function and a type that it does not own. + +### 4.5 Where the rules run + +- The derive rules belong to the functions that declare a derive, such as `Binary`, `Between` and + the order aggregates. So the static optimizer runs them. +- Under the recommendation of XQ2, the type rules come from the dtype. So the static optimizer + finds them as well. +- The executor applies the same rules before it dispatches a kernel. A lazy array that nobody + optimized still gets the rewrite, so the row path and the array path agree (I6). +- After phase 2, no generic kernel applies storage semantics. So a semantic call that reaches a + generic kernel without a rewrite fails. A missing rule gives an error, never a wrong result. + +### 4.6 Examples -### 4.4 Proposed hooks on the extension type +```text +Timestamp(ms) a < Timestamp(ms) b => ext_storage(a) < ext_storage(b) +Timestamp(ms) a < lit(Timestamp(ms) 5) => ext_storage(a) < lit(5i64) +Timestamp(ms) a < lit(5i64) => rejected by typing (D1) +Timestamp(ms) a < Timestamp(ns) b => rejected by typing (X3) +min(Timestamp(ms) a) => min(ext_storage(a)), and the rule builds a + Timestamp(ms) scalar from the result +cast(Timestamp(ms) a, Timestamp(ns)) => Extension(Timestamp(ns), + checked_mul(ext_storage(a), 1_000_000)) +cast(Timestamp(ns) a, Timestamp(ms)) => rejected unless the type defines rounding +Json a = Json b => rejected by typing +``` -```rust -pub trait ExtVTable { - /// Rewrites a cast from this type into an expression, or returns `None`. - fn cast_to( - ext: &ExtDType, - input: Expression, - target: &DType, - ) -> VortexResult> { - Ok(None) - } +The cast rule builds its result with `ExtensionArray::new_unchecked`. Legal input gives legal +output, and `checked_mul` fails on overflow. The result is a wrapper over a pending computation, +which section 6 allows. - /// Rewrites a cast into this type into an expression, or returns `None`. - fn cast_from( - ext: &ExtDType, - input: Expression, - source: &DType, - ) -> VortexResult> { - Ok(None) - } +### 4.7 Array API and enforcement - /// Rewrites a scalar function call that has an operand of this type, or returns `None`. - /// - /// The call holds every argument, so a rewrite handles any arity and operands of other dtypes. - fn rewrite(ext: &ExtDType, call: &ScalarFnCall) -> VortexResult> { - Ok(None) - } -} -``` +Typing runs in `try_new_from_slots` (`vortex-array/src/arrays/scalar_fn/array.rs:102-114`), which +`try_new` and `try_new_with_len` both call. `ArrayRef::apply_bound` uses `try_new_with_len` +(`vortex-array/src/expression.rs:19-42`). Typing never changes the node, so a typed constructor +keeps its return type. -`ScalarFnCall` is a new view of the function, its options and its bound arguments. Aggregates need -the same pair of hooks (`rewrite_aggregate`). Open: whether a session registry keyed by function -and extension type also exists. It lets a crate add support for a function and a type that it -does not own. +The types make the typing entry points of section 4.1 the only paths. The design accepts the API +changes that this needs (X8): -### 4.5 Examples +- `BoundExpression` keeps its variants private. Today they are public, so any code can build a + node with any dtype (`vortex-array/src/expr/bound_expression.rs:42-59`). +- A constructor for a call that typing can reject becomes fallible. `Cast::new` is the main case. + It calls `vortex_expect` on `try_new` (`vortex-array/src/scalar_fn/fns/cast/mod.rs:62-65`), and + typing rejects every cast that no rule covers. So `Cast::new` becomes fallible in phase 2. A + constructor for a structural call, such as `IsNull::new` + (`vortex-array/src/scalar_fn/fns/is_null.rs:35-37`), can stay infallible, because typing never + rejects it. +- Code that runs a function on raw arguments applies typing and the rules first. + `DynamicComparison::execute` calls `Binary` directly (section 4.2). The row encoder of sort keys + runs on raw columns, and today it rejects extension dtypes at every depth + (`vortex-row/src/encoder.rs:112`, `:126-144`). After the derive rule, it receives the output of + `ext_storage_deep`. -```text -Timestamp(ms) a < Timestamp(ms) b => ext_storage(a) < ext_storage(b) -Timestamp(ms) a < lit(Timestamp(ms) 5) => ext_storage(a) < lit(5i64) -Timestamp(ms) a < lit(5i64) => rejected (D1) -min(Timestamp(ms) a) => ext_wrap(min(ext_storage(a)), Timestamp(ms), checked) -cast(Timestamp(ms) a, Timestamp(ns)) => ext_wrap(checked_mul(ext_storage(a), 1_000_000), - Timestamp(ns), unchecked) -cast(Timestamp(ns) a, Timestamp(ms)) => rejected unless the type defines rounding -DecimalLike(10, 1) a < DecimalLike(10, 2) b - => checked_mul(ext_storage(a), 10) < ext_storage(b) - (a rewrite that the type owns) -Json a = Json b => rejected -``` +### 4.8 Simplification -`min` uses a checked wrap, because its result is a scalar and every scalar is valid (section 8). -The unit cast uses an unchecked wrap. Its result stays unvalidated, so a read of a row still -checks the value. +`ext_storage` is infallible today (`vortex-array/src/scalar_fn/fns/ext_storage.rs:87-93`), so these +rules never remove or add a failure: -### 4.6 Array API +| Rule | Valid when | Reason | +| --- | --- | --- | +| `ext_storage(lit(s))` to `lit(s.to_storage_scalar())` | Always | Both read the storage of one valid scalar, and the dtypes match | +| `ext_storage_deep(lit(s))` to a literal of the storage dtype | Always | The same as the first rule, at every depth | +| `ext_storage(Extension(s))` to `s` | Always | The wrapper holds `s` as its storage. The row path then reads storage directly (section 8.3). No such rule exists today (probe R9) | +| `ext_storage(Constant(extension scalar))` to `Constant(storage scalar)` | Always | The same as the first rule, for a constant array | -The array API builds `ScalarFnArray` values directly. The binder must run in -`ScalarFnArray::try_new` (or in the typed constructors such as `Binary::try_new`). It returns the -rewritten tree, so the new array holds `ext_storage` children and an `ext_wrap` parent where -needed. The row path (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42`) then -evaluates the rewritten tree. +`Expression::bind` folds nothing (`vortex-array/src/expr/bound_expression.rs:309-323`), but +simplification turns `cast(lit)` into a literal +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`). When such a fold fails, the cast stays in +the tree. This behavior needs no change. -Open: some rewrites can need session state. The hooks above use only the `ExtDType`, which travels -with the dtype, so the binder does not need a session. +## 5. Wrapping storage -### 4.7 Simplification of wraps +No `ext_wrap` expression exists (X4). A rule that produces extension values builds the extension +array or scalar itself. -`ext_storage` is infallible today (`vortex-array/src/scalar_fn/fns/ext_storage.rs:87-93`). A -checked `ext_wrap` is fallible in the sense of `ScalarFnVTable::is_infallible` -(`vortex-array/src/scalar_fn/vtable.rs:262-279`). It fails when a row that it evaluates holds a -value that the type rejects. A simplification must not remove such a failure. It must also not add -a failure for a row that the original expression does not evaluate. +- A rule uses `ExtensionArray::new_unchecked` when legal input gives legal output. Examples are the + unit cast of section 4.6 and the result of `min`, which is a stored value. +- A rule uses `ExtensionArray::try_new` for storage values that no check has seen. +- A rule builds an extension scalar with `Scalar::try_new`, which always checks (section 8). +- Wrapping is a Rust call, not an expression. So no user expression and no serialized plan can + build a wrapper without a check. +- A lazy computation that produces extension values is a wrapper over a pending computation, for + example the result of the cast in section 4.6. Section 6 keeps pending computations under the + wrapper. -| Rule | Valid when | Reason | +## 6. Physical form + +### 6.1 Decision + +A rewrite rule pulls generic structural arrays out from under the extension wrapper (X5). Code +that looks for a constant, a dictionary or runs then finds them at the top, without any knowledge +of extension types. The compressor, the exporters, pruning and `as_constant()` all depend on this. + +Which arrays the rule pulls out is open (XQ1). These notes recommend three roles: + +| Role | Encodings | Place | | --- | --- | --- | -| `ext_storage(lit(s))` to `lit(s.to_storage_scalar())` | Always | Both read the storage of one valid scalar, and the dtypes match | -| `ext_storage(ext_wrap(y, d, unchecked))` to `y` | `y` has the storage dtype of `d`, ignoring nullability | Both steps are infallible and keep the values. The result dtype is the dtype of `y` | -| `ext_storage(ext_wrap(y, d, checked))` to `y` | Never | The wrap fails on a value that `d` rejects, and `y` alone does not fail | -| `ext_wrap(ext_storage(x), d, unchecked)` to `x` | `d` equals the dtype of `x`, ignoring nullability | Same values and dtype. The result keeps the validation state of `x`, which came from a real check | -| `ext_wrap(ext_storage(x), d, checked)` to `x` | Never at bind time | `x` can be unvalidated, and then the check can fail | -| `ext_wrap(lit(v), d, mode)` to a literal | `Scalar::try_new(d.with_nullability(n), v)` succeeds, where `n` is the nullability of `v` | A literal is a scalar, and every scalar is valid | - -A counterexample for the third rule uses `NonZero` over `u64` and `y = lit(0u64)`. The original -expression fails, because `NonZero` rejects zero. The simplified expression returns `0`. Probe R4 -shows that the check fails: `Scalar::try_new` with `NonZero` and storage `0` returns an error. -`ext_wrap` itself does not exist yet. - -Binding checks a literal argument of `ext_wrap` in both modes (section 5). A literal can also -appear after binding. `Expression::bind` folds nothing -(`vortex-array/src/expr/bound_expression.rs:309-323`), but simplification turns `cast(lit)` into a -literal (`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`). So -`ext_wrap(cast(lit(0i64), u64), NonZero, checked)` binds, and a later fold meets an invalid literal. -When the fold check fails, the wrap stays in the tree, as a failing cast does. Then a checked wrap -fails only for rows that it evaluates, and an unchecked wrap leaves the check to reads. The binder -emits an unchecked wrap only where the values are valid by construction. - -## 5. `ext_wrap` - -- Options: the target `ExtDTypeRef`, and a mode that is `checked` or `unchecked`. -- `return_dtype`: the target dtype, with the nullability of the input. -- Argument: must have the storage dtype of the target, ignoring nullability. Otherwise binding - fails. -- Literal argument: binding checks it with `Scalar::try_new`, in both modes, and an invalid - literal is a bind error. A literal that simplification makes later follows the fold rule of - section 4.7. -- Checked execution: calls `validate_array` (section 8), then builds a validated wrapper. It is - fallible. -- Unchecked execution: builds the wrapper without a check. The result is unvalidated. It is - infallible. -- Row path: both modes build the scalar with `Scalar::try_new`. `Scalar::new_unchecked` checks - values only in debug builds, and it panics there on an invalid value - (`vortex-array/src/scalar/scalar_impl.rs:97-104`). - -## 6. Physical form (open) - -Opacity is a semantic rule. It does not decide where the wrapper sits relative to generic -encodings such as `Constant`, `Dict`, `Chunked`, `Masked` and `Sparse`. The binder works with -either placement, because `ext_storage` works with either. This section compares the two. No -maintainer decision exists. - -### 6.1 Current code +| Structural | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked`, `Interleave` | Above the wrapper | +| Storage | The canonical storage arrays. The FastLanes encodings (BitPacked, FoR, Delta, RLE), ALP, ALP-RD, ByteBool, ZigZag, Pco, Zstd, FSST, OnPair, Sequence, PiecewiseSequence and Patched. Also DecimalByteParts, and `DateTimeParts` after phase 4 | Below the wrapper | +| Pending | Slices, filters and scalar functions that wait to run | Below the wrapper, with the storage that they apply to | + +A storage encoding computes values from a layout that only built-in dtypes have, so it can never +carry an extension dtype. Patched, for example, accepts only primitive dtypes +(`vortex-array/src/arrays/patched/array.rs:187`). A structural encoding holds values of the column +dtype, whatever it is. + +These notes also recommend that a canonical extension array is a wrapper over a canonical storage +array. Then a canonical array never hides a pending computation or a compressed child. Today +`Extension::execute` returns the array unchanged +(`vortex-array/src/arrays/extension/vtable/mod.rs:189-191`), so a canonical extension array can +hold any child. + +### 6.2 Current code - The compressor writes `Constant(extension scalar)` for a constant extension column, and `Masked(Constant(extension scalar))` when some rows are null @@ -397,70 +469,67 @@ maintainer decision exists. `vortex-compressor/src/compressor/constant/mod.rs:175-193`). For other columns, it compares two results and keeps the smaller one (`cascade.rs:175-205`). The first comes from a scheme that matches the extension type. For example, `TemporalScheme` writes `DateTimeParts` for a - `Timestamp` (`vortex-btrblocks/src/schemes/temporal.rs:45-56`, `:124`). The second is - `vortex.ext` over the compressed storage. Probe R8 shows the constant, masked and `vortex.ext` - shapes. Its compressor had no temporal scheme. + `Timestamp` (`vortex-btrblocks/src/schemes/temporal.rs:45-56`, `:124`). When no scheme matches, + the first result is the input itself (`cascade.rs:288-290`). The second is `vortex.ext` over the + compressed storage, and it wins a tie (`cascade.rs:201`). Probe R8 shows the constant, masked + and `vortex.ext` shapes. Its compressor had no temporal scheme. - `ExtensionConstantRule` rewrites `Extension(Constant(storage))` to `Constant(extension scalar)` - (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`). Probe R5 shows the rewrite. + (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`). Probe R5 shows the rewrite. This + rule is the first case of the decision in section 6.1. - `ArrayRef::as_constant()` matches only the `Constant` encoding (`vortex-array/src/array/erased.rs:485-487`). Probe R5 shows that it misses `Extension(Constant)`. +- `ArrayRef::take` wraps its input in a lazy `DictArray` + (`vortex-array/src/array/erased.rs:262-264`). The extension take rule applies `take` to the + storage, so it builds `Extension(Dict(indices, storage))` + (`vortex-array/src/arrays/extension/compute/take.rs`). Under the decision, `take` of an extension + array gives `Dict(indices, Extension(storage))`, so this rule goes. +- The extension mask rule builds a lazy mask expression over the storage + (`vortex-array/src/arrays/extension/compute/mask.rs`). That is a pending computation, so it stays + under the wrapper. - `ext_storage` reads a constant input directly. For other inputs, it executes the input to a canonical `ExtensionArray` (`vortex-array/src/scalar_fn/fns/ext_storage.rs:78-84`). -- Generic rules push infallible scalar functions into dictionary values and into chunks - (`vortex-array/src/arrays/dict/compute/rules.rs:96-138`, - `vortex-array/src/arrays/chunked/compute/rules.rs:35-36`). Probe R6 shows that +- Generic rules push scalar functions into dictionary values and into chunks. The dictionary rule + pushes an infallible function, or any function when every value is referenced + (`vortex-array/src/arrays/dict/compute/rules.rs:96-208`). The chunked rules push a unary + function, or a function with constant siblings + (`vortex-array/src/arrays/chunked/compute/rules.rs:34-115`). Probe R6 shows that `ext_storage` keeps both encodings. Over `Masked`, it decodes the input. Over `Sparse` with extension values, it fails, because sparse canonicalization rejects extension arrays (`encodings/sparse/src/canonical.rs:360-362`). - Constant canonicalization and masked execution have extension arms (`vortex-array/src/arrays/constant/vtable/canonical.rs:204-219`, `vortex-array/src/arrays/masked/execute.rs:178-193`). So does dictionary execution - (`vortex-array/src/arrays/dict/execute.rs:182-193`). + (`vortex-array/src/arrays/dict/execute.rs:182-193`). Run-end decoding rejects extension values + (`encodings/runend/src/array.rs:509`). - `vortex.ext`, `vortex.constant`, `vortex.dict` and `vortex.chunked` are in core edition 2025.05 (`vortex-edition/src/declarations/core/v2025_05.rs:27-33`). `vortex.masked` is in 2025.10 (`v2025_10.rs:24`). -### 6.2 Comparison - -| Aspect | Wrapper above generic encodings | Generic encodings above the wrapper | -| --- | --- | --- | -| Shapes of an extension column | One after reduction: `Extension(child)` | Several: `Extension(x)`, `Constant(ext s)`, `Masked(Constant(ext s))`, `Dict(codes, Extension(v))`, `Chunked[...]` | -| Constructors | Code that builds a `Constant`, `Dict` or `Chunked` with an extension dtype makes a form that reduce rules lift. A constructor can also build the lifted form directly. `ExtensionConstantRule` reverses | No change. `Constant` already sits on top | -| Constant detection | `as_constant()` misses `Extension(Constant)`, so callers need a helper that looks through the wrapper. The compressor writes `Extension(Constant(storage))` | `as_constant()` works as today | -| Storage extraction, array path | `ext_storage` reads the child | `ext_storage` keeps `Dict` and `Chunked` through existing rules. `Masked` decodes, and `Sparse` fails (R6) | -| Storage extraction, row path | One reduce rule, `ext_storage(Extension(s))` to `s`, makes the row path read storage. No such rule fired in probe R9 | The same rule, plus a pushdown for `Masked` and `Sparse`. `Dict` and `Chunked` already push `ext_storage` down (R6) | -| Extension arms in generic encodings | None. Sparse works over storage | `Constant`, `Masked` and `Dict` keep theirs. `Sparse` needs one | -| Statistics | Child statistics are storage statistics | `Constant(ext s)` holds an extension scalar. A reader converts it | -| Wire compatibility | Readers read every old form, and reduce rules normalize it. Old readers read `vortex.ext` over `vortex.constant`. `Decimal` needs a translation on write, because `vortex.decimal` is a leaf | The memory form and the wire form match for both types | - -### 6.3 Recommendation - -These notes recommend the wrapper above generic encodings. This is a recommendation, not a -decision. The reasons: - -- One shape removes EXT-S1, and generic encodings lose their extension arms. -- The compressor already writes this shape for non-constant extension columns, unless a - type-specific scheme such as `DateTimeParts` wins (R8). -- Extension columns need no new wire form, because old readers read `vortex.ext` over - `vortex.constant`. - -The costs: - -- `ExtensionConstantRule` reverses. Constant detection in the compressor and in kernels must look - through the wrapper. -- The change lands best after the binder. Before the binder, a kernel that reads a constant - extension operand with `as_constant()` misses `Extension(Constant)`. The `DateTimeParts` compare - kernel then declines (`encodings/datetime-parts/src/compute/compare.rs:31-33`). The extension - compare kernel still compares the storage - (`vortex-array/src/arrays/extension/compute/compare.rs:31-51`). -- The `Decimal` plugin translates the wrapper form to the legacy decimal forms on write - (`../../decimal/agents/report.md`, section 3.5). +### 6.3 Mechanism and costs + +- One static rule on the `Extension` encoding moves a structural child above the wrapper. For + example, `Extension(Dict(codes, values))` becomes `Dict(codes, Extension(values))`. A static rule + runs in every `.optimize()` call, unlike a session rule (section 4.4). +- Each structural encoding implements one method that rebuilds it with its value children and value + scalars passed through a function. That method gives three things: the lift rule, the pushdown + of `ext_storage`, and decoding of a structural encoding with an extension dtype. Decoding strips + the wrapper, decodes the storage form, and wraps the result again. +- So the extension code that `Constant`, `Masked` and `Dict` have today becomes one shared path. + `RunEnd` and `Sparse`, which fail on extension values today, get support through the same path. +- `ext_storage` then never computes. It passes through the structural encodings and removes the + wrapper. +- Readers apply the lift rule, so old files with `vortex.ext` over `vortex.dict` change shape when + they are read. Old readers decode `Constant`, `Dict`, `Masked` and `Chunked` with extension + values, but not `RunEnd` or `Sparse`. So writers keep the old shape for those two until an + edition allows the new one. +- `Constant(extension scalar)` holds an extension scalar. Readers of statistics convert it to a + storage scalar (S3). ### 6.4 Type-specific encodings Type-specific encodings are a separate question. `DateTimeParts` carries the `Timestamp` dtype -today (`encodings/datetime-parts/src/canonical.rs:28`, EXT-S2). Under the binder, a compare +today (`encodings/datetime-parts/src/canonical.rs:28`, EXT-S2). After the derive rule, a compare reaches it as `ext_storage(column) < lit(i64)`. To keep its compare kernel, `DateTimeParts` becomes an `i64` encoding with the time unit in its metadata. Readers convert the old form on read. An edition gates writes of the new form. `vortex-btrblocks/src/schemes/temporal.rs` chooses @@ -470,10 +539,12 @@ any future encoding for one extension type. ## 7. Statistics - **S1.** Zone maps and file statistics for an extension column store statistics of the storage, - with the storage dtype. The fallback in `vortex-layout/src/layouts/zoned/schema.rs:112-121` - becomes the rule. + with the storage dtype. The storage fallback in `aggregate_state_dtype` + (`vortex-layout/src/layouts/zoned/schema.rs:177-188`) becomes the rule. Today it applies only + when an aggregate rejects the extension dtype. - **S2.** A new stats rewrite rule maps `stat(ext_storage(x), s)` to the storage statistic `s` of - `x`. Bound predicates reference `ext_storage(col)` after binding, so pruning needs only this rule. + `x`. After the derive rules, bound predicates reference `ext_storage(col)`, so pruning needs only + this rule. - **S3.** Readers convert existing extension-typed min and max to storage scalars with `to_storage_scalar()`. The values are already storage values. - **S4.** Engine statistics export (DuckDB `column_statistics.rs`, DataFusion `convert/stats`, JNI @@ -482,7 +553,7 @@ any future encoding for one extension type. - **S5.** Bounded min and max (truncation) apply to the storage dtype, so they also apply to extensions over `Utf8` and `Binary`. -## 8. Validation and trust +## 8. Value validation ### 8.1 Three kinds of validation @@ -490,86 +561,116 @@ any future encoding for one extension type. | --- | --- | --- | --- | | Dtype validation | `validate_dtype`: the metadata and the storage dtype | When a dtype is built or read, for both nullabilities (section 9) | Once for each dtype | | Storage-shape validation | The storage child has the length of the array, which `Extension::validate` checks today, and the storage dtype (U6). A scalar value has the storage dtype (EXT-V7) | When an array or a scalar is built or read | Constant | -| Value validation | `validate_array`: each checked value is a valid value of the type | At the trust boundaries in section 8.3 | One pass over the values | - -### 8.2 Validation state - -Each extension array is validated or unvalidated. - -- An array is validated only after a value check succeeds for all its checked values, for example - `validate_array`. The result of a structural operation is validated only if all its extension - inputs are validated. -- In this design, a `Scalar` is always valid, because `Scalar::try_new` runs the value check. - Today two gaps exist. A scalar can hold a value of the wrong storage type (EXT-V7, fixed by U6). +| Value validation | `validate_array`: each non-null value is a legal value of the type | When `ExtensionArray::try_new` builds an array, and at the other places in section 8.3 | One pass over the values | + +### 8.2 The invariant + +Every extension array that Vortex builds holds only legal values in its non-null rows (X7). A +legal value is a value that `validate_scalar_value` accepts. Data from a file holds legal values +only if the read checks them (XQ3). + +- `ExtensionArray::try_new` checks every non-null value with `validate_array`, and it returns an + error for an illegal value. It takes an `ExecutionCtx`, because the check can execute compressed + storage. `ExtensionArray::new` calls `try_new`. Today `try_new` checks only the storage dtype + (`vortex-array/src/arrays/extension/array.rs:52-73`, EXT-V1). +- `ExtensionArray::new_unchecked` skips the check. Its caller promises that every non-null value + is legal, as with other `new_unchecked` constructors. A debug build checks the promise. +- Kernels that keep values, such as slice, take, filter and mask, use `new_unchecked`. So do rules + that make legal values from legal input, such as a unit cast of `Timestamp`. So does code that + wraps storage it already holds again, such as canonicalization and the compressor + (`vortex-compressor/src/compressor/cascade.rs:198`). A grep counts about 100 call sites of + `ExtensionArray::new` and `try_new` outside test files. Each one moves to the right constructor. +- A `Scalar` is always legal, because `Scalar::try_new` runs the value check. Today two gaps + exist. A scalar can hold a value of the wrong storage type (EXT-V7, fixed by U6). `Scalar::zero_value` skips the check (EXT-V3, fixed by U7). -- A constant array checks its scalar when it is decoded - (`vortex-array/src/arrays/constant/vtable/mod.rs:160-161`). So a constant extension array is - validated. -- Every other array is unvalidated. This includes file reads without a value check, unchecked - wraps, and constructors that skip the value check. -- No constructor marks an array validated without a check. A skipped check is never a proof of - validity. +- Building a scalar from an array checks the value with `Scalar::try_new`, because data from a file + can hold illegal values (XQ3). The check returns an error, never a panic (I7). + +A value check in every call of `ExtensionArray::new` slows common paths. In this design, kernels +use `new_unchecked`, so only the places in section 8.3 pay for the check. Type overrides of +`validate_array` make that check fast. ### 8.3 Where value validation runs -- Checked constructors and checked `ext_wrap`. The result is validated. -- Imports from Arrow, DuckDB, FFI and Python. The result is validated. -- `Scalar::try_new`, for every extension scalar, as today. -- File reads, when a read option asks for it. The result is validated. Open: the default of this - option. -- Row reads from an unvalidated array check each read value, and return an error for an invalid - value (U3). Row reads from a validated array skip the check. -- A consumer that needs every value to be valid, such as an export, validates an unvalidated array - first. +- `ExtensionArray::try_new`, for data that no check has seen. +- Rules that wrap storage values that no check has seen (section 5). +- Imports from Arrow, DuckDB, FFI and Python. They call `try_new`. +- `Scalar::try_new`, for every extension scalar, including a read of one row. +- File reads, if the answer to XQ3 asks for it. `ext_storage` never checks values, on either path. Today the row path checks them. It reads each child as an extension scalar (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`), and that read runs the per-row check (`vortex-array/src/arrays/extension/vtable/operations.rs:19-30`). The array path reads the storage without a check (`vortex-array/src/scalar_fn/fns/ext_storage.rs:83-84`). Probe R9 shows the difference: the array -path returns `[0, 1]` for `NonZero [0, 1]`, and the row path panics. The row path must read the -storage child directly. +path returns `[0, 1]` for `NonZero [0, 1]`, and the row path panics. Under X7, +`ExtensionArray::new` rejects that array, but a file can still hold it. So the row path of +`ext_storage` reads the storage child directly, through the rule `ext_storage(Extension(s))` to +`s` (section 4.8). ### 8.4 The array validator and the scalar validator ```rust pub trait ExtVTable { - /// Checks every value of `storage` that is valid in `storage` and in `ancestors`. + /// Checks every non-null value of `storage`. /// /// The default applies `validate_scalar_value` to each of those values. An override must /// accept exactly the arrays that the default accepts. fn validate_array( ext: &ExtDType, storage: &ArrayRef, - ancestors: &Mask, ctx: &mut ExecutionCtx, ) -> VortexResult<()> { - validate_each_value::(ext, storage, ancestors, ctx) + validate_each_value::(ext, storage, ctx) } } ``` - `validate_scalar_value` stays the rule for one value. Its default calls `unpack_native` (`vortex-array/src/dtype/extension/vtable.rs:53-66`). -- The default `validate_array` is not a no-op. It applies `validate_scalar_value` to each value - that is valid in the array and in every ancestor. -- The caller passes the ancestor mask. Builders pad the rows under a null parent with default - values (`vortex-array/src/builders/fixed_size_list.rs:292-305`). `vortex-spatial` treats - storage behind a null row as arbitrary (`vortex-spatial/src/scalar_fn/row.rs:77-78`). So a - padding value never has to be valid. +- The default `validate_array` is not a no-op. It applies `validate_scalar_value` to each non-null + value. +- A check returns an error for an illegal value and never panics. Today the `Timestamp` check + panics on values outside the range of `jiff::Span`, because it calls the panicking setters + (`vortex-array/src/extension/datetime/timestamp.rs:214-229`, EXT-V8). U5 fixes it. - A type can override `validate_array` with a faster check. The conformance harness compares the override with the default on generated inputs. - Examples of overrides: `Timestamp` checks the storage minimum and maximum against its valid range. `Uuid` checks the version nibble of each value against the metadata. A hypothetical `NormalizedVector` type checks the norm of each row within a tolerance. +### 8.5 Open questions + +- **File reads (XQ3).** Decoding builds arrays through the encoding vtable, not through `try_new`. + A reader can check values, or it can trust the file. These notes recommend a general rule for + Vortex: decoding checks only what memory safety needs, and it trusts semantic constraints. + Decoding already checks lengths, offsets and UTF-8, because Rust requires valid UTF-8 for `str` + (`vortex-array/src/arrays/varbinview/array.rs:234-245`). A strict read mode runs + `validate_array` on every chunk. No operation panics on an illegal value (I7). + `Constant(extension scalar)` already checks its value on decode + (`vortex-array/src/arrays/constant/vtable/mod.rs:160-161`). +- **Values in null slots (XQ4).** Builders fill a non-nullable child under a null parent with + placeholder values (`vortex-array/src/builders/fixed_size_list.rs:294-305`, + `vortex-array/src/builders/mod.rs:156-166`). Arrow leaves the child values under a null parent + undefined. A type such as `NonZero` rejects the placeholder 0. + - These notes recommend that a null slot can hold any value. The value check skips null slots + and rows under a null parent. Every public accessor that takes a child out of its parent masks + the child with the validity of the parent, as `get_item` already does + (`vortex-array/src/scalar_fn/fns/get_item.rs:205-209`). + - This recommendation keeps Arrow imports free of copies. It also allows a storage encoding to + store nulls as a reserved value, such as 0 for `NonZero`. + - The alternative requires a legal value in every slot. Then imports with nulls must copy, and a + type with no zero needs a nullable child. + ## 9. Nullability - **N1, short term.** `ExtDType::try_with_vtable` calls `validate_dtype` for both nullabilities of the storage dtype, and fails if either call fails. The documentation states that the result of - `validate_dtype` must not depend on nullability. `with_nullability` then never validates again. A - type that breaks the rule fails when its dtype is built, not later in a kernel. Calling - `validate_dtype` only with the non-nullable form is the alternative, but it hides such a type. + `validate_dtype` must not depend on nullability. `with_nullability` then builds the new dtype + directly and never validates again. Today it calls `try_with_vtable` + (`vortex-array/src/dtype/extension/typed.rs:91-98`). A type that breaks the rule fails when its + dtype is built, not later in a kernel. Calling `validate_dtype` only with the non-nullable form + is the alternative, but it hides such a type. - **N2, long term.** `ExtDType` stores a non-nullable storage dtype and a `Nullability`, like every other `DType` variant. The storage child has the dtype `storage_dtype.with_nullability(n)`. The wire format does not change, because the serialized storage dtype already carries nullability. @@ -586,10 +687,10 @@ pub trait ExtVTable { ``` - `Scalar::zero_value` and `ScalarValue::try_zero_value` use this hook for extension dtypes. -- Builders pad a non-nullable child under a null parent +- Builders fill a non-nullable child under a null parent (`vortex-array/src/builders/fixed_size_list.rs:303`, `vortex-array/src/builders/mod.rs:156-166`). - Value validation skips those rows (section 8.4), so a type with no zero still works as such a - child. The padding needs only a storage value. + Under the recommendation of XQ4, any value is allowed there. So the builder can keep the storage + zero, and the hook is not needed for padding. ## 11. Scalar identity and order @@ -598,7 +699,7 @@ pub trait ExtVTable { - `Scalar` derives `Eq` (`vortex-array/src/scalar/mod.rs:41`). `PartialEq` and `Hash` compare the dtype without nullability and the value (`vortex-array/src/scalar/scalar_impl.rs:336-356`). - `PartialOrd` returns `None` for different dtypes. Otherwise it compares the stored values - (`scalar_impl.rs:358-401`), so extension values compare by storage for every type (EXT-Q2). + (`scalar_impl.rs:358-474`), so extension values compare by storage for every type (EXT-Q2). Probe R2 shows `Some(Less)` for two `Opaque` values. - Floats use bitwise equality and `total_cmp` (`vortex-array/src/dtype/ptype.rs:328-336`). - `scalar_cmp` computes every comparison operator from `partial_cmp` @@ -606,11 +707,14 @@ pub trait ExtVTable { path use it. - A `Variant` value has no order, so `partial_cmp` returns `None` even when `==` holds (`scalar_impl.rs:449-451`). +- An extension over `Union` storage also gets `None` even when `==` holds. The union arm needs + `DType::Union`, but it receives the extension dtype (`scalar_impl.rs:441-443`). The tuple arm + unwraps extensions (`scalar_impl.rs:469-471`). - An extension scalar can hold a value of the wrong storage type. Then `==` can panic (EXT-V7, probe R2). - A valid `F16` scalar can hold a `U64` value, for compatibility with an old serialized form - (`vortex-array/src/scalar/validate.rs:147-157`). Then `==` panics in one direction and returns - `false` in the other (probe R10). + (`vortex-array/src/scalar/validate.rs:46-50`, `:147-157`). Then `==` panics in one direction + and returns `false` in the other (probe R10). ### 11.2 The Rust rule @@ -622,9 +726,14 @@ returns `None`. ### 11.3 Contract - `PartialEq`, `Eq` and `Hash` describe identity, as today: the same dtype without nullability, - and equal stored values. Integer and decimal values of different widths compare by value - (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-176`, - `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:255-281`). + and equal stored values. Decimal values of different widths compare by value + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:255-281`). Primitive values of different + widths compare by value when one converts to the other. Equality panics when the conversion + fails, for example `U8(5)` against `I8(-1)` + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-138`). A valid primitive value has + the `PType` of its dtype, except the `U64` form of an `F16` value + (`vortex-array/src/scalar/validate.rs:46-50`). So valid scalars of one dtype never reach the + panic after U6. - `PartialOrd` follows three steps: 1. If `a == b`, return `Some(Equal)`. 2. If both have one ordered dtype, ignoring nullability, return the order of the stored values. @@ -640,30 +749,41 @@ returns `None`. These steps keep the Rust rule. Step 1 returns `Some(Equal)` exactly when `==` holds. Under the stored-value order, two values that are not equal never compare as `Equal`, because float -equality is bitwise and float order is `total_cmp`. The steps also fix the `Variant` case. +equality is bitwise and float order is `total_cmp`. The steps also fix the `Variant` and `Union` +cases. ### 11.4 Relation to expression semantics - Identity serves Rust code: maps of constants, deduplication, tests and caches. It never decides an expression result for extension values. - Identity implies semantic equality, because one dtype and one storage value give one value. - The reverse does not hold. Without `eq`, or with a rule across dtypes, two values that are not - identical can be equal. -- Expression comparisons follow the binder. After binding, `scalar_cmp` sees storage scalars at - the top level. It must compute `=` and `!=` with `==`, and the order operators with - `partial_cmp`. Then `=` works on a container with an `eq`-only member, such as - `struct{v: Vector}`. -- The nested comparator follows the container rule of section 3.4. + The reverse does not hold. Without `eq`, two values that are not identical can be equal. +- Expression comparisons follow the derive rules. After the rules run, `scalar_cmp` and the nested + comparator see storage dtypes at every depth (section 4.3). So the order of extension scalars + never decides an expression result, and `scalar_cmp` keeps its current operators. For example, + `=` on `struct{v: Vector}` compares `struct{v: FixedSizeList}` storage, where the built-in + order agrees with `==`. ### 11.5 Order of work Change `PartialOrd` only together with, or after, the phase 2 change that rejects `min` and `max` -for extension types without `ord`. Today `min` and `max` merge partial results with -`partial_min(..).vortex_expect` (`vortex-array/src/aggregate_fn/fns/min/mod.rs:59`, -`vortex-array/src/aggregate_fn/fns/min_max/mod.rs:247`, `:252`). The zone-map writer binds them -for every extension column (`vortex-layout/src/layouts/zoned/writer.rs:217-231`). With the new -order, a `Json` zone that merges two different maxima panics. The merge of several partial results -in one zone is an inference from the code, not a run. +for extension types without `ord`. Today `min`, `max` and their variants merge partial results with +`vortex_expect` on `partial_min` or `partial_max`. The sites are +`vortex-array/src/aggregate_fn/fns/min/mod.rs:59`, `max/mod.rs:59`, `bounded_min/mod.rs:71`, +`bounded_max/mod.rs:86` and `min_max/mod.rs:247`, `:252`. With the new order, a merge of two +different `Json` maxima panics. + +- `ChunkedArrayAggregate` merges one partial for each chunk + (`vortex-array/src/arrays/chunked/compute/aggregate.rs:28-32`). So `max` over a chunked `Json` + column panics. +- DuckDB aggregate pushdown accumulates every split into one partial + (`vortex-duckdb/src/file_reader.rs:298-300`). It does not exclude `Json` + (`vortex-duckdb/src/table_function.rs:525-537`). +- The default zone-map writer is not affected. Each zone is one canonical chunk with its own + accumulator (`vortex-layout/src/layouts/repartition.rs:158-161`, + `vortex-layout/src/layouts/zoned/builder.rs:106-118`). + +These results come from reading the code, not from a run. ## 12. Files to change, by phase @@ -677,23 +797,25 @@ in one zone is an inference from the code, not a run. | 0, U6 | `vortex-array/src/arrays/extension/vtable/mod.rs`, `vortex-array/src/scalar/validate.rs`, `vortex-array/src/scalar/scalar_impl.rs` | | 0, U7 | `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/scalar_value.rs` | | 0, U8 | `vortex-array/src/dtype/extension/erased.rs` | -| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/scalar_fn/vtable.rs`, `vortex-array/src/aggregate_fn/vtable.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/scalar_fn/fns/{binary,between,dynamic.rs,ext_storage.rs}`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-layout/src/layouts/zoned/writer.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | +| 1 | `vortex-array/src/dtype/extension/vtable.rs`, `vortex-array/src/compute/conformance/`, `vortex-array/src/extension/tests/` | +| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/scalar_fn/vtable.rs`, `vortex-array/src/aggregate_fn/{vtable,erased}.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/optimizer/`, `vortex-array/src/scalar_fn/fns/{binary,between,cast,dynamic.rs,ext_storage.rs}`, `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-layout/src/layouts/zoned/writer.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | | 3 | `vortex-array/src/scalar_fn/fns/cast/mod.rs`, `vortex-array/src/extension/datetime/`, `vortex-duckdb/src/convert/expr.rs`, `vortex-datafusion/src/convert/exprs/mod.rs` | -| 4 | The files depend on the chosen physical form (section 6). Both forms change `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/` and `vortex-edition/`. The wrapper on top also changes `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/` and `vortex-compressor/src/compressor/constant/`. Generic encodings on top also change `encodings/sparse/src/canonical.rs` | -| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/builders/extension.rs`, `vortex-array/src/dtype/extension/` | +| 4 | `vortex-array/src/arrays/extension/compute/{rules,take}.rs`, `vortex-array/src/arrays/extension/vtable/mod.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/`, `encodings/sparse/`, `encodings/runend/`, `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/`, `vortex-edition/` | +| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/`, `vortex-array/src/builders/{extension,fixed_size_list,mod}.rs`, `vortex-array/src/dtype/extension/`, `vortex-array/src/extension/datetime/timestamp.rs`, the Arrow, DuckDB, FFI and Python import paths | | 6 | `vortex-arrow/src/session.rs`, `vortex-duckdb/src/convert/`, `vortex-duckdb/src/exporter/`, `vortex-datafusion/src/convert/`, `vortex-python/src/scalar/`, `docs/developer-guide/extending/extension-dtypes.md` | ## 13. Tests - **Path consistency.** A helper in `vortex-array/src/compute/conformance/` takes an expression builder and inputs. It evaluates the expression on the array path, on the row path, and with - constant-encoded inputs. It asserts equal values or an error on all three. The inputs include - unvalidated arrays with invalid values. + constant-encoded inputs. It asserts equal values or an error on all three. +- **Rule coverage.** The same expression gives the same result through `optimize`, + `optimize_ctx`, execution without optimization, and the row path. This test catches a rule that + one path does not run. - **Reference types.** `vortex-array/src/extension/tests/` already has `DivisibleInt`. Add: - - `Opaque` over `i64`, with no derives. Every semantic operation fails when it is bound. - - `DecimalLike` over `i64`, with `eq` and `ord` for each dtype and a rescale rewrite across - scales. A differential test runs each operation on `DecimalLike` and on the built-in - `Decimal`, and compares values or errors. + - `Opaque` over `i64`, with no derives. Typing rejects every semantic operation. + - `DecimalLike` over `i64`, with `eq` and `ord` for each dtype. A differential test runs each + operation on `DecimalLike` and on the built-in `Decimal`, and compares values or errors. - `Refined` over `i64` (for example, positive values only), with every derive and a `validate_array` override. - **Conformance harness for `ExtVTable`.** It checks the metadata round trip, that diff --git a/research/extension-types/agents/findings.md b/research/extension-types/agents/findings.md index 9373d444f87..fee5e0fed61 100644 --- a/research/extension-types/agents/findings.md +++ b/research/extension-types/agents/findings.md @@ -59,6 +59,7 @@ Terms: | EXT-V5 | `Timestamp` accepts dtypes that reject every value | S2 | code | | EXT-V6 | Eight extension types never check values | S3 | code | | EXT-V7 | Extension scalars skip the storage dtype check | S2 | probe | +| EXT-V8 | The `Timestamp` value check panics on out-of-range values | S2 | code | | EXT-N1 | A nullability change can panic | S2 | probe | | EXT-S1 | A constant extension column has two shapes | S3 | code | | EXT-S2 | Some encodings carry the extension dtype themselves | S3 | code | @@ -102,8 +103,8 @@ Terms: - **Expected:** One answer per expression, whatever the encoding and the read path. The type system RFC draft (`vortex-data/rfcs#55`) states this as a rule for every operation. - **Fix:** Give cast and compare one rule. Short term, make both implementations reject the cases - in EXT-C1, EXT-C2 and EXT-Q1. Long term, rewrite semantic operations at bind time so that both - paths evaluate the same expression (see `design-notes.md`). + in EXT-C1, EXT-C2 and EXT-Q1. Long term, rewrite semantic operations into storage operations + with rules that both paths apply (see `design-notes.md`). - **Regression test:** Add a helper that evaluates an expression three ways: on the array path, on the row path, and with constant-encoded inputs. The helper asserts equal values or an error in all three. Run it for every case in the table above. @@ -123,7 +124,7 @@ Terms: `vx.scalar(v, dtype=ext)` (`vortex-python/src/scalar/factory.rs:106`). - **Expected:** An error, unless the source and target are the same extension dtype with a different nullability. A conversion that changes meaning must go through a cast that the - extension type owns (issue #8570), or through explicit `ext_storage` and a future `ext_wrap`. + extension type owns (issue #8570), or through an explicit `ext_storage`. - **Fix:** Remove the wrap. Keep only the nullability case. - **Regression test:** The three cases above must return an error on the row path, the array path, and with constant inputs. @@ -198,7 +199,7 @@ Terms: - **Impact:** The result is wrong for any type whose storage order or storage equality differs from its value order or value equality. Examples in the repository: JSON (key order and whitespace), WKB geometry (two encodings of one geometry), vectors and tensors (no natural order). -- **Fix:** See `design-notes.md` (derives for one dtype that the binder reads). +- **Fix:** See `design-notes.md` (derives for one dtype, which typing and the derive rules read). ### EXT-Q3: `DateTimeParts` compare kernel trusts its operand types @@ -245,8 +246,9 @@ Terms: ### EXT-A2: Zone maps fall back to storage statistics - **Locations:** - - `vortex-layout/src/layouts/zoned/schema.rs:112-121`: when a statistic rejects the extension - dtype, the stats schema uses the storage dtype (comment at line 114). + - `vortex-layout/src/layouts/zoned/schema.rs:177-188`: when an aggregate rejects the extension + dtype, `aggregate_state_dtype` uses the storage dtype. `legacy_stats_table_dtype` does the + same for old files (lines 112-121). - `vortex-layout/src/layouts/zoned/writer.rs:217-231`: bounded min and max apply only to top-level `Utf8` and `Binary`. An extension over `Utf8` or `Binary` (JSON, WKB) stores exact, unbounded bounds. @@ -270,9 +272,10 @@ Terms: Scalar` [L2]. `ext_storage` over `NonZero [0, 1]` returns `[0, 1]` on the array path and panics on the row path [R9]. The row path reads the child as an extension scalar (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`). -- **Fix:** Return an error from these read paths. Make the row path of `ext_storage` read the - storage without a value check. Add value validation at trust boundaries (`design-notes.md`, - "Validation and trust"). +- **Fix:** Return an error from these read paths (U3). Make `ExtensionArray::try_new` check values + with a batch `validate_array`, and add `ExtensionArray::new_unchecked` for kernels + (`design-notes.md`, section 8). Make the row path of `ext_storage` read the storage without a + value check (phase 2). ### EXT-V2: `Extension::validate` never checks the storage dtype @@ -338,7 +341,10 @@ Terms: - `vortex-array/src/dtype/extension/erased.rs:107-109`: `validate_storage_value` calls only the `validate_scalar_value` of the type. - `vortex-array/src/scalar/validate.rs:46-50`: a primitive dtype requires a value of the same - `PType`. Extension storage never gets this check. + `PType`, except the `U64` form of an `F16` value. `Scalar::try_new` does not apply this check + to extension storage. + - `vortex-array/src/scalar/constructor.rs:256`: `Scalar::extension_ref` asserts the storage + dtype. So the gap is a raw `ScalarValue` passed to `Scalar::new` or `Scalar::try_new`. - `vortex-array/src/extension/datetime/timestamp.rs:211`: `Timestamp` casts any primitive value to `i64` before it checks the range. - `vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-160`: `PValue` equality and order @@ -349,13 +355,26 @@ Terms: `I64(5)` `i`. Then `i == f` panics with `upcast`, and `f == i` returns `false`. So `==` is not symmetric. `i.partial_cmp(&f)` also panics. - **Related primitive case** [R10]: a valid `F16` scalar can hold a `U64` value, for compatibility - with an old serialized form (`vortex-array/src/scalar/validate.rs:147-157`). `==` between the - `U64` form and the `F16` form of one value panics in one direction and returns `false` in the - other. + with an old serialized form (`vortex-array/src/scalar/validate.rs:46-50`, `:147-157`). `==` + between the `U64` form and the `F16` form of one value panics in one direction and returns + `false` in the other. The proto decoder already converts the old form + (`vortex-array/src/scalar/proto.rs:343`), so only direct construction makes the `U64` form. - **Reach:** Not measured. The probe builds the scalars with `Scalar::try_new` directly. - **Fix:** Check that an extension value matches the storage dtype before the check of the type. U6 covers it. Convert the `U64` form of an `F16` value to `F16` when the scalar is built. +### EXT-V8: The `Timestamp` value check panics on out-of-range values + +- **Location:** `vortex-array/src/extension/datetime/timestamp.rs:214-229`: `unpack_native` builds + a `jiff::Span` with `seconds`, `milliseconds`, `microseconds` or `nanoseconds`. These setters + call `expect` on their `try_` versions, so they panic on a value outside the range of a `Span`. + For seconds, the limit is 631,107,417,600. +- **Expected:** An error. `Scalar::try_new` and the value check of an array must not panic on + illegal data, for example an Arrow `timestamp[s]` column that holds `i64::MAX`. +- **Status:** code. Nothing ran. +- **Fix:** Use `Span::try_seconds` and the other `try_` setters, and return their error. U5 + covers it. + ### EXT-N1: A nullability change can panic - **Locations:** @@ -381,8 +400,8 @@ Terms: `TODO(connor): This is maybe incorrect unless this is a refinement type?`. - **Impact:** Kernels and extension crates must handle both shapes. `vortex-tensor` handles both by hand. The same choice exists for `Dict` and `Chunked`. -- **Fix:** Choose one physical form. `design-notes.md`, section 6, compares the two options. The - choice is open. +- **Fix:** A rewrite rule pulls structural arrays such as `Constant` and `Dict` out from under the + wrapper (decision X5 in `../../decisions.md`). `design-notes.md`, section 6, has the details. ### EXT-S2: Some encodings carry the extension dtype themselves @@ -595,7 +614,7 @@ any extension type. | `vortex-array/src/scalar/typed_view/extension/mod.rs:87-96` | Storage cast of a scalar, exact storage target only | Semantic (EXT-C2) | | `vortex-array/src/scalar/cast.rs:56-61` | Storage cast and wrap | Semantic (EXT-C1) | | `vortex-array/src/scalar/scalar_value.rs:90-95`, `builders/extension.rs:110` | Storage zero | Semantic (EXT-V3) | -| `vortex-layout/src/layouts/zoned/schema.rs:112-121` | Storage stats dtype | Semantic (EXT-A2) | +| `vortex-layout/src/layouts/zoned/schema.rs:112-121`, `:177-188` | Storage stats dtype | Semantic (EXT-A2) | | `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | Storage export through cast | Semantic (EXT-B2) | | `vortex-datafusion/src/convert/scalars.rs:138-145`, `schema.rs:105` | Storage conversion | Semantic (EXT-B1) | | `vortex-python/src/scalar/into_py.rs:93-95` | Storage value | Semantic (EXT-B3) | From 077f1b7401249036c41ba4df3eacd8ef5e5930d2 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 12:30:02 -0400 Subject: [PATCH 07/12] Record the second decimal review in the type research Add decisions D21 to D24 and update D8, D9, D11, D16, D17 and D20 after a confirmation review of the decimal and Narrow design. Signed-off-by: Connor Tsui --- research/README.md | 4 + research/decimal/01-current-state.md | 7 +- research/decimal/02-design.md | 357 ++++++-- research/decimal/03-roadmap.md | 138 ++- research/decimal/README.md | 65 +- research/decimal/agents/decimal-call-sites.md | 2 +- research/decimal/agents/report.md | 846 +++++++++++++----- research/decisions.md | 97 +- 8 files changed, 1145 insertions(+), 371 deletions(-) diff --git a/research/README.md b/research/README.md index 265619da89f..864c70ac2c0 100644 --- a/research/README.md +++ b/research/README.md @@ -4,6 +4,10 @@ This directory holds research notes about the Vortex type system. The notes are at `5fc5cb8` (2026-10-17). They live on the `ct/ext-types-research` branch and are not meant to be merged. +Reviews on 2026-10-22 and 2026-10-23 updated the Decimal notes and [`decisions.md`](decisions.md). +The decimal, DecimalByteParts, serialization and versioning code that the notes cite is the same on +`develop` at `dbaf5f8` (2026-10-22). + | Topic | Summary for people | Detail for agents | | --- | --- | --- | | Extension types | [`extension-types/`](extension-types/README.md) | [`extension-types/agents/`](extension-types/agents/) | diff --git a/research/decimal/01-current-state.md b/research/decimal/01-current-state.md index 6e438389ff8..853ab43f142 100644 --- a/research/decimal/01-current-state.md +++ b/research/decimal/01-current-state.md @@ -27,18 +27,21 @@ gives file and line references for each one. | Problem | Effect | | --- | --- | | One dtype has up to six physical forms | Kernels must handle mixed widths, widen on the fly, or test whether a constant fits the storage. `DecimalArray::patch` panics when the width cannot hold the dtype. Issue [#5820](https://github.com/vortex-data/vortex/issues/5820) collects the fuzzer crashes | +| The compressor chooses the width from the values | `narrowed_decimal` picks the smallest integer type that holds the values, not the width of a lower precision. So an `i32` buffer of a `Decimal(18, 2)` column can hold 2,000,000,000, which needs precision 10 | +| DecimalByteParts decodes to narrow leaves | Since 2025-05, the compressors put narrow values of 64 bits or less into DecimalByteParts. Its decode reuses the buffer of its most significant part, so the canonical result often has a narrow width. The only narrow `vortex.decimal` node that the compressors write is `I128` at precision 39 to 76 | | The canonical width is a convention, not a contract | 25 call sites compute it. The row codec uses it so that keys "from differently compressed chunks stay memcmp-comparable" | | Arrays never check the precision | Scalars check it, but arrays do not. So out-of-range values can reach kernels, and a comment in the arithmetic kernel says so | | The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel". A throwaway test confirmed the `f32` case | | `Decimal` duplicates `Primitive` | It has its own canonical variant (51 dispatch sites), builder, compare, arithmetic and aggregates. Decimal-specific files hold about 7,600 non-test lines | | DecimalByteParts carries the decimal dtype | It splits wide integers into 64-bit parts, but only decimals can use it | +| The DecimalByteParts cast rule skips a nullability check | For a cast that changes only the nullability, the rule casts the most significant part. It does not first make sure that the change is safe without execution, as other cast rules do with `Validity::trivially_cast_nullability` | | Unknown enum values decode silently | An unknown `values_type` decodes as `I8`. An unknown protobuf `PType` decodes as `U8`, and Variant metadata stores protobuf dtypes in files | | Shared code has decimal special cases | Bloom filters, `IN` lists, size statistics, sparse arrays, run-end decoding and the row codec each have a decimal path | | No 128-bit integer columns | DuckDB `HUGEINT` cannot be imported, although decimal storage already holds 128-bit values | The silent decoding causes no wrong reads today, because no writer emits unknown values. But it -blocks safe additions to both enums, so a fix is part of the urgent changes in -[part 3](03-roadmap.md#urgent-changes). +blocks safe additions to both enums, so a fix is part of the independent changes in +[part 3](03-roadmap.md#independent-changes). ## `Decimal` as a test of the extension design diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index 4d4ae69a1b4..d657348fd78 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -6,21 +6,25 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # 2. Design This part follows [`decisions.md`](../decisions.md), which records the decisions of the maintainer -from 2026-10-19. "Decided" marks a settled choice. "Recommendation" marks a proposal of these notes -that is still open. +from 2026-10-19, 2026-10-22 and 2026-10-23. "Decided" marks a settled choice. "Recommendation" +marks a proposal of these notes that is still open. ## Goal `Decimal` stays a built-in type that behaves like a hardcoded extension type (decided, D1). Each -decimal dtype has exactly one canonical encoding (decided, D3). Decimal kernels reuse the integer -kernels. +decimal dtype has exactly one canonical encoding (decided, D3), and a `Narrow` encoding keeps +narrow storage possible (decided, D4). Issue +[#10701](https://github.com/vortex-data/vortex/issues/10701) tracks this fix. The move of decimal +kernels onto the integer kernels is future work (decided, D20). ## The leaf A canonical decimal array is a leaf: one values buffer and validity, with no data child (decided, D2). `DecimalArray` and the `vortex.decimal` wire node already have this shape. -The precision fixes the width of the buffer: +The precision fixes the width of the buffer (decided, D12). The mapping is the existing +`DecimalType::smallest_decimal_value_type`, which gives the smallest signed width that holds every +value of the precision: ```rust impl DecimalDType { @@ -38,15 +42,15 @@ impl DecimalDType { } ``` -Recommendation (DQ1): use this mapping, which gives the smallest signed width that holds the -digits. It already exists as `DecimalType::smallest_decimal_value_type`. - The leaf keeps three invariants: 1. **Width.** The buffer has the width of the precision. Construction checks it, and the check costs nothing for each row. -2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. Imports from Arrow, - DuckDB, FFI and Python check it. Kernels that make new values check their own results. +2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `DecimalArray::try_new` + checks every valid value, and `new_unchecked` skips that check for kernels that keep values + legal (decided, D14). Imports use `try_new`. File reads trust the precision and never panic. + A read that makes a narrowing copy checks the precision, because it reads every value anyway + (decided, D21). 3. **Leaf.** The array has one buffer and validity, and no data child. So a kernel never receives a width that it does not expect. The `patch` panic and the "does the @@ -55,10 +59,52 @@ constant fit" special cases go away. `DecimalType` already has `I128` and `I256`. So the leaf needs no new `PType`, and this fix can land before the wide integers. +## Imports, exports and Arrow + +An import copies into the canonical width when the source has another width (decided, D13). It +writes zeros in null slots, and it fails on a value outside the precision. Two common imports +copy: + +- Arrow `Decimal128` data at precision 18 or below. This import is zero-copy today. +- DuckDB data at precision 1 to 2, which DuckDB stores as `i16`. + +The Arrow import logic can be configured to handle values outside the precision in another way. + +Exports widen as they do today. The Arrow export uses `Decimal128` for every precision up to 38, +because DataFusion does not fully support `Decimal32` and `Decimal64` yet. The DuckDB export uses +`i16` for precision 1 to 4. To widen an array, Vortex executes a `Narrow` with a wider dtype over +it. + +Arrow has the same model as Vortex. A decimal type has a precision and a scale, and each value is +an unscaled two's complement integer. arrow-rs also rejects a positive scale larger than the +precision. Arrow has one decimal type for each width: + +| Arrow type | Largest precision | Vortex width for that precision | +| --- | --- | --- | +| `Decimal32` | 9 | `i32` | +| `Decimal64` | 18 | `i64` | +| `Decimal128` | 38 | `i128` | +| `Decimal256` | 76 | `i256` | + +In Arrow, the width is part of the type, so `Decimal32(5, 2)` and `Decimal128(5, 2)` are two +types. A Vortex dtype has no width, so Vortex chooses one. From precision 5, the Vortex width is +the width of the smallest Arrow type that holds the precision. Arrow has no 8-bit or 16-bit +decimal, so precision 1 to 4 has no exact Arrow match. When engines support the small Arrow types, +exports for precision 5 to 18 can stop copying. + +Arrow has the same gap as Vortex at the top of each width (see +[Width and precision](#width-and-precision)). Arrow treats a value above the precision as invalid, +but arrow-rs does not check values when it builds an array. A separate call checks them, and +another call turns them into nulls. So an Arrow array can hold invalid values, and the Vortex +import returns an error for them. + ## The integer view +This section is future work (decided, D20). Its recommendations are not reviewed. + The integer view is a `PrimitiveArray` over the same buffer and validity. It needs no copy. -Decimal kernels delegate to integer kernels through it. +Decimal kernels can delegate to integer kernels through it. The `Narrow` work does not need the +view, because `Narrow` kernels pass work to the decimal kernels of the child (decided, D15). For precision 1 to 18, the view uses `I8` to `I64`, which exist today. For precision 19 to 76, the view needs `I128` and `I256` (decided, D5). Until those types exist, the kernels for wide decimals @@ -70,46 +116,208 @@ integer view plays that role. ## `Narrow` Without a narrow form, D3 forces the full width on every column. A `Narrow` encoding solves this -(decided, D4). It stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)`, with -`p' < p` and the same nullability. +(decided, D4). It stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)`. The child +follows three rules (decided, D10): -- `Narrow` has no buffers. Validity comes from the child. +- It has the same scale and the same nullability as the parent. +- Its canonical width is strictly narrower than the canonical width of `p`. So a `Narrow` from + `Decimal(76, s)` to `Decimal(75, s)` is not allowed, because both use `i256`. The integer + `Narrow` has the same rule, and the rule can be relaxed later. +- The producer chooses `p'`: the best precision that the values and the scale allow, or a larger + one. + +`DecimalDType::try_new` rejects a positive scale larger than the precision +(`vortex-array/src/dtype/decimal/mod.rs:68`). So `p'` is at least `s`, and the scale limits how far +a column can narrow: + +| Scale `s` | Narrowest child width | +| --- | --- | +| Up to 2 | `i8` | +| 3 to 4 | `i16` | +| 5 to 9 | `i32` | +| 10 to 18 | `i64` | +| 19 to 38 | `i128` | + +This limit matters only for a column whose values all have fewer unscaled digits than the scale. +Such values have an absolute value below 1. + +A `Narrow` has three properties: + +- It has no buffers. Validity comes from the child. - It decodes by widening the child buffer to the width of `p`. - It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. -Recommendation (DQ2): one `Narrow` encoding for integers and decimals. Integers narrow by width -(issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by -precision. The integer view of a decimal `Narrow` is an integer `Narrow` over the view of the -child. So the integer `Narrow` kernels give decimals their narrow-width speed. +One `Narrow` encoding serves integers and decimals, with one wire ID `vortex.narrow` (decided, +D8). Integers narrow by width (issue +[#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. +An integer child has the same signedness and nullability as its parent, and a strictly smaller +width (decided, D22). So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` +child. The metadata stores only the child `PType` or the child precision. The plugin of the parent +builds the child dtype from it. The child must have the scale and nullability of the parent, so +metadata that stored them would only let a file hold an invalid child. `vortex.narrow` stays in +memory only until the maintainer decides to write it to files (decided, D18). + +`vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level +integer columns (decided, D23). Internal integer children keep changing their own dtype, as they +do today. Examples are list offsets, dictionary codes, run ends and the DecimalByteParts MSP. +Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on +issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not +depend on it. + +`Narrow` kernels pass work to the child, and the fallback is to canonicalize (decided, D15): + +| Operation | Kernel on a decimal `Narrow` | +| --- | --- | +| Compare with a constant | If the constant is outside the range of the child dtype, the answer needs no values. Otherwise, compare the child with the constant at the child precision | +| Min and max | The result of the child, as a value of the parent dtype | +| Sum | The sum of the child, with the result dtype of the parent | +| Filter, take and slice | Run on the child | +| Cast | Pass the cast to the child, with the same target dtype | -Precision and integer width do not line up. An `i32` holds values up to 2,147,483,647, which needs -precision 10. But 9 is the largest precision that maps to `i32`. So a writer needs value bounds to -choose `p'`: +A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the same +scale follows three rules: -- The compressor has them from statistics. -- A cast that widens the precision has them from the source dtype. It returns a `Narrow` over the - source without a copy. -- An old narrow buffer, read without a scan, does not have them. DecimalByteParts holds it instead, - as the next section describes. +- When the canonical width grows, the cast returns a `Narrow` over its input. Nothing is copied + until execution. +- When the canonical width stays the same, for example from `Decimal(5, 2)` to `Decimal(9, 2)`, the + cast changes only the dtype. +- When the precision drops, the cast checks every value, and it fails on a value that does not + fit. -Recommendation (DQ3): decimal `Narrow` gets its own wire ID, unless it lands before the edition of -PR [#10333](https://github.com/vortex-data/vortex/pull/10333) ships. That draft edition already -defines `vortex.narrow` for integers. +### Width and precision -## DecimalByteParts +Precision limits are powers of ten, and integer widths are powers of two, so they do not line up. +The table shows the largest precision for each width: -DecimalByteParts (DBP) stays the compressed decimal form. Its children are integers: a signed most -significant part (MSP), and 64-bit lower parts for wide values. The ordinary integer schemes -compress each part. +| Width | Largest precision for the width | Largest unscaled value at that precision | Largest value of the width | +| --- | --- | --- | --- | +| `i8` | 2 | 99 | 127 | +| `i16` | 4 | 9,999 | 32,767 | +| `i32` | 9 | 999,999,999 | 2,147,483,647 | +| `i64` | 18 | 10^18 - 1 | about 9.2 * 10^18 | +| `i128` | 38 | 10^38 - 1 | about 1.7 * 10^38 | + +A value between the two limits fits the width, but it needs a precision of the next width. For +example, 2,000,000,000 fits `i32`, but it needs precision 10, whose canonical width is `i64`. So a +width alone does not give `p'`, and each producer finds `p'` from the values. A false child label +gives wrong results, because the `Narrow` rules trust the child dtype. Examples are the compare +shortcut and the cast that passes to the child. + +A gap column holds values in that range. In the final state, it uses the next width in memory +(decided, D19): + +| Column | Values | Width in memory today | Width in the final state | +| --- | --- | --- | --- | +| `Decimal(10, 2)` | 100.00 to 327.67 | `i16` | `i32`, at precision 5 | +| `Decimal(18, 2)` | 10,000,000.00 to 21,474,836.47 | `i32` | `i64`, at precision 10 | -Recommendation (DQ4): the dtype of a DBP array fixes the type of its MSP. Then narrowing happens in -only two places: `Narrow` for the precision, and integer encodings such as bit packing for the -bits. The current narrowing step of the compressor (`narrowed_decimal`) has no job left. +Disk size does not change, because integer encodings such as BitPacking store the same bits. +Exports do not change, because they already widen. Pushdown of sum, min and max into the parts of +DecimalByteParts can remove this cost for those operations later. -Readers still accept the narrow MSPs of old files. A reader wraps an old narrow MSP in an integer -`Narrow` of the fixed type, without a copy. So the in-memory MSP always has the fixed type. An old -narrow `vortex.decimal` leaf of up to 64 bits becomes a DBP array in the same way. Its buffer -becomes the child of that integer `Narrow`. +### Reading old narrow data + +Old writers chose the width from the values (`narrowed_decimal`), not from a precision. A reader +finds an old node because its stored width differs from the canonical width of its dtype (decided, +D7). It then converts the node (decided, D9). For a narrower node: + +1. The reader scans the non-null values to find their max. The read plugin cannot use stored + statistics, because the deserializer attaches them after the plugin returns. +2. It chooses the best precision that the values and the scale allow, or a larger one. +3. If the width of that precision is narrower than the canonical width of the column, the result + is a `Narrow` over a child at that width. Otherwise, the result is the canonical array. +4. The result reuses the buffer when its width is the stored width. Otherwise, the result is a + copy. + +For a wider node, the reader makes a narrowing copy. Null slots can hold any value (decided, D14), +so the max ignores them. + +Every narrowing copy follows one rule (decided, D21). It writes zeros in null slots, and it fails +on a valid value outside the precision of the target dtype. Three paths make such a copy: + +- An old node that is wider than its canonical width. +- An old narrow node whose best precision has a smaller width than the stored width. +- DecimalByteParts decoding with an MSP wider than the canonical width. + +The copy reads every value anyway, so the precision check costs nothing extra. For example, an old +`Decimal(4, 2)` node at `i32` with the unscaled value 20,000 fails, because 20,000 is outside +precision 4. A read without a copy trusts the precision and does not scan. Values in a buffer +narrower than the canonical width always fit the precision, so only wider nodes can hold values +above it. + +In the gaps, the result is a copy. For example, take an old `Decimal(4, 2)` node with an `i8` +buffer and values up to 1.27 (unscaled 127). The values need precision 3, and precision 3 and +precision 4 both use `i16`. No legal array keeps the `i8` buffer, so the result is a plain +`Decimal(4, 2)` array at `i16`. + +Old narrow `vortex.decimal` nodes are rare, so the speed of this conversion matters little. + +## DecimalByteParts + +DecimalByteParts (DBP) stays the compressed decimal form: + +- Each unscaled value is cut into 64-bit words of its two's complement bits. The most significant + part (MSP) holds the top word, and up to three lower parts hold the rest. With no lower parts, + the value is the MSP. +- The MSP is any signed integer up to 64 bits, with the nullability of the decimal, and it holds + the validity (decided, D16). Each lower part is a non-nullable `u64`, with zeros in null rows. + Today the constructor accepts a lower part of any unsigned width, and the module doc says that + each part can use a narrower dtype. But the split makes only `u64` parts, and the compressor + never narrows the dtype of a part. D16 tightens the constructor and the doc. This breaks no + file, because no file has lower parts. +- Each part can use any integer encoding. The parts are integers, so they promise only a width, + never a number of digits. +- One in-memory array has two wire forms. `vortex.decimal_byte_parts` (v1) has only the MSP. + `vortex.decimal_byte_parts.v2` adds lower parts, and no edition permits it yet. The writer picks + v1 when there are no lower parts. + +Operations run on the parts wherever they can (decided, D16). Today compare, between, filter, +take, slice, mask and the nullability cast do this. Every other operation decodes DBP first. + +Decoding produces the canonical array of the DBP dtype (decided, D16). Today it reuses the MSP +buffer at the MSP width. So a `Decimal(18, 2)` array with an `i32` MSP decodes to a decimal array +with an `i32` buffer. D3 removes that form, so decoding widens a narrower MSP. The MSP rule also +allows an MSP wider than the canonical width for precision up to 18, for example an `i64` MSP for +`Decimal(4, 2)`. Decoding narrows such an MSP with the copy of D21. A `TODO(connor)` in the code +marks a later optimization: find the max of the MSP, and return a `Narrow` that reuses its buffer. +It waits until benchmarks show large regressions. + +The compressor has one DBP scheme (decided, D17): + +- Today the scheme calls `narrowed_decimal`, which narrows the decimal array to the smallest + integer type that holds the values, and the split then copies its buffer into the MSP. The + canonical fix makes that narrow array illegal. Instead, the scheme takes the canonical array. + It uses the value range to choose the number of lower parts, because v1 has none. It also + narrows the MSP dtype, as the compressor narrows every internal child. The parts get the same + widths as today, so file contents do not change. +- The scheme is not temporary. Writers whose edition does not permit `vortex.narrow` need it. These + include default writers until `vortex.narrow` enters `core`, and writers that select an older + edition. +- When an edition permits `vortex.narrow`, a decimal `Narrow` scheme runs first and narrows the + decimal type. The DBP scheme then compresses the child. So the compressor writes a `Narrow` of + dtype `Decimal(p, s)` over a DBP child of dtype `Decimal(p', s)`. The MSP of that child usually + has the canonical width of `p'`, so decoding reuses its buffer. In the gaps, decoding widens it. + If the best precision has the canonical width of the column, the compressor writes DBP without + `Narrow`. +- For precision 39 to 76 with values that need more than 64 bits, the compressor uses the + lower-parts form when an edition permits it. Otherwise, it writes a plain `i256` array. Today it + writes a plain `i128` array, which D3 forbids. + +The `core` family holds the formats of the default writer +(`docs/specs/versioning/using-editions.md`). So default writers write these wide columns as plain +`i256` arrays until `vortex.narrow` and DBP v2 enter `core`. The maintainer accepts this cost. If +possible, the wire change ships in the same release as the in-memory change, so that writers that +select the preview edition never pay it (decided, D11). + +Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full +width in memory. This applies to existing files and to files that are written in that period. The +maintainer accepts this cost, because the bar of no compute regression applies to the final state +(decided, D19). + +The DBP cast rule has a bug that is independent of these changes. For a cast that changes only the +nullability, the rule casts the MSP without the check that other cast rules make with +`Validity::trivially_cast_nullability`. The fix lands as its own PR +([part 3](03-roadmap.md#independent-changes)). ## Where structural encodings sit @@ -124,6 +332,9 @@ in-memory form back to the legacy wire forms. ## Decimal operations as rules over the view +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + `Decimal` implements its semantics as rewrite rules on arrays that the type owns (decided, X4). Each rule runs integer kernels on the view and builds a decimal result. There is no `ext_wrap` expression. @@ -155,6 +366,9 @@ precision check in one pass. A fused integer kernel keeps that speed. ## Integer lanes or FixedSizeBinary +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + D2 and D3 fix the shape of the leaf. The remaining choice is the type of the values in the buffer, and of the view over it. @@ -214,6 +428,9 @@ Wide integers do not need FixedSizeBinary: ## Wide integers +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + - `PType` gets `I128`, `I256`, `U128` and `U256` (decided, D5), for example as the values 11 to 14. - Two typed sub-enums split the integers. `LaneIntPType` holds the eight integer types up to 64 bits, and `WideIntPType` holds the four wide types. Offsets, codes, take indices and lane @@ -247,6 +464,9 @@ Top-level wide columns need a rule in each engine: ## FixedSizeBinary +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + FixedSizeBinary is a core type for opaque bytes of a fixed length (decided, D6). It refines `Binary` in the same way that `FixedSizeList` refines `List`. @@ -268,26 +488,45 @@ layer is the right place for shared code. PR ## The file format -The leaf is the `vortex.decimal` wire node, so the decimal file format does not change. Readers -handle every existing decimal wire form: +The leaf is the `vortex.decimal` wire node, so the decimal file format does not change. The node +keeps its wire ID (decided, D7). The versioning spec asks for a new ID only when readers must +understand something new (`docs/specs/versioning/design.md`). D3 removes forms and adds none, so +old readers read every new file. The read plugin of `vortex.decimal` finds an old node by its +stored width, and it converts the node. Readers handle every existing decimal wire form: | Wire node | In memory | | --- | --- | | `vortex.decimal` at the canonical width | The leaf, zero-copy | -| `vortex.decimal` at a narrower width of up to 64 bits | DecimalByteParts with the buffer as its MSP, zero-copy | -| `vortex.decimal` at `I128` for precision 39 to 76 | The leaf, after a widening copy | -| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at low precision | The leaf, after a checked narrowing copy | +| `vortex.decimal` at a narrower width | The result of [Reading old narrow data](#reading-old-narrow-data): a `Narrow`, or the canonical array | +| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at a low precision | The leaf, after the narrowing copy of D21 with zeros in null slots. A value outside the precision is an error | | `vortex.decimal` with an unknown width | An error, not a silent `I8` | -| DecimalByteParts v1 and v2 | DecimalByteParts. An old narrow MSP becomes an integer `Narrow` | +| DecimalByteParts v1 and v2 | DecimalByteParts, unchanged. Decoding widens a narrower MSP and narrows a wider MSP | +| `vortex.narrow`, after the wire change | A `Narrow` over its child | Writers produce forms that old readers understand. The leaf becomes `vortex.decimal` at the canonical width. DBP becomes DecimalByteParts v1, which the first core edition already contains, -or v2 where an edition allows it. Decimal `Narrow` is written only where an edition allows its wire -ID. Constant, dictionary and run-end arrays of decimals keep their current wire forms. +or v2 where an edition permits it. `Narrow` is written only where an edition permits +`vortex.narrow`, and the compressor produces it only then. `vortex.narrow` and DBP v2 enter one +preview edition together, and later `core` together (decided, D18). Constant, dictionary and +run-end arrays of decimals keep their current wire forms. So wide `PType` values never reach the wire for decimal columns, and decimal statistics keep the -`Decimal` dtype. Before the reader changes, the compatibility fixtures need narrow, wide and -Arrow-style widths. Today they cover only `i32` and `i64`. +`Decimal` dtype. + +The compat harness (`vortex-test/compat-gen`) publishes fixtures from a released version, and the +current reader must read every published fixture. Only the old writer can make narrow leaves. So +the new fixtures land and are published from a release before the in-memory change (decided, +D24). They cover these cases: + +- Narrow, wide and Arrow-style widths. +- Narrow buffers with gap values. +- High scales at narrow widths. +- Nonzero values in null slots. +- DBP with a narrow MSP. + +The fixtures cover only reads that succeed. The error cases are unit tests, because the harness +expects every read to succeed. Examples are a value outside the precision in a narrowing copy, and +an unknown width. Today the decimal fixture covers only canonical `i32` and `i64` widths. ## `Decimal` stays built in @@ -307,24 +546,16 @@ the unscaled integers, and `price < 5` compares `500` with `5`. ## Open questions -The decisions file lists these questions as open. Each answer is a recommendation of these notes. +The decisions file lists these questions as open. They belong to future work (decided, D20), and +their recommendations are not reviewed. | ID | Question | Recommendation | | --- | --- | --- | -| DQ1 | Which width does each precision get? | The smallest signed width, as above. A wider function, such as the `I128` minimum of the Arrow export, avoids copies at Arrow import but costs memory | -| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes | -| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless it lands before the edition of PR #10333 ships | -| DQ4 | Does the dtype of a DBP array fix the type of its MSP? | Yes | | DQ5 | Are wide integers user-visible, and which kernels come first? | User-visible behind an edition, with a small kernel set first | | DQ6 | Does decimal arithmetic accept different precisions and scales? | Yes, with the Arrow rules. Comparisons keep the rule of equal dtypes | | DQ7 | How does FixedSizeBinary appear on the wire? | A new core dtype tag behind an edition | | CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error | -These notes add three smaller open questions: - -- What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist, - and tests use -128. DuckDB rejects negative scales, and Arrow allows them. -- When DBP decodes an old narrow MSP, does it compute value bounds and return a decimal `Narrow`, - or does it always widen? -- Do file reads check the decimal precision, or trust the file? Recommendation XQ3 for extension - values suggests trust, with checks only for what memory safety needs. +One smaller question is also future work. What is the lower bound of a negative scale? The docs +name a `MIN_SCALE` that does not exist, and tests use -128. DuckDB rejects negative scales, and +Arrow allows them. diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 8d138db7a7b..dba9cd8565c 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -9,7 +9,7 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors Many planned type changes depend on each other. The IDs below match the full graph in section 7 of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. -The decision IDs (D1 to D6, X1 to X9, C1) and the open questions (DQ1 to DQ7, XQ1 to XQ6, CQ1) +The decision IDs (D1 to D24, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) come from [`decisions.md`](../decisions.md). | ID | Work | Kind | @@ -42,6 +42,9 @@ come from [`decisions.md`](../decisions.md). | N27 | DuckDB `HUGEINT` and `UHUGEINT` import and export | Implementation | | N28 | One `Narrow` encoding for integers and decimals (D4) | In progress | +For decimals, N9, N12, N13, N14, N17, N18 and N27 are future work outside the canonical decimal fix +(D20). The decimal part of N5, the value check in `DecimalArray::try_new`, moves into N11 (D14). + ## Graph A solid line is a hard dependency: the later work must not land before the earlier work. A dashed @@ -116,59 +119,114 @@ Two edges need a note. N9 to N12 applies only to precision 19 to 76. The view fo 18 uses existing types, so it can land at any point. N11 does not depend on N9, because the leaf uses `DecimalType`, which already has 128-bit and 256-bit widths. -## Urgent changes +## Independent changes -These changes are small and independent. They are safe to make now: +These changes are small and independent, and they can land in any order. On 2026-10-23, none of +them is on `develop`: 1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a - silent `I8` or `U8`. Readers must have this change before any writer emits a new value. -2. **The DecimalByteParts nullability fix.** The cast rule of DecimalByteParts declines a - nullability change that the validity cannot make trivially. This is the first reusable part of - PR [#10230](https://github.com/vortex-data/vortex/pull/10230). -3. **One owner for the width function.** `DecimalDType::values_type()` replaces the 25 direct - calls to `smallest_decimal_value_type`, and producers take the width from the precision. This - is the first step of N11. -4. **Extension opacity fixes (N1).** These are the urgent changes in + silent `I8` or `U8`. Readers must have this change before any writer emits a new value. The + decimal part must also land before the read plugin converts old nodes, because the detection + of old nodes reads this field. +2. **The DecimalByteParts nullability fix.** For a cast that changes only the nullability, the + cast rule casts the MSP without the check that `Validity::trivially_cast_nullability` makes in + other cast rules. The fix adds that check, as PR + [#10230](https://github.com/vortex-data/vortex/pull/10230) did. It also adds a cast kernel for + the case that needs execution, so that a declined cast still runs on the parts. It lands as its + own PR with tests, and it needs no issue. +3. **Extension opacity fixes (N1).** These are the urgent changes in [part 4 of the extension notes](../extension-types/04-urgent-changes.md). -5. **The integer `Narrow` PRs.** PRs [#10223](https://github.com/vortex-data/vortex/pull/10223) - and [#10332](https://github.com/vortex-data/vortex/pull/10332) can merge as they are. Their only - integer-only check is the private `validate_dtypes`, so decimal support stays additive. +4. **The integer `Narrow` in memory.** The rewrite of PRs + [#10223](https://github.com/vortex-data/vortex/pull/10223) and + [#10332](https://github.com/vortex-data/vortex/pull/10332) goes up as new PRs, and the old PRs + close then. The rewrite makes `Narrow` in-memory only. Its only integer-only check is the + private `validate_dtypes`, so decimal support stays additive. The first part of PR + [#10333](https://github.com/vortex-data/vortex/pull/10333) (`NarrowArray::encode_values`) lands + with it. The fix for dictionary fill codes is its own PR. + [`decisions.md`](../decisions.md#the-open-narrow-pull-requests) lists the state of these PRs. ## One canonical form -N11 fixes the crashes of issue [#5820](https://github.com/vortex-data/vortex/issues/5820). The -decisions file splits the reusable parts of PR #10230 into small PRs. After the urgent changes, -the rest of N11 follows that order: +N11 fixes the crashes of issue [#5820](https://github.com/vortex-data/vortex/issues/5820), and +issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks it. It lands as two +changes: an in-memory change, then a wire change (decided, D11). A part of a change lands as its +own PR only if it needs no temporary code. Otherwise, it lands together with the parts that it +depends on. + +Benchmarks and fixtures come first: + +- Benchmarks for decimal compare, filter, sum, min, max and the Arrow and DuckDB exports, on + DecimalByteParts and on `Narrow`. They include a gap column and the decoding of DecimalByteParts + with a narrow MSP. They land on `develop` before the producers change, so that CodSpeed has a + base. The existing ones are + `vortex-array/benches/{compare,binary_ops,cast_decimal,filter_fixed_width}.rs`, + `encodings/decimal-byte-parts/benches/dbp_assemble.rs` and `vortex-arrow/benches/to_arrow.rs`. +- Compatibility fixtures for narrow, wide and Arrow-style widths, gap values, high scales at + narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP + ([part 2](02-design.md#the-file-format)). They are published from a release before the + in-memory change, because only the old writer can make narrow leaves (decided, D24). The error + cases, such as a value outside the precision in a narrowing copy, are unit tests, because the + compat harness expects every read to succeed. + +The in-memory change has these steps, in order. Each step is its own PR, because none of them +needs temporary code (decided, D11): + +1. Give the width function one owner. `DecimalDType::values_type()` replaces the 25 direct calls + to `smallest_decimal_value_type`. Producers and imports take the width from the precision + (decided, D12 and D13). +2. Add `Narrow` over decimals (N28). This step holds the child rules of D10, the kernels and casts + of D15, and the rule that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for + decimals, the compare kernel and the sum pushdown get decimal code in the same change. +3. Make DecimalByteParts decoding produce the canonical array of its dtype, with the + `TODO(connor)` of D16. Decoding widens a narrower MSP, and it narrows a wider MSP with the copy + of D21. Require `u64` lower parts, and correct the module doc, which allows narrower parts. +4. Make the `vortex.decimal` read plugin convert old nodes (decided, D9 and D21). The plugin scans + the values, because it cannot use stored statistics. +5. Make the DecimalByteParts scheme take the canonical array, and delete `narrowed_decimal` + (decided, D17). The scheme stays after the wire change, for writers whose edition does not + permit `vortex.narrow`. File contents stay the same. The exception is precision 39 to 76 with + values that need more than 64 bits, which becomes a plain `i256` array. +6. Make `DecimalArray::try_new` accept only the canonical width and check every valid value, with + `new_unchecked` for kernels (decided, D14). Builders follow, including `DecimalBuilder::extend`, + which accepts wider buffers (PR [#10155](https://github.com/vortex-data/vortex/pull/10155)). + +Then the `patch` panic and the "does the constant fit" special cases are unreachable. + +The wire change lands in lockstep (decided, D18). If possible, it ships in the same release as the +in-memory change (decided, D11). It holds three parts: + +- The `vortex.narrow` wire format for integers and decimals (decided, D8). This is the second part + of PR #10333. +- One new preview edition that permits `vortex.narrow` and DecimalByteParts with lower parts, + together with the other wire improvements. +- A decimal `Narrow` scheme that narrows the decimal type first, for writers that select that + edition. It runs before the DecimalByteParts scheme, which then compresses the child. -1. Decide DQ3, then merge PR [#10333](https://github.com/vortex-data/vortex/pull/10333). It adds - `vortex.narrow` to the draft edition `preview2026.10.0`. -2. Add `Narrow` over decimals (N28). When `validate_dtypes` opens for decimals, the compare kernel - and the sum pushdown need decimal code in the same change. -3. Make the compressor and DecimalByteParts decoding stop narrowing and return `Narrow` instead. -4. Make readers convert old buffers at other widths. Old narrow leaves of up to 64 bits become - DecimalByteParts arrays without a copy, with an integer `Narrow` as the most significant part. - An old I128 leaf at precision 39 to 76 needs a widening copy. The compatibility fixtures for - narrow, wide and Arrow-style widths come first. -5. Make `DecimalArray::try_new` accept only the canonical width. +Later, both formats enter `core` together. Two optimizations wait until benchmarks show that they +are needed. One is the max pass in DecimalByteParts decoding (D16). The other is the pushdown of +sum, min and max into the MSP for gap columns (D19). -Then the `patch` panic and the "does the constant fit" special cases are unreachable. Decimal -kernels can move onto integer kernels through the view (N12) at any point. +After the in-memory change, decoding a compressed decimal runs at the full width in memory until +the compressor narrows the decimal type first. This applies to existing files, and to files from +writers that do not select an edition with `vortex.narrow`. The maintainer accepts this, because +the bar of no compute regression applies to the final state (decided, D19). ## Order of work | Wave | Work | | --- | --- | -| 0, urgent | Strict enum decoding, the DecimalByteParts nullability fix, one owner for the width function, N1, and PRs #10223 and #10332 | -| 1, one canonical form | DQ3 and PR #10333. N28 for decimals. The rest of N11. N12 for precision 1 to 18 | +| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Narrow` in memory with the first part of PR #10333, and the decimal benchmarks and fixtures | +| 1, one canonical form | N11 as two changes (D11). The in-memory change holds the width owner, N28 for decimals, DecimalByteParts decoding, the read plugin, the DecimalByteParts scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds the `vortex.narrow` wire format, the preview edition with DecimalByteParts lower parts, and the decimal `Narrow` scheme | | 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins | | 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation | | 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 | | 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 | -Waves 1 and 2 can run in parallel. The decimal fix (N28, N11, N12 and N13) does not wait for the -extension work, because the built-in type owns its rules. Only the optional move to an extension -type (N15) waits for N1 to N4. +Waves 1 and 2 can run in parallel. The decimal fix (N28 and N11) does not wait for the extension +work, because the built-in type owns its rules. Only the optional move to an extension type (N15) +waits for N1 to N4. For decimals, waves 3 and 4 are future work (D20), and their recommendations +are not reviewed. ## Shared work between the tracks @@ -178,7 +236,7 @@ type (N15) waits for N1 to N4. N3). Phase 1 of the extension notes adds a decimal-like extension type over `i64`, and that type can run the same tests against the built-in `Decimal`. - The fixed-width structural kernels (N7) serve wide integers and FixedSizeBinary. -- One `Narrow` encoding (N28, recommendation DQ2) serves integers and decimals. +- One `Narrow` encoding (N28) with one wire ID serves integers and decimals (D8). - The placement rule (X5, recommendation XQ1) serves `Decimal` and every extension type. Structural encodings sit above, and storage encodings sit below. `DateTimeParts` under the `Timestamp` type is the temporal twin of DecimalByteParts under `Decimal`. @@ -191,8 +249,12 @@ type (N15) waits for N1 to N4. | A leftover `other => panic!` arm panics on a wide type | Retire those macros, and fuzz decimals and wide integers at every width | | An old reader reads a protobuf wide `PType` as `U8` | A new protobuf `oneof` variant, and strict readers early | | Rewritten arithmetic makes several passes over the data | Fused scaled-integer kernels, checked with the existing decimal and compare benchmarks | -| D3 without `Narrow` forces the full width on every column | `Narrow` for decimals lands before `try_new` rejects other widths | -| A narrow buffer does not give a precision | The compressor uses statistics. Old narrow leaves become DecimalByteParts arrays without a scan | -| Arrow import at another width copies the buffer | Accept the copy, or choose another width function in DQ1 | +| D3 without `Narrow` forces the full width on every column | `Narrow` for decimals lands in the in-memory change, before `try_new` rejects other widths | +| A narrow buffer does not give a precision | Every producer finds the precision from the values. Readers convert old nodes by their max (D9). DecimalByteParts decoding widens a narrow MSP (D16) | +| Compute on DecimalByteParts regresses when its decoding stops returning narrow leaves | In the final state, the compressor narrows the decimal type first, so decoding reuses the buffer. Until then, the maintainer accepts the cost (D19). The max pass of D16 can come later, and new benchmarks measure the cost | +| Gap columns use the next width in memory in the final state | Accepted (D19). Pushdown of sum, min and max into the MSP can come later | +| The conversion of old `vortex.decimal` nodes scans their buffer, because the plugin cannot use stored statistics | Old narrow nodes are rare, and the speed of this path matters little (D9) | +| Default writers use `core`, so they write precision 39 to 76 with values above 64 bits as plain `i256` arrays until `vortex.narrow` and DecimalByteParts with lower parts enter `core` | Accepted (D17). The wire change puts both into one preview edition. If it ships in the same release as the in-memory change, writers that select that edition never pay the cost (D11) | +| Arrow import at another width copies the buffer | Accepted (D12 and D13) | | The public API breaks: `DecimalArray::new::` with a `T` other than the canonical type | Constructors that convert, and errors that name the canonical width. The in-memory ID stays `vortex.decimal`, which the Python bindings use | | Wide integers reach users before engines can export them | Edition gates, and an explicit export rule for each engine | diff --git a/research/decimal/README.md b/research/decimal/README.md index a69d56de7d6..950ead89fc0 100644 --- a/research/decimal/README.md +++ b/research/decimal/README.md @@ -3,34 +3,55 @@ `Decimal` is a built-in dtype with the shape of an extension type: an integer storage value plus a scale. On `develop`, the decimal array does not keep to that shape. One decimal dtype can have six physical forms, and `Decimal` duplicates much of `Primitive`. These notes describe the target -design that the maintainer settled on 2026-10-19, and they put the type work that it needs in -order. [`decisions.md`](../decisions.md) records the decisions and the open questions. +design that the maintainer settled on 2026-10-19, 2026-10-22 and 2026-10-23, and they put the type +work that it needs in order. [`decisions.md`](../decisions.md) records the decisions and the open +questions. Issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks the fix of the +canonical decimal encoding. | Part | Content | | --- | --- | | [1. Current state](01-current-state.md) | How `Decimal` works on `develop`, and what is wrong | -| [2. Design](02-design.md) | The leaf, the integer view, `Narrow`, wide integers, FixedSizeBinary, and the open questions | -| [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the urgent changes, and the order of work | +| [2. Design](02-design.md) | The leaf, imports and Arrow, `Narrow`, DecimalByteParts, the file format, and the future work | +| [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the independent changes, and the order of work | ## Summary - The width of the decimal buffer does not depend on the precision. So one dtype has up to six physical forms, every kernel must handle mixed widths, and some paths panic. - Decided: a decimal array stays a leaf with one buffer and validity. Each decimal dtype has - exactly one canonical encoding, because the precision fixes the width of the buffer. + exactly one canonical encoding, because the precision fixes the width of the buffer. Each + precision gets the smallest signed width that holds it. +- Decided: `DecimalArray::try_new` checks the width and every valid value. Imports copy into the + canonical width, and they fail on a value outside the precision. File reads trust the precision. - Decided: a `Narrow` encoding stores a column as a child with a lower precision and the same - scale. So narrow storage stays possible. These notes recommend one `Narrow` encoding for - integers and decimals. -- Decimal kernels delegate to integer kernels through a zero-copy integer view of the buffer. Each - decimal operation is an integer operation plus bookkeeping for the scale. -- The view needs the new primitive types `I128` and `I256` for precision 19 to 76. The leaf does - not need them, so the fix for one canonical form can come first. -- Decided: Vortex adds `I128`, `I256`, `U128` and `U256`. None of them needs FixedSizeBinary. -- Decided: FixedSizeBinary becomes a core type for opaque bytes, such as UUIDs and hashes. -- The decimal file format does not change. Only decimal `Narrow` needs a new wire ID, behind an - edition. + scale. The child must have a narrower canonical width. One `Narrow` encoding serves integers and + decimals, and its kernels pass work to the child. An integer child keeps the signedness of its + parent. The wire form is only for columns with a fixed logical dtype, and internal integer + children keep changing their own dtype. +- Precision limits are powers of ten, and widths are powers of two. So a width does not give a + precision: an `i32` holds 2,000,000,000, which needs precision 10. Each producer finds the child + precision from the values. +- Decided: a reader finds an old narrow `vortex.decimal` node by its stored width. It scans the + values for their max, and it returns a `Narrow` or a copy. Every narrowing copy writes zeros in + null slots and fails on a value outside the precision. +- Decided: DecimalByteParts stays the compressed form. Operations run on its parts where they can, + and it decodes to the canonical array of its own dtype. One compressor scheme builds it from the + canonical array, so file contents stay the same. +- Decided: `vortex.narrow` stays in memory until the maintainer decides to write it. Then it + enters one preview edition together with DecimalByteParts with lower parts. Only then does a + decimal `Narrow` scheme narrow the decimal type first. Default writers use `core`, so until both + formats enter `core`, they write wide decimals with values above 64 bits as plain `i256` arrays. +- Decided: the bar of no compute regression applies to the final state. Until the compressor + narrows the decimal type first, decoding a compressed decimal runs at the full width in memory. +- Arrow has the same decimal model and the same gap at the top of each width. Its width is part + of the type, and from precision 5 the Vortex widths match the smallest Arrow types. +- Decided: `vortex.decimal` keeps its wire ID, and the decimal file format does not change. - Decided: `Decimal` stays a built-in type that behaves like a hardcoded extension type. A move to an extension type is an optional later step. +- Decided: Vortex adds `I128`, `I256`, `U128` and `U256`, and FixedSizeBinary becomes a core type. + Their designs are future work, outside the canonical decimal fix. So are the integer view, + decimal arithmetic across precisions, and casts that lose information. The recommendations for + this future work are not reviewed. ## Agent documents @@ -46,7 +67,13 @@ order. [`decisions.md`](../decisions.md) records the decisions and the open ques ## Evidence The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. On -2026-10-19 the review also read the draft `Narrow` PRs and PR #10230. One throwaway Rust test -confirmed the `f32` cast disagreement and the `ext_storage` pushdown rules (probes R1 and R6 in the -extension notes). One short Python script checked byte order. Section 10 of the report lists the -claims that are not verified, which include every performance claim. +2026-10-19 the review also read the draft `Narrow` PRs and PR #10230. On 2026-10-22 it read the +versioning spec, the array plugin and serialization code, and the DecimalByteParts code. It also +read the history of the decimal compression scheme, and PRs #10305 and #10314. On 2026-10-23 it +traced which statistics the compressors store on the DecimalByteParts parts, read the decimal +types of arrow-rs 59.3, and read the cast change of PR #10230. A second review on the same day +read when the reader attaches stored statistics, the DecimalByteParts constructor and compressor +scheme, the compat harness, and the edition families. One throwaway Rust test confirmed +the `f32` cast disagreement and the `ext_storage` pushdown rules (probes R1 and R6 in the extension +notes). One short Python script checked byte order. Section 10 of the report lists the claims that +are not verified, which include every performance claim. diff --git a/research/decimal/agents/decimal-call-sites.md b/research/decimal/agents/decimal-call-sites.md index 0fab6f128aa..89cda07df3e 100644 --- a/research/decimal/agents/decimal-call-sites.md +++ b/research/decimal/agents/decimal-call-sites.md @@ -98,7 +98,7 @@ these labels. The leaf design is decision D2 and D3 in `../../decisions.md`. | `vortex-array/src/builders/decimal.rs` | `DecimalBuilder` with six buffer variants | shrink: one width for each dtype, over `PrimitiveBuilder` | | `vortex-array/src/dtype/decimal/*` | `DecimalDType`, `DecimalType`, `NativeDecimalType`, i256 bound tables | keep `DecimalDType`, and keep `DecimalType` only for legacy metadata | | `vortex-array/src/scalar/typed_view/decimal/*` | `DecimalValue`, `DecimalScalar`, Arrow arithmetic rules | shrink: scalar view over a storage integer | -| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | keep as the compressed form. Its dtype fixes the type of its MSP (DQ4) | +| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | keep as the compressed form. Its MSP can stay narrow, and its lower parts are `u64` (D16). Decoding widens a narrower MSP to the canonical width of the dtype, and it narrows a wider one (D21). One compressor scheme builds it from the canonical array (D17) | | `vortex-arrow/src/executor/decimal.rs` | width adaptation to Decimal32/64/128/256 | keep, reads the one canonical width | | `vortex-duckdb/src/exporter/decimal.rs` | width adaptation to DuckDB widths | keep, reads the one canonical width | | `vortex-btrblocks/src/schemes/decimal.rs` | narrow then split | shrink: no narrowing, which moves to `Narrow` and integer encodings | diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index 92d3d9c6fae..265babf967e 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -9,9 +9,14 @@ This file is the detailed report behind the summary in `../README.md`. It is res accepted design, and it changes no code. Paths and line numbers are for `develop` at `5fc5cb8` (2026-10-17), the same commit as the extension-type notes in `../../extension-types/`. -The maintainer reviewed these notes on 2026-10-19. [`decisions.md`](../../decisions.md) records the -decisions (D1 to D6, X1 to X9, C1) and the open questions (DQ1 to DQ7, XQ1 to XQ6, CQ1, CQ2). This -report follows that file. If the two disagree, the decisions file wins. +The maintainer reviewed these notes on 2026-10-19, 2026-10-22 and 2026-10-23. +[`decisions.md`](../../decisions.md) records the decisions (D1 to D20, X1 to X9, C1) and the open +questions (DQ5 to DQ7, XQ1 to XQ6, CQ1, CQ2). This report follows that file. If the two disagree, +the decisions file wins. Issue #10701 tracks the fix of the canonical decimal encoding. + +D20 makes the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions +and scales, and casts that lose information future work. Sections 3.2, 3.4, 4, 5 and 6 describe +that work. Their recommendations are not reviewed. Audience: Vortex maintainers and coding agents who implement the type-system work. @@ -27,10 +32,12 @@ Conventions: - Paths are relative to the repository root. A `file:line` reference points at the commit above. - "Settled" marks a decision of the maintainer in the decisions file. "Recommendation" marks a proposal of these notes that the maintainer has not confirmed. -- "Canonical width" means the buffer width that the precision fixes (D3). These notes recommend - the existing function `DecimalType::smallest_decimal_value_type` for it (DQ1, - `vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, +- "Canonical width" means the buffer width that the precision fixes (D3). D12 settles the existing + function `DecimalType::smallest_decimal_value_type` for it + (`vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, 5-9 to I32, 10-18 to I64, 19-38 to I128, and 39-76 to I256. +- "Gap value" means a value that fits a width but needs a precision whose canonical width is + wider, for example 2,000,000,000 in an `i32` (section 3.3). A gap column holds such values. - "Integer view" means a `PrimitiveArray` over the buffer and the validity of a decimal array, without a copy (section 3.2). - "Storage" means the storage child of an extension array. @@ -73,29 +80,41 @@ Each point names the settled decision or the open question behind it. 1. Keep `DecimalArray` a leaf: one values buffer and validity, and no data child (D2). Give each decimal dtype exactly one canonical encoding, so the precision fixes the buffer width (D3). - Recommendation (DQ1): use the existing smallest-width mapping. -2. Give decimal kernels a zero-copy integer view: a `PrimitiveArray` over the same buffer and + Use the existing smallest-width mapping (D12). `try_new` checks the width and every valid + value, and imports copy into the canonical width (D13, D14). +2. Store narrow columns with a `Narrow` encoding (D4). A `Decimal(p, s)` column holds one child of + dtype `Decimal(p', s)` whose canonical width is narrower (D10). One `Narrow` encoding serves + integers and decimals (D8). An integer child keeps the signedness of its parent (D22). Its + kernels pass work to the child (D15). The wire form is only for columns with a fixed logical + dtype (D23). +3. Keep DecimalByteParts (DBP) as the compressed decimal form. Its MSP is any signed integer up to + 64 bits, and its lower parts are `u64` (D16). Operations run on the parts where they can. DBP + decodes to the canonical array of its own dtype. It widens a narrower MSP, and it narrows a + wider one (D21). +4. Convert old narrow `vortex.decimal` nodes by their max, found with a scan, into a `Narrow` or a + copy (D9). Every narrowing copy writes zeros in null slots and fails on a value outside the + precision (D21). +5. Build DBP with one scheme from the canonical array, with parts as narrow as the values allow, + so that file contents stay the same (D17). Once an edition permits `vortex.narrow`, a decimal + `Narrow` scheme narrows the decimal type first. +6. Keep every decimal wire format. Generic structural encodings sit above the decimal storage + encodings (X5, recommendation XQ1). The wire forms of today already have this shape. + `vortex.decimal` keeps its wire ID (D7). The only new wire ID is `vortex.narrow`, which serves + integers and decimals. It enters one preview edition together with DBP v2 (D18). +7. Keep `Decimal` a built-in type that behaves like a hardcoded extension type (D1). A move to an + extension type is an optional later step. + +Future work (D20), with recommendations that are not reviewed: + +8. Give decimal kernels a zero-copy integer view: a `PrimitiveArray` over the same buffer and validity. Decimal kernels delegate to integer kernels through the view. Rewrite rules that the type owns add the bookkeeping for the scale (X4). -3. Store narrow columns with a `Narrow` encoding (D4). A `Decimal(p, s)` column holds one child of - dtype `Decimal(p', s)` with `p' < p`. Recommendation (DQ2): one `Narrow` encoding for integers - and decimals. -4. Keep DecimalByteParts (DBP) as the compressed decimal form. Recommendation (DQ4): the dtype of a - DBP array fixes the type of its most significant part. Then `Narrow` narrows the precision, and - integer encodings such as bit packing narrow the bits. -5. Keep every decimal wire format. Generic structural encodings sit above the decimal storage - encodings (X5, recommendation XQ1). The wire forms of today already have this shape. Only - decimal `Narrow` needs a wire ID, behind an edition (DQ3). -6. Add `I128`, `I256`, `U128` and `U256` as primitive integer types (D5). Build them as a separate +9. Add `I128`, `I256`, `U128` and `U256` as primitive integer types (D5). Build them as a separate wide-integer tier with typed dispatch. Recommendation (DQ5): user-visible behind an edition, - with a small kernel set first. -7. The integer view needs `I128` and `I256` for precision 19 to 76. The leaf does not need them, - because `DecimalType` already has both widths. So the one-canonical-form fix can land before - the `PType` migration. -8. Add FixedSizeBinary as a core type for opaque bytes (D6). Decimal and wide integers do not - depend on it. Recommendation (DQ7): a new core dtype tag behind an edition. -9. Keep `Decimal` a built-in type that behaves like a hardcoded extension type (D1). A move to an - extension type is an optional later step. + with a small kernel set first. The integer view needs `I128` and `I256` for precision 19 to 76. + The leaf does not need them, because `DecimalType` already has both widths. +10. Add FixedSizeBinary as a core type for opaque bytes (D6). Decimal and wide integers do not + depend on it. Recommendation (DQ7): a new core dtype tag behind an edition. Section 7 gives the dependency graph. Section 8 gives the recommended order of work. @@ -378,7 +397,7 @@ The precision fixes the width of the buffer (D3). The width function gets one ow impl DecimalDType { /// The width of the values buffer for this precision. /// - /// This is the existing `DecimalType::smallest_decimal_value_type` (recommendation DQ1). + /// This is the existing `DecimalType::smallest_decimal_value_type` (D12). pub const fn values_type(&self) -> DecimalType { match self.precision.get() { 1..=2 => DecimalType::I8, @@ -414,19 +433,23 @@ Invariants: - **I1, width.** The buffer width equals `decimal_dtype.values_type()`. `DecimalData::validate` checks it next to the length and alignment checks of today (`vortex-array/src/arrays/decimal/array.rs:301-319`). The check costs nothing for each row. -- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. The places where - data enters establish it. These are the Arrow, DuckDB, FFI, and Python imports, and the value - check in `try_new` (node N5, X7). Kernels that produce new values (arithmetic, rescaling casts, - sums) check their own outputs, as `numeric/decimal.rs` does today with `DecimalValueBounds`. +- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `try_new` checks + every valid value, and `new_unchecked` skips that check for kernels that keep values legal + (D14, the rule of X7 for decimals). The Arrow, DuckDB, FFI, and Python imports use `try_new` + (D13). File reads trust the precision and never panic. Kernels that produce new values + (arithmetic, rescaling casts, sums) check their own outputs, as `numeric/decimal.rs` does today + with `DecimalValueBounds`. - **I3, leaf.** The array has one buffer and one validity slot (`vortex-array/src/arrays/decimal/array.rs:46-51`). It has no data child. The value of a row is `unscaled * 10^(-scale)`. Nothing else is stored. -Recommendation DQ1 keeps the I8 minimum. Section 2.11 lists three other width functions in the -engines. A function with an I128 minimum below precision 39 matches the Arrow export. It keeps -Arrow `Decimal128` imports zero-copy, but it costs up to 16 times the memory at small precisions. -Any of these functions fits the existing wire form, because `vortex.decimal` records its width. +D12 keeps the I8 minimum. Section 2.11 lists three other width functions in the engines. A +function with an I128 minimum below precision 39 matches the Arrow export. It keeps Arrow +`Decimal128` imports zero-copy, but it costs up to 16 times the memory at small precisions. With an +engine width, no precision maps to `i8` or `i16`, so decimal `Narrow` children lose those widths. +From precision 5, the D12 width equals the width of the smallest Arrow decimal type +that holds the precision (section 3.7). ### 3.2 Canonical form and the integer view @@ -438,6 +461,10 @@ Issue #5820 also proposed a decimal array with an integer child "that is separat like list offsets". D2 rejects that shape for the canonical form. The narrow representation lives in a separate `Narrow` encoding instead (section 3.3). +The rest of this section is future work (D20), and its recommendations are not reviewed. The +`Narrow` work does not need the view, because `Narrow` kernels pass work to the decimal kernels of +the child (D15). + The integer view is a `PrimitiveArray` over the same buffer and validity. It costs no copy. Its dtype is `Primitive(I_w, nullability)`, where `I_w` is the signed integer type of the canonical width: @@ -462,71 +489,248 @@ For extension types, `ext_storage` is the explicit unwrap to the storage, and it #### `Narrow` -A `Narrow` encoding stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)` with -`p' < p` (D4). The child has the same scale and the same nullability. `Narrow` has three -properties: +A `Narrow` encoding stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)` (D4). +The child follows three rules (D10): + +- It has the same scale and the same nullability as the parent. +- Its canonical width is strictly narrower than the canonical width of `p`. A `Narrow` from + `Decimal(76, s)` to `Decimal(75, s)` is not allowed, because both use `i256`. The integer + `Narrow` has the same rule: its `validate_dtypes` requires a strictly smaller byte width + (`vortex-array/src/arrays/narrow/mod.rs:81-84` in the local rewrite of #10223). The rule can be + relaxed later. +- The producer chooses `p'`: the best precision that the values and the scale allow, or a larger + one. + +`DecimalDType::try_new` rejects a positive scale larger than the precision +(`vortex-array/src/dtype/decimal/mod.rs:68`), so `p'` is at least `s`. The narrowest child is `i8` +for a scale up to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, and `i128` for 19 to +38. This limit matters only for a column whose values all have fewer unscaled digits than the +scale. arrow-rs has the same scale rule (`arrow-array-59.3.0/src/types.rs:1436`). + +`Narrow` has three properties: - It has no buffers. Validity comes from the child. - It decodes by widening the child buffer to the canonical width of `p`. - It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. -Recommendation (DQ2): one `Narrow` encoding for integers and decimals. Integers narrow by width -(issue #10020), and decimals narrow by precision. The integer view of a decimal `Narrow` is then -an integer `Narrow` over the integer view of the child. So the integer `Narrow` kernels give -decimals their narrow-width speed. - -Precision and integer width do not line up. An `i32` holds values up to 2,147,483,647, which needs -precision 10. But 9 is the largest precision that maps to `i32`. So a narrow buffer alone does not -give `p'`, and a writer needs value bounds to choose it: +One `Narrow` encoding, with one wire ID `vortex.narrow`, serves integers and decimals (D8). +Integers narrow by width (issue #10020), and decimals narrow by precision. An integer child has +the same signedness and nullability as its parent, and a strictly smaller width (D22). This is +the rule of `validate_dtypes` in the local rewrite. `vortex.narrow` stays in memory only until the +maintainer decides to write it to files (D18). + +`vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level +integer columns (D23). Internal integer children keep changing their own dtype with +`PrimitiveArrayExt::narrow`, as they do today. The btrblocks schemes call it for internal children +of the patch, sparse, RLE, temporal, VarBin, FSST and OnPair schemes +(`vortex-btrblocks/src/schemes/{patches,temporal}.rs`, `integer/{sparse,rle}.rs`, +`float/sparse.rs`, `binary/varbin.rs`, `string/{fsst,onpair,sparse}.rs`). The structural compressor +calls it for list offsets (`vortex-compressor/src/compressor/structural.rs:54`). No call narrows a +DBP part. +Issue #10733 keeps open whether `List` and `ListView` offsets get a fixed `I64` dtype with +`Narrow`. + +`Narrow` kernels pass work to the child, and the fallback is to canonicalize (D15): + +| Operation | Kernel on a decimal `Narrow` | +| --- | --- | +| Compare with a constant | A constant outside the range of the child dtype gives the answer without the values. Otherwise, the child is compared with the constant at the child precision | +| Min and max | The result of the child, as a value of the parent dtype | +| Sum | The sum of the child, with the result dtype of the parent (`sum_decimal_dtype`, `aggregate_fn/fns/sum/mod.rs:81-86`) | +| Filter, take and slice | Run on the child | +| Cast | Passes to the child with the same target dtype. A `Narrow` holds exactly the values of its child, and a cast depends only on the values and the target dtype | -- The compressor has them from statistics. -- A cast that widens the precision has them from the source dtype. It returns a `Narrow` over the - source without a copy. -- An old narrow buffer, read without a scan, does not have them. DBP holds it instead, as the next - part describes. +A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the same +scale returns a `Narrow` over its input when the canonical width grows. It changes only the dtype +when the width stays the same, and it checks every value when the precision drops. To widen an +array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it. -The integer `Narrow` is a draft stack today: PRs #10223, #10332 and #10333. Their only integer-only -check is the private `validate_dtypes`, so decimal support stays additive. When that check opens -for decimals, the compare kernel and the sum pushdown need decimal code in the same change. +Precision and integer width do not line up, because precision limits are powers of ten and widths +are powers of two. The table gives the largest precision for each width: -PR #10333 adds `vortex.narrow` to the draft edition `preview2026.10.0`, with one metadata byte that -holds the child `PType`. Recommendation (DQ3): decimal `Narrow` gets a second wire ID, unless it -lands before that edition ships in a release. A second wire ID needs no change to the open PRs. -The editions spec says that a change to what readers must understand needs a new wire ID -(`docs/specs/versioning/editions.md:118-121`). +| Width | Largest precision for the width | Largest unscaled value at that precision | Largest value of the width | +| --- | --- | --- | --- | +| `i8` | 2 | 99 | 127 | +| `i16` | 4 | 9,999 | 32,767 | +| `i32` | 9 | 999,999,999 | 2,147,483,647 | +| `i64` | 18 | 10^18 - 1 | about 9.2 * 10^18 | +| `i128` | 38 | 10^38 - 1 | about 1.7 * 10^38 | + +A gap value fits the width but needs a precision of the next width. For example, 2,000,000,000 +fits `i32`, but it needs precision 10, whose canonical width is `i64`. The largest value of each +width is always a gap value. So a narrow buffer alone does not give `p'`, and each producer finds +`p'` from the values. + +A false label gives wrong results, because the `Narrow` rules trust the child dtype. In the +`Narrow` stack, the compare kernel answers a constant that does not cast to the child dtype +without reading the values (`vortex-array/src/arrays/narrow/compare.rs`). For integers this is +correct, because an `i32` child never holds a value above `i32::MAX`. For a decimal child with a +false label, `x < 15,000,000.00` on a `Decimal(9, 2)` child returns true for the row +20,000,000.00. The cast rule passes a cast to the child when the child fits the target +(`narrow/rules.rs`). So a cast to `Decimal(9, 2)` returns that child unchecked. + +In the final state, a gap column uses the next width in memory (D19). For example, a +`Decimal(10, 2)` column with values from 100.00 to 327.67 has unscaled values from 10,000 to +32,767. Today DBP decodes it to an `i16` buffer. In the final state, its child has precision 5 and +an `i32` buffer. Disk size does not change, and exports already widen. Pushdown of sum, min and max +into the MSP can remove the cost for those operations later. + +#### Reading old narrow data + +Old writers chose the width from the values. `narrowed_decimal` +(`vortex-array/src/arrays/decimal/utils.rs:124-143`) picks the smallest integer type that holds +the minimum and the maximum, not the width of a lower precision. + +- **Detection (D7).** A reader compares the stored `values_type` with the canonical width of the + dtype. This costs nothing for each row, and it does not depend on which writer wrote the node. + An unknown `values_type` must fail first (section 8.3, P1), or the reader treats it as an old + `i8` node. +- **Narrower node (D9).** The reader scans the non-null values for their max. It chooses the best + precision that the values and the scale allow, or a larger one. If the canonical width of that + precision is narrower than the canonical width of the column, the result is a `Narrow` over a + child at that width. Otherwise, the result is the canonical array. The result reuses the buffer + when its width is the stored width, and it copies otherwise. Null slots can hold any value + (D14), so they do not count. +- **Wider node (D9).** The reader makes a narrowing copy (D21). +- **Narrowing copies (D21).** Every narrowing copy writes zeros in null slots, and it fails on a + valid value outside the precision of the target dtype. Three paths make one: a wider node, a + narrower node whose best precision has a smaller width than the stored width, and DBP decoding + with an MSP wider than the canonical width. The copy reads every value anyway, so the precision + check costs nothing extra. A read without a copy trusts the precision. A buffer narrower than + the canonical width cannot hold a value above the precision, because the largest value of each + narrower width is below `10^p - 1`. + +In a gap, the result is a copy. An old `Decimal(4, 2)` node with an `i8` buffer and values up to +1.27 holds unscaled values up to 127, which need precision 3. Precision 3 and precision 4 both use +`i16`, so no legal array keeps the `i8` buffer. The result is a plain `Decimal(4, 2)` array at +`i16`. + +The `vortex.decimal` read plugin returns a `Narrow` array. So it overrides `is_supported_encoding` +(`vortex-array/src/array/plugin.rs:154`), as the ALP plugin does for `Patched` +(`encodings/alp/src/alp/plugin.rs:90`). The deserializer asserts this +(`vortex-array/src/serde.rs:413`). `Decimal` and `Narrow` both live in `vortex-array`, so no other +crate needs to replace the plugin. + +The plugin does not see stored statistics. The deserializer attaches them after the plugin returns +(`vortex-array/src/serde.rs:419-424`). So the plugin scans the buffer, which is already in memory +(D9). A change that passes the statistics to plugins would change the `ArrayPlugin` trait for +every plugin, and the maintainer rejected it for this rare path. + +Such nodes are rare, and the speed of the conversion matters little (D9). The btrblocks +`DecimalScheme` always applies (`vortex-btrblocks/src/schemes/decimal.rs:122`). It narrows first +(`:133`), returns the narrowed leaf only for `I128` and `I256` in v1 mode (`:134-137`), and puts +every other width into DBP. This routing exists since commit `1b89b2f950` (2025-05-28). The compact +compressor did the same from commit `0036b1566d` (2025-09-08). On `develop` it is no longer +separate code. `with_compact` keeps the same `DecimalScheme` and changes only the excluded +schemes: the default mode excludes Pco, and the compact mode allows it +(`vortex-btrblocks/src/builder.rs:46-63`, `:190-193`). So a narrow `vortex.decimal` node is `I128` +at precision 39 to 76, or it comes from a write that skipped the decimal scheme. + +#### The integer `Narrow` PRs + +The integer `Narrow` is a draft stack: PRs #10223, #10332 and #10333. Section 8.3 and the decisions +file give their state on 2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory +only. It goes up as new PRs, and the old PRs close then. Its only integer-only check is the private +`validate_dtypes`, so decimal support stays additive. When that check opens for decimals, the +compare kernel and the sum pushdown need decimal code in the same change. + +The pushed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one +metadata byte that holds the child `PType`. Under D8, the metadata stores the child dtype: the +child `PType` for an integer parent, or the child precision for a decimal parent. The parent dtype +tells the reader which of the two it holds. Every release that permits an ID must read every valid +instance of it (invariant 6 in `docs/specs/versioning/design.md`). So the decimal reader ships in +the same release as the first edition that permits `vortex.narrow`. The second part of #10333 +moves into the wire change, which puts `vortex.narrow` and DBP v2 into one preview edition (D11, +D18). #### DecimalByteParts -DBP stays the compressed decimal form. Its children are integers: a signed most significant part -(MSP), and unsigned 64-bit lower parts for wide values. The ordinary integer schemes compress each -part (section 2.8). - -Recommendation (DQ4): the dtype of a DBP array fixes the type of its MSP. These notes propose the -types that `split_decimal` produces for canonical input -(`encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:84-121`). For precision 1 to 18, -the MSP has the canonical width. For precision 19 to 76, the MSP is `i64`, with one lower part for -`i128` values and three for `i256` values. - -Narrowing then happens in only two places. `Narrow` narrows the precision, and integer encodings -such as bit packing narrow the bits of the MSP. So `narrowed_decimal` -(`vortex-array/src/arrays/decimal/utils.rs:124-143`) has no job left. Today the btrblocks -`DecimalScheme` calls it first (`vortex-btrblocks/src/schemes/decimal.rs:133`). - -Readers still accept the narrow MSPs of old files. The DBP read plugin wraps an old narrow MSP in -an integer `Narrow` of the fixed type, without a copy. So the in-memory MSP always has the type -that the dtype fixes. The integer `Narrow` of PR #10223 carries the old width. - -A reader reads an old narrow `vortex.decimal` leaf as DBP in the same way. This applies when the -leaf is narrower than the canonical width and has 64 bits or less. Its buffer becomes the child of -that integer `Narrow`. -Today `DecimalParts::from_narrow` wraps such a buffer directly as the MSP -(`encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:60-65`). - -Today DBP decoding with no lower parts builds a leaf at the MSP width -(`encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-93`). Under D3 it must decode -to the canonical width. Under DQ4 this needs no special case for old files, because the integer -`Narrow` widens the old MSP when it decodes. When the decoder knows value bounds, it returns a -decimal `Narrow` instead (step 4 in section 8.3). +DBP stays the compressed decimal form. Its semantics: + +- **Value.** Each unscaled value is its two's complement bits, cut into 64-bit words. The value + is `msp * 2^(64k) + lower[0] * 2^(64(k-1)) + ... + lower[k-1]`, with `k` from 0 to 3 + (`encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:6-16`). +- **Parts.** The most significant part (MSP) is any signed integer up to 64 bits, with the + nullability of the decimal, and it holds the validity (`array.rs:97-99`). Each lower part is a + non-nullable `u64` (D16), and `split_decimal` writes zeros in its null rows. Today the + constructor accepts a lower part of any unsigned width (`array.rs:111-118`), and the module doc + says that each part can use a narrower dtype (`mod.rs:16`). But `split_decimal` makes only `u64` + parts (`split.rs:68-71`), and the compressor never narrows the dtype of a part. D16 tightens the + constructor and the doc. No file has lower parts: no edition permits v2, and the v2 compat + fixture is disabled (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:47`). + So the change breaks no file. Each part can use any integer encoding. The parts are integers, so + they promise only a width, never a number of digits. +- **Split.** `split_decimal` (`split.rs:98`) keeps a buffer of `i8` to `i64` as the MSP, with no + lower parts (`split.rs:60`). It splits `i128` into an `i64` MSP and one `u64` part, and `i256` + into an `i64` MSP and three `u64` parts. +- **Wire.** One in-memory array, whose ID is `vortex.decimal_byte_parts.v2` + (`plugin/mod.rs:38`), has two wire forms. v1 (`vortex.decimal_byte_parts`, `plugin/mod.rs:32`) + has exactly one child, and its metadata holds the MSP `PType` (`plugin/v1.rs:28`, `:70`). It is + in `core2025.05.0`. v2 adds the lower parts and their `PType`s, and no edition permits it yet. + The serializer picks v1 when there are no lower parts (`plugin/mod.rs:73`). +- **Decoding.** DBP is not canonical. `execute` decodes each part to a primitive array + (`array.rs:289-301`) and assembles a decimal leaf. With no lower parts, the leaf reuses the MSP + buffer at the MSP width (`assemble.rs:82-93`). One lower part gives `i128`, and two or three give + `i256`. +- **Kernels.** Cast (nullability only), filter, mask, slice and take reduce into the parts + (`rules.rs:14-18`). Compare and between run on the parts (`compute/kernel.rs:16-25`). Every + other operation, such as sum, min, max and arithmetic, decodes DBP first. + +The compressor narrows the values before it splits them. So DBP for `Decimal(18, 2)` often has an +`i32` MSP, and its decoded result is a leaf with an `i32` buffer. D3 removes that form. Under D3: + +- **Reading.** No change. A narrow MSP is already correct, because the MSP is an integer (D16). +- **Decoding (D16).** Decoding produces the canonical array of the DBP dtype. It widens an MSP + that is narrower than that width. For precision up to 18, the MSP rule also allows an MSP wider + than the canonical width, and decoding narrows it with the copy of D21. The btrblocks scheme + never writes such an MSP, because it splits the narrowed buffer. A `TODO(connor)` in + `assemble.rs` marks a later optimization: find the max of the MSP, and return a `Narrow` that + reuses its buffer. It waits until benchmarks show large regressions. +- **Operations (D16).** They run on the parts wherever they can. Decoding is the fallback. +- **Writing (D17).** The compressor has one DBP scheme. It takes the canonical array, uses the + value range to choose the number of lower parts (v1 has none), and narrows the MSP dtype as the + compressor narrows every internal child. The parts get the same widths as today, so file + contents do not change. The scheme stays after the wire change, for writers whose edition does + not permit `vortex.narrow`. When an edition permits it, a decimal `Narrow` scheme runs first, and + the DBP scheme compresses the child. The result is a `Narrow` of dtype `Decimal(p, s)` over a + DBP child of dtype `Decimal(p', s)`. The MSP of that child usually has the canonical width of + `p'`, so decoding reuses its buffer. In the gaps, decoding widens it. + +The scheme cannot keep `narrowed_decimal`. Today the scheme narrows the canonical array +(`vortex-btrblocks/src/schemes/decimal.rs:133`) and returns the narrowed leaf for `I128` and `I256` +in v1 mode (`:134-137`). Under D3 and D14, that leaf is illegal in memory. + +So DBP decoding needs no check of the values, and DBP needs no conversion at read time. + +Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full +width in memory. This applies to existing files, and to files from writers that do not select an +edition with `vortex.narrow`. The maintainer accepts this cost, because the bar of no compute +regression applies to the final state (D19). + +The stored statistics of the MSP do not help here. A review on 2026-10-23 traced them, and the +2026-10-22 recommendation DQ8, to always store min and max on the MSP, is dropped (D20): + +- The compressor computes min and max on the canonical MSP to estimate the integer schemes + (`vortex-btrblocks/src/schemes/integer/bitpacking.rs:129`, `for_.rs:155`). +- A BitPacked MSP keeps them (`bitpacking.rs:155`, `:201-206`). +- A FoR MSP has none, because the FoR node copies statistics from a newly built array + (`for_.rs:233-234`). Its inner BitPacked child holds the min and max of the shifted values only. +- A Pco MSP has none (`vortex-btrblocks/src/schemes/integer/pco.rs:55-69`). +- The layout writer computes statistics on the uncompressed chunk, but it does not pass them to + the DBP node or its MSP (`vortex-layout/src/layouts/compressed.rs:73`, `:114-127`). +- A read of part of a chunk slices it (`vortex-layout/src/layouts/flat/reader.rs:150-152`), and a + slice keeps only IsConstant, IsSorted and IsStrictSorted (`vortex-array/src/array/erased.rs:238-249`). +- FastLanes registers no min or max kernel, so `min_max` on a FoR or BitPacked array without + statistics decompresses it (`encodings/fastlanes/src/lib.rs:77-91`). + +The DBP cast rule has a bug that is independent of D3. For a cast that changes only the +nullability, it casts the MSP for every target nullability (`compute/cast.rs:16-35`). The contract +of cast rules in the optimizer asks them to call `Validity::trivially_cast_nullability`, to return +`None` when that call cannot prove the cast safe, and to pair the rule with a cast kernel for +execution (`vortex-array/src/validity.rs:447-500`). DBP has no cast kernel (`rules.rs:14`). PR +#10230 added the guard in 8 lines. The fix lands as its own PR, with a cast kernel and tests, and +it needs no issue (section 8.3, P2). Whether the current rule gives a wrong validity before +execution is not verified. #### Placement of structural encodings @@ -558,6 +762,9 @@ fails on a `Sparse` input with extension values. ### 3.4 Decimal semantics as rewrite rules +This section is future work (D20), and its recommendations are not reviewed. The casts between +precisions with the same scale are settled in D15 (section 3.3). + `Decimal` implements its semantics as rewrite rules on arrays that the type owns (X4). Each rule runs integer kernels on the integer view and builds a decimal result. There is no `ext_wrap` expression, because a rule builds its result array directly. @@ -634,21 +841,35 @@ Two points matter for the design: ### 3.5 Wire compatibility -The leaf is the `vortex.decimal` wire node, so it needs no new wire ID and no new dtype tag. Only -decimal `Narrow` needs a wire ID (DQ3). The `ArrayPlugin` mechanism already lets one in-memory -array read several historical wire forms (`vortex-array/src/array/plugin.rs:102-157`). +The leaf is the `vortex.decimal` wire node, so it needs no new wire ID and no new dtype tag +(D7). The versioning spec asks for a new ID only for "a change to what a reader must understand" +(`docs/specs/versioning/design.md`). D3 removes forms and adds none: + +- Every canonical leaf is a valid instance of the frozen `vortex.decimal` contract, so old readers + read new files. +- The format selection rule makes a serializer choose "the oldest supported writable format that + preserves the array's representation". For a canonical leaf that is always `vortex.decimal`, so + a writer never selects a v2 ID. +- Invariant 3 keeps readers of frozen formats forever, so a v2 ID removes no reader code. +- The existing v2 IDs each add a shape that old readers cannot parse. `fastlanes.for.v2` stores a + reference for each chunk as a child, and `fastlanes.bitpacked.v2` gives each block its own bit + width (`encodings/fastlanes/src/{for_,bitpacking}/plugin/mod.rs`). + `vortex.decimal_byte_parts.v2` adds lower parts. D3 is the opposite case. + +The only new ID is `vortex.narrow` (D8). The `ArrayPlugin` mechanism already lets one plugin read +several historical wire forms, and return an in-memory array of another encoding +(`vortex-array/src/array/plugin.rs:102-157`). Reading: | Wire node | In-memory result | | --- | --- | | `vortex.decimal`, `values_type` equals the canonical width | The leaf, zero-copy | -| `vortex.decimal`, `values_type` narrower and at most 64 bits (compressor narrowing) | DBP with an integer `Narrow` over the buffer as its MSP, zero-copy (section 3.3) | -| `vortex.decimal`, `values_type` I128 at precision 39 to 76 (compressor narrowing) | The leaf after a widening copy. DBP has no 128-bit MSP | -| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | The leaf after a checked narrowing copy. A value that does not fit the canonical width is an error, not a panic | +| `vortex.decimal`, `values_type` narrower (compressor narrowing) | The conversion of D9: a `Narrow` over a child with the best precision, or the canonical array in a gap (section 3.3) | +| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | The leaf after the narrowing copy of D21, with zeros in null slots. A value outside the precision is an error, not a panic | | `vortex.decimal`, unknown `values_type` | Error. Decode with `DecimalType::try_from(raw)`, not the prost getter | -| `vortex.decimal_byte_parts` (v1) and `vortex.decimal_byte_parts.v2` | DBP. An old narrow MSP becomes an integer `Narrow` of the fixed type | -| Decimal `Narrow`, with the wire ID of DQ3 | `Narrow` over its child | +| `vortex.decimal_byte_parts` (v1) and `vortex.decimal_byte_parts.v2` | DBP, unchanged. Decoding widens a narrower MSP (D16) and narrows a wider one (D21) | +| `vortex.narrow`, after the wire change | `Narrow` over its child | Writing: @@ -656,8 +877,8 @@ Writing: | --- | --- | --- | | The leaf | `vortex.decimal`, `values_type` equals the canonical width | Yes | | DBP with an MSP only | `vortex.decimal_byte_parts` v1 | Yes, core 2025.05 | -| DBP with lower parts | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2 | -| Decimal `Narrow` | The wire ID of DQ3 | Only when an edition allows it. Otherwise the compressor does not produce it | +| DBP with lower parts | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2. It enters one preview edition together with `vortex.narrow` (D18) | +| `Narrow` | `vortex.narrow` | Only when an edition allows it. Otherwise the compressor does not produce it (D17) | | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked` or `Masked` over decimal values | The same nodes as today | Yes | Consequences: @@ -668,32 +889,47 @@ Consequences: UHUGEINT, node N27) need new wire values and edition gating. - Decimal statistics and zone-map entries keep the `Decimal` dtype tag. They stay readable. - A writer without an edition for `Narrow` or DBP v2 loses one compact form. Today the v1 - compressor writes `vortex.decimal` at I128 for precision 39 to 76 when the values fit - (`vortex-btrblocks/src/schemes/decimal.rs:133-137`). Under D3, that writer stores I256 values. -- Extend the compat fixtures before the reader changes. Add narrowed widths (I8 at p 10) and wide - widths (I128 at p 20, I256 at p 40). Also add Arrow-style wider widths (I128 at p 5) and the - narrow 128-bit form (I128 at p 40). + compressor writes `vortex.decimal` at I128 for precision 39 to 76 when the values need more than + 64 bits (`vortex-btrblocks/src/schemes/decimal.rs:133-137`). Under D3, that writer stores I256 + values (D17). The `core` family holds the formats of the default writer + (`docs/specs/versioning/using-editions.md:47`), so default writers pay this cost until + `vortex.narrow` and DBP v2 enter `core`. A writer that selects the preview edition of D18 uses + DBP v2 instead. +- Extend the compat fixtures before the reader changes (D24). The harness publishes fixtures from + a released version, and the current reader must read all of them + (`vortex-test/compat-gen/README.md`). Only the old writer can make narrow leaves, so the new + fixtures are published from a release before the in-memory change. Add narrowed widths (I8 at + p 10) and wide widths (I128 at p 20, I256 at p 40). Also add Arrow-style wider widths (I128 at + p 5) and the narrow 128-bit form (I128 at p 40). Add gap values, such as 100 in an I8 buffer at + p 10, so that the copy path of D9 has a fixture. Add a high scale at a narrow width, such as I8 + at `Decimal(18, 10)`. Add nonzero values in null slots, and DBP with a narrow MSP. +- Test the error cases with unit tests, not compat fixtures, because the harness expects every + read to succeed. Examples are a value outside the precision in a narrowing copy, and an unknown + `values_type`. ### 3.6 What happens to `values_type` narrowing -Narrowing leaves the canonical form. Two mechanisms replace it: +Narrowing leaves the canonical form. Three mechanisms replace it: -- **Precision.** `Narrow` stores a column at a lower precision with the same scale (section 3.3). - The compressor picks `p'` from the value bounds in its statistics. -- **Bits.** The integer encodings narrow the MSP of DBP. FoR with BitPacking, ZigZag, Delta, RLE, - and Dict already narrow integers better than a fixed smaller width does. +- **Parts.** One DBP scheme builds the parts from the canonical array, as narrow as the values + allow (D17). The parts get the same widths as today, so file contents do not change. +- **Precision.** Once an edition permits `vortex.narrow`, the compressor narrows the decimal type + first. A `Narrow` stores the column at a lower precision with the same scale (section 3.3). The + compressor finds `p'` from the values. +- **Bits.** The integer encodings compress the MSP of DBP. FoR with BitPacking, ZigZag, Delta, + RLE, and Dict already narrow integers better than a fixed smaller width does. -`narrowed_decimal` and the first step of the btrblocks `DecimalScheme` go away. The compressor -returns a `Narrow` over a leaf or over a DBP array instead (step 4 in section 8.3). Legacy reads -keep working: old narrow leaves become DBP arrays, and old narrow MSPs become integer `Narrow` -arrays (section 3.5). +`narrowed_decimal` and the first step of the btrblocks `DecimalScheme` go away (section 8.3). +Legacy reads keep working. A reader converts old narrow leaves into a `Narrow` or a copy (D9). Old +DBP arrays read unchanged, and their decoding widens a narrow MSP (D16). `Narrow` replaces the lazy `widen` and `checked_narrow` nodes of an earlier version of these notes. It carries the generic comparison rule that DBP already has (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). A comparison of a narrow column with a constant compares the child when the constant fits the child. Otherwise the result is a constant. PR #10223 adds this rule for integers. The rule helps every narrow integer -column, not only decimals. +column, not only decimals. For a decimal child, the rule is correct only when the child dtype is +true. Every producer of a decimal `Narrow` finds the child precision from the values (D9, D10). ### 3.7 Builders, scalars, engines, and the public API @@ -708,13 +944,29 @@ column, not only decimals. at the canonical width. - **Accessors.** `DecimalArray::buffer::()` keeps its signature, and `T` is always the type of the canonical width. `values_type()` always returns the canonical width. `unscaled()` is new and - returns the integer view. `DecimalArray::new::` fails for any other `T` (step 6 in section - 8.3). + returns the integer view (future work, D20). `DecimalArray::new::` fails for any other `T` + (step 6 of the in-memory change in section 8.3). - **Engines.** The Arrow, DuckDB, and CUDA exporters read the leaf as today. Arrow export already widens buffers below 128 bits for `Decimal128` (`vortex-arrow/src/executor/decimal.rs:152-165`). - Arrow import copies when the Arrow width differs from the canonical width, for example for - `Decimal128` data at precision 18 or below. Today that import is zero-copy - (`vortex-arrow/src/convert.rs:204-277`). DQ1 decides this cost. + It uses `Decimal128` for every precision up to 38, because DataFusion does not fully support + `Decimal32` and `Decimal64` (`vortex-arrow/src/dtype.rs:337-346`). Arrow import copies when the + Arrow width differs from the canonical width, for example for `Decimal128` data at precision 18 + or below. Today that import is zero-copy (`vortex-arrow/src/convert.rs:204-277`). D13 accepts + this cost. The DuckDB export uses `i16` for precision 1 to 4 + (`vortex-duckdb/src/exporter/decimal.rs:124`). +- **Arrow model.** Arrow has the same model as Vortex: a precision, a scale, and unscaled two's + complement integers. In Arrow, the width is part of the type. `Decimal32`, `Decimal64`, + `Decimal128` and `Decimal256` hold precision up to 9, 18, 38 and 76 + (`arrow-data-59.3.1/src/decimal.rs:894`, `:820`, `:706`, `:56`). From precision 5, the D12 width + equals the width of the smallest Arrow type for the precision. Arrow has no 8-bit or 16-bit + decimal. The format comment that arrow-rs ships still says that only 128-bit and 256-bit values + are used (`arrow-ipc-59.3.1/src/gen/Schema.rs:3100-3102`). +- **Arrow gap.** Arrow has the same gap at the top of each width. arrow-rs does not check decimal + values when it builds an array: `with_precision_and_scale` checks only the precision and scale + (`arrow-array-59.3.0/src/array/primitive_array.rs:1642-1648`). `validate_decimal_precision` + checks the values (`:1652`), and `null_if_overflow_precision` turns invalid values into nulls + (`:1671`). So the Vortex import fails on such values by default (D13), and the Arrow import + logic can be configured to handle them in another way. - **Python.** The binding keys its `DecimalArray` subclass on the in-memory encoding id (`vortex-python/src/arrays/native.rs:174-175`). The id stays `vortex.decimal`. @@ -1126,16 +1378,16 @@ needs an **optional** node. | N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode` | | N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) | | N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types get `cast_to` and `cast_from` hooks. CQ1, the default for inexact values, is open | -| N5 | Value checks in `try_new` | impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. The decimal leaf runs its own precision check. Checks: decimal precision, UTF-8, FixedSizeBinary length | +| N5 | Value checks in `try_new` | impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. Checks: UTF-8 and FixedSizeBinary length. The decimal precision check moves into N11 (D14) | | N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes | | N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width | | N8 | Scaled-integer kernels | impl | `rescale`, `check_range`, fused scaled arithmetic and compare | | N9 | Wide-integer tier (I128, I256, U128, U256) | impl | Section 5 and D5. Recommendation (DQ5): user-visible behind an edition, with a small kernel set first | | N10 | Wire safety and core dtype gating | impl | Strict enum decoding, protobuf oneof for wide PTypes, edition kind for core dtype features | -| N11 | One canonical width for each decimal dtype | impl, urgent | D3. Producers take the width from the precision, readers convert old buffers, and `DecimalArray::try_new` accepts only the canonical width. Removes the crash class of issue #5820 | -| N12 | Zero-copy integer view | impl | Section 3.2. Precision 1 to 18 needs no new `PType`, so that part can land at any point. Precision 19 to 76 needs N9 | -| N13 | Decimal semantics as rewrite rules | impl | Section 3.4. Decimal kernels delegate to integer kernels through the view. Most decimal-specific kernels go away | -| N14 | Limb split for wide-integer columns | impl, optional | The split of DBP, offered as an encoding of wide-integer columns. DBP itself keeps its decimal dtype (DQ4) | +| N11 | One canonical width for each decimal dtype | impl, urgent | D3, tracked by issue #10701. An in-memory change and then a wire change (D11). Producers take the width from the precision, readers convert old buffers, DBP decodes to its own canonical width, and `DecimalArray::try_new` accepts only the canonical width and checks every value. Removes the crash class of issue #5820 | +| N12 | Zero-copy integer view | impl, future work (D20) | Section 3.2. Precision 1 to 18 needs no new `PType`, so that part can land at any point. Precision 19 to 76 needs N9 | +| N13 | Decimal semantics as rewrite rules | impl, future work (D20) | Section 3.4. Decimal kernels delegate to integer kernels through the view. Most decimal-specific kernels go away | +| N14 | Limb split for wide-integer columns | impl, optional, future work (D20) | The split of DBP, offered as an encoding of wide-integer columns. DBP itself keeps its decimal dtype (D16) | | N15 | Decimal as an extension type | impl, optional, later | D1 keeps the built-in type. A later change can remove it if the two become identical. Wire tag unchanged | | N16 | Wire translation layer for well-known types | research, then impl | Rust representation and wire representation chosen separately | | N17 | FixedSizeBinary core type | impl | D6 and section 6. Recommendation (DQ7): a new core dtype tag behind an edition | @@ -1148,7 +1400,7 @@ needs an **optional** node. | N24 | Refinement types (RFC 0055) | research only | UUID within FixedSizeBinary(16), Utf8 within Binary | | N26 | Scalar value model (RFC 0056) | research, then impl | Decides which types keep a host `Scalar` | | N27 | DuckDB HUGEINT and UHUGEINT import and export | impl | First user-visible wide-integer columns | -| N28 | One `Narrow` encoding | impl, in flight | D4. Integers first (PRs #10223, #10332, #10333, issue #10020), then decimals. Recommendation (DQ2): one encoding. DQ3 decides the decimal wire ID | +| N28 | One `Narrow` encoding | impl, in flight | D4. Integers first (PRs #10223, #10332, #10333, issue #10020), then decimals. D8 settles one encoding with one wire ID, `vortex.narrow`. D10 and D15 settle the decimal child and its kernels. The wire format waits for the wire change (D18) | (There is no N25. The Variant interaction is an edge on N10.) @@ -1269,8 +1521,8 @@ soft: the target works without the source, but it is cheaper, safer, or faster w and constant support without new per-type code. Take and filter already come from `FixedWidthArray`. 14. N28 to N11: the last step of N11 makes `DecimalArray::try_new` accept only the canonical - width. Before that step, the compressor and DBP decoding must return `Narrow` instead of - narrow leaves. Without `Narrow`, D3 forces the full width on every column (D4). + width. Before that step, the read plugin needs decimal `Narrow` to convert old narrow nodes + (D9). Without `Narrow`, D3 forces the full width on every column (D4). 15. N11 to N12 (soft): after N11, all arrays of one decimal dtype have one view dtype. Before N11, a delegated binary kernel must widen one side first. 16. N9 to N12: the view for precision 19 to 76 is `Primitive(I128)` or `Primitive(I256)`. The @@ -1285,8 +1537,9 @@ soft: the target works without the source, but it is cheaper, safer, or faster w 21. N10 to N17 (soft): needed only if FixedSizeBinary gets a new core wire tag. DQ7 recommends one, and then this edge becomes hard. 22. N12 to N13: the rules run over the integer view. -23. N28 to N13 (soft): the view of a decimal `Narrow` is an integer `Narrow`. So the decimal rules - get the narrow-width speed of the integer `Narrow` kernels. +23. N28 to N13 (soft): decimal `Narrow` kernels pass work to the decimal kernels of the child + (D15). Once N13 moves those kernels onto integer kernels, a decimal `Narrow` gets the speed of + the integer kernels without its own code. 24. N13 to N15: once decimal semantics are rules over the view, the optional move to an extension changes the API, not the behavior. 25. N16 to N15: the `Decimal` wire tag must translate to and from the extension, recursively. @@ -1314,11 +1567,15 @@ soft: the target works without the source, but it is cheaper, safer, or faster w ### 7.4 Critical path and parallel work -- **Critical path for one canonical form:** the integer `Narrow` PRs, then N28 for decimals, then - the rest of N11. The DBP nullability fix and the first step of N11 come before them. N11 does - not wait for N9. -- **Critical path for delegated decimal kernels:** N9, N12, N13. N8 must land before N13. N7 is - soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at any point. +- **Critical path for one canonical form:** strict decoding of the decimal width and the integer + `Narrow` in memory, then the benchmarks and fixtures, then N11 as two changes (D11). The + in-memory change holds the width owner, N28 for decimals, DBP decoding, the read plugin, the DBP + scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds + the `vortex.narrow` wire format, the preview edition with DBP v2, and the decimal `Narrow` + scheme. N11 does not wait for N9. +- **Critical path for delegated decimal kernels (future work, D20):** N9, N12, N13. N8 must land + before N13. N7 is soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at + any point. - **The optional move (N15):** after N13, with N1, N2 then N3, N4, and N16 joining at N15. - **Parallel tracks:** - Extension correctness: N1, N2, N3, N4, N5. @@ -1337,7 +1594,7 @@ onto the graph. | Extension research item | Node here | | --- | --- | -| U1 to U8, the urgent changes | N1, in wave 0. U5 and U6 are dtype and storage checks, not the value validation of N5 | +| U1 to U8, the urgent changes | N1, item P3 in section 8.3. U5 and U6 are dtype and storage checks, not the value validation of N5 | | U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 35 describes | | Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 | | Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show | @@ -1346,7 +1603,7 @@ onto the graph. | Phase 5, value checks in `ExtensionArray::try_new`, and nullability (X7) | N5 and N6 | | Phase 6, authoring and boundaries | N23 | -The decimal fix (N28, N11, N12, N13) does not wait for the extension phases. The built-in Decimal +The decimal fix (N28 and N11) does not wait for the extension phases. The built-in Decimal owns its rules (D1, section 3.4). The optional move (N15) needs phases 0, 2 and 3 first (N1, N3 and N4). @@ -1363,10 +1620,52 @@ The maintainer settled the main choices on 2026-10-19: - FixedSizeBinary is a core type for opaque bytes (D6). - `Decimal` stays a built-in type (D1). +On 2026-10-22 the maintainer settled the wire IDs: + +- `vortex.decimal` keeps its wire ID (D7). +- One `vortex.narrow` ID serves integers and decimals (D8). + +On 2026-10-23 the maintainer settled the rest of the canonical decimal fix, and revised D9 to D11: + +- Each precision gets the smallest signed width (D12). Imports copy into it and fail on a value + outside the precision (D13). `try_new` checks the width and every valid value (D14). +- A reader converts an old narrow `vortex.decimal` node by its max into a `Narrow` or a copy (D9). +- A decimal `Narrow` child has a strictly narrower canonical width, and the producer chooses its + precision (D10). `Narrow` kernels pass work to the child, and casts of a `Narrow` pass to the + child (D15). +- In DBP, the MSP is any signed integer up to 64 bits, and the lower parts are `u64`. DBP decodes + to the canonical array of its own dtype (D16). +- The compressor builds DBP from the canonical array, so file contents stay the same (D17). +- `vortex.narrow` stays in memory until the maintainer decides to write it. Then it enters one + preview edition together with DBP v2 (D18). +- The bar of no compute regression applies to the final state, and gap columns use the next + width in memory (D19). +- The work lands as an in-memory change and a wire change. PRs #10305 and #10314 are ignored for + now (D11). +- The integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and + scales, and casts that lose information are future work. The plan to always store min and max on + the MSP is dropped (D20). + +A second review on 2026-10-23 confirmed these decisions and added four: + +- Every narrowing copy writes zeros in null slots and fails on a value outside the precision + (D21). +- An integer `Narrow` child keeps the signedness of its parent (D22). +- `vortex.narrow` is only for columns with a fixed logical dtype (D23). +- The compat fixtures are published from a release before the in-memory change, and the error + cases are unit tests (D24). + +It also changed five decisions. The read plugin scans instead of using stored statistics (D9). +The `Narrow` metadata stores only the child precision (D8). Each in-memory step is its own PR +(D11). The DBP constructor and module doc require `u64` lower parts (D16). One DBP scheme serves +every edition, and default writers pay the `i256` cost until both formats enter `core` (D17). + These notes add the integer view of section 3.2, so that decimal kernels delegate to integer -kernels. Section 9 lists the open questions with the recommendations of these notes. +kernels. D20 makes it future work. Section 9 lists the open questions with the recommendations of +these notes. -Reasons for integer lanes and the integer view, in order of weight: +Reasons for integer lanes and the integer view, in order of weight. They support the future work +of D20, and they are not reviewed: 1. Every decimal operation is an integer operation plus scale bookkeeping. The view turns the decimal kernels into rules over shared integer kernels. FixedSizeBinary values keep every @@ -1386,91 +1685,120 @@ Reasons for integer lanes and the integer view, in order of weight: | A leftover `other => panic!` arm panics on a wide type | Retire the panic macros. Fuzz decimals and wide integers at every width | | Old readers silently read a protobuf wide PType as `U8` | Use a new protobuf oneof. Ship strict readers early (N10) | | Rewritten arithmetic makes several passes | Fused scaled-integer kernels (N8). Compare before and after with the existing `vortex-array/benches/binary_ops.rs`, `cast_decimal.rs`, and `compare.rs` | -| D3 without `Narrow` forces the full width on every column | Land `Narrow` for decimals before `try_new` rejects other widths (N28 before the end of N11) | -| A narrow buffer does not give a precision, because precision and integer width do not line up | The compressor uses statistics. Old narrow leaves become DBP arrays without a scan | -| Arrow import at a width other than the canonical width copies the buffer | Accept the copy, or choose another width function in DQ1. Arrow export already copies for widths below 128 bits | -| A writer without an edition for `Narrow` or DBP v2 stores precision 39 to 76 at 32 bytes, even when values fit 16 bytes | Allow `Narrow` or DBP v2 in an edition | +| D3 without `Narrow` forces the full width on every column | Land `Narrow` for decimals in the in-memory change, before `try_new` rejects other widths | +| A narrow buffer does not give a precision, because precision and integer width do not line up | Every producer finds the precision from the values. Readers convert old nodes by their max (D9). DBP decoding widens a narrow MSP (D16) | +| Compute on DBP regresses when its decoding stops returning narrow leaves | In the final state, the compressor narrows the decimal type first, so decoding reuses the buffer. Until then, the maintainer accepts the cost (D19). The max pass of D16 can come later. New benchmarks measure the cost (section 8.3, B1) | +| Gap columns use the next width in memory in the final state | Accepted (D19). Pushdown of sum, min and max into the MSP can come later | +| The conversion of old `vortex.decimal` nodes scans their buffer, because the plugin does not see stored statistics | Old narrow nodes are rare, and the speed of this path matters little (D9) | +| Arrow import at a width other than the canonical width copies the buffer | Accepted (D12, D13). Arrow export already copies for widths below 128 bits | +| A writer without an edition for `Narrow` or DBP v2 stores precision 39 to 76 at 32 bytes, even when values fit 16 bytes. Default writers use `core`, so this lasts until both formats enter `core` | Accepted (D17). The wire change puts `vortex.narrow` and DBP v2 into one preview edition (D18). If it ships in the same release as the in-memory change, writers that select that edition never pay the cost (D11) | | Public API breaks: `DecimalArray::new::` with a `T` other than the canonical type, and `DecimalBuilder::new::` | Constructors that convert, and errors that name the canonical width. Keep the in-memory id `vortex.decimal`, which the Python bindings key on (`vortex-python/src/arrays/native.rs:174-175`) | | Wide integers become visible to users before engines can export them | Writer gating (N10) and an explicit engine policy (section 5.6) | ### 8.3 Sequence -PR #10230 gives decimals an integer child, which D2 replaces. The decisions file splits its -reusable parts into six smaller PRs, in this order. This section calls them steps 1 to 6: - -1. The nullability fix in DBP. -2. Producers take the width from the precision. -3. `Narrow` over decimals. -4. The compressor and DBP decoding stop narrowing and return `Narrow` instead. -5. Readers convert old buffers at other widths. -6. `DecimalArray::try_new` accepts only the canonical width. - -Decimal kernels can move onto integer kernels through the zero-copy view at any point. - -Wave 0, small and independent: - -1. Strict decoding: `DecimalType::try_from(raw)` in the decimal deserializer - (`vortex-array/src/arrays/decimal/vtable/mod.rs:170`) and `PType::try_from(raw)` in the - protobuf dtype reader (`vortex-array/src/dtype/serde/proto.rs:36`). Part of N10. -2. Step 1: the nullability fix in the DBP cast rule. Today the rule casts the MSP for every - nullability change - (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`). PR #10230 makes - it decline a change that the validity cannot make trivially. -3. Step 2: producers take the width from the precision, which is the first part of N11. Add - `DecimalDType::values_type()` and route the 25 `smallest_decimal_value_type` calls through - it, so that the width function has one owner. -4. N1: the urgent extension changes U1 to U8. -5. Merge the integer `Narrow` PRs #10223 and #10332 as they are. Their only integer-only check is - the private `validate_dtypes`, so decimal support stays additive. - -Wave 1, one canonical form: - -6. Decide DQ3, then merge PR #10333. It adds `vortex.narrow` to the draft edition - `preview2026.10.0`, with one metadata byte that holds the child `PType`. -7. Add compat fixtures for narrow, wide, and Arrow-style widths, and for the narrow 128-bit form - (section 3.5). -8. Step 3: `Narrow` over decimals (N28). When `validate_dtypes` opens for decimals, the compare +PR #10230 gave decimals an integer child, which D2 replaces. It closed on 2026-10-19. Its fix to the +DBP cast rule becomes P2 below. The plan below replaces its other steps. + +Independent changes, in any order. On 2026-10-23, none of these is on `develop`: + +- **P1.** Strict decoding: `DecimalType::try_from(raw)` in the decimal deserializer + (`vortex-array/src/arrays/decimal/vtable/mod.rs:170`) and `PType::try_from(raw)` in the + protobuf dtype reader (`vortex-array/src/dtype/serde/proto.rs:36`). Part of N10. The decimal + part must land before step 4 of the in-memory change. +- **P2.** The DBP nullability fix (section 3.3). It adds the guard of #10230 to the cast rule + (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`), a cast kernel for + the case that needs execution, and tests for a nullable-to-non-nullable cast with nulls, without + nulls, and with only nulls. It is its own PR, with no issue. +- **P3.** N1: the urgent extension changes U1 to U8. +- **P4.** The integer `Narrow` in memory. The rewrite of #10223 and #10332 goes up as new PRs, and + the old PRs close then. The first part of PR #10333 (`NarrowArray::encode_values`) lands with it. + The fix for dictionary fill codes is its own PR. + +Benchmarks and fixtures, before the producers change: + +- **B1.** Benchmarks for decimal compare, filter, sum, min, max and the Arrow and DuckDB exports, + on DBP and on `Narrow`. They include a gap column and the decoding of DBP with a narrow MSP. + Record a CodSpeed base. The related benchmarks of today are + `vortex-array/benches/{compare,binary_ops,cast_decimal,filter_fixed_width}.rs`, + `encodings/decimal-byte-parts/benches/dbp_assemble.rs` and `vortex-arrow/benches/to_arrow.rs`. +- **B2.** The compat fixtures of section 3.5, published from a release before step 1 (D24). The + error cases are unit tests. + +The in-memory change (D11). Each step is its own PR, because none of them needs temporary code: + +1. Add `DecimalDType::values_type()` and route the 25 `smallest_decimal_value_type` calls through + it, so that the width function has one owner (D12). Producers and imports take the width from + the precision. Imports copy, write zeros in null slots, and fail on a value outside the + precision (D13). +2. `Narrow` over decimals (N28): the child rules of D10, the kernels and casts of D15, and the rule + that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for decimals, the compare kernel and the sum pushdown get decimal code in the same change. -9. Step 4: the compressor and DBP decoding return `Narrow`. `narrowed_decimal` goes away. -10. Step 5: readers convert old buffers. Old narrow MSPs, and old narrow leaves of 64 bits or - less, become DBP arrays with an integer `Narrow` as the MSP, without a copy. An old I128 leaf - at precision 39 to 76 needs a widening copy. -11. Step 6: `DecimalArray::try_new` accepts only the canonical width, which completes N11. The - `patch` panic, the fill-null upcast, and the "does not fit" constant tests become unreachable - and can go. Update tests that build narrow arrays on purpose, for example - `decimal_narrow_storage` (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). -12. N12 for precision 1 to 18, at any point in this wave. - -Wave 2, foundations, in parallel with wave 1: - -13. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. -14. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for +3. DBP: require `u64` lower parts, and correct the module doc (`mod.rs:16`). Make decoding produce + the canonical array of the DBP dtype, with the `TODO(connor)` of D16 in `assemble.rs`. Decoding + widens a narrower MSP, and it narrows a wider one with the copy of D21. +4. The `vortex.decimal` read plugin converts old nodes (D9, D21, section 3.3). It scans the values, + because it cannot use stored statistics. P1 must be on `develop` first. +5. The DBP scheme takes the canonical array, and `narrowed_decimal` goes away (D17). The scheme + stays after the wire change, for writers whose edition does not permit `vortex.narrow`. File + contents stay the same. The exception is precision 39 to 76 with values above 64 bits, which + becomes a plain `i256` array. +6. `DecimalArray::try_new` accepts only the canonical width and checks every valid value, with + `new_unchecked` for kernels (D14). The `patch` panic, the fill-null upcast, and the "does not + fit" constant tests become unreachable and can go. Update tests that build narrow arrays on + purpose, for example `decimal_narrow_storage` + (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). Builders follow, including + `DecimalBuilder::extend`, which accepts wider buffers (PR #10155). + +The wire change, in lockstep (D18), and if possible in the same release as the in-memory change +(D11): + +7. The `vortex.narrow` wire format for integers and decimals (D8), which is the second part of PR + #10333. +8. One new preview edition that permits `vortex.narrow` and DBP v2, together with the other wire + improvements. +9. A decimal `Narrow` scheme that narrows the decimal type first, for writers that select that + edition (D17). It runs before the DBP scheme, which then compresses the child. + +Later: + +10. `vortex.narrow` and DBP v2 enter `core` together (D18). +11. The max pass in DBP decoding (D16), and the pushdown of sum, min and max into the MSP for gap + columns (D19), if benchmarks show that they are needed. + +The waves below are future work for decimals (D20), and their recommendations are not reviewed. +Wave 2 also holds extension-type work, which D20 does not cover. + +Wave 2, foundations, in parallel with the canonical decimal fix: + +12. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. +13. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for the Timestamp unit casts of N4. -15. N4: cast rules (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. -16. N2, then N3: the rewrite rules that types own, then derives for one dtype. Use the decimal +14. N4: cast rules (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. +15. N2, then N3: the rewrite rules that types own, then derives for one dtype. Use the decimal rules of section 3.4 as acceptance tests. -17. N23: plugin registries for DuckDB and DataFusion type mapping. +16. N23: plugin registries for DuckDB and DataFusion type mapping. Wave 3, new types: -18. N9 in the order of section 5.3, with all four wide types. Recommendation (DQ5): user-visible +17. N9 in the order of section 5.3, with all four wide types. Recommendation (DQ5): user-visible columns only behind an edition. -19. N10: the core dtype edition kind and the protobuf oneof. -20. N17: FixedSizeBinary, after DQ7. -21. N5: the value check in `ExtensionArray::try_new`, and the precision check in - `DecimalArray::try_new`, starting at import boundaries. +18. N10: the core dtype edition kind and the protobuf oneof. +19. N17: FixedSizeBinary, after DQ7. +20. N5: the value check in `ExtensionArray::try_new`. The decimal check is step 6 of the in-memory + change. Wave 4, decimal kernels: -22. N12 for precision 19 to 76. -23. N13: rules over the view. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, +21. N12 for all precisions. Precision 19 to 76 needs N9. +22. N13: rules over the view. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, `decimal/utils.rs`, and the decimal aggregates. -24. N14, which is optional. -25. N18 and N27. +23. N14, which is optional. +24. N18 and N27. Wave 5, optional: -26. N16, then N15 if the built-in type and an extension type become identical. Revisit N19 and +25. N16, then N15 if the built-in type and an extension type become identical. Revisit N19 and N20 with the results of N22. ## 9. Open questions @@ -1478,20 +1806,16 @@ Wave 5, optional: ### 9.1 Questions in the decisions file The decisions file lists these questions as open. Each answer below is a recommendation of these -notes, which the maintainer has not confirmed. +notes, which the maintainer has not confirmed. DQ5, DQ6, DQ7 and CQ1 belong to future work (D20). | ID | Question | Recommendation | | --- | --- | --- | -| DQ1 | Which width does each precision get? | The smallest signed width that holds the digits. This is the existing mapping (`vortex-array/src/dtype/decimal/types.rs:46-56`), with an I8 minimum. Section 3.1 lists the cost of other functions | -| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes. Integers narrow by width (issue #10020). Decimals narrow by precision | -| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless decimal `Narrow` lands before the edition of PR #10333 ships in a release | -| DQ4 | Does the dtype of a DBP array fix the type of its MSP? | Yes. A reader wraps an old narrow MSP in an integer `Narrow` of the fixed type | | DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. First compare, arithmetic, cast, sum, min, max, hash and structural operations | | DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes (X3) | | DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | | XQ1 | Which arrays does the rule of X5 pull out, and what stays under the wrapper? | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. For `Decimal`, the wire forms already have this shape (section 3.3) | | XQ2 | Where do the rules of X4 live? | On `ExtVTable`, written over `ReduceNode`. The built-in `Decimal` owns the same kind of rules | -| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, and never panic. For decimals, these notes read this as a width check on conversion, with trust for the precision | +| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, and never panic. For decimals, D14 settles this: file reads trust the precision. A read that makes a narrowing copy reads the values anyway, so it checks the precision (D21) | | XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. The answer affects N18 | | CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error. Lossy steps such as `trunc` and `round` are separate expressions. The decimal-to-integer rule of section 3.4 keeps the current truncation until CQ1 is settled | @@ -1499,23 +1823,83 @@ notes, which the maintainer has not confirmed. 1. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist (`vortex-array/src/dtype/decimal/mod.rs:47`). Tests use -128. DuckDB rejects negative scales, - and Arrow allows them. + and Arrow allows them. This question is future work (D20). 2. PRs #10467 and #10468 leave two decisions open. One is the order of intervals: field-wise as in Arrow, or normalized as in DuckDB. The other is the edition membership of `vortex.duration` and `vortex.interval`. 3. Does `is_int()` mean any integer or a lane integer? Section 5.3 proposes to remove it, so that each of the 53 call sites chooses. -4. When DBP decodes an old narrow MSP, does it compute value bounds and return a decimal `Narrow`, - or does it always widen? Statistics of the MSP can make the bounds cheap. ## 10. Evidence log and limits Code read at commit `5fc5cb8`: the files cited in sections 2 to 6, plus the supporting inventories. Every count in this report comes from `grep` over the workspace, excluding `target`. -The maintainer decisions of 2026-10-19 are recorded in [`decisions.md`](../../decisions.md). -Sections 1 and 3 to 9 follow them. Section 3 replaced a wrapper design with one integer child, -which D2 rejects. +The maintainer decisions of 2026-10-19, 2026-10-22 and 2026-10-23 are recorded in +[`decisions.md`](../../decisions.md). Sections 1 and 3 to 9 follow them. Section 3 replaced a +wrapper design with one integer child, which D2 rejects. + +The review of 2026-10-23 read these files in the worktree of this branch, which has the code of +`5fc5cb8`: + +- `vortex-array/src/dtype/decimal/mod.rs:68`: a positive scale larger than the precision is + rejected, which limits how far a column can narrow (section 3.3). +- `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`, + `vortex-array/src/validity.rs:435-511` and `rules.rs:14`: the DBP cast rule and the contract + that it skips. The diff of PR #10230 for `cast.rs` adds the guard. +- `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:111-118` and `split.rs:68-71`: + the constructor accepts any unsigned lower part, but the split makes only `u64` parts. +- The statistics trace of section 3.3: `vortex-compressor/src/compressor/cascade.rs:82-102` and + `:302-327`, `vortex-btrblocks/src/schemes/integer/{bitpacking,for_,pco}.rs`, + `vortex-layout/src/layouts/compressed.rs:73` and `:114-127`, + `vortex-layout/src/layouts/flat/reader.rs:150-152`, `vortex-array/src/array/erased.rs:238-249` + and `encodings/fastlanes/src/lib.rs:77-91`. A subagent read them, and the main review checked + the cited lines. +- `vortex-btrblocks/src/builder.rs:46-63` and `:190-193`: the compact mode reuses the + `DecimalScheme`. +- `vortex-arrow/src/dtype.rs:337-346` and `vortex-duckdb/src/exporter/decimal.rs:124`: the export + widths. +- The integer `Narrow` check in the local rewrite of #10223 + (`vortex-array/src/arrays/narrow/mod.rs:73-88` in the worktree `vortex3`). + +The second review of 2026-10-23 read these files in the same worktree: + +- `vortex-array/src/serde.rs:413-424`: the encoding assertion and the stored statistics come after + the plugin returns, so the plugin cannot use them (D9). +- `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:16`, `array.rs:97-118` and + `split.rs:98-118`: the module doc and the constructor allow narrower lower parts, and the split + keeps a buffer of `i8` to `i64` as the MSP. +- `vortex-btrblocks/src/schemes/decimal.rs:100-166`: the DBP scheme narrows first, and it returns + the narrowed leaf for `I128` and `I256` in v1 mode. +- Every call of `PrimitiveArrayExt::narrow` in `vortex-btrblocks/src` and + `vortex-compressor/src`: none narrows a DBP part (D23). +- `vortex-test/compat-gen/README.md`, `src/fixtures/arrays/synthetic/arrays/decimal.rs`, + `encodings/decimal_byte_parts.rs` and `encodings/mod.rs:47`: how the harness publishes and + checks fixtures, the decimal fixture at canonical `i32` and `i64` widths, the DBP fixtures with + `i64` MSPs, and the disabled v2 fixture (D24). +- `docs/specs/versioning/using-editions.md:47`: the `core` family holds the formats of the default + writer (D17). + +The review of 2026-10-22 read these files at `5fc5cb8`. They are the same on `develop` at +`dbaf5f8`, except that the `min_max` short-circuit moved from line 86 to line 89: + +- `docs/specs/versioning/design.md` and `docs/specs/versioning/editions.md`: the rule for new + wire IDs, the format selection rule, the compatibility invariants, and the process for a new + component. +- `vortex-array/src/array/plugin.rs`, `vortex-array/src/session/mod.rs:68` and + `encodings/alp/src/alp/plugin.rs:90`: how a plugin reads several wire forms and returns another + encoding. +- `vortex-array/src/serde.rs:283`, `:413` and `:419-424`: which statistics a node writes, the + encoding assertion, and when the reader attaches stored statistics. +- `vortex-array/src/aggregate_fn/fns/min_max/mod.rs:86`: `min_max` returns cached statistics + first. +- Every file under `encodings/decimal-byte-parts/src/decimal_byte_parts/`: the DBP semantics of + section 3.3. +- `vortex-btrblocks/src/schemes/decimal.rs:122-137`, and the history of `narrowed_decimal` + (commits `1b89b2f950` and `0036b1566d`): where narrow decimal nodes come from. +- `encodings/fastlanes/src/{for_,bitpacking}/plugin/mod.rs`: what the existing v2 IDs add. +- The `Narrow` compare kernel and cast rule on the `Narrow` stack + (`vortex-array/src/arrays/narrow/{compare,rules}.rs`): the rules that trust the child dtype. RFC drafts read from the `vortex-data/rfcs` repository: @@ -1547,8 +1931,14 @@ GitHub, read only: | [PR #10468](https://github.com/vortex-data/vortex/pull/10468) | Interval extension type. The file list and the new `interval.rs` | | [PR #10223](https://github.com/vortex-data/vortex/pull/10223) | `Narrow` integer encoding, draft. The description, the diff, and the comments, read on 2026-10-19 | | [PR #10332](https://github.com/vortex-data/vortex/pull/10332) | Arithmetic and selection through `Narrow`, draft, stacked on #10223. Same depth, same date | -| [PR #10333](https://github.com/vortex-data/vortex/pull/10333) | `Narrow` through compression, and `vortex.narrow` in `preview2026.10.0`, draft, stacked on #10332. Same depth, same date | -| [PR #10230](https://github.com/vortex-data/vortex/pull/10230) | Canonical decimals with one integer child, draft, stacked on #10333. The description, the diff, and the comments, read on 2026-10-19 | +| [PR #10333](https://github.com/vortex-data/vortex/pull/10333) | `Narrow` through compression, and `vortex.narrow` in `preview2026.10.0`, draft, stacked on #10332. Same depth, same date. On 2026-10-22, the review read the CodSpeed reports of #10223, #10332 and #10333, and the diff of #10333 again | +| [PR #10230](https://github.com/vortex-data/vortex/pull/10230) | Canonical decimals with one integer child, stacked on #10333. The description, the diff, and the comments, read on 2026-10-19. It closed on 2026-10-19. On 2026-10-23, the review read its diff of the DBP cast rule again | +| [PR #10305](https://github.com/vortex-data/vortex/pull/10305) | Decimal storage width at most the canonical width, open. The description, read on 2026-10-22 and 2026-10-23 | +| [PR #10314](https://github.com/vortex-data/vortex/pull/10314) | Bounded decimal dispatch macros, draft, stacked on #10305. The description, read on 2026-10-22 and 2026-10-23 | +| [PR #10155](https://github.com/vortex-data/vortex/pull/10155) | `DecimalBuilder::extend` from wider buffers for DuckDB imports, merged. The commit message, read on 2026-10-22 | +| [Issue #10701](https://github.com/vortex-data/vortex/issues/10701) | Tracking issue for the canonical decimal fix, opened by the maintainer on 2026-10-22. The body, read on 2026-10-23. Its question about the gap of section 3.3 was answered on 2026-10-23 | +| [Issue #10733](https://github.com/vortex-data/vortex/issues/10733) | Tracking issue for `Narrow` over integers and decimals, opened by the maintainer on 2026-10-23. The body, read on 2026-10-23. It keeps the question about `List` and `ListView` offsets open | +| [Epic #10730](https://github.com/vortex-data/vortex/issues/10730) | Hardening of the type system, opened by the maintainer on 2026-10-23. The body, read on 2026-10-23 | | [Epic #10020](https://github.com/vortex-data/vortex/issues/10020) | Narrow integer encoding. The title and the first part of the body, read on 2026-10-19 | Verified outside the repository: @@ -1556,6 +1946,11 @@ Verified outside the repository: - prost-derive 0.14.4 enum getters fall back to the first variant on unknown values (`src/field/scalar.rs:286-300`, `src/lib.rs:308` in the cargo registry). - arrow-buffer 59.3.0 `i256` layout and order (`src/bigint/mod.rs:59-66`, `:171-183`). +- arrow-rs 59.3, the version of the workspace: the precision limits of the four decimal types + (`arrow-data-59.3.1/src/decimal.rs`), the scale rule (`arrow-array-59.3.0/src/types.rs:1436`), + the construction without a value check (`arrow-array-59.3.0/src/array/primitive_array.rs:1642-1671`), + and the format comment on decimal widths (`arrow-ipc-59.3.1/src/gen/Schema.rs:3100-3102`). +- A search of the GitHub issues on 2026-10-23 found no issue for the DBP cast rule. - Byte order against integer order, with the script in appendix A. Verified with a run: `../../extension-types/agents/probe_followup.rs` ran in a debug build of @@ -1580,9 +1975,20 @@ Not verified, because no run covered them: - The alignment of `i128` on targets other than x86_64 and aarch64, for example wasm32. - Whether the fuzzer reports in issue #5820 that end in a null decimal sum come from narrow storage. This report does not rely on that. -- Whether a decimal `Narrow` reaches the speed of the integer `Narrow` kernels. No benchmark - covers it. The benchmark comment on PR #10230 measures a different design. +- Whether a decimal `Narrow` reaches the speed of today's narrow leaves. No benchmark covers it. + The benchmark comment on PR #10230 measures a different design. - The cost of the copies at Arrow import and in readers that convert old buffers. +- The in-memory cost of the full-width DBP decoding between the in-memory change and the decimal + `Narrow` scheme (D19). +- The size of the `i256` cost of D17 for default writers. No data shows how many columns have + precision 39 to 76 and values above 64 bits. +- How often columns hold gap values. The gap is a large part of each integer range, so these notes + do not assume that it is rare. +- Which integer scheme usually wins for a real MSP, and so how often an MSP has stored min and + max. +- Whether the DBP cast rule gives a wrong validity before execution. No test covers it. +- The cause of the `take_map` regressions in the CodSpeed report of PR #10333. The benchmark + does not call the narrowing code that the PR changes. ## Appendix A: byte order script diff --git a/research/decisions.md b/research/decisions.md index 5dae6193b0c..7ffd12a5aa7 100644 --- a/research/decisions.md +++ b/research/decisions.md @@ -6,8 +6,8 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # Decisions This file records the decisions that the maintainer made while reviewing these notes on -2026-10-19. It also lists the questions that stay open. The other documents follow this file. If a -document disagrees with it, this file wins. +2026-10-19, 2026-10-22 and 2026-10-23. It also lists the questions that stay open. The other +documents follow this file. If a document disagrees with it, this file wins. "Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that the maintainer has not confirmed yet. @@ -41,6 +41,9 @@ the maintainer has not confirmed yet. ## Decimal and new physical types +Issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks the fix of the canonical +decimal encoding. + ### Settled | ID | Decision | Reason | @@ -51,15 +54,38 @@ the maintainer has not confirmed yet. | D4 | A `Narrow` encoding stores a decimal column as a child with a lower precision and the same scale | Without it, D3 forces the full width on every column, which costs performance | | D5 | Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types | Wide decimals need an integer view, and wide integers are useful as columns | | D6 | FixedSizeBinary becomes a core type for opaque bytes of a fixed length, such as UUIDs and hashes | Its defining operations (equality, hashing, byte order, cheap scalars) are core operations. Arrow and Parquet have it natively | +| D7 | `vortex.decimal` keeps its wire ID. A reader finds an old node when its stored width differs from the canonical width of its dtype | D3 removes forms and adds none. The versioning spec asks for a new ID only when readers must understand something new | +| D8 | One `Narrow` encoding, with one `vortex.narrow` wire ID, covers integers and decimals. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. The metadata stores only the child `PType` of an integer child, or only the child precision of a decimal child. It stores no scale and no nullability. This settles DQ2 and DQ3 | A release that permits an ID must read every valid instance of it. So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The child must have the scale and nullability of the parent, so metadata that stored them would only let a file hold an invalid child. The format is in no edition yet, so it can still change | +| D9 | A reader converts an old `vortex.decimal` node. For a narrower node, it scans the non-null values for their max, and it chooses the best precision that the values and the scale allow, or a larger one. The result is a `Narrow` over a child at the width of that precision, or the canonical array when that width is the canonical width of the column. The result copies the buffer when its width differs from the stored width. For a wider node, the reader makes a narrowing copy (D21) | A width does not give a precision. The read plugin cannot use stored statistics, because the deserializer attaches them after the plugin returns. Old narrow nodes are rare, so the speed of this path matters little | +| D10 | A decimal `Narrow` child has the same scale and nullability as its parent, and a canonical width strictly narrower than the canonical width of the parent. The producer chooses the child precision: the best precision that the values and the scale allow, or a larger one | A `Narrow` that does not reduce the width has no use. The integer `Narrow` has the same rule. The rule can be relaxed later | +| D11 | The canonical-width work lands as an in-memory change, and then as a wire change (D18). Each step of the in-memory change is its own PR, because none of the steps needs temporary code. The wire change lands in lockstep, if possible in the same release as the in-memory change. PRs [#10305](https://github.com/vortex-data/vortex/pull/10305) and [#10314](https://github.com/vortex-data/vortex/pull/10314) stay open, and the work ignores them for now | Small PRs are easier to review, but not when a split needs temporary code or changes file contents twice. In one release, writers that select the preview edition never pay the `i256` cost of D17 | +| D12 | Each precision has the smallest signed width that holds its values: `i8` for precision 1 to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, `i128` for 19 to 38 and `i256` for 39 to 76. This settles DQ1 | It is the existing `smallest_decimal_value_type`. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision | +| D13 | Imports copy into the canonical width. They write zeros in null slots, and they fail on a value outside the precision. The Arrow import logic can be configured to handle such values in another way. To widen an array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it | arrow-rs does not check decimal values when it builds an array. An error is better than a silent change of the data | +| D14 | `DecimalArray::try_new` checks the width and every valid value. `new_unchecked` skips the value check, for kernels that keep values legal. File reads trust the precision and never panic | This is the rule of X7 for decimals, and the recommendation XQ3 for file reads | +| D15 | `Narrow` kernels pass work to the child, with canonicalization as the fallback. A `Narrow` over a `Narrow` becomes one `Narrow`. A cast of a `Narrow` passes to the child with the same target dtype. Another decimal cast to a precision with the same scale returns a `Narrow` when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops | A `Narrow` holds exactly the values of its child, so work on the child gives the same result on less data | +| D16 | In DecimalByteParts, the MSP is any signed integer up to 64 bits, and every lower part has the dtype `u64`. The constructor and the module doc become stricter. This settles DQ4. Operations run on the parts wherever they can. Decoding produces the canonical array of the DecimalByteParts dtype. It widens a narrower MSP, and it narrows a wider MSP with the copy of D21. A `TODO(connor)` marks a later optimization: find the max of a narrow MSP, and return a `Narrow` that reuses its buffer | A narrow MSP needs no conversion at read time. The stricter constructor breaks no file, because no file has lower parts: no edition permits v2, and its compat fixture is disabled. The compressor never narrows the dtype of a part. Simple decoding is correct, and the optimization waits until benchmarks show large regressions | +| D17 | The compressor has one DecimalByteParts scheme. The scheme takes the canonical array, uses the value range to choose the number of lower parts, and narrows the MSP dtype as the compressor narrows every internal child. This replaces `narrowed_decimal` and keeps file contents. The scheme stays after the wire change, for writers whose edition does not permit `vortex.narrow`. When an edition permits `vortex.narrow`, a decimal `Narrow` scheme runs before it and narrows the decimal type first. In an edition without the lower-parts format, precision 39 to 76 with values that need more than 64 bits is written as a plain `i256` array. The `core` family holds the formats of the default writer, so default writers pay this cost until both formats enter `core` | The canonical fix makes a narrow decimal array illegal in memory, so the scheme cannot call `narrowed_decimal`. File contents do not change before the wire change. The `i256` cost affects only very wide decimals with values above 64 bits. The only way around it writes the old narrow form into new files, and it needs a special serializer | +| D18 | `vortex.narrow` stays in memory only until the maintainer decides to write it to files. Every new wire format enters a preview edition before `core`. `vortex.narrow` and DecimalByteParts with lower parts enter one preview edition together, with the other wire improvements, and later `core` together. This settles DQ9 | In-memory changes land first, so the edition change stays small | +| D19 | The bar of no compute regression applies to the final state, not to the steps in between. Missing benchmarks land before the changes that they cover. A gap column holds values that fit a width but need the precision of the next width. In the final state, such a column uses the wider width in memory, and that cost is accepted. Pushdown of sum, min and max into the MSP can come later | A true child label needs the next width for such values. Disk size and exports do not change | +| D20 | The canonical decimal work does not include the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and scales, casts that lose information, or the lower bound of a negative scale. Their recommendations are unreviewed future work. Epic [#10730](https://github.com/vortex-data/vortex/issues/10730) tracks wide integers and FixedSizeBinary. Issue [#9902](https://github.com/vortex-data/vortex/issues/9902) is its sub-issue for FixedSizeBinary, and wide integers get a tracking issue when that work is scheduled. The recommendation to always store min and max on the MSP (DQ8) is dropped | The canonical fix needs none of them. Stored min and max exist only on a BitPacked MSP, and a slice drops them | +| D21 | Every narrowing copy writes zeros in null slots, and it fails on a valid value outside the precision of the target dtype. Three paths make such a copy: an old node wider than its canonical width, an old narrow node whose best precision has a smaller width than the stored width, and DecimalByteParts decoding with an MSP wider than the canonical width. A read without a copy trusts the precision and does not scan | The copy reads every value anyway, so the precision check costs nothing extra, and it matches the import rule of D13. A check without a copy needs a scan that the read does not otherwise make | +| D22 | An integer `Narrow` child has the same signedness and nullability as its parent, and a strictly smaller width. So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` child | Mixed signedness needs more code in casts, in the compare shortcut and in the arithmetic bounds, and it saves at most one width. Part 1 of PR [#10333](https://github.com/vortex-data/vortex/pull/10333) also keeps the signedness of internal buffers | +| D23 | `vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level integer columns. Internal integer children keep changing their own dtype, as they do today. These include list offsets, dictionary codes, run ends, patch and sparse indices, `DateTimeParts`, VarBin and FSST offsets, and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend on it | The parent chooses the dtype of an internal child, so the child can narrow its dtype without a `Narrow`. A column has a dtype that the user chose | +| D24 | The compat fixtures land before the in-memory change, and they are published from a release before it, because only the old writer can make narrow leaves. They cover only reads that succeed: narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP. Error cases, such as a value outside the precision in a narrowing copy, are unit tests | The compat harness (`vortex-test/compat-gen`) expects the current reader to read every published fixture | + +D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D24 from 2026-10-23. On +2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the +largest precision whose canonical width is `W`, as the child precision of old data, and they +planned the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it +changed D8, D9, D11, D16, D17 and D20. [Part 2](decimal/02-design.md) of the Decimal notes explains +D7 to D19 and D21 to D24. ### Open +These questions belong to future work (D20). Their recommendations are not reviewed. + | ID | Question | Recommendation | | --- | --- | --- | -| DQ1 | Which width does each precision get? | The smallest signed width that holds the digits, which is the existing mapping in `vortex-array/src/dtype/decimal/types.rs:46-56` | -| DQ2 | Is `Narrow` one encoding for integers and decimals? | Yes. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)). Decimals narrow by precision | -| DQ3 | Does decimal `Narrow` share the wire ID `vortex.narrow`? | No, unless decimal `Narrow` lands before the edition of PR #10333 ships in a release. A second wire ID needs no change to the open PRs | -| DQ4 | Does the dtype of a DecimalByteParts array fix the type of its most significant part? | Yes. Narrowing then happens only through `Narrow` and integer encodings. A reader wraps the narrow part of an old file in an integer `Narrow` of the fixed type, without a copy | | DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. A small kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations | | DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes | | DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | @@ -76,29 +102,44 @@ the maintainer has not confirmed yet. | ID | Question | Recommendation | | --- | --- | --- | -| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error, like `TryFrom`. Lossy steps such as `trunc` and `round` are separate, named expressions | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error, like `TryFrom`. Lossy steps such as `trunc` and `round` are separate, named expressions. For decimals, this is future work (D20) | | CQ2 | How do struct casts match fields? | By name, with unique field names (issue [#6900](https://github.com/vortex-data/vortex/issues/6900)). Extra source fields return an error | ## The open `Narrow` pull requests -This review read the draft stack on 2026-10-19. - -- PR [#10223](https://github.com/vortex-data/vortex/pull/10223) and PR - [#10332](https://github.com/vortex-data/vortex/pull/10332) can merge as they are. Their only - integer-only check is the private `validate_dtypes`, so decimal support stays additive. When - that check opens for decimals, the compare kernel and the sum pushdown need decimal code in the - same change. -- PR [#10333](https://github.com/vortex-data/vortex/pull/10333) adds `vortex.narrow` to the draft - edition `preview2026.10.0`, with one metadata byte that holds the child `PType`. Question DQ3 - decides it before the merge. The editions spec says that a change to what readers must - understand needs a new wire ID (`docs/specs/versioning/editions.md:118-121`). -- PR [#10230](https://github.com/vortex-data/vortex/pull/10230) gives decimals an integer child, - which D2 replaces. Split its reusable parts into smaller PRs, in this order: - 1. The nullability fix in DecimalByteParts. - 2. Producers take the width from the precision. - 3. `Narrow` over decimals. - 4. The compressor and DecimalByteParts decoding stop narrowing and return `Narrow` instead. - 5. Readers convert old buffers at other widths. - 6. `DecimalArray::try_new` accepts only the canonical width. - - Decimal kernels can move onto integer kernels through a zero-copy view at any point. +This review read the draft stack on 2026-10-19 and again on 2026-10-22. The state below is from +2026-10-23. + +- The pushed PRs [#10223](https://github.com/vortex-data/vortex/pull/10223), + [#10332](https://github.com/vortex-data/vortex/pull/10332) and + [#10333](https://github.com/vortex-data/vortex/pull/10333) date from 2026-10-06. They are 215 + commits behind `develop`, and #10223 and #10333 conflict with it. +- A rewrite of #10223 and #10332 exists, but it is not pushed. It makes `Narrow` an in-memory + encoding with no serialized form, and it merges into `develop` without conflicts. It goes up as + new PRs, and #10223 and #10332 close then. +- The rewrite also fixes a fill code that the dictionary `fill_null` kernel truncates. That bug + exists on `develop` without `Narrow`, so the fix becomes its own PR. +- The only integer-only check of #10223 and #10332 is the private `validate_dtypes`, so decimal + support stays additive. When that check opens for decimals, the compare kernel and the sum + pushdown need decimal code in the same change. +- The CodSpeed reports on #10223 and #10332 compare against `3c87da0`, because `cc018ee` had no + run. Their gains, such as those of the FSST benchmarks, do not come from `Narrow`. The rebased + PRs need new reports. +- PR #10333 splits in two: + 1. The first part replaces `PrimitiveArrayExt::narrow` with `NarrowArray::encode_values`. + Internal buffers then keep their signedness in every preset. Golden snapshots and file + contents change, but no wire format changes. This part lands with the rewrite. + 2. The second part adds the serialization of D8, the edition entry, and `Narrow` children in the + compressor. It moves into the wire change (D11 and D18). +- The CodSpeed report on #10333 shows `take_map` regressions of 22% and 11% in Simulation mode. + The `take_map` benchmark builds `u64` patch indices directly and does not call the narrowing + code. The cause is not known. The new PRs get fresh CodSpeed runs, and the investigation + happens there. +- PR [#10230](https://github.com/vortex-data/vortex/pull/10230) gave decimals an integer child, + which D2 replaces. It closed on 2026-10-19. Its fix to the DecimalByteParts cast rule becomes its + own PR, with no issue. The PR adds the guard of #10230, a cast kernel for the case that needs + execution, and tests. [Part 3](decimal/03-roadmap.md) of the Decimal notes replaces the other + steps of #10230. +- PR #10305 limits the decimal storage width to the canonical width, and PR #10314 adds bounded + dispatch macros on top of it. Another contributor owns both, and both are open. The work + ignores them for now (D11). From 77c9829ae9adb147f25a98b05f51926e40281eca Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 12:48:54 -0400 Subject: [PATCH 08/12] Record the closed Narrow PRs in the type research #10223, #10332 and #10333 closed on 2026-10-23, and new PRs replace them. Note that the local rewrite needs a rebase over #10681 and new branch names. Signed-off-by: Connor Tsui --- research/decimal/03-roadmap.md | 19 +++++++++--------- research/decimal/agents/report.md | 27 ++++++++++++++----------- research/decisions.md | 33 ++++++++++++++++++------------- 3 files changed, 44 insertions(+), 35 deletions(-) diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index dba9cd8565c..7af2f37132e 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -137,14 +137,15 @@ them is on `develop`: own PR with tests, and it needs no issue. 3. **Extension opacity fixes (N1).** These are the urgent changes in [part 4 of the extension notes](../extension-types/04-urgent-changes.md). -4. **The integer `Narrow` in memory.** The rewrite of PRs - [#10223](https://github.com/vortex-data/vortex/pull/10223) and - [#10332](https://github.com/vortex-data/vortex/pull/10332) goes up as new PRs, and the old PRs - close then. The rewrite makes `Narrow` in-memory only. Its only integer-only check is the - private `validate_dtypes`, so decimal support stays additive. The first part of PR - [#10333](https://github.com/vortex-data/vortex/pull/10333) (`NarrowArray::encode_values`) lands - with it. The fix for dictionary fill codes is its own PR. - [`decisions.md`](../decisions.md#the-open-narrow-pull-requests) lists the state of these PRs. +4. **The integer `Narrow` in memory.** The draft PRs + [#10223](https://github.com/vortex-data/vortex/pull/10223), + [#10332](https://github.com/vortex-data/vortex/pull/10332) and + [#10333](https://github.com/vortex-data/vortex/pull/10333) closed on 2026-10-23. A rewrite of + #10223 and #10332 goes up as new PRs after a rebase. The rewrite makes `Narrow` in-memory only. + Its only integer-only check is the private `validate_dtypes`, so decimal support stays + additive. A new PR with the first part of #10333 (`NarrowArray::encode_values`) lands after it. + The fix for dictionary fill codes is its own PR. + [`decisions.md`](../decisions.md#the-narrow-pull-requests) lists the state of these PRs. ## One canonical form @@ -216,7 +217,7 @@ the bar of no compute regression applies to the final state (decided, D19). | Wave | Work | | --- | --- | -| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Narrow` in memory with the first part of PR #10333, and the decimal benchmarks and fixtures | +| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Narrow` in memory followed by the `encode_values` PR, and the decimal benchmarks and fixtures | | 1, one canonical form | N11 as two changes (D11). The in-memory change holds the width owner, N28 for decimals, DecimalByteParts decoding, the read plugin, the DecimalByteParts scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds the `vortex.narrow` wire format, the preview edition with DecimalByteParts lower parts, and the decimal `Narrow` scheme | | 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins | | 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation | diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index 265babf967e..c7db6028581 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -628,20 +628,21 @@ at precision 39 to 76, or it comes from a write that skipped the decimal scheme. #### The integer `Narrow` PRs -The integer `Narrow` is a draft stack: PRs #10223, #10332 and #10333. Section 8.3 and the decisions -file give their state on 2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory -only. It goes up as new PRs, and the old PRs close then. Its only integer-only check is the private -`validate_dtypes`, so decimal support stays additive. When that check opens for decimals, the -compare kernel and the sum pushdown need decimal code in the same change. - -The pushed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one +The integer `Narrow` was a draft stack: PRs #10223, #10332 and #10333. The maintainer closed all +three on 2026-10-23, and new PRs replace them. Section 8.3 and the decisions file give the state on +2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory only. It goes up as new +PRs after a rebase over PR #10681, which removed `array_session()`. Its only integer-only check is +the private `validate_dtypes`, so decimal support stays additive. When that check opens for +decimals, the compare kernel and the sum pushdown need decimal code in the same change. + +The closed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one metadata byte that holds the child `PType`. Under D8, the metadata stores the child dtype: the child `PType` for an integer parent, or the child precision for a decimal parent. The parent dtype tells the reader which of the two it holds. Every release that permits an ID must read every valid instance of it (invariant 6 in `docs/specs/versioning/design.md`). So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The second part of #10333 -moves into the wire change, which puts `vortex.narrow` and DBP v2 into one preview edition (D11, -D18). +comes back as a new PR in the wire change, which puts `vortex.narrow` and DBP v2 into one preview +edition (D11, D18). #### DecimalByteParts @@ -1711,9 +1712,10 @@ Independent changes, in any order. On 2026-10-23, none of these is on `develop`: the case that needs execution, and tests for a nullable-to-non-nullable cast with nulls, without nulls, and with only nulls. It is its own PR, with no issue. - **P3.** N1: the urgent extension changes U1 to U8. -- **P4.** The integer `Narrow` in memory. The rewrite of #10223 and #10332 goes up as new PRs, and - the old PRs close then. The first part of PR #10333 (`NarrowArray::encode_values`) lands with it. - The fix for dictionary fill codes is its own PR. +- **P4.** The integer `Narrow` in memory. #10223, #10332 and #10333 closed on 2026-10-23. The + rewrite of #10223 and #10332 goes up as new PRs after a rebase. A new PR with the first part of + #10333 (`NarrowArray::encode_values`) lands after it. The fix for dictionary fill codes is its + own PR. Benchmarks and fixtures, before the producers change: @@ -1935,6 +1937,7 @@ GitHub, read only: | [PR #10230](https://github.com/vortex-data/vortex/pull/10230) | Canonical decimals with one integer child, stacked on #10333. The description, the diff, and the comments, read on 2026-10-19. It closed on 2026-10-19. On 2026-10-23, the review read its diff of the DBP cast rule again | | [PR #10305](https://github.com/vortex-data/vortex/pull/10305) | Decimal storage width at most the canonical width, open. The description, read on 2026-10-22 and 2026-10-23 | | [PR #10314](https://github.com/vortex-data/vortex/pull/10314) | Bounded decimal dispatch macros, draft, stacked on #10305. The description, read on 2026-10-22 and 2026-10-23 | +| [PR #10681](https://github.com/vortex-data/vortex/pull/10681) | Removal of `array_session()`, merged on 2026-10-23. The commit message and its diff of `dict/compute/fill_null.rs`, read on 2026-10-23 | | [PR #10155](https://github.com/vortex-data/vortex/pull/10155) | `DecimalBuilder::extend` from wider buffers for DuckDB imports, merged. The commit message, read on 2026-10-22 | | [Issue #10701](https://github.com/vortex-data/vortex/issues/10701) | Tracking issue for the canonical decimal fix, opened by the maintainer on 2026-10-22. The body, read on 2026-10-23. Its question about the gap of section 3.3 was answered on 2026-10-23 | | [Issue #10733](https://github.com/vortex-data/vortex/issues/10733) | Tracking issue for `Narrow` over integers and decimals, opened by the maintainer on 2026-10-23. The body, read on 2026-10-23. It keeps the question about `List` and `ListView` offsets open | diff --git a/research/decisions.md b/research/decisions.md index 7ffd12a5aa7..7414de56c5a 100644 --- a/research/decisions.md +++ b/research/decisions.md @@ -105,32 +105,37 @@ These questions belong to future work (D20). Their recommendations are not revie | CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error, like `TryFrom`. Lossy steps such as `trunc` and `round` are separate, named expressions. For decimals, this is future work (D20) | | CQ2 | How do struct casts match fields? | By name, with unique field names (issue [#6900](https://github.com/vortex-data/vortex/issues/6900)). Extra source fields return an error | -## The open `Narrow` pull requests +## The `Narrow` pull requests This review read the draft stack on 2026-10-19 and again on 2026-10-22. The state below is from 2026-10-23. -- The pushed PRs [#10223](https://github.com/vortex-data/vortex/pull/10223), +- The draft PRs [#10223](https://github.com/vortex-data/vortex/pull/10223), [#10332](https://github.com/vortex-data/vortex/pull/10332) and - [#10333](https://github.com/vortex-data/vortex/pull/10333) date from 2026-10-06. They are 215 - commits behind `develop`, and #10223 and #10333 conflict with it. -- A rewrite of #10223 and #10332 exists, but it is not pushed. It makes `Narrow` an in-memory - encoding with no serialized form, and it merges into `develop` without conflicts. It goes up as - new PRs, and #10223 and #10332 close then. + [#10333](https://github.com/vortex-data/vortex/pull/10333) date from 2026-10-06. The maintainer + closed all three on 2026-10-23, and new PRs replace them. Issue + [#10733](https://github.com/vortex-data/vortex/issues/10733) tracks the new PRs. +- A rewrite of #10223 and #10332 exists in five local commits, but it is not pushed. It makes + `Narrow` an in-memory encoding with no serialized form. Since PR + [#10681](https://github.com/vortex-data/vortex/pull/10681) merged on 2026-10-23, the rewrite + needs a rebase. That PR removed `array_session()`, which the rewrite calls 47 times, and it + conflicts with the rewrite in the test module of `dict/compute/fill_null.rs`. The local branch + has the name `ct/narrow-pushdown`, which is the head branch of the closed #10332, so the new PRs + use new branch names. - The rewrite also fixes a fill code that the dictionary `fill_null` kernel truncates. That bug exists on `develop` without `Narrow`, so the fix becomes its own PR. - The only integer-only check of #10223 and #10332 is the private `validate_dtypes`, so decimal support stays additive. When that check opens for decimals, the compare kernel and the sum pushdown need decimal code in the same change. - The CodSpeed reports on #10223 and #10332 compare against `3c87da0`, because `cc018ee` had no - run. Their gains, such as those of the FSST benchmarks, do not come from `Narrow`. The rebased - PRs need new reports. -- PR #10333 splits in two: - 1. The first part replaces `PrimitiveArrayExt::narrow` with `NarrowArray::encode_values`. + run. Their gains, such as those of the FSST benchmarks, do not come from `Narrow`. The new PRs + need new reports. +- The work of PR #10333 comes back as two new PRs: + 1. The first PR replaces `PrimitiveArrayExt::narrow` with `NarrowArray::encode_values`. Internal buffers then keep their signedness in every preset. Golden snapshots and file - contents change, but no wire format changes. This part lands with the rewrite. - 2. The second part adds the serialization of D8, the edition entry, and `Narrow` children in the - compressor. It moves into the wire change (D11 and D18). + contents change, but no wire format changes. This PR lands after the rewrite. + 2. The second PR adds the serialization of D8, the edition entry, and `Narrow` children in the + compressor. It is part of the wire change (D11 and D18). - The CodSpeed report on #10333 shows `take_map` regressions of 22% and 11% in Simulation mode. The `take_map` benchmark builds `u64` patch indices directly and does not call the narrowing code. The cause is not known. The new PRs get fresh CodSpeed runs, and the investigation From 043178d67e8408acd2f1dfc622e6d2dbaf1e9ec7 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 14:47:50 -0400 Subject: [PATCH 09/12] Add the implementation plan for Narrow and the canonical decimal fix Split the work into 22 PRs with one brief for each, and record decisions D25 to D35 from the planning review. Signed-off-by: Connor Tsui --- research/README.md | 2 + research/decimal/02-design.md | 84 +- research/decimal/03-roadmap.md | 52 +- research/decimal/04-implementation.md | 201 ++ research/decimal/README.md | 10 +- research/decimal/agents/briefs/01-fixes.md | 577 ++++++ .../agents/briefs/02-integer-narrow.md | 907 ++++++++ .../agents/briefs/03-canonical-decimal.md | 1839 +++++++++++++++++ research/decimal/agents/briefs/04-wire.md | 487 +++++ research/decimal/agents/report.md | 42 +- research/decisions.md | 50 +- 11 files changed, 4174 insertions(+), 77 deletions(-) create mode 100644 research/decimal/04-implementation.md create mode 100644 research/decimal/agents/briefs/01-fixes.md create mode 100644 research/decimal/agents/briefs/02-integer-narrow.md create mode 100644 research/decimal/agents/briefs/03-canonical-decimal.md create mode 100644 research/decimal/agents/briefs/04-wire.md diff --git a/research/README.md b/research/README.md index 864c70ac2c0..1b4d7cb090b 100644 --- a/research/README.md +++ b/research/README.md @@ -5,6 +5,8 @@ at `5fc5cb8` (2026-10-17). They live on the `ct/ext-types-research` branch and a merged. Reviews on 2026-10-22 and 2026-10-23 updated the Decimal notes and [`decisions.md`](decisions.md). +Part 4 of the Decimal notes holds the implementation plan for `Narrow` and the canonical decimal +fix. The decimal, DecimalByteParts, serialization and versioning code that the notes cite is the same on `develop` at `dbaf5f8` (2026-10-22). diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index d657348fd78..c2cfaf9c61f 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -48,9 +48,10 @@ The leaf keeps three invariants: costs nothing for each row. 2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `DecimalArray::try_new` checks every valid value, and `new_unchecked` skips that check for kernels that keep values - legal (decided, D14). Imports use `try_new`. File reads trust the precision and never panic. - A read that makes a narrowing copy checks the precision, because it reads every value anyway - (decided, D21). + legal (decided, D14). The check reads the validity, so `try_new` takes an `ExecutionCtx` + (decided, D30). Imports check their own values and then use the unchecked constructors. File + reads trust the precision and never panic. A read that makes a narrowing copy checks the + precision, because it reads every value anyway (decided, D21). 3. **Leaf.** The array has one buffer and validity, and no data child. So a kernel never receives a width that it does not expect. The `patch` panic and the "does the @@ -62,13 +63,16 @@ land before the wide integers. ## Imports, exports and Arrow An import copies into the canonical width when the source has another width (decided, D13). It -writes zeros in null slots, and it fails on a value outside the precision. Two common imports -copy: +writes zeros in null slots, and it fails on a value outside the precision. An import at the +canonical width stays zero-copy, but it scans every valid value once for the precision check +(decided, D30). Two common imports copy: - Arrow `Decimal128` data at precision 18 or below. This import is zero-copy today. - DuckDB data at precision 1 to 2, which DuckDB stores as `i16`. The Arrow import logic can be configured to handle values outside the precision in another way. +No import options exist yet. Until they do, the import fails, and the error names arrow-rs +`null_if_overflow_precision`. Exports widen as they do today. The Arrow export uses `Decimal128` for every precision up to 38, because DataFusion does not fully support `Decimal32` and `Decimal64` yet. The DuckDB export uses @@ -152,17 +156,32 @@ D8). Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. An integer child has the same signedness and nullability as its parent, and a strictly smaller width (decided, D22). So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` -child. The metadata stores only the child `PType` or the child precision. The plugin of the parent -builds the child dtype from it. The child must have the scale and nullability of the parent, so -metadata that stored them would only let a file hold an invalid child. `vortex.narrow` stays in -memory only until the maintainer decides to write it to files (decided, D18). - -`vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level -integer columns (decided, D23). Internal integer children keep changing their own dtype, as they -do today. Examples are list offsets, dictionary codes, run ends and the DecimalByteParts MSP. -Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on -issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not -depend on it. +child. The metadata stores only the child `PType` or the child precision, in a prost message with +a typed `oneof` (decided, D32). The plugin of the parent builds the child dtype from it. The child +must have the scale and nullability of the parent, so metadata that stored them would only let a +file hold an invalid child. `vortex.narrow` stays in memory only until the maintainer decides to +write it to files (decided, D18). + +Until then, a writer that meets a `Narrow` writes its executed canonical array (decided, D26). This +also holds later for writers whose editions do not permit `vortex.narrow`. Decimal `Narrow` appears +in ordinary pipelines, through widening casts and old files, so IPC or a writer without +compression would fail without it. A plugin cannot return the wire form of another encoding, so +the fallback lives in the serializer. The code documents it as a workaround, which stays until +`vortex.narrow` is in `core`. + +`Narrow` has two producer functions (decided, D28). `encode` builds a `Narrow` over the narrowest +child of the same signedness. `encode_values` returns only that child, so it changes the dtype. +`encode_values` is `#[deprecated]`, and it replaces `PrimitiveArrayExt::narrow` at every call site +with `#[expect(deprecated)]`. So every narrowing that changes a dtype is marked for a later +decision. + +`vortex.narrow` is for arrays with a fixed logical dtype (decided, D23). Exactly which arrays the +compressor wraps is deferred to the design of the compressor scheme (decided, D27). Internal integer +children keep changing their own dtype, through `encode_values`. Examples are list offsets, +dictionary codes, run ends and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a +fixed `I64` dtype with `Narrow` is an open question on issue +[#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend +on it. `Narrow` kernels pass work to the child, and the fallback is to canonicalize (decided, D15): @@ -263,8 +282,10 @@ DecimalByteParts (DBP) stays the compressed decimal form: the validity (decided, D16). Each lower part is a non-nullable `u64`, with zeros in null rows. Today the constructor accepts a lower part of any unsigned width, and the module doc says that each part can use a narrower dtype. But the split makes only `u64` parts, and the compressor - never narrows the dtype of a part. D16 tightens the constructor and the doc. This breaks no - file, because no file has lower parts. + never narrows the dtype of a part. D16 tightens the constructor and the doc. The constructor + also rejects lower parts that assemble wider than the canonical width (decided, D29): precision + up to 18 allows none, 19 to 38 at most one, and 39 to 76 any count. This breaks no file, because + no file has lower parts. - Each part can use any integer encoding. The parts are integers, so they promise only a width, never a number of digits. - One in-memory array has two wire forms. `vortex.decimal_byte_parts` (v1) has only the MSP. @@ -293,12 +314,13 @@ The compressor has one DBP scheme (decided, D17): - The scheme is not temporary. Writers whose edition does not permit `vortex.narrow` need it. These include default writers until `vortex.narrow` enters `core`, and writers that select an older edition. -- When an edition permits `vortex.narrow`, a decimal `Narrow` scheme runs first and narrows the - decimal type. The DBP scheme then compresses the child. So the compressor writes a `Narrow` of - dtype `Decimal(p, s)` over a DBP child of dtype `Decimal(p', s)`. The MSP of that child usually - has the canonical width of `p'`, so decoding reuses its buffer. In the gaps, decoding widens it. - If the best precision has the canonical width of the column, the compressor writes DBP without - `Narrow`. +- When an edition permits `vortex.narrow`, a `Narrow` scheme runs first and narrows the decimal type + (decided, D33). It is a real scheme at first. Its design settles its precedence with the DBP + scheme, which returns `AlwaysUse`, and the cascade level that it uses. The DBP scheme then + compresses the child. So the compressor writes a `Narrow` of dtype `Decimal(p, s)` over a DBP + child of dtype `Decimal(p', s)`. The MSP of that child usually has the canonical width of `p'`, so + decoding reuses its buffer. In the gaps, decoding widens it. If the best precision has the + canonical width of the column, the compressor writes DBP without `Narrow`. - For precision 39 to 76 with values that need more than 64 bits, the compressor uses the lower-parts form when an edition permits it. Otherwise, it writes a plain `i256` array. Today it writes a plain `i128` array, which D3 forbids. @@ -325,6 +347,12 @@ Generic structural encodings sit above the type layer (decided, X5). These notes list for them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave` (XQ1). Decimal storage encodings, DBP and `Narrow`, sit below them. +For `Narrow`, reduce rules enforce this (decided, D25). They move `Dict`, `RunEnd`, `Sparse`, +`Constant`, `Masked`, `Chunked` and `Interleave` above a `Narrow`, for example +`Narrow(Dict(codes, values))` becomes `Dict(codes, Narrow(values))`. No rule moves `Narrow` above +them, so the rules cannot loop. The Arrow and DuckDB exporters and the dict layout then find these +encodings at the root. A follow-up issue covers the same rule for every encoding. + The decimal wire forms already follow this rule. A constant decimal column is a `vortex.constant` node with a decimal scalar, or `vortex.masked` over that node when some rows are null. Run-end and dictionary arrays already hold decimal values. So `Decimal` needs no plugin that translates an @@ -506,7 +534,8 @@ stored width, and it converts the node. Readers handle every existing decimal wi Writers produce forms that old readers understand. The leaf becomes `vortex.decimal` at the canonical width. DBP becomes DecimalByteParts v1, which the first core edition already contains, or v2 where an edition permits it. `Narrow` is written only where an edition permits -`vortex.narrow`, and the compressor produces it only then. `vortex.narrow` and DBP v2 enter one +`vortex.narrow`, and the compressor produces it only then. Elsewhere the serializer writes the +canonical array of a `Narrow` (decided, D26). `vortex.narrow` and DBP v2 enter one preview edition together, and later `core` together (decided, D18). Constant, dictionary and run-end arrays of decimals keep their current wire forms. @@ -515,8 +544,9 @@ So wide `PType` values never reach the wire for decimal columns, and decimal sta The compat harness (`vortex-test/compat-gen`) publishes fixtures from a released version, and the current reader must read every published fixture. Only the old writer can make narrow leaves. So -the new fixtures land and are published from a release before the in-memory change (decided, -D24). They cover these cases: +the new fixtures are published from a release before the PRs that change how files are read: +DecimalByteParts decoding and the read plugin (decided, D24 and D31). Library changes do not wait. +The fixtures cover these cases: - Narrow, wide and Arrow-style widths. - Narrow buffers with gap values. diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 7af2f37132e..82858b3e25f 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -9,7 +9,7 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors Many planned type changes depend on each other. The IDs below match the full graph in section 7 of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. -The decision IDs (D1 to D24, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) +The decision IDs (D1 to D35, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) come from [`decisions.md`](../decisions.md). | ID | Work | Kind | @@ -122,13 +122,15 @@ uses `DecimalType`, which already has 128-bit and 256-bit widths. ## Independent changes These changes are small and independent, and they can land in any order. On 2026-10-23, none of -them is on `develop`: +them is on `develop`. [Part 4](04-implementation.md) splits these changes and the canonical form +into PRs, with one brief for each PR. 1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a silent `I8` or `U8`. Readers must have this change before any writer emits a new value. The decimal part must also land before the read plugin converts old nodes, because the detection - of old nodes reads this field. + of old nodes reads this field. The same PR fixes eleven `PType` fields of array and layout + metadata with the same fallback (decided, D34). 2. **The DecimalByteParts nullability fix.** For a cast that changes only the nullability, the cast rule casts the MSP without the check that `Validity::trivially_cast_nullability` makes in other cast rules. The fix adds that check, as PR @@ -142,9 +144,11 @@ them is on `develop`: [#10332](https://github.com/vortex-data/vortex/pull/10332) and [#10333](https://github.com/vortex-data/vortex/pull/10333) closed on 2026-10-23. A rewrite of #10223 and #10332 goes up as new PRs after a rebase. The rewrite makes `Narrow` in-memory only. - Its only integer-only check is the private `validate_dtypes`, so decimal support stays - additive. A new PR with the first part of #10333 (`NarrowArray::encode_values`) lands after it. - The fix for dictionary fill codes is its own PR. + It lands as five stacked PRs (decided, D35). The first adds `encode` and the deprecated + `encode_values`, and a later PR moves the existing narrowing call sites to `encode_values` + (decided, D28). The last PR of the stack adds the rules that move structural encodings above + `Narrow` (decided, D25). The fix for dictionary fill codes is its own PR, before that last PR. + Decimal support needs real kernel changes, because some kernels assume an integer child. [`decisions.md`](../decisions.md#the-narrow-pull-requests) lists the state of these PRs. ## One canonical form @@ -165,8 +169,9 @@ Benchmarks and fixtures come first: `encodings/decimal-byte-parts/benches/dbp_assemble.rs` and `vortex-arrow/benches/to_arrow.rs`. - Compatibility fixtures for narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP - ([part 2](02-design.md#the-file-format)). They are published from a release before the - in-memory change, because only the old writer can make narrow leaves (decided, D24). The error + ([part 2](02-design.md#the-file-format)). A release publishes them before the PRs that change + how files are read, because only the old writer can make narrow leaves (decided, D24 and D31). + Library changes do not wait for it. The error cases, such as a value outside the precision in a narrowing copy, are unit tests, because the compat harness expects every read to succeed. @@ -174,23 +179,26 @@ The in-memory change has these steps, in order. Each step is its own PR, because needs temporary code (decided, D11): 1. Give the width function one owner. `DecimalDType::values_type()` replaces the 25 direct calls - to `smallest_decimal_value_type`. Producers and imports take the width from the precision - (decided, D12 and D13). + to `smallest_decimal_value_type`. Producers and imports take the width from the precision, and + every import checks its values (decided, D12, D13 and D30). 2. Add `Narrow` over decimals (N28). This step holds the child rules of D10, the kernels and casts of D15, and the rule that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for decimals, the compare kernel and the sum pushdown get decimal code in the same change. 3. Make DecimalByteParts decoding produce the canonical array of its dtype, with the `TODO(connor)` of D16. Decoding widens a narrower MSP, and it narrows a wider MSP with the copy - of D21. Require `u64` lower parts, and correct the module doc, which allows narrower parts. + of D21. Require `u64` lower parts, and reject lower parts that assemble wider than the + canonical width (decided, D29). Correct the module doc, which allows narrower parts. 4. Make the `vortex.decimal` read plugin convert old nodes (decided, D9 and D21). The plugin scans the values, because it cannot use stored statistics. 5. Make the DecimalByteParts scheme take the canonical array, and delete `narrowed_decimal` - (decided, D17). The scheme stays after the wire change, for writers whose edition does not - permit `vortex.narrow`. File contents stay the same. The exception is precision 39 to 76 with + (decided, D17). The MSP narrows with the deprecated `encode_values` (decided, D28). The scheme + stays after the wire change, for writers whose edition does not permit `vortex.narrow`. File + contents stay the same. The exception is precision 39 to 76 with values that need more than 64 bits, which becomes a plain `i256` array. 6. Make `DecimalArray::try_new` accept only the canonical width and check every valid value, with - `new_unchecked` for kernels (decided, D14). Builders follow, including `DecimalBuilder::extend`, - which accepts wider buffers (PR [#10155](https://github.com/vortex-data/vortex/pull/10155)). + `new_unchecked` for kernels (decided, D14). `try_new` takes an `ExecutionCtx` (decided, D30). + Builders follow, including `DecimalBuilder::extend`, which accepts wider buffers. This step is + three PRs: the width rule, the value check, and the removal of dead code. Then the `patch` panic and the "does the constant fit" special cases are unreachable. @@ -199,10 +207,11 @@ in-memory change (decided, D11). It holds three parts: - The `vortex.narrow` wire format for integers and decimals (decided, D8). This is the second part of PR #10333. -- One new preview edition that permits `vortex.narrow` and DecimalByteParts with lower parts, - together with the other wire improvements. -- A decimal `Narrow` scheme that narrows the decimal type first, for writers that select that - edition. It runs before the DecimalByteParts scheme, which then compresses the child. +- One new preview edition that permits `vortex.narrow` and DecimalByteParts with lower parts. +- A `Narrow` scheme that narrows the decimal type first, for writers that select that edition. It + runs before the DecimalByteParts scheme, which then compresses the child (decided, D33). Its + design pass settles which arrays it narrows (decided, D27), its precedence with the + DecimalByteParts scheme, and its cascade level. Later, both formats enter `core` together. Two optimizations wait until benchmarks show that they are needed. One is the max pass in DecimalByteParts decoding (D16). The other is the pushdown of @@ -224,7 +233,8 @@ the bar of no compute regression applies to the final state (decided, D19). | 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 | | 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 | -Waves 1 and 2 can run in parallel. The decimal fix (N28 and N11) does not wait for the extension +[Part 4](04-implementation.md) gives the order of the PRs inside waves 0 and 1. Waves 1 and 2 can +run in parallel. The decimal fix (N28 and N11) does not wait for the extension work, because the built-in type owns its rules. Only the optional move to an extension type (N15) waits for N1 to N4. For decimals, waves 3 and 4 are future work (D20), and their recommendations are not reviewed. @@ -251,6 +261,8 @@ are not reviewed. | An old reader reads a protobuf wide `PType` as `U8` | A new protobuf `oneof` variant, and strict readers early | | Rewritten arithmetic makes several passes over the data | Fused scaled-integer kernels, checked with the existing decimal and compare benchmarks | | D3 without `Narrow` forces the full width on every column | `Narrow` for decimals lands in the in-memory change, before `try_new` rejects other widths | +| An in-memory `Narrow` reaches a writer that cannot write `vortex.narrow` | The serializer writes the canonical array (D26). The code documents this as a workaround until `vortex.narrow` is in `core` | +| Moving the narrowing call sites to `encode_values` changes the signedness of internal children in new files | Readers on `develop` accept any integer type there, and the PR checks that the latest release reads its files (D28) | | A narrow buffer does not give a precision | Every producer finds the precision from the values. Readers convert old nodes by their max (D9). DecimalByteParts decoding widens a narrow MSP (D16) | | Compute on DecimalByteParts regresses when its decoding stops returning narrow leaves | In the final state, the compressor narrows the decimal type first, so decoding reuses the buffer. Until then, the maintainer accepts the cost (D19). The max pass of D16 can come later, and new benchmarks measure the cost | | Gap columns use the next width in memory in the final state | Accepted (D19). Pushdown of sum, min and max into the MSP can come later | diff --git a/research/decimal/04-implementation.md b/research/decimal/04-implementation.md new file mode 100644 index 00000000000..de905fb6d93 --- /dev/null +++ b/research/decimal/04-implementation.md @@ -0,0 +1,201 @@ + + +# 4. Implementation plan + +This part splits the `Narrow` encoding (issue +[#10733](https://github.com/vortex-data/vortex/issues/10733)) and the canonical decimal fix (issue +[#10701](https://github.com/vortex-data/vortex/issues/10701)) into PRs. Each PR has a brief in +[`agents/briefs/`](agents/briefs/), written so that a coding agent can implement the PR from the +brief alone. The briefs follow [`decisions.md`](../decisions.md), including D25 to D35 from the +planning review on 2026-10-23. + +## How to use the briefs + +- Give a coding agent one brief, `decisions.md`, and the repository rules in `AGENTS.md` and + `STYLE.md`. The brief names the branch, the files, the steps, the tests and the acceptance checks. +- The briefs cite `develop` at `b70206818f`. Line numbers move as PRs land, so the agent reads the + cited code before it changes it. Code from the local integer `Narrow` rewrite is cited with its + commit, for example `f7408481ce:vortex-array/src/arrays/narrow/mod.rs`. +- The acceptance checks are commands for the maintainer to run. An agent does not run them unless + the maintainer asks for it. +- Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author + line. +- WR3 needs a design pass before an agent starts it. Its brief lists the open questions. + +| Brief | PRs | +| --- | --- | +| [`01-fixes.md`](agents/briefs/01-fixes.md) | FX1 to FX3, independent fixes | +| [`02-integer-narrow.md`](agents/briefs/02-integer-narrow.md) | NR1 to NR6, the integer `Narrow` | +| [`03-canonical-decimal.md`](agents/briefs/03-canonical-decimal.md) | CD1 to CD8c, the canonical decimal fix and decimal `Narrow` | +| [`04-wire.md`](agents/briefs/04-wire.md) | WR1 to WR3, the wire change, and LT1 to LT3 | + +## The PRs + +| ID | PR | Hard dependencies | Size | +| --- | --- | --- | --- | +| FX1 | Strict decoding of stored `DecimalType` and `PType` values, in three commits | None | Medium | +| FX2 | DecimalByteParts nullability cast follows the cast-rule contract | None | Small | +| FX3 | Dictionary `fill_null` no longer truncates the fill code | None | Small | +| NR1 | In-memory integer `Narrow`, with `encode`, the deprecated `encode_values`, the serialization workaround and a flatten rule | None | Large | +| NR2 | Benchmarks for compute on integer `Narrow` | NR1 | Medium | +| NR3 | Aggregates run on the child | NR2 | Small | +| NR4 | Arithmetic, `between`, `zip` and casts run on the child | NR3 | Large | +| NR5 | Structural encodings move above `Narrow`, and dictionary codes use the child | NR4, FX3 | Medium | +| NR6 | The existing narrowing call sites move to `encode_values` | NR1 | Medium | +| CD1 | Decimal benchmarks | None | Medium | +| CD2 | Compat fixtures for old decimal forms | None | Medium | +| CD3 | One width function, imports that check and copy, and the shared helpers | None | Medium | +| CD4 | Decimal `Narrow` in memory | NR1, NR3, NR4, CD1, CD3 | Large | +| CD5 | DecimalByteParts decodes to the canonical width | CD1, CD3, released CD2 fixtures | Medium | +| CD6 | The `vortex.decimal` read plugin converts old nodes | FX1, CD3, CD4, released CD2 fixtures | Medium | +| CD7 | One DecimalByteParts scheme on the canonical array | NR1 | Medium | +| CD8a | Every decimal array has the canonical width | CD3, CD5, CD6, CD7 | Large | +| CD8b | `DecimalArray::try_new` checks every valid value, with an `ExecutionCtx` | CD8a | Large | +| CD8c | Remove the code for mixed decimal widths | CD8a | Medium | +| WR1 | `vortex.narrow` wire format for integers and decimals | NR1, CD3, CD4, after CD8a | Medium | +| WR2 | One preview edition with `vortex.narrow` and DecimalByteParts v2 | WR1, CD5, CD7 | Small | +| WR3 | A `Narrow` scheme in the compressor (needs a design pass) | WR1, WR2, NR1, NR5, CD3, CD4, CD7 | Large | + +Later work: LT1 puts both formats into `core` and decides the end of the serialization +workaround. LT2 is the max pass in DecimalByteParts decoding (D16). LT3 is the pushdown of sum, +min and max into the MSP (D19). LT2 and LT3 wait until benchmarks show that they are needed. + +## Graph + +Each arrow is a hard dependency. The graph leaves out dependencies that follow from other arrows. +The dashed arrows are the release gate of the compat fixtures (D31). + +```mermaid +flowchart TD + subgraph FIX["Independent fixes"] + FX1["FX1 Strict decoding"] + FX2["FX2 DBP nullability cast"] + FX3["FX3 Dict fill code"] + end + subgraph NR["Integer Narrow"] + NR1["NR1 Encoding"] + NR2["NR2 Benchmarks"] + NR3["NR3 Aggregates"] + NR4["NR4 Arithmetic and casts"] + NR5["NR5 Structural rules"] + NR6["NR6 Call sites"] + end + subgraph CD["Canonical decimal"] + CD1["CD1 Benchmarks"] + CD2["CD2 Compat fixtures"] + REL(["Release with CD2"]) + CD3["CD3 Width and imports"] + CD4["CD4 Decimal Narrow"] + CD5["CD5 DBP decoding"] + CD6["CD6 Read plugin"] + CD7["CD7 DBP scheme"] + CD8a["CD8a Canonical width"] + CD8b["CD8b Value check"] + CD8c["CD8c Dead code"] + end + subgraph WR["Wire change"] + WR1["WR1 Wire format"] + WR2["WR2 Preview edition"] + WR3["WR3 Narrow scheme"] + end + + NR1 --> NR2 --> NR3 --> NR4 --> NR5 + FX3 --> NR5 + NR1 --> NR6 + CD2 --> REL + REL -.-> CD5 + REL -.-> CD6 + CD1 --> CD4 + CD3 --> CD4 + NR1 --> CD4 + NR3 --> CD4 + NR4 --> CD4 + CD1 --> CD5 + CD3 --> CD5 + FX1 --> CD6 + CD3 --> CD6 + CD4 --> CD6 + NR1 --> CD7 + CD3 --> CD8a + CD5 --> CD8a + CD6 --> CD8a + CD7 --> CD8a + CD8a --> CD8b + CD8a --> CD8c + CD8a --> WR1 + CD4 --> WR1 + WR1 --> WR2 + CD5 --> WR2 + CD7 --> WR2 + WR2 --> WR3 + NR5 --> WR3 +``` + +FX2 has no edge. It can land at any time. + +## Order of work + +Several agents can work at once. A wave starts when its dependencies have merged. + +| Wave | PRs | Note | +| --- | --- | --- | +| 1 | FX1, FX2, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | +| 2 | NR2, NR6, CD7, CD5 | NR6 merges after NR5 (D35). CD5 waits for the release with CD2 | +| 3 | NR3, then NR4 | One stack. Merge NR1 and NR2 before reading the CodSpeed reports of NR3 and NR4 | +| 4 | NR5, CD4 | NR5 waits for FX3 | +| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | +| 6 | CD8a | The last in-memory step that the wire change needs | +| 7 | CD8b, CD8c, WR1 | Independent of each other | +| 8 | WR2, then WR3 | WR3 after its design pass. If possible, the wire stack ships in the same release as the in-memory change (D11) | + +The critical path runs NR1, NR2, NR3, NR4, CD4, CD6, CD8a, WR1, WR2 and WR3. The release that +publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wave 1. + +## Rules for every PR + +- No PR adds temporary code (D11). The serialization workaround of D26 is the one documented + exception. It stays until `vortex.narrow` is in `core`. +- Benchmarks land before the changes that they cover (D19): NR2 before NR3 to NR5, and CD1 before + CD4, CD5 and CD7. +- Every call of the deprecated `encode_values` carries `#[expect(deprecated)]` with a + `TODO(connor)` reason (D28). +- No `Narrow` sits directly above a generic structural encoding after optimization (D25). +- A narrowing copy writes zeros in null slots and fails on a valid value outside the precision + (D21). Reads without a copy trust the precision and never panic (D14). + +## Deferred + +- Which arrays the compressor narrows (D27). The WR3 design pass decides it, together with the + precedence between the `Narrow` scheme and the DecimalByteParts scheme and the cascade level + (D33). +- Other members of the preview edition. WR2 adds only `vortex.narrow` and + `vortex.decimal_byte_parts.v2`. +- Arrow import options for invalid decimal values (D13). The import fails, and the error names + arrow-rs `null_if_overflow_precision`. + +## Follow-up issues + +These problems came up during planning and are outside the planned PRs. Each one gets its own +issue. + +1. Generic structural encodings always sit above other encodings. NR5 does this for `Narrow` only. +2. The decimal `min_max` aggregate panics on a valid value outside the precision, which a file read + can produce. +3. The cast rules of RunEnd, ZigZag, FoR and RLE push nullability casts down without the + `trivially_cast_nullability` guard that FX2 adds to DecimalByteParts. +4. The compat harness: no CI job runs `--mode all`, one failing fixture aborts the whole check, and + the README describes `--git-ref` wrongly. +5. Stale versioning docs about the decimal scheme and per-edition scheme configuration. + +## Risks + +| Risk | Mitigation | +| --- | --- | +| CD5 changes the CUDA decoder, which few reviewers know | A minimal change in its own commit, with a CUDA reviewer. It becomes its own PR if it grows | +| NR6 writes signed internal children that earlier releases have not read | Readers on `develop` accept any integer type there. NR6 runs the latest release on its files | +| The `take_map` regressions on #10333 have no known cause | NR6 compares CodSpeed runs on NR1, NR3, NR4 and NR6, and reads the flame graph diff | +| WR3 has open design questions | A design pass before any code. The other wire PRs do not depend on it | +| The decimal critical path is long | Wave 1 starts seven PRs at once, and CD5, CD7 and NR6 need only early PRs | diff --git a/research/decimal/README.md b/research/decimal/README.md index 950ead89fc0..e74dbc06f75 100644 --- a/research/decimal/README.md +++ b/research/decimal/README.md @@ -13,6 +13,7 @@ canonical decimal encoding. | [1. Current state](01-current-state.md) | How `Decimal` works on `develop`, and what is wrong | | [2. Design](02-design.md) | The leaf, imports and Arrow, `Narrow`, DecimalByteParts, the file format, and the future work | | [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the independent changes, and the order of work | +| [4. Implementation plan](04-implementation.md) | The PRs of the `Narrow` and canonical decimal work, their dependencies and waves, with one brief for each PR | ## Summary @@ -43,6 +44,10 @@ canonical decimal encoding. formats enter `core`, they write wide decimals with values above 64 bits as plain `i256` arrays. - Decided: the bar of no compute regression applies to the final state. Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full width in memory. +- Decided: the work lands as 22 PRs in four tracks. Each PR has a brief that a coding agent can + implement alone. Generic structural encodings always sit above `Narrow`. A writer that cannot + write `vortex.narrow` writes the canonical array, as a documented workaround. Which arrays the + compressor narrows is deferred. - Arrow has the same decimal model and the same gap at the top of each width. Its width is part of the type, and from precision 5 the Vortex widths match the smallest Arrow types. - Decided: `vortex.decimal` keeps its wire ID, and the decimal file format does not change. @@ -55,6 +60,7 @@ canonical decimal encoding. ## Agent documents +- [`agents/briefs/`](agents/briefs/): one brief for each PR of part 4, in four files. - [`agents/report.md`](agents/report.md): the full report. It has file and line references, API sketches, the wire plan, the complete dependency graph with a reason for each edge, and the evidence log. @@ -73,7 +79,9 @@ read the history of the decimal compression scheme, and PRs #10305 and #10314. O traced which statistics the compressors store on the DecimalByteParts parts, read the decimal types of arrow-rs 59.3, and read the cast change of PR #10230. A second review on the same day read when the reader attaches stored statistics, the DecimalByteParts constructor and compressor -scheme, the compat harness, and the edition families. One throwaway Rust test confirmed +scheme, the compat harness, and the edition families. A planning review on the same day had five +agents draft the PR briefs from the code at `b70206818f`, and the review checked their main claims +in the code. One throwaway Rust test confirmed the `f32` cast disagreement and the `ext_storage` pushdown rules (probes R1 and R6 in the extension notes). One short Python script checked byte order. Section 10 of the report lists the claims that are not verified, which include every performance claim. diff --git a/research/decimal/agents/briefs/01-fixes.md b/research/decimal/agents/briefs/01-fixes.md new file mode 100644 index 00000000000..8dd20d60896 --- /dev/null +++ b/research/decimal/agents/briefs/01-fixes.md @@ -0,0 +1,577 @@ + + +# Independent fixes: FX1, FX2 and FX3 + +These three PRs depend on no other PR. FX1 blocks CD6, and FX3 blocks NR5. Each one starts from +`develop` at `b70206818f`. Every `file:line` below points at that checkout, unless a commit is +named. + +Rules for all three PRs: + +- Each commit ends with `Signed-off-by: Connor Tsui `. Add no AI + co-author line. +- The user chooses which checks to run. The acceptance checks list the commands, scoped to the + affected crates. The pinned nightly is `nightly-2026-09-10` (`.github/workflows/ci.yml:25`). +- Tests return `VortexResult<()>` where they can and use named `rstest` cases. Tests that need a + session share the module `static SESSION: LazyLock` that PR #10681 introduced. +- Every "before the fix" result in these briefs comes from reading the code, not from a run. + +### FX1: Strict decoding of stored `DecimalType` and `PType` values + +- Goal: A reader returns an error for an unknown stored `DecimalType` or `PType` value. Today the + prost getters decode such a value silently as `I8` or `U8`. CD6 detects old decimal nodes by the + stored width, so that field must be strict first. +- Depends on: None. Soft: None. Blocks: CD6. +- Decisions: D34 (one PR for all three commits). FX1 is the prerequisite of the D7 detection in + CD6. It follows the read rule of D14: a file read never panics. It is the first half of node N10 + in the roadmap. +- Branch: `ct/strict-enum-decoding`, based on `develop`. +- Scope: Commit 1 makes the decimal `values_type` strict. Commit 2 makes the protobuf dtype `PType` + strict, and it adds a test for the flatbuffer path, which is already strict. Commit 3 makes the + eleven `PType` fields of array and layout metadata strict. +- Out of scope: New `PType` values and the protobuf `oneof` for wide integers (D5, D20). Edition + gates for core dtypes (the second half of N10). The detection of old decimal nodes (CD6). Paths + that are already strict (listed below). The renaming of the test modules named `test`. The + unused `DictLayoutMetadata` in `vortex-layout/src/layouts/dict/writer.rs:537-550` (see Notes). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs` | `DecimalMetadata::get_values_type`, the strict decode in `deserialize` (`:151-186`), tests | +| `vortex-array/src/dtype/serde/proto.rs` | Strict `PType` decode at `:36`, tests | +| `vortex-array/src/dtype/serde/flatbuffers.rs` | The raw value in the error at `:559`, a test | +| `vortex-array/src/arrays/dict/array.rs` | `DictMetadata::get_codes_ptype` beside the struct (`:29-43`), a test module | +| `vortex-array/src/arrays/dict/vtable/mod.rs` | Call `get_codes_ptype` instead of the getter at `:180` | +| `vortex-array/src/arrays/varbin/vtable/mod.rs` | `VarBinMetadata::get_offsets_ptype` (`:60-64`), call at `:180`, a test module | +| `vortex-array/src/arrays/list/vtable/mod.rs` | `ListMetadata::get_offset_ptype` (`:48-54`), call at `:178`, a test module | +| `vortex-array/src/arrays/listview/vtable/mod.rs` | `ListViewMetadata::get_offset_ptype` and `get_size_ptype` (`:56-64`), calls at `:198` and `:205`, a test module | +| `encodings/runend/src/array.rs` | `RunEndMetadata::get_ends_ptype` (`:62-70`), call at `:155`, a test in `mod tests` (`:574`) | +| `encodings/fastlanes/src/rle/vtable/mod.rs` | `RLEMetadata::get_indices_ptype` and `get_values_idx_offsets_ptype` (`:48-62`), calls at `:178` and `:185`, tests in `mod tests` (`:329`) | +| `encodings/alp/src/alp_rd/array.rs` | `ALPRDMetadata::get_left_parts_ptype` (`:61-73`), call at `:182`, a test in `mod test` (`:563`) | +| `vortex-layout/src/layouts/dict/mod.rs` | `DictLayoutMetadata::get_codes_ptype` (`:163-178`), call at `:83`, a test module | +| `vortex-layout/src/layouts/list/mod.rs` | `ListLayoutMetadata::get_offsets_ptype` (`:228-240`), one call that serves `:87` and `:105`, a test module | + +Design notes: + +The cause. prost-derive 0.14.4, the version in `Cargo.lock`, generates each enumeration getter as +`TryFrom::try_from(raw).unwrap_or(default)` (`prost-derive-0.14.4/src/field/scalar.rs:291-300` in +the cargo registry). The default is the first variant (`src/lib.rs:308`). `DecimalType` and `PType` +both derive `prost::Enumeration` (`vortex-array/src/dtype/decimal/types.rs:25`, +`vortex-array/src/dtype/ptype.rs:33`). So an unknown value reads as `I8` or `U8`. + +Silent fallbacks on `develop`, all fixed by FX1: + +| Read site | Field | Commit | +| --- | --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:170`, `:177` | `DecimalMetadata.values_type` | 1 | +| `vortex-array/src/dtype/serde/proto.rs:36` | `pb::Primitive.r#type` | 2 | +| `vortex-array/src/arrays/dict/vtable/mod.rs:180` | `DictMetadata.codes_ptype` | 3 | +| `vortex-array/src/arrays/varbin/vtable/mod.rs:180` | `VarBinMetadata.offsets_ptype` | 3 | +| `vortex-array/src/arrays/list/vtable/mod.rs:178` | `ListMetadata.offset_ptype` | 3 | +| `vortex-array/src/arrays/listview/vtable/mod.rs:198`, `:205` | `ListViewMetadata.offset_ptype`, `size_ptype` | 3 | +| `encodings/runend/src/array.rs:155` | `RunEndMetadata.ends_ptype` | 3 | +| `encodings/fastlanes/src/rle/vtable/mod.rs:178`, `:185` | `RLEMetadata.indices_ptype`, `values_idx_offsets_ptype` | 3 | +| `encodings/alp/src/alp_rd/array.rs:182` | `ALPRDMetadata.left_parts_ptype` | 3 | +| `vortex-layout/src/layouts/dict/mod.rs:83` | `DictLayoutMetadata.codes_ptype` | 3 | +| `vortex-layout/src/layouts/list/mod.rs:87`, `:105` | `ListLayoutMetadata.offsets_ptype` | 3 | + +Already strict, no change: + +| Path | Item | +| --- | --- | +| `vortex-array/src/dtype/serde/flatbuffers.rs:543-562` | Flatbuffer `PType` | +| `vortex-array/src/patches.rs:103-124` | `PatchesMetadata` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v1.rs:71` | DecimalByteParts v1 | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs:83`, `:93` | DecimalByteParts v2 | +| `encodings/fastlanes/src/bitpacking/plugin/v2.rs:98` | BitPacked v2 | +| `encodings/datetime-parts/src/array.rs:74-88` | DateTimeParts | +| `encodings/fsst/src/array.rs:85-88`, `:269` | FSST | +| `encodings/onpair/src/array.rs:554-563` | OnPair | +| `vortex-array/src/scalar_fn/fns/operators.rs:84` | Expression `BinaryOp` | + +Variant metadata. `VariantMetadataProto.shredded_dtype` +(`vortex-array/src/arrays/variant/vtable/mod.rs:51-55`, decoded at `:154-159`) and +`ParquetVariantMetadataProto.typed_value_dtype` (`encodings/parquet-variant/src/vtable.rs:66-77`, +decoded at `:212-216`) store protobuf dtypes on disk. Neither one holds an enum field directly. Both +call `DType::from_proto`, so commit 2 covers them. Commit 2 also covers the other callers of +`DType::from_proto`: scalars (`vortex-array/src/scalar/proto.rs:198`), cast options +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:91`), `variant_get` options +(`vortex-array/src/scalar_fn/fns/variant_get/mod.rs:88`), the indexed layout +(`vortex-layout/src/layouts/indexed/mod.rs:393`), zstd buffers +(`encodings/zstd/src/zstd_buffers.rs:578`), `vortex-tensor` and `vortex-json`. + +Rules for the change: + +- Each metadata struct gets one `get_()` accessor per enum field. It returns + `VortexResult` or `VortexResult`. This follows + `DateTimePartsMetadata::get_days_ptype` and `FSSTMetadata::get_uncompressed_lengths_ptype`. The + plain field name is taken by the prost getter. +- Put each accessor in an `impl` block right below its struct. Give it the smallest visibility that + its caller needs. +- Each `deserialize` calls the accessor right after it decodes the metadata. The call comes before + any other check and before any buffer or child read. So an unknown value always fails first. +- Keep the `enumeration` attributes. They document the wire type, and the setters such as + `set_codes_ptype` use them. The generated `pb::Primitive::r#type()` getter also stays, because + generated files are never edited by hand. +- Decode the protobuf field with `pb::PType::try_from(p.r#type)`, then convert with the existing + `From for PType` (`proto.rs:229-246`). Do not use `PType::try_from(i32)` here. It ties + the wire numbers to the discriminants of the Rust enum. The numbers match today + (`vortex-array/proto/dtype.proto:11-23`, `ptype.rs:37-60`), but the two enums are separate. +- proto3 omits a field that holds 0, and a missing field decodes as 0. Zero is `U8` or `I8`, which + are valid values. So a file from a current writer reads the same way after FX1. +- Every error uses the `Serde` category. It states the requirement first and ends with the raw + value. + +| Site | Message | +| --- | --- | +| Decimal metadata | `DecimalMetadata values_type must be a known DecimalType, got {raw}` | +| Protobuf dtype | `protobuf Primitive type must be a known PType, got {raw}` | +| Flatbuffer dtype | `flatbuffer Primitive ptype must be a known PType, got {raw}` | +| Array and layout metadata | `{Struct} {field} must be a known PType, got {raw}`, for example `DictMetadata codes_ptype must be a known PType, got 11` | + +```rust +impl DecimalMetadata { + fn get_values_type(&self) -> VortexResult { + DecimalType::try_from(self.values_type).map_err(|_| { + vortex_err!( + Serde: "DecimalMetadata values_type must be a known DecimalType, got {}", + self.values_type + ) + }) + } +} + +// In `DType::from_proto`. +DtypeType::Primitive(p) => { + let ptype = pb::PType::try_from(p.r#type).map_err(|_| { + vortex_err!(Serde: "protobuf Primitive type must be a known PType, got {}", p.r#type) + })?; + Ok(Self::Primitive(ptype.into(), p.nullable.into())) +} +``` + +Steps: + +1. `fix(array): decode decimal values_type strictly`. Add `DecimalMetadata::get_values_type`. In + `Decimal::deserialize`, call it right after `DecimalMetadata::decode` at `:160`. Use the result + at `:170` and `:177`. Add the decimal tests. +2. `fix(array): decode protobuf PType strictly`. Change `proto.rs:36` as sketched. Put the raw value + into the error at `flatbuffers.rs:559`. Add the protobuf and flatbuffer tests. +3. `fix: decode PType metadata fields strictly`. Add the accessors of commit 3 in the table above. + Call each one at the top of its `deserialize`. In `vortex-layout/src/layouts/list/mod.rs`, bind + the result once and use it at `:87` and `:105`. Add the accessor tests. + +Tests: + +- `vortex-array/src/arrays/decimal/vtable/mod.rs`, `mod tests` (`:220`): + - `test_deserialize_rejects_unknown_values_type`, with named cases `first_unused` (6), + `negative` (-1) and `max` (`i32::MAX`). Encode `DecimalMetadata { values_type: raw }`. Call + `Decimal.deserialize` with one host buffer of 4 bytes, the dtype `Decimal(2, 0)` non-nullable, + length 4 and no children. Assert an error that contains `must be a known DecimalType`. Before + the fix, case `first_unused` returns `Ok` with an `I8` array. + - `test_serde_each_values_type`, with one named case for each width at a precision with that + canonical width: `i8` at `Decimal(2, 0)`, `i16` at `(4, 0)`, `i32` at `(9, 2)`, `i64` at + `(18, 2)`, `i128` at `(38, 2)` and `i256` at `(76, 2)`. Round-trip as `test_array_serde` does + (`:241-273`). Assert the decoded `values_type()` and use `assert_arrays_eq!`. Every case uses + the canonical width, so CD8a keeps all of them valid. +- `vortex-array/src/dtype/serde/proto.rs`, `mod tests` (`:286`): + - `test_from_proto_rejects_unknown_ptype`, with named cases `first_unused` (11), `negative` (-1) + and `max` (`i32::MAX`). Assert an error that contains `must be a known PType`. Before the fix, + every case returns `Primitive(U8)`. + - `test_from_proto_rejects_unknown_ptype_in_struct_field`. A struct dtype with one field whose + `PType` is 11. This is the shape of a shredded Variant dtype. Assert an error. +- `vortex-array/src/dtype/serde/flatbuffers.rs`, `mod test` (`:565`): + `test_unknown_ptype_from_flatbuffer`. Assert that `PType::try_from(fb::PType(11))` returns an + error that contains `got 11`. The path is strict today, so this test covers the message only. +- Commit 3: one test per struct, named `test_get__rejects_unknown_ptype`. Use named cases + where a struct has two fields (`ListViewMetadata`, `RLEMetadata`). Each test builds the struct + with the field set to 11 and asserts an error that names the struct and the field. One more case + with a valid value asserts `Ok`. Put the test in the file that defines the struct. Add a + `#[cfg(test)] mod tests` at the bottom of `dict/array.rs`, `varbin/vtable/mod.rs`, + `list/vtable/mod.rs`, `listview/vtable/mod.rs`, `vortex-layout/src/layouts/dict/mod.rs` and + `vortex-layout/src/layouts/list/mod.rs`. +- No existing test changes. + +Benchmarks: None. Each read adds one integer match per enum field. CodSpeed is expected to show +no change. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp` +- `cargo nextest run -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp -p vortex-parquet-variant` +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode all`. It reads every published + fixture, so it shows that no published file holds a value outside the enums. +- An unknown stored value returns a `Serde` error. No read path panics on it. +- No read path in the table above calls a prost enumeration getter. + +Review focus: + +- Every read site in the table calls its accessor, and the call comes before any other check. +- `rg -n 'metadata\.\w+_ptype\(\)|metadata\.values_type\(\)' | rg -v '\.get_'` finds no read + path. +- The error messages follow the table, with the raw value last. +- The wire format and the setters do not change. +- An old file with a value outside the enum now fails. Before FX1 it read as `U8` or `I8`. Whether + such a file exists is Not verified. The compat check covers the published fixtures. +- CD6 keeps `get_values_type()` as the first step of its detection. + +Size: medium. Thirteen fields in five crates, each with a small test. + +Open questions: None. CD6 needs only commit 1, but the maintainer chose one PR for all three +commits (D34). + +### FX2: DecimalByteParts nullability cast follows the cast-rule contract + +- Goal: The DecimalByteParts (DBP) cast rule rewrites a nullability cast only when + `Validity::trivially_cast_nullability` proves it safe. A new cast kernel handles the declined + case on the parts. Today the rule casts the most significant part (MSP) for every target + nullability, and DBP has no cast kernel. +- Depends on: None. Soft: CD5 (same crate, other files). Blocks: None. +- Decisions: None by ID. `decisions.md`, section "The `Narrow` pull requests", settles this PR: the + guard of #10230, a cast kernel for the case that needs execution, tests, and no issue. +- Branch: `ct/dbp-nullability-cast`, based on `develop`. +- Scope: The guard in `CastReduce`, a `CastKernel` for casts that change only the nullability, its + registration, and tests. +- Out of scope: Casts that change the precision or the scale stay unsupported (the `TODO(mk)` at + `compute/cast.rs:19`). DBP decoding and `u64` lower parts (CD5). The same push-down pattern in + other encodings (see Notes). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs` | The guard in `CastReduce` (`:16-35`), a new `CastKernel`, one shared private helper, tests, and a test session with `crate::initialize` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/kernel.rs` | Register `CastExecuteAdaptor(DecimalByteParts)` under `Cast.id()` (`:14-26`) | + +`rules.rs:14` keeps the cast rule as it is. + +Design notes: + +The contract (`vortex-array/src/validity.rs:430-494`). A cast rule runs in the optimizer and cannot +execute. It calls `trivially_cast_nullability` and returns `Ok(None)` when that call returns +`Ok(None)`. A cast kernel runs at execution and calls `cast_nullability`, which can read the +validity array. FSST (`encodings/fsst/src/compute/cast.rs:48-91`, registered at +`encodings/fsst/src/kernel.rs:25`) and BitPacked +(`encodings/fastlanes/src/bitpacking/compute/cast.rs:56`, `:74`) show the pair. + +DBP has no validity of its own. Its validity is the validity of the MSP +(`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:200`, `:404-409`). So a cast of the +MSP is the only way to change it. `with_msp` keeps the lower parts (`array.rs:346-352`). + +Outcomes for a cast that changes only the nullability: + +| MSP validity | Target | Rule | Kernel | +| --- | --- | --- | --- | +| Any | Nullable | DBP over the MSP cast to nullable | Same result | +| `NonNullable`, `AllValid`, or length 0 | Non-nullable | DBP over the MSP cast to non-nullable | Same result | +| `AllInvalid`, length above 0 | Non-nullable | Error `Cannot cast AllInvalid to NonNullable` | Error, if an unoptimized cast reaches it | +| `Array` | Non-nullable | `Ok(None)` | Error when a null exists, else DBP over the MSP cast to non-nullable | + +```rust +/// Rebuilds `array` over its MSP cast to `nullability`. +/// +/// The MSP carries the validity of a DBP array, so the cast of the MSP sets the validity of the +/// result. The lower parts are non-nullable and stay as they are. +fn with_msp_nullability( + array: ArrayView<'_, DecimalByteParts>, + nullability: Nullability, +) -> VortexResult; + +impl CastReduce for DecimalByteParts { + fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { + // TODO(mk): Support non-nullability changes as well, e.g. precision. + if !dtype.eq_ignore_nullability(array.dtype()) { + return Ok(None); + } + let nullability = dtype.nullability(); + if array + .validity()? + .trivially_cast_nullability(nullability, array.len())? + .is_none() + { + return Ok(None); + } + + with_msp_nullability(array, nullability).map(Some) + } +} + +impl CastKernel for DecimalByteParts { + fn cast( + array: ArrayView<'_, Self>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !dtype.eq_ignore_nullability(array.dtype()) { + return Ok(None); + } + let nullability = dtype.nullability(); + array.validity()?.cast_nullability(nullability, array.len(), ctx)?; + + with_msp_nullability(array, nullability).map(Some) + } +} +``` + +- The guard is the change of #10230 to this file, as `gh pr diff 10230 --repo vortex-data/vortex` + shows. FX2 keeps it as it is. +- The kernel returns `Ok(None)` for every other cast. The generic `Cast::execute` then decodes the + DBP, as it does today (`vortex-array/src/scalar_fn/fns/cast/mod.rs:126-159`). +- The kernel discards the `Validity` that `cast_nullability` returns, because the MSP stores the + validity. A comment beside that call must say this. For a Primitive MSP with a validity array, + the MSP cast stays lazy, because the Primitive rule declines + (`vortex-array/src/arrays/primitive/compute/cast.rs:62-67`). The Primitive kernel checks again + when the DBP executes. It reads the minimum of the same validity array, which is a cached + statistic. That this second check costs only a cache lookup is Not verified. +- The existing check `let DType::Decimal(_, target_nullability) = dtype` at `cast.rs:24-26` goes + away. `eq_ignore_nullability` against a decimal dtype already implies it. + +Is a wrong validity observable before execution today? It is not shown from the code: + +- The current rule returns a DBP over `Cast(msp)` with a non-nullable dtype (`cast.rs:28-33`). +- `validity()` of an array with a non-nullable dtype returns `Validity::NonNullable` without asking + the encoding (`vortex-array/src/array/mod.rs:304-319`). +- The lazy `Cast(dbp)` that the rule replaces has the same dtype, so it reports the same validity. + `Cast::validity` also reports all rows valid for a non-nullable target + (`vortex-array/src/scalar_fn/fns/cast/mod.rs:183-195`). +- The null check still runs when the MSP cast executes. The existing test + `test_cast_decimal_byte_parts_nullable_to_non_nullable` (`cast.rs:92-108`) expects that error. +- So the defect is a contract violation. The rule rewrites a cast that it cannot prove safe, and it + depends on the cast of each MSP encoding to keep the null check. +- These MSP cast rules were read: Primitive, BitPacked, FoR, ZigZag, RunEnd, Delta, Sparse, + Sequence, Constant, Dict and RLE. Each one keeps the check, pushes it to a child, or declines. + Whether another MSP encoding drops the check is Not verified. + +After FX2, `cast` of a DBP whose MSP has a validity array returns a lazy `Cast` over the DBP, not a +DBP. Execution gives the same values or the same error as today. No caller in the workspace +downcasts a cast result to DBP (checked with a search for `is::` and +`as_opt::`). + +Steps: + +1. `fix(decimal-byte-parts): guard the nullability cast rule`. Add `with_msp_nullability`, the + guard, the `CastKernel`, and the registration in `compute/kernel.rs`. Add the tests. + +Tests, all in the `mod tests` of `compute/cast.rs` (`:37-135`): + +- Change the module session to the one in `compute/mod.rs:35-39`, which calls + `crate::initialize(&session)`. Without it, the execution context does not hold the new kernel. +- Add one fixture helper at the top. It takes a `Validity` and builds a DBP at + `DecimalDType::new(10, 2)` over `PrimitiveArray::new(buffer![100i32, 200, 300], validity)`. +- Build each validity array with `Validity::Array(BoolArray::from_iter([..]).into_array())`. + `Validity::from_iter` and `PrimitiveArray::from_option_iter` turn all-true input into `AllValid` + and all-false input into `AllInvalid` (`vortex-array/src/validity.rs:506-532`). So they cannot + build these cases. Each validity-array case asserts that the fixture reports `Validity::Array`, + so that it reaches the guard that it claims to test. +- `test_cast_rule_declines_validity_array`, with named cases `with_nulls` (`[true, false, true]`), + `without_nulls` (`[true, true, true]`) and `only_nulls` (`[false, false, false]`). Assert that + `::cast` returns `None` for the non-nullable target. The direct + call follows `encodings/fastlanes/src/bitpacking/array/tests.rs:191-192`. +- `test_cast_kernel_rejects_nulls`, with named cases `with_nulls` and `only_nulls` as above. Assert + that `::cast` returns an error. Also assert that `cast` followed + by `execute::` returns an error. +- `test_cast_kernel_without_nulls`. Case `without_nulls`. Assert that the kernel returns a DBP with + a non-nullable dtype. Assert with `assert_arrays_eq!` that `cast` gives the non-nullable + `DecimalArray` of the same values. +- `test_cast_rule_rejects_all_invalid`. MSP validity `AllInvalid`. Assert that `cast` to the + non-nullable dtype returns an error that contains `Cannot cast AllInvalid to NonNullable`. This + error comes from the rule, before execution. +- `test_cast_rule_reduces`, with named cases `nullable_all_valid_to_non_nullable` and + `non_nullable_to_nullable`. Assert that `cast` returns a DBP (`is::()`) with + the target dtype and the same values. This shows that the rule still avoids execution where it + can. +- `test_cast_kernel_keeps_lower_parts`. Build a DBP with + `DecimalByteParts::try_new_with_lower_parts` from an `i64` MSP `[1, 0, -1]` with an all-true + validity array and one `u64` lower part `[5, 6, 7]`, at `DecimalDType::new(38, 2)`. Assert with + `assert_arrays_eq!` that the cast to non-nullable gives the `i128` values + `[(1 << 64) + 5, 6, (-1 << 64) + 7]`. +- Keep `test_cast_decimal_byte_parts_nullability`, + `test_cast_decimal_byte_parts_nullable_to_non_nullable` and the conformance cases. + +Benchmarks: None. No benchmark covers DBP casts. A cast with a validity array now runs its null +check in the DBP kernel first, and the MSP kernel then finds the cached minimum. CodSpeed is +expected to show no change. + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-decimal-byte-parts` +- `cargo nextest run -p vortex-decimal-byte-parts` +- The rule returns `Ok(None)` whenever `trivially_cast_nullability` returns `Ok(None)`. +- A cast to non-nullable of a DBP with nulls fails, with or without a validity array. +- A cast to non-nullable without nulls keeps the DBP and its lower parts. + +Review focus: + +- The kernel is registered under `Cast.id()` for `DecimalByteParts`. +- The guard runs before any cast of the MSP. +- Both the rule and the kernel return `Ok(None)` for a cast that changes more than the + nullability. +- The comment beside the discarded `Validity` in the kernel is correct. +- The test session registers the crate, so the kernel tests do not pass through the decode + fallback. + +Size: small. One file of logic, one registration and focused tests. + +Open questions: None. + +### FX3: Dictionary `fill_null` no longer truncates the fill code + +- Goal: The dictionary `fill_null` kernel stops truncating the index of the fill value into the + codes type. When the index does not fit the codes dtype, the kernel declines, and the generic + path fills the decoded array. +- Depends on: None. Soft: None. Blocks: NR5. With NR5, `optimize()` unwraps a `Narrow` over + dictionary codes to a narrow child + (`af4b663059:vortex-array/src/arrays/narrow/tests/merge.rs:29-41`). This makes the bug easy to + reach. NR5 also carries the `Narrow` regression test of this fix. +- Decisions: None by ID. `decisions.md`, section "The `Narrow` pull requests", makes this fix its + own PR. +- Branch: `ct/dict-fill-code`, based on `develop`. +- Scope: The fix of local commit `7e2e5e4d7a` to + `vortex-array/src/arrays/dict/compute/fill_null.rs`, and its `Narrow`-free regression test, + adapted to the current test module. +- Out of scope: The `Narrow` test + `7e2e5e4d7a:vortex-array/src/arrays/narrow/tests/merge.rs:45-62`, which NR5 owns. Any change to + the "no fill value found" branch at `fill_null.rs:45-55`. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/dict/compute/fill_null.rs` | Checked fill code at `:57-76`, two unused imports removed (`:19`, `:21`), a regression test in `mod tests` (`:90-131`) | + +Design notes: + +How the truncation happens on `develop`, without `Narrow`: + +1. The kernel compares every dictionary value with the fill value (`fill_null.rs:33-40`). This + includes values that no code references. +2. It takes the first match as a `usize` index (`:43-44`). +3. It converts the index with `as P` into the integer type of the codes (`:61-69`). The + `#[expect(clippy::cast_possible_truncation)]` at `:63-66` assumes that the index fits. +4. A dictionary can hold more values than its codes type can index. `DictArray::try_new` and + `Dict::validate` check only the codes type, the lengths and the dtypes + (`dict/array.rs:303-315`, `dict/vtable/mod.rs:97-116`). +5. So `u8` codes `[0, null]` over the 301 values `0..=300` form a valid dictionary. With fill value + 300, `300 as u8` is 44, and the null row reads value 44. With fill value 256, it reads value 0. +6. With `i8` codes and index 128, the code becomes -128. Whether the read then returns an error or + panics is Not verified. + +Whether a producer on `develop` other than direct construction builds such a dictionary is Not +verified. + +The fix, as in `7e2e5e4d7a`: + +```rust +let codes = array.codes(); +// An unreferenced dictionary entry can exceed the codes' range. +let Ok(fill_code) = + Scalar::from(existing_fill_value_index).cast(&codes.dtype().as_nonnullable()) +else { + return Ok(None); +}; + +// Fill nulls in both the codes and the values. ... +let codes = codes.clone().fill_null(fill_code)?; +``` + +- `Scalar::from(usize)` builds a `u64` scalar + (`vortex-array/src/scalar/convert/primitive.rs:199-203`). The cast to an integer dtype is + checked. `PValue::cast` returns an error for a value that does not fit + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:287-303`). +- `Ok(None)` declines the kernel. `FillNull::execute` then decodes the dictionary and fills the + result (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:153-163`). +- The first match has the smallest index. If it does not fit, no later match fits. So declining on + the first match loses no fast path. +- Remove the imports `match_each_integer_ptype` (`:19`) and `ScalarValue` (`:21`). Only the + replaced code uses them. + +Steps: + +1. `fix(array): avoid truncating dictionary fill codes`. Apply the kernel change and the test in + one commit, as `7e2e5e4d7a` does. + +Tests: + +- Port `fill_unreferenced_value` from `7e2e5e4d7a:vortex-array/src/arrays/dict/compute/fill_null.rs` + into `mod tests` at `fill_null.rs:90-131`. PR #10681 rewrote this module (commit `d834a11faa`), + so adapt the test: + - Use the module `SESSION` (`:109`) with `SESSION.create_execution_ctx()`, not `array_session()`. + - Drop the explicit `.execute::` before `assert_arrays_eq!`. The macro executes + the lazy result (`AGENTS.md`, Tests), and PR #10458 removed the same pattern from this module. + - Add `use rstest::rstest;` and `use vortex_error::VortexResult;`. +- The test reaches the dictionary kernel. `VortexSession::empty()` installs the built-in kernels on + first use (`vortex-array/src/optimizer/kernels.rs:336-349`), including this one + (`dict/vtable/kernel.rs:25`). The codes are partially valid, so `short_circuit` does not answer + first (`vortex-array/src/scalar_fn/fns/fill_null/kernel.rs:67-93`). +- Cases. Each dictionary has the codes `[0, null]` and the values `0..=fill`. The results before the + fix come from reading the code, not from a run. + +| Case | Codes type | Fill | Before the fix | After the fix | +| --- | --- | --- | --- | --- | +| `u8_max` | `u8` | 255 | `[0, 255]`, passes | `[0, 255]` | +| `u8_overflow` | `u8` | 256 | `[0, 0]`, fails | `[0, 256]` | +| `u8_truncation` | `u8` | 300 | `[0, 44]`, fails | `[0, 300]` | +| `i8_max` | `i8` | 127 | `[0, 127]`, passes | `[0, 127]` | +| `i8_overflow` | `i8` | 128 | Code -128, error or panic Not verified | `[0, 128]` | + +- `nullable_codes_fill_in_values` stays as it is. + +Benchmarks: None. The change adds one checked scalar cast per kernel call. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array dict::compute::fill_null` +- With the kernel change reverted, `u8_overflow` and `u8_truncation` fail. + +Review focus: + +- No `as` conversion of the index remains, and the `#[expect(clippy::cast_possible_truncation)]` is + gone. +- The declined case gives the same result as the "no fill value found" branch. +- The test style matches the module after PR #10681. + +Size: small. One kernel branch and one parameterized test. + +Open questions: None. + +## Notes from planning + +These facts came up while the briefs were drafted. They do not change the scope of the PRs. + +1. FX1 decodes the protobuf field with `pb::PType::try_from` and the existing + `From for PType`, not with `PType::try_from(i32)`. The wire numbers then stay tied to + the protobuf enum. The numbers of the two enums match today. +2. The flatbuffer `PType` path is already strict + (`vortex-array/src/dtype/serde/flatbuffers.rs:559`). FX1 only adds the raw value to its message + and a test. +3. No Variant metadata field holds an enum. Variant and Parquet-variant metadata store protobuf + dtypes, so the protobuf fix covers them. +4. `vortex-layout/src/layouts/dict/writer.rs:537-550` defines a second `DictLayoutMetadata` that no + code uses. Its comment says to read the field with the generated getter, which FX1 makes wrong + advice. Its removal belongs in a separate change. +5. FX2 fixes a contract violation, not a wrong result that a caller can see. The rule does not + change the validity that a caller sees before execution, and the null check still runs at + execution. +6. The cast rules of RunEnd (`encodings/runend/src/compute/cast.rs:15-33`), ZigZag + (`encodings/zigzag/src/compute/cast.rs:14-25`), FoR + (`encodings/fastlanes/src/for_/compute/cast.rs`) and RLE + (`encodings/fastlanes/src/rle/compute/cast.rs`) push nullability casts down without the same + guard. A follow-up issue tracks them. FX2 does not change them. +7. The cast tests in `compute/cast.rs:60` use `VortexSession::empty()` without + `crate::initialize`. FX2 changes that session, or the tests never reach the new kernel. +8. Commit `7e2e5e4d7a` has two tests. `fill_unreferenced_value` already avoids `Narrow`, and FX3 + ports it. The `Narrow` test belongs to NR5, so NR5 lands after FX3. +9. On `develop`, the FX3 bug is shown only for a dictionary built directly. Whether the compressor + or a file read builds a dictionary with more values than its codes type can index is Not + verified. +10. The test modules in `vortex-array/src/dtype/serde/flatbuffers.rs:565` and + `encodings/alp/src/alp_rd/array.rs:563` are named `test`, which `AGENTS.md` discourages. FX1 + adds tests to them but does not rename them. diff --git a/research/decimal/agents/briefs/02-integer-narrow.md b/research/decimal/agents/briefs/02-integer-narrow.md new file mode 100644 index 00000000000..85168337d7c --- /dev/null +++ b/research/decimal/agents/briefs/02-integer-narrow.md @@ -0,0 +1,907 @@ + + +# Integer `Narrow` PR briefs: NR1 to NR6 + +These briefs cover the integer `Narrow` track. NR1, NR3 and NR4 land the local rewrite in +`/Users/connor/spiral/vortex-data/vortex3` (base `61ff28d970`) after a rebase onto `develop` at +`b70206818f`. NR1 also gets `encode_values`, the serialization workaround and a flatten rule. NR2 +is new. NR5 keeps only the dictionary-codes rule of the rewrite, and it adds the rules that move +structural encodings above `Narrow` (D25). NR6 moves the existing narrowing call sites to +`encode_values` (D28). All rewrite commits and `adc1e83b9d` are reachable from every worktree, +because the worktrees share one bare repository. + +No build, test or benchmark ran for these briefs. Every statement that a commit builds or passes +comes from reading the code, and it is marked "Not verified" where it matters. + +## Shared context for the track + +### Commit map + +| Local commit | Subject | PR | Files | +| --- | --- | --- | --- | +| `f7408481ce` | feat(array): add in-memory Narrow integer encoding | NR1 | `vortex-array/src/arrays/narrow/{mod,vtable,encoding,compare,rules}.rs`, `narrow/tests/{mod,encoding,compute}.rs`, `vortex-array/src/arrays/mod.rs`, `vortex-array/src/session/mod.rs` | +| (none) | new benchmarks | NR2 | `vortex-array/benches/narrow.rs`, `vortex-array/Cargo.toml` | +| `8d548646ee` | feat(array): forward Narrow integer aggregates | NR3 | `narrow/aggregates.rs`, `narrow/mod.rs`, `narrow/tests/{mod,aggregates}.rs`, `vortex-array/src/aggregate_fn/session.rs`, `encodings/fastlanes/tests/narrow.rs` | +| `74aa17644e` | feat(array): push arithmetic and scalar operations through Narrow | NR4 | `narrow/numeric.rs`, `narrow/{rules,encoding,mod}.rs`, `narrow/tests/{mod,numeric,pushdown}.rs` | +| `af4b663059` | feat(array): retain Narrow storage through merges and indices | NR5, in part | Only `TakeIndicesReduce` in `narrow/rules.rs` and `test_narrowed_take_indices`. The chunked and interleave merge rules in `narrow/merge.rs` are dropped, because they put `Narrow` above `Chunked` and `Interleave` (D25) | +| `7e2e5e4d7a` | fix(array): avoid truncating dictionary fill codes | FX3, plus one test for NR5 | `vortex-array/src/arrays/dict/compute/fill_null.rs` (FX3), `narrow/tests/merge.rs` test `test_fill_null_after_narrowing_take_indices` (NR5, in `narrow/tests/structural.rs`) | +| `adc1e83b9d` | feat(compress): retain narrow integer children through compression | `encode_values` to NR1, the call-site switch to NR6, the rest to WR1 to WR3 | See the NR1 and NR6 briefs | + +Each commit builds on its predecessor only, by reading: + +- `f7408481ce` needs nothing but `develop`. Its tests use only NR1 code. +- `8d548646ee` needs NR1. It does not need NR2. +- `74aa17644e` needs NR1 code only. Its diff context in `narrow/mod.rs` and `narrow/tests/mod.rs` + names `mod aggregates`, so a cherry-pick without NR3 conflicts textually. NR3 is its base. +- The part of `af4b663059` that NR5 keeps needs NR4, because `TakeIndicesReduce` sits in the NR4 + version of `rules.rs`. +- NR4 rewrites the cast and fill-null rules of NR1. The NR1 versions are correct subsets, so the + churn is acceptable. + +None of these statements is verified by a build. + +### Rebase onto `develop` + +`git merge-tree --write-tree --merge-base=61ff28d970 b70206818f ` reports no textual +conflict for `f7408481ce`, `8d548646ee`, `74aa17644e` and `af4b663059`. It reports one conflict +for `7e2e5e4d7a`, in `vortex-array/src/arrays/dict/compute/fill_null.rs`. That commit is FX3, not +part of this track. + +These changes on `develop` since `61ff28d970` touch the APIs that the rewrite uses: + +| Change on `develop` | Commit | Effect on the rewrite | +| --- | --- | --- | +| `vortex_array::array_session()` removed | `d834a11faa` (#10681) | NR1 to NR5 have 39 calls and 6 `use crate::array_session` lines. FX3 adds 2 more calls. Together these are the 47 mentions that the notes count. Every one must change, with the pattern below | +| Interleave uses named slots | `96ac9f2cc5` (#10688) | `InterleaveArray::try_new(values, array_indices, row_indices)` and `InterleaveArrayExt` keep their signatures. The slot order is the same: selectors at 0 and 1, values from 2. `InterleaveSlots` is now `pub(super)` (`vortex-array/src/arrays/interleave/array.rs:24`). NR5 drops `narrow/merge.rs`, which used the literal `2`. The NR5 pull-up rule for `Interleave` builds a new array with `InterleaveArray::try_new` | +| Storage pushdown in the aggregate session | `d7562aa0e6` (#10566), `3bf14ba647` (#10616) | `AggregateFnSession::default` gained `register_storage_ordered` calls (`vortex-array/src/aggregate_fn/session.rs:127-129`). The NR3 insertion merges cleanly. `register_aggregate_kernel` and the fallback lookup are unchanged (`:199-224`) | +| `fill_null` on an empty input | `633c52deab` (#10627) | `FillNull` returns an empty array for a non-constant fill on an empty input (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:145`). The `Narrow` rule receives a `Scalar`, so it is not affected | +| Kernel for constant dictionary codes | `a169338930` (#10643) | A new execute-parent kernel handles `Constant` codes (`vortex-array/src/arrays/dict/take.rs:181-205`). The NR5 rule for `Narrow` codes is a reduce rule, so the two do not overlap | +| Deprecated APIs and `legacy_session` removed | `85ae34d5f0` (#10595) | The rewrite uses none of the removed APIs | +| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064` | `ListViewArray::try_new` and `Patches` take a context. The rewrite uses neither | + +`VTable`, `OperationsVTable`, `ValidityChild`, the kernel registry, `CompareKernel`, +`execute_compare`, `execute_numeric`, the reduce adaptors, `ChunkedSlots::CHUNKS_OFFSET`, +`DictSlots` and `BitBuffer::full_in` keep the signatures that the rewrite uses at `b70206818f`. +The execute-parent registry keeps a list for each key +(`vortex-array/src/optimizer/kernels.rs:263-276`). So the compare kernel of NR1 and the numeric +kernel of NR4 can both register for `Binary` on `Narrow`. + +#### Session replacement pattern + +PR #10681 gave each test module one session. `VortexSession::empty()` creates the vortex-array +session variables on first use, and the default `KernelSession` runs `arrays::initialize` +(`vortex-array/src/optimizer/kernels.rs:336-349`), so the `Narrow` kernels are present. + +```rust +// Module level, as in vortex-array/src/arrays/dict/compute/fill_null.rs:109 on develop. +use std::sync::LazyLock; + +use vortex_session::VortexSession; + +static SESSION: LazyLock = LazyLock::new(VortexSession::empty); +``` + +| Before | After | +| --- | --- | +| `use crate::array_session;` | Delete | +| `let mut ctx = array_session().create_execution_ctx();` | `let mut ctx = SESSION.create_execution_ctx();` | +| `let session = array_session();` | Delete, and use `SESSION` | +| `session.aggregate_fns()` | `SESSION.aggregate_fns()` | +| `session.array_serialize(&array)?` | `SESSION.array_serialize(&array)?` | +| `array.serialize(&ctx, &session, &opts)` | `array.serialize(&ctx, &SESSION, &opts)` | +| `Narrow.deserialize(.., &session)` | `Narrow.deserialize(.., &SESSION)` | + +A test outside vortex-array that needs another crate registers it in the closure, as +`encodings/fastlanes/src/bitpacking/array/tests.rs:43-47` does: + +```rust +static SESSION: LazyLock = LazyLock::new(|| { + let session = VortexSession::empty(); + vortex_fastlanes::initialize(&session); + session +}); +``` + +No `Narrow` test changes its session after setup, so every test can share the static. + +### Stack and branches + +The closed PRs used `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and +`ct/narrow-compression` (#10333). All three still exist on `origin`. The new branches avoid them. + +| PR | Branch | Base | +| --- | --- | --- | +| NR1 | `ct/narrow-in-memory` | `develop` | +| NR2 | `ct/narrow-benchmarks` | NR1 | +| NR3 | `ct/narrow-aggregate-kernels` | NR2 | +| NR4 | `ct/narrow-arithmetic-kernels` | NR3 | +| NR5 | `ct/narrow-structural-rules` | NR4, after FX3 is on `develop` | +| NR6 | `ct/narrow-encode-values` | `develop` after NR1 merges. Merge it after NR5, so that the stack lands first (D35) | + +NR1 to NR5 form one linear stack (D35). NR3 needs no code from NR2, but it sits on NR2 so that the +benchmarks exist before the kernels change. NR6 needs only NR1. CodSpeed baselines come from pushes +to `develop` (`.github/workflows/codspeed.yml:13-17`). How CodSpeed picks a base for a PR whose base +branch is another PR branch is not verified. The safe order is to merge NR1 and NR2, then rebase NR3 +to NR5 on `develop` before their CodSpeed reports are read. + +### Integer-only sites in the rewrite (input for CD4) + +The private `validate_dtypes` is the only check that rejects a decimal dtype. Many other sites +assume integers. Most of them decline a decimal with `Ok(None)` and fall back to +canonicalization, which is safe. Five sites give a panic, a wrong result or a likely execution loop +once `validate_dtypes` accepts decimals. CD4 must change those five in the same PR. + +| Site | Assumption | Effect with a decimal `Narrow` | Safe for decimals | +| --- | --- | --- | --- | +| `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73-88` `validate_dtypes` | `PType::try_from`, integer, signedness, byte width | The gate. It rejects decimals | Gate | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:52-61` | `as_ptype()` and `DType::Primitive` for two `Narrow` operands | Panic: `DType::as_ptype` panics on a non-primitive dtype (`vortex-array/src/dtype/dtype_impl.rs:209-215`) | No | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:89` | `below = lhs.dtype().is_signed_int() && i64::try_from(&constant)? < 0` | Wrong result: `is_signed_int` is false for a decimal (`dtype_impl.rs:226-231`), so a negative constant below the child range answers `Lt`, `Lte`, `Gt` and `Gte` wrongly | No | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:76-86` | Constant cast to the child dtype | Correct when the child label is true (D10): the decimal scalar cast checks the precision (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:158-182`) | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:316-346` `BetweenReduce` | Constant cast to the child, else two compares | Inherits the wrong `below` result through `two_compares` | No | +| `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | Accumulate at the child dtype, then cast the partial | Wrong result for `Sum` and `SumV2`: the decimal sum state is `Decimal(p + 10, s)` (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`), so a child state overflows to null before the parent state does | No | +| `f7408481ce:vortex-array/src/arrays/narrow/vtable.rs:111-116` `execute` | Yields `values.cast(dtype)` | Today the decimal cast executes a copy (`vortex-array/src/arrays/decimal/compute/cast.rs`). D15 says a decimal cast to a wider canonical width returns a `Narrow` over its input. If CD4 implements that as a reduce rule, `execute` returns the same `Narrow` again and execution does not converge. Not verified | Depends on CD4 | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:111-143` `CastReduce` | `PType::try_from` for target and child | Declines decimals. CD4 adds the decimal cast rules of D15 | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:146-194` fill null, `integer_storage_type` | `PType::try_from`, `i64::try_from`, `min_*_ptype_for_value` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:240-278` `zip_values` | `PType::try_from`, `DType::Primitive` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | Not reachable for decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/numeric.rs:70` | `PType::try_from(array.dtype())` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/encoding.rs:29` `encode` | Takes a `PrimitiveArray`, rejects non-integers | No decimal producer. CD4 adds one | Yes | + +## Briefs + +### NR1: In-memory integer `Narrow` encoding + +- Goal: Add `Narrow`, an in-memory encoding that stores an integer column as a strictly narrower + integer child with the same signedness and nullability. Selection, compare, mask, cast and fill + null pass to the child. `Narrow` has no wire form yet, so the serializer writes its canonical + array instead (D26). +- Depends on: None. Soft: None. Blocks: NR2, NR3, NR4, NR5, NR6, CD4, CD7. +- Decisions: D8 (one encoding with ID `vortex.narrow`), D15, D18, D22, D23, D26, D28. +- Branch: `ct/narrow-in-memory`, based on `develop` (`b70206818f` or later). +- Scope: + - Cherry-pick `f7408481ce`, and replace `array_session()` in its tests. + - Add `NarrowArray::encode_values`, marked `#[deprecated]` (D28), from `adc1e83b9d`. + - Add the serialization workaround of D26: a plugin hook that lowers an array before it is + serialized, which `Narrow` implements by executing itself to its canonical array. + - Add a reduce rule that flattens a `Narrow` child, so D15 holds after physical rewrites. +- Out of scope: Aggregate kernels (NR3). Arithmetic, between, zip, float and cross-signedness casts + (NR4). The rules that move structural encodings above `Narrow`, and the dictionary-codes rule + (NR5). The switch of the existing narrowing call sites to `encode_values` (NR6). The + `vortex.narrow` wire form and editions (WR1, WR2). Compressor changes (WR3). Decimals (CD4). The + dictionary fill-code fix (FX3). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/mod.rs` | New, from `f7408481ce`. `Narrow`, `NarrowArray`, `NarrowSlots`, `try_new`, `validate_dtypes`, `initialize` | +| `vortex-array/src/arrays/narrow/vtable.rs` | New, from `f7408481ce`. Add `VTable::reduce` that flattens a `Narrow` child, and the lowering hook | +| `vortex-array/src/arrays/narrow/encoding.rs` | New, from `f7408481ce`. `NarrowArray::encode`, plus `encode_values` from `adc1e83b9d:vortex-array/src/arrays/narrow/encoding.rs:45-90` | +| `vortex-array/src/arrays/narrow/compare.rs` | New, from `f7408481ce`. Compare kernel | +| `vortex-array/src/arrays/narrow/rules.rs` | New, from `f7408481ce`. Slice, filter, take, mask, cast and fill-null rules | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | New | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | New. 9 `array_session()` calls and 1 import to replace. Add the flatten, `encode_values` and serialization tests | +| `vortex-array/src/arrays/narrow/tests/compute.rs` | New. 5 calls and 1 import to replace | +| `vortex-array/src/array/plugin.rs:113-158` | Add the lowering hook to `ArrayPlugin`, with a default of `Ok(None)`. The blanket impl for `VTable` (`:165-205`) forwards it | +| `vortex-array/src/serde.rs:169-190` | `ArraySerializationTree::try_new` asks the hook when `array_serialize` returns `None` | +| `vortex-array/src/arrays/mod.rs:85-91` | Add `pub mod narrow;` with `pub use narrow::Narrow;` and `pub use narrow::NarrowArray;` between `masked` and `null` | +| `vortex-array/src/arrays/mod.rs:144-145` | Add `narrow::initialize(session);` between `map` and `null` | +| `vortex-array/src/session/mod.rs:31-32` | Import `crate::arrays::Narrow` | +| `vortex-array/src/session/mod.rs:107-108` | `this.register(Narrow);` after `Masked` | + +Design notes: + +```rust +pub struct Narrow; +pub type NarrowArray = Array; +#[array_slots(Narrow)] +pub struct NarrowSlots { pub values: ArrayRef } + +impl NarrowArray { + pub fn try_new(values: ArrayRef, dtype: DType) -> VortexResult; + + /// Stores `array` as a `Narrow` over a child of the smallest integer type with the same + /// signedness, or returns it unchanged when no narrower type holds its values. + pub fn encode(array: PrimitiveArray, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Returns `array` at the smallest integer type with the same signedness, without a `Narrow`. + /// Floats return unchanged. + #[deprecated(note = "changes the dtype; a later change decides whether each caller needs a Narrow")] + pub fn encode_values(array: PrimitiveArray, ctx: &mut ExecutionCtx) + -> VortexResult; +} +pub(super) fn validate_dtypes(storage: &DType, logical: &DType) -> VortexResult<()>; +``` + +- `validate_dtypes` implements D22: both dtypes are integer primitives, the signedness is equal, + the child byte width is strictly smaller, and the nullability is equal. `VTable::validate` and + `try_new` both call it. +- `encode` and `encode_values` share one private, non-deprecated function, so `encode` itself needs + no `#[expect(deprecated)]`. That function takes `min_max` of the valid values, picks the smallest + type of the same signedness for the min and for the max, and takes the wider of the two. An + empty or all-null input gets the 1-byte type of the same signedness. A 1-byte input returns + unchanged. `encode` fails on a float input, and `encode_values` returns a float unchanged. An + `I64` column with values 0 to 200 gets an `I16` child (D22). +- `encode_values` has no caller in this PR. CD7 (the DecimalByteParts MSP) and NR6 (the existing + call sites) call it, each with `#[expect(deprecated, reason = "...")]` (D28). +- `try_new` flattens a `Narrow` child. Physical rewrites keep only the slot dtype + (`vortex-array/src/array/erased.rs:595-620`), so a rewrite can replace a child with a `Narrow` of + the same dtype. The new `reduce` returns `NarrowArray::try_new(child.values().clone(), dtype)` + when the child is a `Narrow`, and `None` otherwise. +- `execute` yields `values.cast(dtype)`, so the child can fuse decoding and widening. `scalar_at` + casts the child scalar. Validity comes from the child. CD4 changes `execute` for decimals, + because the decimal cast of D15 would loop. +- The serialization workaround (D26): + - Today `ArraySerializationTree::try_new` fails with "Array … does not support serialization" + when `array_serialize` returns `None` (`vortex-array/src/serde.rs:171-175`). + - A plugin cannot return the wire form of another encoding. `array_serialize` checks that the + returned ID is one of the plugin's `serialized_ids` (`vortex-array/src/session/mod.rs:144-151`). + - So the fallback lives in the serializer. A new `ArrayPlugin` method, for example + `lower_for_serialization(&self, array, session) -> VortexResult>`, returns + `Ok(None)` by default. When `array_serialize` returns `None`, `try_new` calls it, and it builds + the tree from the returned array. Without a lowered array, it fails as today. + - `Narrow` lowers itself by executing to its canonical array with + `session.create_execution_ctx()`. No other encoding implements the hook. + - The doc of the hook and a comment in the `Narrow` impl say that this is a workaround. It stays + until `vortex.narrow` is in `core` (LT1). WR1 also uses it when the selected editions do not + permit `vortex.narrow`. +- `deserialize` fails with "Narrow deserialization is not supported", because no wire form exists + yet. `ArraySession::default` registers `Narrow` like `PiecewiseSequence` + (`vortex-array/src/session/mod.rs:108`). No compressor produces a `Narrow` in this PR. +- Compare kernel: two `Narrow` operands compare at the wider child width. A constant that casts to + the child dtype compares against the child. A constant outside the child range gives a constant + answer with the child validity. `Eq` is false and `NotEq` is true. `Lt`, `Lte`, `Gt` and `Gte` + follow from whether the constant is below the range. A null constant gives all nulls. +- Cast rule in this PR: take an integer target of the same signedness. If it is no wider than the + child, the cast goes to the child with the same target. If it is wider, the rule returns a + `Narrow` over the child cast to the target nullability (D15 as amended). Other targets fall + back. NR4 widens this rule. +- Fill null declines when the fill value does not fit the child dtype, so canonical execution + handles it. + +Steps: + +1. Create `ct/narrow-in-memory` from `develop` in a new worktree. Do not reuse the `vortex3` + worktree or its branch. +2. Run `git cherry-pick f7408481ce`. It applies without conflicts. +3. Replace `array_session()` in `narrow/tests/encoding.rs` and `narrow/tests/compute.rs` with the + module `SESSION` pattern of the shared section. +4. Add `encode_values` and the shared private function, with their tests. +5. Add the lowering hook in `plugin.rs` and `serde.rs`, and the `Narrow` impl, with its tests. +6. Add `VTable::reduce` to `vortex-array/src/arrays/narrow/vtable.rs` and its test. +7. Keep the cherry-picked commit `feat(array): add in-memory Narrow integer encoding`, and add steps + 3 to 6 as new commits. Each commit has the line + `Signed-off-by: Connor Tsui ` and no co-author line. + +Tests: + +- From `f7408481ce`, in `narrow/tests/encoding.rs`: `test_signed_encoding` and + `test_unsigned_encoding` (`rstest` width boundaries), `test_null_payloads_and_empty`, + `test_reject_invalid_dtypes`, `test_validate_slots`, `test_flatten_and_scalar_dtype`, + `test_encoded_child_execution`, `test_smallest_width_unchanged`, `test_reject_float_encoding`, + `test_deserialization_unsupported`. +- `test_serialization_unsupported` becomes `test_serialization_writes_canonical`. Serialize a + nullable and a non-nullable `Narrow`, and decode the bytes. The result is a canonical primitive + array with the logical dtype and equal values, and its wire ID is the primitive ID. +- New, in `narrow/tests/encoding.rs`: + - `test_encode_values_keeps_signedness`, an `rstest`. `[0i64, 127]` gives `I8`, `[0i64, 200]` + gives `I16`, `[-1i32, 200]` gives `I16`, an empty `i32` gives `I8`, all-invalid `u32` gives + `U8`, and `[0u64, u64::MAX]` stays `U64`. Validity is kept in every case. + - `test_encode_values_preserves_floats`, from `adc1e83b9d`. + - `test_reduce_flattens_nested_narrow`. Build a `Narrow` with `try_from_parts` over a `Narrow` + child, and call `optimize()`. Check that the result is one `Narrow` over the inner child, with + the same values. +- In `vortex-array/src/serde.rs` tests: an encoding without a wire form and without the hook still + fails with "does not support serialization". +- From `f7408481ce`, in `narrow/tests/compute.rs`: `test_selection_and_fill_null`, + `test_cast_preserves_checked_nullability`, `test_fill_null_preserves_result_dtype`, + `test_arithmetic_uses_logical_width`, and `test_comparison`. The last one is an `rstest` over six + operators. It covers constants on both sides of the child range, both operand orders, and a + second `Narrow`. + +Benchmarks: None in this PR. NR2 adds them. No change on existing benchmarks is expected. Record +the `take_patches` results for the NR6 investigation. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array narrow` and `cargo nextest run -p vortex-array serde` +- `git grep -n array_session -- vortex-array/src/arrays/narrow` prints nothing. +- Serializing a `Narrow` writes its canonical array, and the read-back array has equal values. +- No file writer or compressor produces `vortex.narrow`. +- `encode_values` carries `#[deprecated]` and has no caller yet. + +Review focus: The constant shortcut of the compare kernel for both operand orders and null +constants. Checked nullability in the cast rule. Nullability of `take` and `mask` results, which +follow the child. The fill-null fallback. The new `reduce` rule. The lowering hook: it runs only +when an array has no wire form, and only `Narrow` implements it. Registration of the compare kernel +through `arrays::initialize`. + +Size: Large. The rewrite commit has about 550 lines of source and 435 lines of tests, mostly a +new module. `encode_values`, the hook and their tests come on top. + +Open questions: None. + +### NR2: Benchmarks for compute on integer `Narrow` + +- Goal: Add one benchmark file that runs each `Narrow` operation on three inputs. They are the + `Narrow`, the equivalent wide canonical array, and the `Narrow` decoded inside the timed region. + CodSpeed on NR3 to NR5 then shows each kernel against the fallback. +- Depends on: NR1. Soft: None. Blocks: NR3, NR4 and NR5 (their CodSpeed evidence). +- Decisions: D19 (benchmarks land before the changes that they cover), D15. +- Branch: `ct/narrow-benchmarks`, based on NR1. +- Scope: A new `vortex-array/benches/narrow.rs` and its `[[bench]]` entry. +- Out of scope: Kernel changes (NR3 to NR5). Workflow changes. Decimal benchmarks (CD1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/benches/narrow.rs` | New | +| `vortex-array/Cargo.toml:335-336` | Add `[[bench]]` with `name = "narrow"` and `harness = false` after the `probe` entry | + +Design notes: + +- Follow the conventions of `vortex-array/benches/compare.rs:22-76` and + `vortex-array/benches/filter_fixed_width.rs:101-126`: + - An SPDX header and a module doc. + - The `MiMalloc` global allocator. + - A `main` that forces `SESSION` and then calls `divan::main()`. + - A `SESSION` with `ArraySession`, `KernelSession`, `DTypeSession`, `ScalarFnSession`, + `AggregateFnSession`, `MemorySession` and `StatsSession`. + - Each result wrapped in `divan::black_box`. +- Do not tag any case with `#[vortex_bench_support::cpu_features]`. These kernels are dispatch + work, so the cases stay in simulation. +- The name `narrow` matches no prefix in `.github/workflows/codspeed.yml:92-104`, so the cases run + in the "builders" shard. No workflow change is needed. +- Data: `LEN = 8_192` rows, so the wide input holds 64 KiB, as `filter_fixed_width.rs:57-64` + recommends. Values are `i16` from `StdRng::seed_from_u64`, uniform in `-10_000..=10_000`. The + `Narrow` input has logical dtype non-nullable `I64` over that `i16` child. The wide input holds + the same values as `i64`. +- Build every input inside `with_inputs`, so setup is not timed. Build fresh arrays for each + iteration. `sum` caches its result on the array + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:63-65`), and the NR4 arithmetic kernel calls + `min_max` on both operands, so reused inputs hide work. + +```rust +#[derive(Clone, Copy, Debug)] +enum Input { + /// The canonical `i64` array with the same values. + Wide, + /// A `Narrow` with logical dtype `i64` over an `i16` child. + Narrow, + /// The `Narrow`, executed to a `PrimitiveArray` inside the timed region, then the operation. + Decoded, +} + +const INPUTS: &[Input] = &[Input::Wide, Input::Narrow, Input::Decoded]; + +#[divan::bench(args = INPUTS)] +fn compare_in_range(bencher: Bencher, input: Input) { /* ... */ } +``` + +Workloads. Each one ends with `execute::`, or returns the aggregate scalar: + +| Bench | Operation | PR whose kernel changes the `Narrow` case | +| --- | --- | --- | +| `compare_in_range` | `binary(Constant(0i64), Lt)` | NR1 (child compare) | +| `compare_out_of_range` | `binary(Constant(100_000i64), Lt)`, above `i16::MAX` | NR1 (constant answer) | +| `between_in_range` | `between(Constant(-1_000i64), Constant(1_000i64))`, both bounds non-strict | NR4 | +| `filter` | `filter(mask)`, random mask with density 0.5 | NR1 | +| `take` | `take(indices)`, `LEN / 4` random `u32` indices | NR1 | +| `slice` | `slice(LEN / 4..3 * LEN / 4)` | NR1 | +| `sum` | `sum(&array, ctx)` | NR3 | +| `min_max` | `min_max(&array, ctx, NumericalAggregateOpts::default())` | NR3 | +| `add_constant` | `binary(Constant(1i64), Add)` | NR4 | +| `add_narrow` | `Narrow + Narrow`, second operand from another seed | NR4 | +| `add_wide` | `Narrow + wide i64 array`, only `Narrow` and `Decoded` inputs | NR4 | +| `chunked` | `ChunkedArray` of four inputs of `LEN / 4` rows | None. It guards the chunk kernels, which push work into each `Narrow` chunk | +| `interleave` | `InterleaveArray` of two inputs of `LEN / 2` rows, `array_indices = i % 2` as `u8`, `row_indices = i / 2` as `u32` | None. It guards the interleave kernels | + +- `Decoded` for `chunked` and `interleave` decodes each input, then builds the parent. +- `Wide` for `add_narrow` adds two `i64` arrays. `add_wide` omits `Wide`, because it is the same + case. + +Steps: + +1. Create `ct/narrow-benchmarks` on top of NR1. +2. Add `vortex-array/benches/narrow.rs` and the `[[bench]]` entry. +3. Commit as `bench(array): add Narrow compute benchmarks`, with the sign-off line. + +Tests: None. The benchmark compiles under `cargo clippy --all-targets`. + +Benchmarks: This PR only adds cases, so CodSpeed reports them as new. Qualitative expectations for +later PRs: + +- NR3: `sum[Narrow]` and `min_max[Narrow]` move from about the `Decoded` level toward or below + `Wide`. +- NR4: `add_constant`, `add_narrow`, `add_wide` and `between_in_range` change for `Narrow`. The + direction for `add_wide` is not known, because the NR4 kernel scans the wide operand for bounds. +- `chunked` and `interleave` stay flat. NR5 changes no tree that they build. +- `Wide` and `Decoded` cases must stay flat in NR3 to NR5. A move there points at noise or at a + side effect of session registration. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo bench -p vortex-array --bench narrow` runs every case. +- Each case stays under 1 ms in CodSpeed simulation. + +Review focus: The three inputs hold the same logical values. Inputs are built outside the timed +closure and fresh for each iteration. `Decoded` decodes inside the timed closure. + +Size: Medium. One file with about 13 workloads. + +Open questions: None. + +### NR3: Aggregates forward to the child + +- Goal: Register one fallback aggregate kernel for `Narrow`. It runs order-preserving and + sum-like aggregates on the child, and it casts the partial state to the state dtype of the + parent. +- Depends on: NR1. Soft: NR2. Blocks: NR4 (stack), CD4. +- Decisions: D15. +- Branch: `ct/narrow-aggregate-kernels`, based on NR2. +- Scope: Cherry-pick `8d548646ee`. Replace `array_session()` in its tests. +- Out of scope: Decimal aggregates (CD4). Pushdown of sum, min and max into the MSP (LT3). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/aggregates.rs` | New, from `8d548646ee` | +| `vortex-array/src/arrays/narrow/mod.rs` | `mod aggregates;` and `pub(crate) use aggregates::register_aggregate_kernels;`, module doc names aggregates | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod aggregates;` | +| `vortex-array/src/arrays/narrow/tests/aggregates.rs` | New. 2 calls and 1 import to replace | +| `vortex-array/src/aggregate_fn/session.rs:51-52` | `use crate::arrays::narrow;` | +| `vortex-array/src/aggregate_fn/session.rs:135-136` | `narrow::register_aggregate_kernels(&this);` after the `Dict` kernels | +| `encodings/fastlanes/tests/narrow.rs` | New integration test with a `BitPacked` child. Replace `vortex_array::array_session()` with the closure pattern | + +Design notes: + +```rust +pub(crate) fn register_aggregate_kernels(session: &AggregateFnSession); +struct NarrowAggregateKernel; // impl DynAggregateKernel +``` + +- The kernel registers with aggregate id `None`, so it is the fallback for `Narrow`. A kernel for an + exact aggregate wins over it (`vortex-array/src/aggregate_fn/session.rs:199-224`). +- Supported: `MinMax`, `Min`, `Max`, `Sum`, `SumV2`, `IsConstant`, `IsSorted`, `First`, `Last`, + `Count`, `NullCount`, `AllNull`, `AllNonNull`. Every other aggregate returns `None` and uses the + default path. `UncompressedSizeInBytes` must stay on the default path, because it depends on the + logical width. +- Algorithm: get the state dtype of the parent. Accumulate the child with an accumulator for the + child dtype. Flush, and cast the partial to the parent state dtype. +- Integer correctness: exact widening keeps values and order. The integer `Sum` state is `I64` for + every signed input and `U64` for every unsigned input + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:114-122`). So the child sum overflows exactly + when the parent sum does. +- Integer-only assumption: a decimal sum state depends on the input precision + (`sum/mod.rs:81-86`). Add a comment at the accumulator line that says this kernel is correct for + integers only, and that decimals need the parent state precision. CD4 owns that change. + +Steps: + +1. Run `git cherry-pick 8d548646ee` on the NR2 branch. +2. Replace `array_session()` in `narrow/tests/aggregates.rs` with the module `SESSION`. +3. Replace the session in `encodings/fastlanes/tests/narrow.rs` with the closure pattern. Remove + the redundant `#![cfg(test)]` at the top of that integration test. +4. Add the integer-only comment in `aggregates.rs`. +5. Amend the commit with the sign-off line. + +Tests: + +- `test_aggregate_partial_states` (`rstest`): six batch shapes (signed, unsigned, descending + boundary, equal boundary, empty, all null) times 14 bound aggregates. It checks the kernel + partial, the running partial and the final value against the canonical input. +- `test_unsupported_aggregate_keeps_logical_width`: `UncompressedSizeInBytes` returns `None` from + the kernel and reports the logical width. +- `encodings/fastlanes/tests/narrow.rs::test_narrow_bitpacked_child`: filter, take, compare, sum and + execution on a `BitPacked` child. + +Benchmarks: NR2 `sum` and `min_max` with the `Narrow` input are expected to improve. All other +cases stay flat. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-fastlanes --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-fastlanes` +- `cargo nextest run -p vortex-array narrow` +- `cargo nextest run -p vortex-fastlanes --test narrow` + +Review focus: The supported list. Each listed aggregate must give the same result on the child as +on the widened values. The cast of partial states that hold extrema or boundary values. The +fallback registration does not shadow kernels for exact aggregates. + +Size: Small. About 100 lines of source and 235 lines of tests. + +Open questions: None. + +### NR4: Arithmetic and scalar operations forward to the child + +- Goal: Execute checked add, subtract, multiply and divide at the smallest width that holds both + operands and the result interval. Push `between`, `zip`, `fill_null` and more casts through + `Narrow`. +- Depends on: NR3 (stack). Soft: NR2. Blocks: NR5. +- Decisions: D15, D22. +- Branch: `ct/narrow-arithmetic-kernels`, based on NR3. +- Scope: Cherry-pick `74aa17644e`. Replace `array_session()` in its tests. Fix one module doc. +- Out of scope: Structural rules and the dictionary-codes rule (NR5). Decimal arithmetic (D20). + Decimal casts (CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/numeric.rs` | New, from `74aa17644e`. Execute-parent kernel for `Binary` | +| `vortex-array/src/arrays/narrow/rules.rs` | New cast rule, fill null through `integer_storage_type`, `ZipReduce` and `ZipFalseReduce`, `BetweenReduce` | +| `vortex-array/src/arrays/narrow/encoding.rs` | `integer_types` helper, used by `encode` and `numeric.rs` | +| `vortex-array/src/arrays/narrow/mod.rs` | `mod numeric;`, `initialize` registers the numeric kernel | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod numeric;` and `mod pushdown;` | +| `vortex-array/src/arrays/narrow/tests/numeric.rs` | New. 9 calls and 1 import to replace | +| `vortex-array/src/arrays/narrow/tests/pushdown.rs` | New. 7 calls and 1 import to replace. Its module doc names concatenation and interleave, which these tests do not cover. Remove those words | + +Design notes: + +- Numeric kernel: for `Add`, `Sub`, `Mul` and `Div`, unwrap `Narrow` operands to their children. + Take `min_max` bounds of both operands as `i128`. Compute the result interval. Pick the smallest + type of the same signedness and no wider than the logical type that holds both operands and the + interval. Cast both operands to it, run the checked `execute_numeric`, and wrap the result with + `wrap_values`. When no narrower type holds the interval, use the logical type, so the checked + kernel reports a real overflow exactly as canonical execution does. An empty parent gives an + empty canonical array. An all-null operand gives a null constant. +- Cast rule (D15): a float target goes to the child with the same target. An integer target + first picks a storage type. It is the child type with the signedness of the target. For an + unsigned child and a signed target, it is the next wider signed type. A target no wider than the + storage type goes to the child with the same target. A wider target returns a `Narrow` over the + child cast to the storage type. Non-primitive targets fall back. +- Fill null: `integer_storage_type` picks the child type, or the smallest type that holds the fill + value, without reading values. The child is cast to it before the fill. A fill value that needs + the logical width gives a plain array. +- Zip: both branches. The other branch is a `Narrow` or a constant. Otherwise fall back. +- Between: when both bounds are constants that fit the child, run `between` on the child. + Otherwise combine two compares with `And`, so each compare uses the NR1 kernel. + +Steps: + +1. Run `git cherry-pick 74aa17644e` on the NR3 branch. +2. Replace `array_session()` in `narrow/tests/numeric.rs` and `narrow/tests/pushdown.rs`. +3. Fix the module doc of `narrow/tests/pushdown.rs`. +4. Amend the commit with the sign-off line. + +Tests: + +- `narrow/tests/numeric.rs`: + - `test_signed_arithmetic`: `rstest` over width growth, skipped widths, full width, division at + `i8::MIN` and across zero, and an overestimated interval. + - `test_signed_domain`: every `i8` value against small divisors, for all four operators. + - `test_logical_errors`: overflow and division by zero match the canonical errors. + - `test_unsigned_arithmetic`, `test_unsigned_underflow`, `test_nulls_and_empty`, + `test_encoded_child_and_operand_order`, `test_overflow_at_each_logical_width`, + `test_logical_overflow_behind_null`. +- `narrow/tests/pushdown.rs`: + - `test_between_bounds`: `rstest` over bounds inside and outside the child range, and both + strictness values. + - `test_cast_child`: unsigned to signed, signed to unsigned, and float. + - `test_fill_width`, `test_zip_width`, `test_between_nullable_bounds`, + `test_zip_false_branch_and_null_mask`, `test_signedness_cast_checks_values_and_nulls`. + +Benchmarks: NR2 `add_constant`, `add_narrow`, `add_wide` and `between_in_range` change for the +`Narrow` input. If `add_wide[Narrow]` is slower than `add_wide[Decoded]`, report it. A possible +fix is to decline operands that are neither `Narrow` nor constant. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array narrow` +- Every arithmetic result equals the canonical result, including errors and nulls. + +Review focus: Interval arithmetic for division across zero and for `i64::MIN / -1`. Overflow +checks stay with the checked kernel at the logical width. The cast rule for unsigned children and +signed targets. The `min_max` pass on a wide operand. Null payloads outside the chosen width. + +Size: Large. About 450 lines of source and 425 lines of tests. + +Open questions: None. + +### NR5: Structural encodings move above `Narrow`, and dictionary codes use the child + +- Goal: Keep generic structural encodings above `Narrow` (D25). Reduce rules turn a `Narrow` over + `Dict`, `RunEnd`, `Sparse`, `Constant`, `Masked`, `Chunked` or `Interleave` into that encoding + over `Narrow` children. Dictionary codes that are a `Narrow` use its child directly. +- Depends on: NR4 and FX3. Soft: NR2. Blocks: WR3. +- Decisions: D25, D15, D23. +- Branch: `ct/narrow-structural-rules`, based on NR4, after FX3 is on `develop`. +- Scope: Seven pull-up rules, the dictionary-codes rule `TakeIndicesReduce` from `af4b663059`, and + the `Narrow` regression test from `7e2e5e4d7a`. +- Out of scope: The chunked and interleave merge rules of `af4b663059` (`narrow/merge.rs`). They + put `Narrow` above `Chunked` and `Interleave`, so they would loop against the new rules. The same + pull-up rule for every other encoding, which a follow-up issue tracks. Decimal tests (CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/rules.rs` | The pull-up rules for `Dict`, `Constant`, `Masked`, `Chunked` and `Interleave`. `TakeIndicesReduce` for `Dict` codes, from `af4b663059` | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod structural;` | +| `vortex-array/src/arrays/narrow/tests/structural.rs` | New. The pull-up tests, `test_narrowed_take_indices` from `af4b663059`, and `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`, with the module `SESSION` | +| `encodings/runend/src/` | A parent-reduce rule that moves `RunEnd` above a `Narrow` parent, with a test | +| `encodings/sparse/src/` | A parent-reduce rule that moves `Sparse` above a `Narrow` parent, with a test | + +Design notes: + +| Input | Output | +| --- | --- | +| `Narrow(Dict(codes, values))` | `Dict(codes, Narrow(values))` | +| `Narrow(RunEnd(ends, values))` | `RunEnd(ends, Narrow(values))` | +| `Narrow(Sparse(indices, values, fill))` | `Sparse(indices, Narrow(values), fill)`, with `fill` cast to the parent dtype | +| `Narrow(Constant(value))` | `Constant(value)`, with `value` cast to the parent dtype | +| `Narrow(Masked(child, validity))` | `Masked(Narrow(child), validity)` | +| `Narrow(Chunked(c1, ..., cn))` | `Chunked(Narrow(c1), ..., Narrow(cn))` | +| `Narrow(Interleave(selectors, v1, ..., vn))` | `Interleave(selectors, Narrow(v1), ..., Narrow(vn))` | + +- Every output is valid. Each new `Narrow` wraps an array of the child dtype, so `validate_dtypes` + holds. A cast of a scalar from the child dtype to the parent dtype is exact. +- Where the rules live. `Dict`, `Constant`, `Masked`, `Chunked` and `Interleave` are in + `vortex-array`, so their rules are `Narrow` reduce rules that match the child encoding. `RunEnd` + and `Sparse` are in their own crates, which depend on `vortex-array`. So their rules are + parent-reduce rules on `RunEnd` and `Sparse` with `Narrow` as the parent. Whether the rule + registry supports both shapes as described is Not verified. Read how `vortex-array/src/arrays/` + registers parent rules before the first commit. +- No rule moves `Narrow` above a structural encoding. A rule in the other direction would loop. +- Dictionary codes: a `Narrow` codes child is replaced by its child, with the same `DictData`. + Narrowed codes can make an unreferenced fill index exceed the codes range. FX3 makes the + dictionary `fill_null` kernel decline in that case, so FX3 must be on `develop` first. +- The rules never read the child dtype, so they also hold for decimal `Narrow`. CD4 adds the + decimal tests. +- WR3 can wrap a column root in `Narrow`, because these rules then move a structural encoding back + to the top. The dict-layout probe and the Arrow and DuckDB exporters then see `Dict` and `RunEnd` + at the root, as today. + +Steps: + +1. Confirm that FX3 is on `develop`. Rebase NR1 to NR4 if needed. +2. Add the five rules in `vortex-array`, with their tests. +3. Add the `RunEnd` and `Sparse` rules in their crates, with one test each. +4. Port `TakeIndicesReduce` and `test_narrowed_take_indices` from `af4b663059`. +5. Port `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`. + +Tests: + +- `test_pull_up_structural`, an `rstest` over the five `vortex-array` shapes. Each result has the + structural encoding at the root and a `Narrow` below it. Its values equal the input, checked with + `assert_arrays_eq!`. +- `test_pull_up_nullable`: nullable children, a null constant, and `Masked` with nulls. +- `test_pull_up_fixed_point`: `optimize()` on a `Chunked` of `Narrow` chunks returns the same tree. + So no rule pair loops. +- In `encodings/runend` and `encodings/sparse`: `Narrow` over the encoding optimizes to the encoding + over `Narrow` values, with equal values. +- `test_narrowed_take_indices`, from `af4b663059`. +- `test_fill_null_after_narrowing_take_indices`, from `7e2e5e4d7a`. It fails without FX3. + +Benchmarks: NR2 `chunked[Narrow]` and `interleave[Narrow]` keep their baseline. Their `Narrow` +input already is a `Chunked` or `Interleave` of `Narrow` chunks, so the new rules change nothing +for them. Kernels push work into each chunk. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-runend -p vortex-sparse --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-runend -p vortex-sparse` +- `cargo nextest run -p vortex-array narrow`, `cargo nextest run -p vortex-runend` and + `cargo nextest run -p vortex-sparse` +- After `optimize()`, no `Narrow` sits directly above one of the seven encodings. + +Review focus: The fill and constant casts to the parent dtype. Nullability of each new `Narrow`. +No rule pair loops. The dictionary rule copies `DictData`. + +Size: Medium. Seven small rules in three crates, and their tests. + +Open questions: None. + +### NR6: The existing narrowing call sites move to `encode_values` + +- Goal: Replace `PrimitiveArrayExt::narrow` with the deprecated `NarrowArray::encode_values` of NR1 + at every call site. Each call site carries `#[expect(deprecated)]` with a reason, so every + narrowing that changes a dtype is marked for a later decision (D28). Internal children keep + their signedness. +- Depends on: NR1. Soft: NR5 (merge after the stack, D35). Blocks: None. +- Decisions: D28, D22, D23, D11. +- Branch: `ct/narrow-encode-values`, based on `develop` after NR1 merges. +- Scope: The call sites of part 1 of `adc1e83b9d`, plus the matching lines of `0d94e44a91`. The + removal of `PrimitiveArrayExt::narrow`, with its tests ported. Regenerated golden snapshots. A + check that the latest release reads files that this PR writes. +- Out of scope: `encode_values` itself (NR1). Part 2 of `adc1e83b9d` (WR1 to WR3). Which arrays a + compressor wraps in a `Narrow` (deferred, D27). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | Port the narrowing tests of `PrimitiveArrayExt::narrow` to `encode_values` | +| `vortex-array/src/arrays/primitive/array/mod.rs:157-241` | Remove `PrimitiveArrayExt::narrow` and imports that become unused | +| `vortex-array/src/arrays/primitive/array/cast.rs:38-213` | Remove the narrowing tests, which move to `narrow/tests/encoding.rs` | +| `vortex-array/src/builders/dict/mod.rs:19,87` | Dictionary codes | +| `encodings/runend/src/compress.rs:17,91` | Run ends | +| `encodings/fastlanes/src/rle/array/mod.rs:128,442` | Test call site | +| `vortex-btrblocks/src/schemes/patches.rs:9,21,42` | Patch indices and chunk offsets | +| `vortex-btrblocks/src/schemes/binary/varbin.rs:21,87` | VarBin offsets | +| `vortex-btrblocks/src/schemes/float/sparse.rs:13,101` | Sparse indices | +| `vortex-btrblocks/src/schemes/integer/rle.rs:13,65,78` | RLE indices and offsets | +| `vortex-btrblocks/src/schemes/integer/sparse.rs:15,192` | Sparse indices | +| `vortex-btrblocks/src/schemes/string/fsst.rs:17,86,100` | FSST lengths and code offsets | +| `vortex-btrblocks/src/schemes/string/onpair.rs:13,145` | OnPair children | +| `vortex-btrblocks/src/schemes/string/sparse.rs:13,109` | Sparse indices | +| `vortex-btrblocks/src/schemes/temporal.rs:16,95,105,115` | DateTimeParts days, seconds, subseconds | +| `vortex-compressor/src/builtins/dict/binary.rs:20,121` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/float.rs:22,139` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/integer.rs:22,131` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/string.rs:20,121` | Dictionary codes | +| `vortex-compressor/src/compressor/structural.rs:19,54,87,99` | List offsets, ListView offsets and sizes | +| `vortex-btrblocks/tests/narrow.rs` | New. Only `test_internal_signed_buffers_roundtrip` from `0d94e44a91`, with the core edition and the module `SESSION` | +| `vortex-btrblocks/tests/snapshots/*.snap` | Regenerated | + +The first number in each call-site row is the `PrimitiveArrayExt` import. Remove an import only +when it becomes unused: `vortex-btrblocks/src/schemes/integer/bitpacking.rs:16`, +`float/alp.rs:18` and `float/alprd.rs:16` import the trait for other methods. + +`07fafc2a5b` pins dictionary compression in `test_array_stream_no_double_dict_encode`. `develop` +reworked that test with far-apart values (`vortex-file/src/tests.rs:1599-1623`). Leave +`07fafc2a5b` out unless that test fails with NR6. + +Design notes: + +- Each call site follows this shape. The reason names the open question, so a search finds every + site later: + + ```rust + #[expect( + deprecated, + reason = "TODO(connor): decide whether this child narrows its dtype or becomes a Narrow" + )] + let offsets = NarrowArray::encode_values(offsets, ctx)?; + ``` + +- What "internal buffers keep their signedness" changes in file contents. The old + `PrimitiveArrayExt::narrow` (`vortex-array/src/arrays/primitive/array/mod.rs:157-241`) picks + `U8`, `U16` or `U32` whenever no value is negative (`:189`), whatever the input signedness. It + also returns a zeroed `u8` buffer for an empty or all-null input (`:163-167`). `encode_values` + keeps the signedness: + - A signed child with only non-negative values gets a signed type. It is the same width when the + max fits the signed type. For example, `u16` becomes `i16` for list offsets below 32,768. It is + the next width when the max is between the signed and the unsigned limit. For example, values + 128 to 255 become `i16` instead of `u8`. + - A 1-byte signed child stays `i8` instead of becoming `u8`. + - An empty or all-null signed child becomes `i8` instead of `u8`. + - Unsigned children, children with negative values, and `u64` values above `i64::MAX` do not + change. + - Signed sources in practice: list offsets, ListView offsets and sizes, VarBin offsets, FSST + lengths and code offsets, OnPair children, and the DateTimeParts parts. Unsigned sources do not + change: patch and sparse indices, RLE indices and offsets, dictionary codes and run ends. + - The encoding IDs and metadata formats do not change. The dtype of these children changes in new + files, and an uncompressed child can grow by one width. +- Readers on `develop` accept signed children. List, ListView, VarBin, FSST, DateTimeParts and + OnPair require only `is_int()` for these children + (`vortex-array/src/arrays/list/array.rs:211-215`, `listview/array.rs:239-246`, + `varbin/array.rs:238-239`, `encodings/fsst/src/array.rs:842-848`, + `encodings/datetime-parts/src/array.rs:318-328`, `encodings/onpair/src/array.rs:419-426`). + ListView reinterprets signed offsets and sizes as unsigned before it uses them + (`listview/array.rs:270-280`). Whether earlier releases behave the same is Not verified, so an + acceptance check runs the latest release on the new files. +- The closed #10333 changed five snapshots this way. List offsets went from `u16` to `i16`. FSST + lengths and code offsets went from `u8` to `i8` and from `u32` to `i32`. Their byte counts did + not change. `develop` now writes `fastlanes.bitpacked.v2` and has more snapshot variants, so the + snapshots must be regenerated, not copied. + +Steps: + +1. Create `ct/narrow-encode-values` from `develop` after NR1 merges. +2. Replace every call site in the table, each with the `#[expect(deprecated)]` shape above. Remove + imports that become unused. +3. Remove `PrimitiveArrayExt::narrow` and port its tests. +4. Add `vortex-btrblocks/tests/narrow.rs` with the round-trip test only. +5. Regenerate the golden snapshots with + `INSTA_UPDATE=always cargo test -p vortex-btrblocks --all-features --test golden`. +6. Commit as `feat(compress): keep integer signedness when narrowing internal children`, with the + sign-off line. The PR body must say that `PrimitiveArrayExt::narrow` is removed from the public + API. + +Tests: + +- Port each case of `vortex-array/src/arrays/primitive/array/cast.rs:58-212` to an `rstest` on + `encode_values` in `narrow/tests/encoding.rs`. Expectations change where signedness changes: + - `[0i64, 127]` gives `I8` (was `U8`). + - `[Some(0i32), None, Some(127)]` gives nullable `I8` (was `U8`). + - An empty `i32` or `i64` gives `I8` (was `U8`). + - Unchanged: all-invalid `u32` gives `U8`, `[-1i32, 200]` gives `I16`, `[-100i16, 0, 100]` gives + `I8`, `[0u64, u64::MAX]` stays `U64`, and floats stay unchanged. + - Validity is kept in every case. +- `vortex-btrblocks/tests/narrow.rs::test_internal_signed_buffers_roundtrip`: lists and strings + compress with the compact preset, serialize, decode and match the input, with no `vortex.narrow` + in the array context. +- Regenerated `vortex-btrblocks/tests/snapshots/*.snap`. Only signed internal children change. + +Benchmarks: Expect changes in the btrblocks `compress` and `compress_listview` benchmarks, because +child dtypes change. The closed #10333 reported `take_map` regressions of 22% and 11% in Simulation +mode. The plan to investigate them: + +- `take_map` is in `vortex-array/benches/take_patches.rs:110-129`. It builds `Patches` with `u64` + indices and `u64` values directly (`:131-140`), draws random `u64` take indices, and times + `Patches::take_map` (`vortex-array/src/patches.rs:919-970`). No narrowing code runs. +- Step 1: compare the `take_map` cases on NR1, NR3 and NR4. These PRs add session registrations + and kernels but no narrowing code. A move there points at registry growth or code layout. +- Step 2: compare `take_map` on NR6 with its base. NR6 changes none of the code that `take_map` + runs, so a move there also points at code layout. +- Step 3: for a case that moves, open the CodSpeed flame graph diff. Find the symbols whose cost + changed, for example `hashbrown` inserts and lookups, or allocator calls. +- Step 4: check the `take_map` history on `develop` for variance. `HashMap` is `hashbrown` with + `DefaultHashBuilder` (`vortex-utils/src/aliases/hash_map.rs:5-9`). If that builder seeds per + process, the hash layout differs between runs. Not verified. +- Report the result in the NR6 PR. Do not change `take_map` in NR6. + +Acceptance checks: + +- The commands below pass: + + ```bash + cargo clippy -p vortex-array -p vortex-btrblocks -p vortex-compressor -p vortex-runend \ + -p vortex-fastlanes --all-targets --all-features -- -D warnings + cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-btrblocks -p vortex-compressor \ + -p vortex-runend -p vortex-fastlanes + cargo nextest run -p vortex-array narrow + cargo nextest run -p vortex-btrblocks --all-features + cargo nextest run -p vortex-compressor -p vortex-runend -p vortex-fastlanes + ``` + +- `git grep -n "\.narrow(" -- '*.rs'` finds no call of `PrimitiveArrayExt::narrow`. +- Every call of `encode_values` carries `#[expect(deprecated)]` with the `TODO(connor)` reason. +- Snapshot diffs show only dtype labels of signed internal children, plus byte counts where a + value range needs the next signed width. +- The latest release reads files that this PR writes: + 1. With this PR, run `cargo run -p vortex-compat --release -- generate --output `. + 2. In a separate worktree at the latest release tag, build `vortex-compat` and run its `check` + on ``. Read the modes in `vortex-test/compat-gen/src/check.rs:21-30` first, and use the + one that skips fixtures that only one side has. `compat.py --git-ref` does not build from the + ref (`vortex-test/compat-gen/scripts/compat.py:922-925`), so build the binary by hand. + +Review focus: Every byte-count change in the snapshot diff. Parents that receive a signed child +where they received an unsigned one before. The public API removal. Null payloads in the all-null +case, where the old code wrote zeros and the new code casts. + +Size: Medium. About 20 call sites, ported tests and snapshots. + +Open questions: None. + +## Notes from planning + +1. The claim that "the only integer-only check is the private `validate_dtypes`" holds only for + explicit dtype checks. Opening it for decimals without other changes gives a panic in the + compare kernel for two `Narrow` operands, wrong compare and between results for negative + constants below the child range, and early overflow in decimal `Sum` and `SumV2`. The table + "Integer-only sites" lists every site, and CD4 owns each fix. +2. `Narrow::execute` yields `values.cast(dtype)`. The decimal cast of D15 returns a `Narrow` for a + widening cast, so a decimal `Narrow` would execute to itself. CD4 widens decimals explicitly. + Any later integer cast rule that returns a `Narrow` has the same hazard. +3. The D15 text said that a cast of a `Narrow` passes to the child with the same target. For a + wider integer target, the rewrite returns a new `Narrow` over the child. The values are equal + and the data stays narrow. D15 now says this. +4. The rewrite flattens nested `Narrow` arrays only in `try_new`. Physical rewrites check only the + slot dtype (`vortex-array/src/array/erased.rs:595-620`), so NR1 adds a `VTable::reduce` flatten + rule. +5. The commits hold 41 `array_session()` calls and 6 `use` lines. NR1 to NR4 and the kept part of + NR5 hold 39 calls and 6 `use` lines, and FX3 holds 2 calls. +6. The report cites `narrow/mod.rs:81-84` for the width rule. That matches the tip `af4b663059`. + In `f7408481ce` the same lines are `:76-79`. +7. `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and `ct/narrow-compression` + (#10333) still exist on `origin`. The new branch names avoid all three. +8. `Narrow` is registered in `ArraySession::default`, while the in-memory `Filter`, `Slice` and + `Interleave` are not. `PiecewiseSequence` is registered and also has no wire form, so the + rewrite follows a precedent. +9. Part 2 of #10333 wraps every integer primitive that reaches `compress_canonical` in a `Narrow`, + including internal children. It also declares a preview edition with only `vortex.narrow`. WR2 + and WR3 must not reuse either as they are. diff --git a/research/decimal/agents/briefs/03-canonical-decimal.md b/research/decimal/agents/briefs/03-canonical-decimal.md new file mode 100644 index 00000000000..71499d2256b --- /dev/null +++ b/research/decimal/agents/briefs/03-canonical-decimal.md @@ -0,0 +1,1839 @@ + + +# Canonical decimal: CD1 to CD8c + +These briefs cover issue #10701: one canonical width for each decimal dtype, and decimal +`Narrow` in memory. The strict construction step is split into CD8a, CD8b and CD8c. + +- A `file:line` without a commit points at the `develop` checkout at `b70206818f`. +- A reference with a commit, for example `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73`, + points at the local integer `Narrow` rewrite in `/Users/connor/spiral/vortex-data/vortex3`. The + NR PRs rebase that code, so those line numbers will move. +- In the rewrite, commit `f7408481ce` holds the encoding, `validate_dtypes`, the compare kernel and + the cast, fill-null, filter, mask, slice and take rules (NR1). Commit `8d548646ee` holds the + aggregate kernel (NR3). Commit `74aa17644e` holds arithmetic, between, zip and the wider + fill-null and cast rules (NR4). Commit `af4b663059` holds merges and indices (NR5). +- Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author + line. +- Verification commands are the user's choice. The acceptance checks list them for the user. + +Two terms recur. The canonical width of a dtype is `DecimalDType::values_type()`, which CD3 adds +(D12). The stored width of an array is `DecimalArray::values_type()`. The two are equal for every +array only after CD8a. + +## Shared helpers + +Each helper has one owner. Later PRs use it and do not add their own. + +| Helper | Owner | Users | Contract | +| --- | --- | --- | --- | +| `DecimalDType::values_type()` | CD3 | Every later PR | The canonical width of the precision (D12) | +| `copy_to_canonical_width` | CD3 | CD3 imports, CD4 execution, CD5 decoding, CD6 old nodes | Public in `vortex_array::arrays::decimal`. A narrowing copy writes zeros in null slots and fails on a valid value outside the target precision (D13, D21). A widening copy checks nothing | +| `check_precision` | CD3 | The zero-copy imports of CD3, `try_new` in CD8b | Public. Fails on a valid value outside the precision. It takes a `Mask`, so it needs no `ExecutionCtx` | +| The best child dtype for a range | CD4 | `NarrowArray::encode_decimal`, CD6, the `Narrow` scheme of WR3 | Public. From the min and max of the valid values, or no range, and a parent `DecimalDType`, it returns the dtype with the same scale and the smallest precision that holds the range and the scale (D10) | +| `NarrowArray::encode_decimal` | CD4 | WR3 | A `Narrow` over the child at the best precision, or the input when that precision has the canonical width of the input | +| `NarrowArray::encode_values` | NR1 | CD7 (the DecimalByteParts MSP), NR6 | Deprecated (D28). The narrowed child with the same signedness, without a `Narrow` | + +No decimal twin of `encode_values` exists. A caller that needs the child calls `encode_decimal` and +takes the values child of the `Narrow`. + +### CD1: Decimal benchmarks before the in-memory change + +- Goal: Add benchmarks for the decimal code that CD4, CD5 and CD7 change. CodSpeed then has a + base before those PRs land (D19). No library behavior changes. +- Depends on: None. Soft: None. Blocks: CD4, CD5, CD7. +- Decisions: D19 (benchmarks land first), D16 (DecimalByteParts decoding, changed by CD5), D17 + (the DecimalByteParts scheme, changed by CD7), D15 (the widening cast, changed by CD4). +- Branch: `ct/decimal-benchmarks`, based on `develop`. +- Scope: New benchmarks on DecimalByteParts (DBP) arrays with a narrow MSP and on a gap column. + Further benchmarks cover a compressed DBP tree, the cast that grows the canonical width, and the + Arrow and DuckDB exports. One `#[doc(hidden)]` re-export in `vortex-duckdb` lets a bench reach + the exporter. +- Out of scope: Benchmarks on a decimal `Narrow` (CD4, because `Narrow` does not exist on + `develop`). Changes to the CodSpeed shards in `.github/workflows/codspeed.yml` (decided: no CI + change). Changes to existing benchmark inputs, including the non-canonical input in `to_arrow.rs` + (CD8a). + +Existing decimal benchmarks at `b70206818f`: + +| Bench | Decimal cases | In CodSpeed | +| --- | --- | --- | +| `vortex-array/benches/compare.rs:166-171`, `:270-276` | `compare_decimal`: two `i128` arrays at `Decimal(38, 2)`, array against array | Yes, simulation, compute shard | +| `vortex-array/benches/binary_ops.rs:286-331`, `:394-403` | add, mul and div on `i64` at `Decimal(18, 2)` and nullable `i128` at `Decimal(38, 2)` | Yes, compute shard | +| `vortex-array/benches/cast_decimal.rs:96-133` | `in_place_*`: `Decimal(10, 2)` to `Decimal(18, 2)`, both `i64`. `copy_*`: `Decimal(18, 2)` to `Decimal(10, 2)` | Yes, compute shard | +| `vortex-array/benches/cast_primitive.rs:112-160` | integer to `Decimal(38, 2)`, `Decimal(9, 2)` and `Decimal(9, 0)` | Yes, compute shard | +| `vortex-array/benches/filter_fixed_width.rs:144-158` | filter on `i128` at `Decimal(19, 0)` and `i256` at `Decimal(39, 0)` | Yes, selection shard | +| `vortex-array/benches/expr/case_when_bench.rs:67-84` | `case_when_decimal_product` on `i64` at `Decimal(15, 2)` | Yes, compute shard (`expr_case_when`) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:35-101` | assembly kernels for `i128` and `i256`, including an `i8` MSP | Walltime CPU-feature legs only (`#[cpu_features]`) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:102-194` | `dbp_assemble`, `dbp_assemble_narrowed` on arrays | No, `#[cfg(not(codspeed))]` "due to noise" | +| `encodings/decimal-byte-parts/benches/dbp_split.rs` | split kernels, split on arrays | Kernels on CPU-feature legs, arrays not in CodSpeed | +| `vortex-arrow/benches/to_arrow.rs:100-122` | one `Decimal(19, 2)` column inside a struct export | No, `vortex-arrow` is in no shard (`codspeed.yml:52-61`) | + +No DuckDB decimal export bench exists. The only DuckDB bench is +`vortex-duckdb/benches/bool_export.rs`, a bit-unpack microbenchmark that needs no live DuckDB +vector. No benchmark covers sum, min or max on decimals, or decoding of a DBP array with a narrow +MSP in CodSpeed. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/benches/dbp_compute.rs` | New. Decode, compare, filter, sum and min-max on DBP arrays | +| `encodings/decimal-byte-parts/Cargo.toml` | Add `[[bench]] name = "dbp_compute"`, next to the entries at `:36-42` | +| `vortex/benches/common_encoding_tree_throughput.rs` | Add a DBP tree with a bit-packed `i32` MSP to the `decompress` arguments at `:454-466` | +| `vortex-array/benches/cast_decimal.rs` | Add casts that grow the canonical width, `i32` input | +| `vortex-arrow/benches/to_arrow.rs` | Add a `decimal_export` bench with its own cases. Leave `array()` at `:100-122` as it is | +| `vortex-arrow/Cargo.toml` | Add `vortex-decimal-byte-parts` to `[dev-dependencies]` at `:39-46` | +| `vortex-duckdb/src/lib.rs` | Add `#[doc(hidden)] pub mod _benchmarking` that re-exports `ArrayExporter` and `ConversionCache` from the private `exporter` module (`:29`) | +| `vortex-duckdb/benches/decimal_export.rs` | New. Exports decimal columns into a DuckDB `DataChunk` | +| `vortex-duckdb/Cargo.toml` | Add `[[bench]] name = "decimal_export"`, next to `:65-67` | +| `vortex-btrblocks/benches/compress.rs` | Optional. A decimal compress case inside the existing `#[cfg(not(codspeed))]` module | + +Design notes: + +- DBP inputs for `dbp_compute.rs`, built in memory with `DecimalByteParts::try_new` + (`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:132-137`). One `Display` enum + names the cases for `divan` arguments: + + | Case | Dtype | MSP | Values | Why | + | --- | --- | --- | --- | --- | + | `msp_i32_p18` | `Decimal(18, 2)` | `i32` | seeded, within plus or minus 999,999,999 | The common old compressor shape. In the final state its child is `Decimal(9, 2)` | + | `msp_i32_p18_nullable` | `Decimal(18, 2)`, nullable | `i32` | as above, seeded nulls, zeros in null slots | Validity paths | + | `gap_msp_i16_p10` | `Decimal(10, 2)` | `i16` | 10,000 to 32,767 | The gap column of D19 | + | `msp_i64_p38` | `Decimal(38, 2)` | `i64` | within plus or minus (10^18 - 1) | No lower parts, canonical width `i128` | + | `msp_i64_p18_canonical` | `Decimal(18, 2)` | `i64` | within plus or minus (10^18 - 1) | Control: the MSP has the canonical width | + +- Operations in `dbp_compute.rs`, each a `#[divan::bench(args = ...)]` over the cases: + - `decode`: execute to `DecimalArray`. + - `compare_constant`: `Gte` against a constant that fits the MSP, executed to a `BoolArray`. + - `compare_constant_out_of_range`: `Gte` against a constant above the MSP range, on the + non-nullable cases only. The DBP kernel answers it without reading values only when all rows + are valid (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). + - `filter`: a seeded random mask, then execute to `Canonical`. + - `sum`: `vortex_array::aggregate_fn::fns::sum::sum` + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:50`). + - `min_max`: `vortex_array::aggregate_fn::fns::min_max::min_max` + (`vortex-array/src/aggregate_fn/fns/min_max/mod.rs:62`). +- These benches run in CodSpeed simulation (shard 7, "FastLanes & decimals"). Do not gate them with + `#[cfg(not(codspeed))]`. Size the arrays so that each case stays near the 1 ms simulation budget + that other array benches state (for example `vortex-array/benches/compare.rs:77-78`). +- The compressed tree in `common_encoding_tree_throughput.rs` mirrors the golden decimal tree + (`vortex-btrblocks/tests/snapshots/golden__regular__decimal_prices.snap`): a `Decimal(12, 2)` DBP + whose `i32` MSP is `fastlanes.bitpacked`. Build it by hand with `BitPacked::encode`, as + `setup::datetime_for_bp` (`:346`) does. A hand-built tree does not move when compressor + heuristics change. +- `cast_decimal.rs`: add `widen_non_nullable` and `widen_nullable`. Each casts an `i32`-backed + `Decimal(9, 2)` array to `Decimal(18, 2)` and executes to `Canonical`. The values stay below + 10^9, as the existing helper at `:66-83` already does. A second helper, or a generic one, builds + the `i32` buffer. +- `to_arrow.rs`: `decimal_export` exports one decimal array to a `Decimal128` field with + `SESSION.arrow().execute_arrow(array, Some(&field), ctx)`, as `string_export` does. Cases: + canonical `i64` at `Decimal(18, 2)`, canonical `i128` at `Decimal(38, 2)`, DBP `msp_i32_p18`, + DBP `gap_msp_i16_p10`, and a nullable variant. +- `decimal_export.rs` (DuckDB): wraps one decimal column in a `StructArray`, builds an + `ArrayExporter`, and exports into `DataChunk::new([LogicalType::decimal_type(p, s)?])` until the + exporter is exhausted. Two cases export without a copy: canonical `i64` at `Decimal(18, 2)` and + canonical `i16` at `Decimal(4, 2)`. Canonical `i8` at `Decimal(2, 1)` copies, because DuckDB + uses `i16`. The DBP cases are `msp_i32_p18` and `gap_msp_i16_p10`. The bench links DuckDB, so it + runs locally only. `ConversionCache` construction is Not verified. +- Optional `compress.rs` case: compress a canonical `Decimal(12, 2)` `i64` array with the default + `BtrBlocksCompressor`. It guards the write path that CD7 changes, outside CodSpeed. + +Steps: + +1. Add `dbp_compute.rs` and its `[[bench]]` entry. +2. Add the compressed DBP tree to `common_encoding_tree_throughput.rs`. +3. Add the widening cases to `cast_decimal.rs`. +4. Add `decimal_export` to `to_arrow.rs` and the dev-dependency. +5. Add the DuckDB `_benchmarking` module, `decimal_export.rs` and its `[[bench]]` entry. +6. Optional: add the decimal compress case. + +Tests: None. Benchmarks only. + +Benchmarks: + +- In this PR, every case is new, so CodSpeed shows additions only. +- After CD5, `decode`, `filter`, `sum` and `min_max` in `dbp_compute.rs` get slower for the cases + with a narrow MSP. Decoding then adds a widening copy, which D19 accepts. Compare cases stay the + same, because they run on the MSP. `msp_i64_p18_canonical` stays the same. The compressed tree + decompress gets slower by one widening copy. +- After CD7, these in-memory inputs do not change. The golden snapshot and the optional compress + case cover the scheme. +- After CD4, `widen_*` in `cast_decimal.rs` stays similar or gets faster, because the execution of + the returned `Narrow` widens without a per-value check. +- The Arrow and DuckDB export benches have no CodSpeed base. Run them locally with walltime before + and after CD5 and CD7. + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings`, and the + same for `vortex`, `vortex-array`, `vortex-arrow`, `vortex-duckdb` and `vortex-btrblocks`. +- `cargo +nightly-2026-09-10 fmt -p ` for each crate above. +- `cargo bench -p vortex-decimal-byte-parts --bench dbp_compute` runs every case. +- `cargo bench -p vortex --bench common_encoding_tree_throughput`, + `cargo bench -p vortex-array --bench cast_decimal`, `cargo bench -p vortex-arrow --bench + to_arrow` and `cargo bench -p vortex-duckdb --bench decimal_export` run. +- Every new bench input is a canonical decimal leaf, or a DBP array with an MSP that D16 allows. + No new input builds a `DecimalArray` at a non-canonical width, so CD8a does not break them. + +Review focus: The new inputs must stay legal through CD8a (canonical leaves and DBP MSPs only). The +CodSpeed benches must stay near the simulation budget. The DuckDB `_benchmarking` module must +expose only what the bench needs. + +Size: medium. Several bench files and one re-export, no library code. + +Open questions: None. The export benches run locally, because `vortex-arrow` and `vortex-duckdb` +have no CodSpeed shard (`codspeed.yml:52-61`), and the maintainer chose no CI change. If CodSpeed +shows noise on the new DBP benches, move them behind `#[cfg(not(codspeed))]`, as +`dbp_assemble.rs:102-104` does. + +### CD2: Compat fixtures for old decimal forms + +- Goal: Add compat fixtures for every old decimal form that the current writer can make. The + harness then compares old files against canonical expected arrays. A release before CD5 and CD6 + publishes them (D24, D31). +- Depends on: None. Soft: None. Blocks: CD5 and CD6 (the release gate of D31), CD8a (removes the + legacy overrides). +- Decisions: D24, D9 and D21 (the reads that the fixtures cover), D16 (DBP with a narrow MSP). +- Branch: `ct/decimal-compat-fixtures`, based on `develop`. +- Scope: A harness hook that separates the written form of a fixture from its expected array. Seven + new fixtures. Documentation of the hook in `DESIGN.md`. +- Out of scope: Error cases, such as a valid value outside the precision in a narrowing copy (unit + tests in CD3 and CD6, D24). Reader changes (CD5, CD6). The v2 DBP fixture, which stays disabled + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`). + +How the check works today: + +- `check` calls `fixture.write(tmp_dir, ctx)` for every fixture + (`vortex-test/compat-gen/src/check.rs:69-71`). For a flat fixture, `write` calls `build()` with + the current code and writes the result with the current flat writer + (`vortex-test/compat-gen/src/fixtures/mod.rs:90-100`). +- It then reads the published file and the fresh file with the current reader + (`check.rs:174-193`), and compares them with `assert_arrays_eq!` (`check.rs:195`). +- `assert_arrays_eq!` requires equal dtypes and lengths, and compares logical values of valid rows + only (`vortex-array/src/arrays/assertions.rs:84-113`, + `vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs:55-93`). Mixed stored widths compare + correctly (`all_non_distinct/decimal.rs:16-42`). So a canonical expected array matches an old + narrow file with the same values. +- So yes, the check rebuilds the expected array with the current code. A `build()` that makes a + narrow decimal leaf fails at runtime after CD8a. The `?` at `check.rs:70` then aborts the whole + check, not one fixture. The unit test `roundtrip_fixtures_to_bytes` also calls `build()` + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs:32-42`). +- `--git-ref` only sets the version. The code builds fixtures from the current tree + (`vortex-test/compat-gen/scripts/compat.py:922-925`, `:948-951`). The README says otherwise + (finding 1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-test/compat-gen/src/fixtures/mod.rs` | Add `FlatLayoutFixture::build_written` (trait at `:38-57`) and `Fixture::write_expected` (trait at `:26-35`). `FlatLayoutAdapter` (`:79-101`) writes `build_written` in `write` and `build` in `write_expected` | +| `vortex-test/compat-gen/src/check.rs` | `:70` calls `write_expected` instead of `write` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs` | The round-trip test (`:32-42`) writes `build_written` and compares the read-back array with `build`. Add the stored-width test | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal_widths.rs` | New. `decimal_canonical_widths.vortex` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal_legacy.rs` | New. A shared column spec and the five legacy-form fixtures | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/mod.rs` | Register the new fixtures in `fixtures()` (`:25-40`) | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_narrow_msp.rs` | New. `decimal_byte_parts_narrow_msp.vortex` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs` | Register it in `fixtures()` (`:38-63`) | +| `vortex-test/compat-gen/DESIGN.md` | Document `build_written` in "Fixture Trait" and "The oracle question" | + +Design notes: + +- Harness hook, signatures only: + + ```rust + pub trait FlatLayoutFixture { + /// Build the arrays that `generate` writes. Defaults to [`Self::build`]. + /// + /// Override this only to write an old physical form that `build` cannot return. The + /// result has the dtype and the logical values of [`Self::build`]. + fn build_written(&self, ctx: &mut ExecutionCtx) -> VortexResult + } + + pub trait Fixture { + /// Write the file that `check` compares with a published file. Defaults to [`Self::write`]. + fn write_expected( + &self, + dir: &Path, + ctx: &mut ExecutionCtx, + ) -> VortexResult> + } + ``` + +- `generate` and `publish` write `build_written`. `check` writes `build`. So the published file + holds the old form, and the check compares it with a canonical array that never changes. This + keeps the contract of `DESIGN.md` that `build()` never changes its output. +- `check_expected_encodings` runs on the written form. `DatasetFixtureAdapter` keeps the default + `write_expected`. +- The legacy fixtures share one column spec: name, `DecimalDType`, stored `DecimalType`, the raw + value of every slot as `i128`, and an optional validity. `written()` builds + `DecimalArray::try_new::` at the stored width with the raw slot values. `expected()` builds + the canonical width with zeros in null slots. The canonical width comes from + `DecimalType::smallest_decimal_value_type` until CD3 migrates that call (CD3 hand-off). +- Survival through CD3 to CD8a: `build()` builds only canonical arrays, so `check` never builds an + illegal array. `build_written` uses `DecimalArray::try_new` and returns its error. After CD8a that + error stops `generate` and the round-trip test, which forces CD8a to delete the overrides. The DBP + fixture and the canonical-width fixture need no override, because D16 keeps a narrow MSP legal. +- After CD8a, later releases publish canonical files under the same names. The old forms stay in the + store under the versions published before CD8a. +- Fixtures. Each column holds the listed boundary values. Columns in one fixture have equal length. + The last column gives the result that CD5 and CD6 must produce, assuming that CD6 picks the best + precision of D9. + + `decimal_canonical_widths.vortex`, no override: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p2_s1` | `i8` | `Decimal(2, 1)` | plus or minus 99 | Zero-copy leaf | + | `i16_p4_s2` | `i16` | `Decimal(4, 2)` | plus or minus 9,999 | Zero-copy leaf | + | `i128_p19_s2` | `i128` | `Decimal(19, 2)` | plus or minus (10^19 - 1) | Zero-copy leaf | + | `i128_p38_s2` | `i128` | `Decimal(38, 2)` | plus or minus (10^38 - 1) | Zero-copy leaf | + | `i256_p39_s2` | `i256` | `Decimal(39, 2)` | plus or minus (10^39 - 1) | Zero-copy leaf | + | `i256_p76_s0` | `i256` | `Decimal(76, 0)` | plus or minus (10^76 - 1) | Zero-copy leaf | + | `i128_p38_s2_nullable` | `i128` | `Decimal(38, 2)`, nullable | with nulls | Zero-copy leaf | + + `decimal_narrow_storage.vortex`, written narrower than canonical: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p10_s2` | `i8` | `Decimal(10, 2)` | plus or minus 99 | `Narrow` over `Decimal(2, 2)`, buffer reused | + | `i16_p18_s2` | `i16` | `Decimal(18, 2)` | plus or minus 9,999 | `Narrow` over `Decimal(4, 2)`, reused | + | `i32_p18_s2` | `i32` | `Decimal(18, 2)` | plus or minus 999,999,999 | `Narrow` over `Decimal(9, 2)`, reused | + | `i64_p30_s2` | `i64` | `Decimal(30, 2)` | plus or minus (10^18 - 1) | `Narrow` over `Decimal(18, 2)`, reused | + | `i128_p40_s0` | `i128` | `Decimal(40, 0)` | plus or minus (10^38 - 1) | `Narrow` over `Decimal(38, 0)`, reused. This is the old v1 compressor form (`vortex-btrblocks/src/schemes/decimal.rs:133-137`) | + | `i8_p76_s0` | `i8` | `Decimal(76, 0)` | plus or minus 99 | `Narrow` over `Decimal(2, 0)`, reused | + + `decimal_gap_values.vortex`, values that fit the width but need the next precision: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p10_s2_gap` | `i8` | `Decimal(10, 2)` | 100 to 127, and -128 | `Narrow` over `Decimal(3, 2)` at `i16`, copy | + | `i8_p4_s2_gap` | `i8` | `Decimal(4, 2)` | up to 127 | Canonical leaf at `i16`, copy (the example in `02-design.md`) | + | `i16_p10_s2_gap` | `i16` | `Decimal(10, 2)` | 10,000 to 32,767 | `Narrow` over `Decimal(5, 2)` at `i32`, copy | + | `i32_p18_s2_gap` | `i32` | `Decimal(18, 2)` | 1,000,000,000 to `i32::MAX` | Canonical leaf at `i64`, copy | + | `i64_p38_s0_gap` | `i64` | `Decimal(38, 0)` | 10^18 to `i64::MAX` | Canonical leaf at `i128`, copy | + + `decimal_high_scale.vortex`, where the scale limits the child precision: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p18_s10` | `i8` | `Decimal(18, 10)` | plus or minus 99 | Canonical leaf at `i64`, copy, because the child needs precision 10 | + | `i8_p18_s5` | `i8` | `Decimal(18, 5)` | plus or minus 99 | `Narrow` over `Decimal(5, 5)` at `i32`, copy | + | `i16_p38_s20` | `i16` | `Decimal(38, 20)` | plus or minus 9,999 | Canonical leaf at `i128`, copy | + | `i8_p4_s4` | `i8` | `Decimal(4, 4)` | plus or minus 99 | Canonical leaf at `i16`, copy | + + `decimal_wide_storage.vortex`, Arrow-style widths wider than canonical: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i128_p5_s2` | `i128` | `Decimal(5, 2)` | plus or minus 99,999 | Canonical leaf at `i32`, narrowing copy | + | `i128_p18_s2` | `i128` | `Decimal(18, 2)` | plus or minus (10^18 - 1) | Canonical leaf at `i64` | + | `i64_p4_s2` | `i64` | `Decimal(4, 2)` | plus or minus 9,999 | Canonical leaf at `i16` | + | `i32_p2_s1` | `i32` | `Decimal(2, 1)` | plus or minus 99 | Canonical leaf at `i8` | + | `i256_p38_s2` | `i256` | `Decimal(38, 2)` | plus or minus (10^38 - 1) | Canonical leaf at `i128` | + + `decimal_null_slots.vortex`, nonzero values in null slots: + + | Column | Stored | Dtype | Null-slot values | Read result | + | --- | --- | --- | --- | --- | + | `canonical_i32_p9_s2` | `i32` | `Decimal(9, 2)` | `i32::MAX` | Zero-copy leaf. No override, legal under D14 | + | `narrow_i8_p10_s2` | `i8` | `Decimal(10, 2)` | 127 and -128, valid values within plus or minus 99 | `Narrow` over `Decimal(2, 2)`, reused, because the max ignores null slots | + | `narrow_i8_p4_s2` | `i8` | `Decimal(4, 2)` | 127, valid values within plus or minus 99 | `Narrow` over `Decimal(2, 2)`, reused. Without null slots ignored, this is a gap | + | `wide_i128_p5_s2` | `i128` | `Decimal(5, 2)` | 10^30 | Canonical leaf at `i32`, zeros in null slots, no error | + + `decimal_byte_parts_narrow_msp.vortex`, DBP v1, no override. The existing DBP fixture uses only + `i64` MSPs (`.../encodings/decimal_byte_parts.rs:35-108`): + + | Column | MSP | Dtype | Values | Decoded after CD5 | + | --- | --- | --- | --- | --- | + | `msp_i8_p10_s2` | `i8` | `Decimal(10, 2)` | full `i8` range | `i64` leaf | + | `msp_i16_p10_s2_gap` | `i16` | `Decimal(10, 2)` | 10,000 to 32,767 | `i64` leaf | + | `msp_i32_p18_s2` | `i32` | `Decimal(18, 2)` | plus or minus 999,999,999 | `i64` leaf | + | `msp_i32_p18_s2_gap` | `i32` | `Decimal(18, 2)` | 1,000,000,000 to `i32::MAX` | `i64` leaf | + | `msp_i64_p38_s2` | `i64` | `Decimal(38, 2)` | plus or minus (10^18 - 1) | `i128` leaf | + | `msp_i32_p18_s2_nullable` | `i32`, nullable | `Decimal(18, 2)` | nonzero null slots | `i64` leaf | + | `msp_i64_p4_s2_wide` | `i64` | `Decimal(4, 2)` | plus or minus 9,999 | `i16` leaf, narrowing copy (D16, D21) | + +- The last DBP column, an MSP wider than the canonical width, is beyond the list of D24. It is a + read that succeeds under D16, so the brief includes it. +- Publication. `publish.yml:119-127` calls `compat-gen-upload.yml` with the release tag on every + published release. That job checks out the tag and runs + `uv run vortex-test/compat-gen/scripts/compat.py publish --yes --git-ref `, which uploads + every fixture to `v/arrays/`. So: + 1. Merge CD2. + 2. Cut a release from a `develop` commit that has CD2 and not CD3. + 3. Confirm that the upload job passed, and that + `uv run vortex-test/compat-gen/scripts/compat.py list --version ` lists the seven + new names. + 4. Merge CD5 and CD6 only after that. The library PRs CD3 and CD4 do not wait (D31). +- Fallback if CD5 or CD6 cannot wait for a release: check out the latest release tag, apply CD2, and + run `compat.py publish --update`. It uploads new fixtures only, and its hash check proves that the + other fixtures did not change (`compat.py:472-545`). Whether CD2 applies cleanly to that tag is + Not verified. +- Every PR runs `compat.py check --mode last` (`publish-dry-runs.yml:110-114`). In the PR of CD2, + the new names are missing from the store, so `superset` mode skips them (`check.rs:117-134`). + +Steps: + +1. Add `build_written` and `write_expected`, switch `check` to `write_expected`, and update the + round-trip test and `DESIGN.md`. +2. Add `decimal_canonical_widths.vortex`. +3. Add the column spec and the five legacy-form fixtures, with the stored-width test. +4. Add `decimal_byte_parts_narrow_msp.vortex`. +5. Generate and check locally. + +Tests: + +- `roundtrip_fixtures_to_bytes` (`synthetic/mod.rs:32-42`): for each fixture, check that + `build_written` and `build` have equal dtypes and logical values in memory. Then write + `build_written`, read it back, and compare it with `build`. +- New `legacy_decimal_fixtures_keep_stored_widths` in the same module: every decimal leaf of each + `build_written` has the stored width of its column spec. Without this test, a fixture can write + canonical data and test nothing. +- The read results in the tables are expectations for CD5 and CD6. Those PRs test them. + +Benchmarks: None. + +Acceptance checks: + +- `cargo nextest run -p vortex-compat` +- `cargo clippy -p vortex-compat --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-compat` +- `cargo run -p vortex-compat --release -- generate --output `, then + `cargo run -p vortex-compat --release -- check --dir --mode exact`. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` passes, with the new names + skipped. +- After the release, `compat.py list --version ` shows the seven new names. + +Review focus: `build()` builds only canonical arrays. Each written form holds the intended stored +width and values. Null-slot values in the written forms are nonzero where the table says so. +`check` uses `write_expected` and `generate` uses `write`. + +Size: medium. One small harness change and seven data fixtures. + +Open questions: None. The next regular release publishes the fixtures (D31). The DBP fixture +keeps `msp_i64_p4_s2_wide`, because it is a read that succeeds. + +### CD3: One width owner, and imports copy into the canonical width + +- Goal: `DecimalDType::values_type()` becomes the one owner of the width of each precision (D12). + The Arrow and DuckDB imports copy into that width, with zeros in null slots and an error on a + valid value outside the precision (D13). One shared helper performs every such copy (D21). +- Depends on: None. Soft: CD1, CD2. Blocks: CD4, CD5, CD6, CD8a. +- Decisions: D12, D13, D21 (the helper), D30 (imports check their own values), D3 (as far as + imports go). +- Branch: `ct/decimal-width-owner`, based on `develop`. +- Scope: Add `DecimalDType::values_type()`. Remove `DecimalType::smallest_decimal_value_type` and + migrate all its calls. Add the shared helpers `copy_to_canonical_width` and `check_precision`. + Change the Arrow and DuckDB imports, so that every import checks its values (D30). +- Out of scope: Builders, including `DecimalBuilder::extend` and `append_decimal_array` + (`vortex-array/src/builders/decimal.rs:298-306`, `:160-178`), and a strict `try_new` (CD8a, CD8b). + DBP decoding (CD5). The read plugin (CD6). `narrowed_decimal` and the compressor (CD7). The + zero-copy widening cast (CD4). Exports. + +Count of `smallest_decimal_value_type` at `b70206818f`: 26 matches, one of them the definition +(`vortex-array/src/dtype/decimal/types.rs:46`), so 25 calls. This matches the notes at `5fc5cb8`. +No call exists outside Rust code. If CD2 or another PR adds calls before CD3, CD3 migrates them +too. Run `rg -n smallest_decimal_value_type` at the base of CD3. + +| # | Call | Use | New code | +| --- | --- | --- | --- | +| 1 | `vortex-array/src/dtype/decimal/types.rs:73` | `is_compatible_decimal_value_type` | `self >= dtype.values_type()` | +| 2 | `vortex-array/src/dtype/dtype_impl.rs:320` | element byte width | `decimal.values_type().byte_width()` | +| 3 | `vortex-array/src/array/haystack/mod.rs:119` | IN-list width | `decimal.values_type()` | +| 4 | `vortex-array/src/builders/mod.rs:416` | builder factory width | `decimal_type.values_type()` | +| 5 | `vortex-array/src/scalar/arbitrary.rs:164` | arbitrary scalar | `decimal_type.values_type()` | +| 6 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186` | `DecimalValue::zero` | `decimal_type.values_type()` | +| 7 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209` | `DecimalValue::normalize` | `decimal_type.values_type()` | +| 8 | `vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112` | scalar work width | `work.values_type()` | +| 9 | `vortex-array/src/arrays/constant/vtable/canonical.rs:97` | constant canonicalization | `decimal_type.values_type()` | +| 10 | `vortex-array/src/arrays/constant/vtable/canonical.rs:356` | constant canonicalization | `decimal.values_type()` | +| 11 | `vortex-array/src/arrays/decimal/compute/cast.rs:155` | cast target width | `to_decimal_dtype.values_type()` | +| 12 | `vortex-array/src/arrays/arbitrary.rs:146` | arbitrary array | `decimal.values_type()` | +| 13 | `vortex-array/src/arrays/primitive/compute/cast.rs:144` | integer to decimal cast | `decimal_dtype.values_type()` | +| 14 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98` | arithmetic work width | `work_dtype.values_type()` | +| 15 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416` | arithmetic result width | `decimal_dtype.values_type()` | +| 16 | `vortex-array/src/canonical.rs:188` | empty canonical array | `decimal_type.values_type()` | +| 17 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | size statistic | `array.decimal_dtype().values_type()` | +| 18 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365` | size statistic | `decimal_type.values_type()` | +| 19 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103` | size statistic | `decimal.values_type()` | +| 20 | `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49` | sum accumulation width | `dtype.values_type()` | +| 21 | `vortex-cuda/src/kernel/arrays/constant.rs:73` | CUDA constant | `decimal_dtype.values_type()` | +| 22 | `fuzz/src/array/sum/mod.rs:54` | fuzz oracle | `output_dtype.values_type()` | +| 23 | `vortex-row/src/codec.rs:583` | row key width in `decimal_key_type` | `dt.values_type()`, keep the function and its doc | +| 24 | `encodings/sparse/src/canonical.rs:316` | sparse canonicalization | `decimal_dtype.values_type()` | +| 25 | `vortex-duckdb/src/convert/vector.rs:314` | DuckDB import narrowing | Removed. The import uses the helper | + +Producers and imports that create a width other than the canonical width today: + +| Producer | Location | Width today | Owner | +| --- | --- | --- | --- | +| Arrow import | `vortex-arrow/src/convert.rs:179-224` | The Arrow width, zero-copy. Wider than canonical for `Decimal32` at precision 1 to 4, `Decimal64` at 1 to 9, `Decimal128` at 1 to 18, `Decimal256` at 1 to 38 | CD3 | +| Python, C and C++ FFI, JNI, DataFusion, Parquet | `vortex-python/src/arrays/from_arrow.rs:40`, `:59`, `vortex-ffi/src/array.rs:510`, `vortex-jni/src/writer.rs:148`, `vortex-datafusion/src/lib.rs:209`, `vortex-datafusion/src/convert/scalars.rs:385` | All route through `ArrowSession::from_arrow_array`, which ends in `from_arrow_dyn` (`vortex-arrow/src/session.rs`, leaf fallback) | CD3, through the Arrow import. The Parquet route through arrow-rs is Not verified for every reader | +| DuckDB import | `vortex-duckdb/src/convert/vector.rs:303-338` | Canonical for every precision. Precision 1 to 2 narrows DuckDB `i16` to `i8` with a lossy `as_()` cast (`:312-322`, PR #10155) | CD3 | +| `DecimalBuilder::extend` and `append_decimal_array` | `vortex-array/src/builders/decimal.rs:298-306`, `:160-178` | Builder width from `T`. Accepts wider source buffers and panics on a null slot that does not fit | CD8a | +| User constructors | `vortex-array/src/arrays/decimal/array.rs:406-534` | Any `T` | CD8a | +| DBP decoding | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-93` | The MSP width | CD5 | +| Old `vortex.decimal` nodes | `vortex-array/src/arrays/decimal/vtable/mod.rs` | The stored width | CD6 | +| Compressor | `vortex-btrblocks/src/schemes/decimal.rs:133` (`narrowed_decimal`) | Narrower | CD7 | +| Zero-copy widening cast | `vortex-array/src/arrays/decimal/compute/cast.rs:122-143` | The source width | CD4 | +| Kernels that keep the input width | for example `vortex-array/src/arrays/decimal/compute/fill_null.rs:55-72` | The input width, or wider | CD8a | + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/dtype/decimal/mod.rs` | Add `DecimalDType::values_type()` in the impl block (`:43-110`). Tests in the module test block | +| `vortex-array/src/dtype/decimal/types.rs` | Remove `smallest_decimal_value_type` (`:45-57`). Update `is_compatible_decimal_value_type` (`:71-74`) | +| The 23 other files of the table above | Mechanical migration | +| `vortex-array/src/arrays/decimal/utils.rs` | Add `copy_to_canonical_width` and `check_precision`, with tests | +| `vortex-arrow/src/convert.rs` | The four `from_arrow_decimal*` functions (`:179-224`) copy when the Arrow width differs. Update the tests at `:1011-1110` | +| `vortex-duckdb/src/convert/vector.rs` | The decimal arm (`:303-338`) uses the helper. Add tests to the module at `:436` | + +Design notes: + +- The width owner, as `02-design.md:29-43` proposes: + + ```rust + impl DecimalDType { + /// The canonical width of the values buffer for this precision. + pub const fn values_type(&self) -> DecimalType { + match self.precision.get() { + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, + } + } + } + ``` + + Every `DecimalDType` built through `try_new` has a precision from 1 to 76 + (`vortex-array/src/dtype/decimal/mod.rs:48-77`). The serde derive at `:36` bypasses `try_new` + (finding 9). +- The shared helper, in `vortex-array/src/arrays/decimal/utils.rs` and exported as + `vortex_array::arrays::decimal::copy_to_canonical_width` through `pub use utils::*` + (`vortex-array/src/arrays/decimal/mod.rs:22-23`): + + ```rust + /// Copies unscaled decimal values into a new buffer at the canonical width of `target`. + pub fn copy_to_canonical_width( + values: &[F], + validity: &Mask, + target: DecimalDType, + ) -> VortexResult + ``` + + - The result holds `values.len()` values of `target.values_type()`. + - It returns an error when `validity.len()` differs from `values.len()`. + - Narrowing copy, when `target.values_type()` is narrower than `F`: every valid value must be + within `F::MIN_BY_PRECISION[p]` and `F::MAX_BY_PRECISION[p]` for the precision `p` of `target` + (`vortex-array/src/dtype/decimal/types.rs:102-108`). Otherwise the copy fails, and the error + names the row, the value and `target`. Null slots of the result hold zeros. + - Widening copy, when `target.values_type()` is wider than `F`: each value is widened, and no + value is checked. The caller must know that every valid value fits `target`. This holds when + `target` has at least the precision of the source dtype, or when the caller chose `target` from + the values (CD6). + - Same width: a plain copy without a check. A caller that owns a `Buffer` reuses it instead. + - The helper never rescales. The caller keeps the scale. + - Use the lane kernels for the checked loop (`vortex_compute::lane_kernels::IndexedSourceExt`, as + `vortex-array/src/arrays/decimal/compute/cast.rs:217-241` does), then write zeros in the null + rows. Follow the hidden-cost rules in `STYLE.md`: no per-row `Mask::value(i)`. + - `converted_buffer` (`utils.rs:40-89`) stays. It checks the width, not the precision, and keeps + null-slot values that fit. Its callers are `vortex-array/src/array/haystack/mod.rs:123`, `:216` + and `vortex-row/src/codec.rs:958`. +- Users of the helper: CD3 (Arrow and DuckDB imports), CD4 (`Narrow` execution and the decimal + encoder), CD5 (DBP decoding in both directions), CD6 (old nodes). CD8c can use it in + `append_decimal_array`. +- Arrow import. One private function in `vortex-arrow/src/convert.rs` serves all four + `from_arrow_decimal*` functions: + - It builds the dtype with `DecimalDType::try_new`, not `new` (`:184`, `:195`, `:206`, `:217`). + An invalid precision or scale from FFI then returns an error, not a panic. + - It builds a `Mask` from the Arrow `NullBuffer`, which is all valid when absent. + - When the Arrow native width equals `decimal_dtype.values_type()`, it keeps the zero-copy + buffer, calls `check_precision`, and builds the array with the unchecked constructor. This + covers `Decimal32` at precision 5 to 9, `Decimal64` at 10 to 18, `Decimal128` at 19 to 38, and + `Decimal256` at 39 to 76. + - Otherwise it calls `copy_to_canonical_width` and builds the array with the unchecked + constructor. `Decimal256` keeps the transmute of `:219-222` before the copy. + - The error adds context with `.with_context()`: the import rejects values outside the precision, + and arrow-rs `PrimitiveArray::null_if_overflow_precision` turns them into nulls before the + import (`arrow-array-59.3.1/src/array/primitive_array.rs:1672`). + - Doc comments change from "Zero-copy conversion" to "zero-copy when the Arrow width is the + canonical width, otherwise a checked copy". +- Options hook (D13). No import options exist. `ArrowExportOptions` + (`vortex-arrow/src/options/mod.rs:22-51`) covers exports only, and the import API takes no options + (`ArrowSession::from_arrow_array`, `from_arrow_dyn`). The smallest scope is the default of D13: + an error, with the arrow-rs pre-processing named in the error. An import options type is future + work. +- DuckDB import. In the decimal arm, each DuckDB width calls `copy_to_canonical_width` on the vector + slice with the mask of `:307`. Precision 1 to 2 then becomes a checked narrowing copy from `i16` + into `i8`, with zeros in null slots. The other widths are equal widths, so the call is a plain + copy, as `Buffer::copy_from` is today, and the import then calls `check_precision`. The `i128` + path keeps `Buffer::from_iter(vector_i128_values(vector, len))` (`:331-334`), which already builds + a new buffer at the canonical width. +- Every import checks its values in CD3, so D13 holds for every import from CD3 on. A zero-copy + import now scans every valid value once. The imports use the unchecked constructors after their + own check, so the `ExecutionCtx` that CD8b adds to `try_new` never reaches the import API (D30). +- `check_precision(values: &[T], validity: &Mask, decimal_dtype: DecimalDType) -> VortexResult<()>` + runs a branchless fold first, and it finds the failing row only on failure, as + `vortex-array/src/arrays/decimal/utils.rs:50-80` does. The error names the row, the value and the + dtype. + +Steps: + +1. Add `DecimalDType::values_type()` with tests. +2. Migrate the 25 calls, remove `smallest_decimal_value_type`, and update + `is_compatible_decimal_value_type`. +3. Add `copy_to_canonical_width` with tests. +4. Change the Arrow import and its tests. +5. Change the DuckDB import and add its tests. + +Tests: + +- `vortex-array/src/dtype/decimal/mod.rs` tests: `test_values_type_boundaries`, an `rstest` over + the precisions 1, 2, 3, 4, 5, 9, 10, 18, 19, 38, 39 and 76. Each case expects the width of D12. +- `vortex-array/src/arrays/decimal/utils.rs`, a new `tests` module: + - `test_copy_to_canonical_width_widens`: `rstest` over the widening pairs, values unchanged. + - `test_copy_to_canonical_width_narrowing_boundaries`: `rstest` over the narrowing pairs. + Plus or minus (10^p - 1) passes, and plus or minus 10^p fails. + - `test_copy_to_canonical_width_zeros_null_slots`: null slots hold values outside the target + width and precision, the copy succeeds, and the null slots are zero. + - `test_copy_to_canonical_width_error_names_row`: the error names the first failing valid row. + - `test_copy_to_canonical_width_all_null`, and `..._length_mismatch` (an error). + - `test_check_precision_boundaries`: plus or minus (10^p - 1) passes, and plus or minus 10^p + fails, for each width. + - `test_check_precision_ignores_null_slots`: a value outside the precision in a null slot + passes. +- `vortex-arrow/src/convert.rs` tests: + - `test_decimal_import_canonical_width`: `rstest` over `Decimal32(4, 2)`, `Decimal64(9, 2)`, + `Decimal128(5, 2)`, `Decimal128(18, 2)`, `Decimal128(38, 2)`, `Decimal256(38, 2)` and + `Decimal256(40, 2)`. The result has `values_type() == decimal_dtype.values_type()` and equal + values. The equal-width cases keep the Arrow buffer pointer. + - `test_decimal_import_zeros_null_slots`: an Arrow `Decimal128(5, 2)` array with 10^30 in a null + slot imports, and the null slot holds zero. + - `test_decimal_import_rejects_value_outside_precision`: `Decimal128(5, 2)` with a valid + 1,000,000 fails. + - `test_decimal_import_zero_copy_rejects_value_outside_precision`: `Decimal128(20, 2)` with a + valid 10^20 fails. This is the zero-copy path. + - `test_decimal_import_rejects_invalid_precision`: an invalid precision or scale returns an error. + - The existing `test_decimal128_array_conversion` and `test_decimal256_array_conversion` + (`:1013`, `:1063`) build their expected arrays at `i128` for precision 10 and `i256` for + precision 38. Change the expected arrays to the canonical widths (`i64` and `i128`), and assert + the width. D12 fixes those widths, so this change does not follow the code under test. +- `vortex-duckdb/src/convert/vector.rs` tests, in the style of `test_vector_with_nulls` (`:676`): + - `test_decimal_vector_conversion`: `rstest` over the precisions 1, 2, 3, 4, 9, 18 and 38. The + width is canonical and the values are equal. + - `test_decimal_vector_null_slot_garbage`: precision 2 with 300 in a null slot imports, and the + slot holds zero. + - `test_decimal_vector_value_outside_precision`: precision 2 with a valid 150 fails. + - `test_decimal_vector_equal_width_outside_precision`: precision 4 with a valid 10,000 fails. + +Benchmarks: None. The Arrow import of `Decimal128` at precision 18 or below now copies, which D13 +accepts. `vortex-arrow` is in no CodSpeed shard. + +Acceptance checks: + +- `cargo nextest run -p vortex-array`, `-p vortex-arrow`, `-p vortex-duckdb`, `-p vortex-row` and + `-p vortex-sparse`. +- `cargo test --doc -p vortex-array` +- `cargo clippy -p --all-targets --all-features -- -D warnings` for `vortex-array`, + `vortex-arrow`, `vortex-duckdb`, `vortex-row`, `vortex-sparse` and `vortex-fuzz`. For + `vortex-cuda`, run it where the CUDA toolchain exists. Whether `vortex-cuda` builds without it is + Not verified. +- `cargo +nightly-2026-09-10 fmt -p ` for each crate above. +- `rg -n smallest_decimal_value_type` returns nothing. +- Every Arrow and DuckDB decimal import returns an array at the canonical width of its dtype. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` still passes, including the + CD2 fixtures and the TPC-H and ClickBench fixtures, whose decimals come through the Arrow import. + +Review focus: The helper checks precision, not width, on a narrowing copy. Null slots never cause an +error. The equal-width Arrow import stays zero-copy. Every import checks its values, then uses the +unchecked constructors. The migration is +mechanical, and every call names the right dtype (the work dtype in arithmetic, the target dtype in +casts). The changed Arrow test expectations follow D12. + +Size: medium. Twenty-five mechanical sites, one helper, and two imports with tests. + +Open questions: None. No import options exist, and the maintainer chose none for now. The import +fails on an invalid value, and the error names `null_if_overflow_precision`. Zero-copy imports check +their values in CD3 (D30). + +### CD4: Decimal `Narrow` in memory + +- Goal: The `Narrow` encoding accepts decimal children under D10, and its kernels and the decimal + cast follow D15. Compare, sum, min and max run on the narrower child, with the result of the + parent. +- Depends on: NR1, NR3, NR4, CD3, CD1. Soft: NR2 (bench file layout), NR5 (its structural rules + also hold for decimals). Blocks: CD6, CD8a, WR1, WR3. +- Decisions: D10, D15, D4, D8 (one encoding for integers and decimals, in memory only), D19, D25, + D26. +- Branch: `ct/decimal-narrow`, based on `develop` after NR1, NR3, NR4, CD1 and CD3 merge. +- Scope: Open `validate_dtypes` for decimals, and add decimal execution and a decimal encoder. Add + decimal branches to the compare, aggregate, cast and fill-null kernels. Change the decimal cast + kernel for D15. Add tests and benchmarks. +- Out of scope: The `vortex.narrow` wire form (WR1). The compressor scheme (WR3). The read plugin + (CD6). A strict `try_new` (CD8a). Arithmetic, zip and merges on decimal `Narrow`, which decline + and fall back. A cast rule that wraps a DBP or other encoded decimal in a `Narrow` without + decoding. Pushdown into the MSP (LT3). Exports keep their own widening code. Replacing it with a + `Narrow` execution (D13) is not part of CD4. + +Places in the rewrite that assume an integer child. CD4 handles each one: + +| # | Place | Assumption | CD4 action | +| --- | --- | --- | --- | +| 1 | `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73-88` `validate_dtypes` | `PType::try_from`, `is_int`, signedness and byte width | Add the decimal rule of D10 | +| 2 | `af4b663059:.../narrow/mod.rs:4-10`, `:35-54` | Docs say integer only | Update the docs | +| 3 | `af4b663059:.../narrow/vtable.rs:111-116` `execute` | Yields `values.cast(dtype)` | Widen explicitly for decimals, or the D15 cast loops (design notes) | +| 4 | `af4b663059:.../narrow/vtable.rs:130-139` `scalar_at` | None, a scalar cast | Test only | +| 5 | `af4b663059:.../narrow/encoding.rs:23-63` `NarrowArray::encode` | Takes a `PrimitiveArray` | Add `NarrowArray::encode_decimal` | +| 6 | `af4b663059:.../narrow/compare.rs:52-70` | Narrow against Narrow uses `as_ptype()`, which panics for decimals | Decimal branch at the wider child precision | +| 7 | `af4b663059:.../narrow/compare.rs:88-95` | Out-of-range sign uses `is_signed_int()` and `i64::try_from` | Decimal sign from the `DecimalValue` | +| 8 | `af4b663059:.../narrow/compare.rs:111-122` `compare_values` | Fast path only for `Primitive` | Also take it for `Decimal` leaves, if `execute_compare` accepts them (Not verified) | +| 9 | `af4b663059:.../narrow/rules.rs:111-144` `CastReduce` | `PType::try_from`, declines decimals | D15: pass the cast to the child | +| 10 | `af4b663059:.../narrow/rules.rs:146-194` `FillNullReduce`, `integer_storage_type` | `PType::try_from`, declines decimals | Decimal branch: fill the child when the fill value fits its precision, otherwise decline | +| 11 | `af4b663059:.../narrow/rules.rs:240-279` `zip_values` | `PType::try_from`, declines | Keep the decline. Test the fallback | +| 12 | `af4b663059:.../narrow/rules.rs:316-346` `BetweenReduce` | None, a scalar cast and `two_compares` | Test only. Relies on 6 and 7 | +| 13 | `af4b663059:.../narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | None. A decimal cannot be a code | +| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | Decimal `Sum` and `SumV2` use the result dtype of the parent (design notes) | +| 15 | `af4b663059:.../narrow/numeric.rs:69-72` | `PType::try_from`, declines | Keep the decline. Test the fallback | +| 16 | `af4b663059:.../narrow/merge.rs:112-148` `stored_inputs` | `PType::try_from`, declines | Keep the decline. Test the fallback | +| 17 | `af4b663059:.../narrow/merge.rs:77-94` interleave indices | Indices are integers | None | + +Files. The `narrow` module comes from NR1 to NR4, so its lines are not on `b70206818f`: + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/mod.rs` | `validate_dtypes` for decimals, docs | +| `vortex-array/src/arrays/narrow/vtable.rs` | Decimal `execute` | +| `vortex-array/src/arrays/narrow/encoding.rs` | `NarrowArray::encode_decimal` and the precision helper | +| `vortex-array/src/arrays/narrow/compare.rs` | Decimal branches | +| `vortex-array/src/arrays/narrow/rules.rs` | Decimal `CastReduce` and `FillNullReduce` | +| `vortex-array/src/arrays/narrow/aggregates.rs` | Decline decimal `Sum` and `SumV2` | +| `vortex-array/src/aggregate_fn/fns/sum/mod.rs` | `try_accumulate` (`:218-254`) sums a decimal `Narrow` child with the result dtype of the parent | +| `vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs` | The same in `try_accumulate` (`:221-244`) | +| `vortex-array/src/arrays/decimal/compute/cast.rs` | The D15 rule in `CastReduce` (`:45-82`) and `CastKernel` (`:84-170`) | +| `vortex-array/src/arrays/narrow/tests/decimal.rs` | New tests, registered in `narrow/tests/mod.rs` | +| The `Narrow` bench file of NR2, or `vortex-array/benches/` | Decimal `Narrow` benches | + +Design notes: + +- `validate_dtypes(child, parent)` under D10: + - Two primitive dtypes follow the integer rule of D22, unchanged. + - Two decimal dtypes need the same scale, the same nullability, and + `child.values_type() < parent.values_type()`. `DecimalType` orders by width + (`vortex-array/src/dtype/decimal/types.rs:25-42`). + - Any other pair fails, with an error that names both dtypes. + - A strictly narrower canonical width implies a lower precision. `DecimalDType::try_new` already + rejects a positive scale above the precision (`vortex-array/src/dtype/decimal/mod.rs:68-74`), so + a child precision is always at least the scale. No extra check is needed. + - `NarrowArray::try_new` flattens a `Narrow` over a `Narrow` + (`af4b663059:.../narrow/mod.rs:55-61`), which gives one `Narrow` (D15). The check of the inner + child holds by transitivity. + - `try_new` trusts the child dtype. It reads no values. +- Execution. The integer path yields a cast to the parent dtype. For decimals that cast returns a + `Narrow` again under the new rule below, so execution never ends. The decimal path therefore + executes the child to a `DecimalArray`, computes its validity mask, and calls + `copy_to_canonical_width` (CD3) with the parent `DecimalDType`. The result keeps the validity of + the child. The call dispatches on the stored width of the child (`DecimalArray::values_type()`), + so a child at a non-canonical width before CD8a is still correct. A child stored at the parent + width reuses its buffer. A child stored wider than the parent width gets a checked narrowing + copy. +- Encoder, the decimal twin of `NarrowArray::encode`: + + ```rust + impl NarrowArray { + /// Stores a decimal array at the best precision that its valid values and its scale allow. + /// + /// Returns the input unchanged when that precision has the canonical width of the input + /// dtype. + pub fn encode_decimal(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult + } + ``` + + - It takes the min and max of the valid values with `min_max`. + - A public function returns the best child dtype (see Shared helpers): the smallest precision + `p`, at least 1 and at least a positive scale, with `MIN_BY_PRECISION[p] <= min` and + `max <= MAX_BY_PRECISION[p]`. An empty or all-null array gets that lower bound. + - When the child width is narrower than the parent width, it copies with + `copy_to_canonical_width` and wraps the child in a `Narrow`. + - This is the producer rule of D10. CD6 and WR3 reuse the function, so that one function owns + the child label. A false label gives wrong answers in the compare shortcut and in casts. +- Compare with a constant (the constant has the parent dtype, `af4b663059:.../compare.rs:48-50`): + - If the constant casts to the child dtype, compare the child with it. The decimal scalar cast + checks the precision (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:158-182`, + `:208-217`). + - Otherwise the constant is outside the range of the child precision. The answer needs no values: + `Eq` is false, `NotEq` is true. For `Lt` and `Lte` the answer is true when the constant is + positive. For `Gt` and `Gte` it is true when the constant is negative. + - The sign comes from the decimal value of the constant, for example `as_i256() < i256::ZERO`. + - The result keeps the validity of the child, cast to the result nullability, as the integer path + does (`af4b663059:.../compare.rs:96-107`). +- Compare two decimal `Narrow` arrays: cast both children to the higher child precision at the + common scale, then compare. The narrower child becomes a `Narrow` or a relabeled leaf under the + cast rule. The comparison then falls back to its canonical form at the child width. +- Sum. The result dtype of a decimal sum depends on the input precision (`sum_decimal_dtype`, + `vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`). A child accumulator uses + `sum_decimal_dtype(child)`, which overflows where the parent result does not. So: + - The NR3 kernel declines decimal `Sum` and `SumV2` (`Ok(None)`). + - `Sum::try_accumulate` and `SumV2::try_accumulate` handle a batch that is a decimal `Narrow`. + They execute the child to a columnar form (`execute::`, as + `vortex-array/src/aggregate_fn/accumulator.rs:169` does) and pass it to their own `accumulate` + with the arguments of the parent. The accumulator dispatches kernels before `try_accumulate` + (`accumulator.rs:107-138`), so the decline comes first. + - `accumulate_decimal` reads any input width into the width of the result dtype + (`vortex-array/src/aggregate_fn/fns/sum/decimal.rs:27-61`). `multiply_constant` handles a + constant child of any precision (`vortex-array/src/aggregate_fn/fns/sum/constant.rs:69-89`). + So the sum never widens the child buffer. + - Alternative: keep the NR3 path, and decline when the child partial overflows. It is simpler, + but its fallback widens a child of any length. +- Min, max, first and last: the NR3 path stands. The child partial casts to the parent partial + dtype, so each value becomes a value of the parent dtype. `IsSorted`, `IsConstant` and the counts + need no change. Whether the struct partial of `MinMax` casts field by field is Not verified, so + CD4 tests it. +- Casts (D15): + - A cast of a decimal `Narrow` passes to the child with the same target dtype: decimal, integer + or float. When the target equals the dtype of the array, the rule returns the array. + - Decimal leaf, same scale, a higher precision, and a canonical width that grows: return + `NarrowArray::try_new(input, target)`. The input first takes the target nullability. In + `CastReduce` this uses `trivially_cast_nullability`, and the rule declines when that returns + `None`. In `CastKernel` it uses `cast_nullability` with the context. + - Same scale, a higher precision, and the same canonical width: change only the dtype, as the + zero-copy path at `cast.rs:122-143` does. + - A lower precision checks every value, as today. A scale change rescales, as today. + - Before CD8a, an input can have a stored width other than its canonical width. The rule compares + the canonical widths of the two dtypes. For such an input with an equal canonical width, the + kernel keeps its current behavior. So only the case where the canonical width grows changes. + - The existing tests `cast_upcast_precision_succeeds` (`cast.rs:969-996`) and + `cast_widening_same_physical_type_is_zero_copy` (`:999-1029`) still pass. The first now goes + through a `Narrow` and its execution. +- Fill null: when the fill value casts to the child dtype, fill the child and wrap the result. + Otherwise decline. Canonical execution then handles a fill value that needs a wider precision. +- Filter, take, slice and mask need no decimal code (D15). Their rules wrap the child result with + `wrap_values` (`af4b663059:.../narrow/rules.rs:73-79`), which now validates decimal pairs. + +Steps: + +1. Open `validate_dtypes` for decimals and update the docs. +2. Add decimal execution and `encode_decimal`. +3. Add the decimal compare branches. +4. Route decimal `Sum` and `SumV2` through the result dtype of the parent. +5. Add the D15 cast rules on `Narrow` and on the decimal leaf, and the fill-null branch. +6. Add the conformance and fallback tests. +7. Add the benchmarks. + +Tests, in `vortex-array/src/arrays/narrow/tests/decimal.rs`, as functions that return +`VortexResult<()>`: + +- `test_try_new_decimal_dtypes`: an `rstest` over pairs of child and parent dtypes. + - Accepted: `Decimal(9, 2)` under `Decimal(18, 2)`, and `Decimal(38, 0)` under `Decimal(76, 0)`. + - Rejected for the same width: `Decimal(10, 2)` under `Decimal(18, 2)`, and `Decimal(75, 0)` + under `Decimal(76, 0)`. + - Rejected for other reasons: another scale, another nullability, or a higher child precision. + - Rejected for mixed types: an integer child under a decimal parent, and the reverse. +- `test_narrow_over_narrow_flattens`: one `Narrow` remains, with the inner child. +- `test_decimal_execute_widens`: nullable and non-nullable, values and validity equal the canonical + array, and the width is the canonical width of the parent. Include a child at a non-canonical + stored width. +- `test_decimal_execute_terminates`: a decimal `Narrow` executes to `Canonical`. This is the + regression test for the cast loop. +- `test_encode_decimal_best_precision`: an `rstest` over values and scales. Cases include a + positive scale that raises the precision, negative values, all nulls and an empty array. One case + needs the canonical width and returns no `Narrow`. +- `test_decimal_compare_constant`: an `rstest` over all six operators. The constant is inside the + child range, above it, below it, or null. One case uses a nullable child. Each result equals the + comparison on the canonical array with `assert_arrays_eq!`. +- `test_decimal_compare_two_narrow`: two children with different precisions. +- `test_decimal_between`: bounds inside and outside the child range. +- `test_decimal_sum`: `Sum` and `SumV2` equal the canonical results, with the dtype + `sum_decimal_dtype(parent)`. Include nulls and all nulls. +- `test_decimal_sum_uses_parent_precision`: a `Narrow` of `Decimal(18, 0)` over a `ConstantArray` + of `Decimal(2, 0)` with the value 99. Its length pushes the child sum past precision 12, the + precision of `sum_decimal_dtype(Decimal(2, 0))`. The sum is the exact product, not null. The + constant child is never materialized. +- `test_decimal_min_max_first_last`: results are values of the parent dtype and equal the canonical + results. +- `test_decimal_cast_rules`: an `rstest` over the leaf rules. + - The canonical width grows: the result is a `Narrow` before execution. + - The width stays: the result reuses the buffer pointer. + - The precision drops: values are checked, and a value that does not fit fails. + - The nullability changes, with and without nulls. +- `test_decimal_narrow_cast_passes_to_child`: decimal, integer and `f64` targets equal the casts of + the canonical array. +- `test_decimal_fill_null`: a fill value that fits the child, and one that needs a wider precision. +- `test_decimal_fallbacks`: arithmetic, zip, chunked and interleave on decimal `Narrow` arrays + equal the canonical results. +- `test_decimal_structural_pull_up`: the NR5 rules move `Dict`, `Constant` and `Chunked` above a + decimal `Narrow`, with equal values. +- `test_decimal_narrow_serializes_as_canonical`: a decimal `Narrow` serializes as a canonical + `vortex.decimal` node at the parent width (D26). +- Conformance: run `test_array_consistency`, `test_cast_conformance`, `test_filter_conformance`, + `test_take_conformance` and `test_mask_conformance` (`vortex-array/src/compute/conformance/`) on + nullable and non-nullable decimal `Narrow` arrays. + +Benchmarks (CodSpeed compute shard). Follow the file layout of NR2 (hand-off). Each case has a +canonical baseline in the same file: + +| Case | `Narrow` input | Baseline | +| --- | --- | --- | +| `p18_child_p9` | `encode_decimal` of `Decimal(18, 2)` values within plus or minus 999,999,999, child `Decimal(9, 2)` at `i32` | The canonical `i64` array | +| `gap_p10_child_p5` | `Decimal(10, 2)` values from 10,000 to 32,767, child `Decimal(5, 2)` at `i32` | The canonical `i64` array | +| `p38_child_p18` | `Decimal(38, 2)` values within plus or minus (10^18 - 1), child `Decimal(18, 2)` at `i64` | The canonical `i128` array | + +Operations: + +- `compare_constant`, with a constant inside the child range. +- `compare_constant_out_of_range`. +- `sum` and `min_max`. +- `filter`, with a seeded mask, then execute. +- `execute`, which widens to canonical. +- `cast_widen`, to `Decimal(38, 2)` or to `Decimal(76, 2)` for the `p38` case, then execute. + +Expected in CodSpeed: `compare_constant`, `sum`, `min_max` and `filter` are equal to or faster than +the baseline, because the child is narrower. `compare_constant_out_of_range` is much faster, +because it reads no values. `execute` costs one widening copy. Compare it with the DBP `decode` of +CD1 when the D19 cost comes up. The CD1 case `widen_*` in `cast_decimal.rs` stays similar or gets +faster. + +Acceptance checks: + +- `cargo nextest run -p vortex-array`, `-p vortex-decimal-byte-parts` and `-p vortex-arrow`. +- `cargo test --doc -p vortex-array` +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- Every decimal `Narrow` kernel result equals the result on the canonical array. +- `NarrowArray::try_new` rejects every pair that D10 forbids. +- A cast of a decimal leaf to a precision with the same scale and a wider canonical width returns a + `Narrow` and reads no values. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` passes. + +Review focus: The child label (D10) comes only from a dtype or from `encode_decimal`. Execution +never goes through the cast rule. The sum overflows only against the precision of the parent. The +compare shortcut gets the sign right for negative constants and keeps nulls. Casts that wrap set +the nullability first. Inputs at a non-canonical stored width before CD8a still give correct +results. `DecimalArray::values_type()` (stored) and `DecimalDType::values_type()` (canonical) have +the same name. + +Size: large. One encoding opens for a second type, with four kernels, the cast kernel and broad +tests. + +Open questions: None. Whether a cast of a DBP array, or of another encoded decimal, to a wider +canonical width wraps it in a `Narrow` without decoding is left to the design pass of WR3. + +### CD5: DecimalByteParts decodes to the canonical width + +- Goal: Decoding DecimalByteParts produces the canonical decimal array of its dtype, and every lower + part has the dtype `u64`. Today decoding reuses the MSP buffer at the MSP width, a form that D3 + removes. +- Depends on: CD1, CD3, and the published CD2 fixtures (D31). Soft: FX2. Blocks: CD8a. +- Decisions: D16, D21, D29, D3, D12, D14. +- Branch: `ct/dbp-canonical-decode`, based on `develop` after CD3 merges, or stacked on CD3. +- Scope: + - Decoding to the canonical width in `assemble_decimal` and in the CUDA decoder. + - The `u64` rule for lower parts, and the bound on the lower-part count (D29), in the + constructor, the v2 reader and the assembler. + - The module and type docs, and the tests and the bench that build narrow lower parts. +- Out of scope: + - The max pass that returns a `Narrow` (LT2). This PR adds only its `TODO(connor)`. + - Sum, min and max pushdown into the MSP (LT3). + - The compressor scheme (CD7). + - The nullability cast fix in `compute/cast.rs` (FX2). + - Strict `DecimalArray` construction (CD8a). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:16` | The doc line that allows narrower parts changes. The MSP is a signed integer of at most 64 bits, and every lower part is `u64` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:62-63` | `LOWER_PART_DTYPE` doc: the dtype of every lower part, not "before any narrowing" | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:57-58` | Type doc: only the MSP can use a narrower dtype | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:105-118` | `DecimalBytePartsData::validate` requires `part.dtype() == &LOWER_PART_DTYPE`, and a lower-part count whose assembled width is at most the canonical width (D29) | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:139-149`, `:167-173` | Constructor docs and the `new_unchecked` safety text name `u64` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:372` | Comment on "narrowed lower value" goes | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:32-160` | Canonical-width assembly (design notes), the same `u64` check as the constructor, the `TODO(connor)`, and removal of the lower-part cast and its `TODO(mk)` at `:107-116` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs:92-104` | The v2 reader requires `PType::U64` for every lower part | +| `encodings/decimal-byte-parts/src/lib.rs:7-14` | Crate doc: narrowing applies to the MSP only | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs`, `assemble.rs`, `plugin/tests.rs` | Tests (see Tests) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:158-179` | `dbp_assemble_narrowed` uses `u64` lower parts | +| `vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:43-118` | Canonical width on the GPU, and tests | +| `vortex-cuda/src/arrow/canonical.rs:507` | Move `decimal_cast` to a crate-private module that the export and the DecimalByteParts decoder share | +| `vortex-cuda/kernels/src/decimal_cast.cu:46-102` | Add an `int16_t` output and the `decimal_cast__i16` kernels | + +Design notes: + +`assemble_decimal(msp, lower_parts, decimal_dtype, ctx)` keeps its signature. It returns a decimal +array whose width is `T = decimal_dtype.values_type()`. The assembled width `A` is the MSP width +with no lower parts, `i128` with one lower part, and `i256` with two or three. + +| Lower parts | Case | Result | +| --- | --- | --- | +| 0 | `A == T` | Reuse the MSP buffer. No copy and no scan | +| 0 | `A < T` | Widening copy into `T`. This branch holds the `TODO(connor)` | +| 0 | `A > T` | `copy_to_canonical_width` into `T` (only for precision 1 to 9, because the MSP is at most `i64`) | +| 1 | `T == i128` | Assemble into `i128`, as today | +| 1 | `T == i256` | Assemble straight into `i256` with `assemble_wide_decimal::`, which sign-extends the MSP | +| 2 or 3 | `T == i256` | Assemble into `i256`, as today | + +- The constructor rejects a lower-part count whose assembled width is wider than the canonical + width (D29). So precision up to 18 allows no lower parts, precision 19 to 38 allows at most one, + and precision 39 to 76 allows any count. Assembly with lower parts then never needs a narrowing + copy. No file breaks, because no edition permits v2. +- A path without a copy trusts the precision (D14). A narrowing copy checks it (D21). A widening + copy from a narrower width cannot leave the precision. +- Build every result with `DecimalArray::new_unchecked`. The SAFETY comment says that the width is + the canonical width and that reads trust the precision. So CD8b adds no scan to decoding. +- The `TODO(connor)` at the widening branch reads: + + ```rust + // TODO(connor): Find the max of a narrow MSP and return a `Narrow` over a child that reuses + // its buffer, once benchmarks show that this widening copy costs too much. + ``` + +- The `u64` rule: the constructor, `assemble_decimal` and the v2 reader reject `u8`, `u16` and `u32` + lower parts. The error text names `u64`. No file breaks: no edition permits v2, and the v2 compat + fixture is disabled + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`). +- `scalar_at` (`array.rs:360-401`) keeps returning `DecimalValue::I64` with no lower parts. A scalar + accepts any width that fits its precision (`vortex-array/src/scalar/validate.rs:58-60`). +- CUDA: `decimal_byte_parts.rs:49-65` reuses the MSP buffer at the MSP width today. Keep the error + for lower parts at `:45-47`. With `A == T`, reuse the buffer. With `A < T`, widen on the device + with `decimal_cast::`. Its kernel has outputs `i32`, `i64`, `i128` and `i256` + (`decimal_cast.cu:80-102`), so precision 3 to 4 with an `i8` MSP needs the new `i16` output. With + `A > T`, return an error, as for lower parts. A device narrowing copy needs an overflow flag that + synchronizes with the host (`vortex-cuda/src/arrow/canonical.rs:433-437`). +- Keep the CUDA change minimal and in its own commit, and ask for a review from someone who + knows the CUDA code. If it grows beyond the device widening and the new `i16` output, move it + to its own PR. + +Steps: + +1. Require `u64` lower parts in the constructor, the assembler and the v2 reader. Fix the docs. + Remove the lower-part cast. Update the tests and the bench that build narrow lower parts. +2. Make `assemble_decimal` return the canonical width, as the table shows, with the `TODO(connor)`. + Add the CPU tests. +3. In a separate commit, make the CUDA decoder return the canonical width. Add the `i16` kernel + output and the GPU tests. + +Tests: + +- New in `assemble.rs`: + - `test_assemble_returns_canonical_width` (`rstest`): MSP ptype and precision `(I8, 2)` reuse, + `(I8, 4)`, `(I32, 18)`, `(I64, 38)`, `(I16, 76)` widen, and `(I64, 9)`, `(I32, 2)` narrow, each + with and without nulls. It asserts `values_type()` against `decimal_dtype.values_type()` and the + values. + - `test_assemble_reuses_msp_buffer`: pointer equality when the MSP width is the canonical width. + - `test_assemble_narrowing_copy_zeroes_null_slots`: an `i64` MSP at `Decimal(2, 0)` with 1000 in a + null slot gives 0 in that slot. + - `test_assemble_narrowing_copy_rejects_value_outside_precision`: an `i64` MSP at `Decimal(2, 0)` + with the valid value 100 fails. 100 fits `i8` but not precision 2. + - `test_assemble_widening_sign_extends`: an `i8` MSP with -1 at precision 40 gives `i256` -1. + - `test_assemble_lower_parts_canonical_width` (`rstest`) with these cases: + - One lower part at precision 40 gives `i256`. + - One lower part at precision 38 gives `i128`. + - Three lower parts at precision 76 give `i256`. +- New in `array.rs`: cases `u8_lower_part`, `u16_lower_part` and `u32_lower_part` in + `test_rejects_invalid_parts` (`:661-679`). Also `one_lower_part_at_p18` and + `three_lower_parts_at_p38`, which break the bound of D29. +- New in `plugin/tests.rs`: case `v2_narrow_lower_part` with `PType::U32` in + `decoder_rejects_malformed_payloads` (`:151-207`). +- Changed: + - `array.rs:579-651` `test_independently_narrowed_parts` becomes + `test_narrow_msp_with_u64_lower_parts`. It keeps every MSP ptype and lower-part count, with the + high bit set in each word, and drops the narrow lower ptypes. + - `plugin/tests.rs:66-85` cases `narrowed_lower_parts` and `nullable_mixed_lower_parts` keep their + narrow MSPs and use `u64` lower parts. + - `plugin/tests.rs:46-60`: case `wider_i64_storage` builds an `i64` MSP with no lower parts at + `Decimal(2, 0)`, so it covers the narrowing decode without a non-canonical `DecimalArray`. Cases + `wider_i128_storage` and `wider_i256_storage` become rejection cases, because one or three + lower parts at precision 2 break the bound of D29. + - `plugin/tests.rs:61-65` builds three lower parts at precision 38. It moves to precision 76. + - `plugin/tests.rs:182-187` case `v2_signed_lower_part` expects the new message. + - `assemble.rs:369-386` `test_split_narrow_decimal_reuses_values` uses `i32` at `Decimal(9, 0)` + and asserts reuse on split and on decode. + - `assemble.rs:388-409` `test_assemble_rejects_invalid_lower_dtype` gets a `u32` case. + - CUDA `decimal_byte_parts.rs:90-118` asserts the canonical width. New cases: `i8` at precision 3 + (the new `i16` kernel), `i64` at precision 40, `i32` at precision 9 (reuse), and `i64` at + precision 4 (error). +- No existing test asserts a narrow decoded width. The `values_type()` asserts at `array.rs:703`, + `:707`, `:716`, `:729`, `assemble.rs:266` and `prop_tests.rs:144` expect canonical widths. Outside + the crate, a search for `values_type()` asserts found no decoded DecimalByteParts width. Not + verified for tests that read buffers with `buffer::()` after a decode. +- `prop_tests.rs` needs no change, because `draw_encoded` draws `u64` lower parts (`:113-119`). + +Benchmarks: CD1's DecimalByteParts decoding with a narrow MSP gets slower, because a widening copy +replaces the reuse. Compute on decoded DecimalByteParts runs at the canonical width, so it gets +slower for columns with a narrow MSP. D19 accepts both until the compressor narrows the decimal type +first (WR3). An Arrow export of such a column copies twice: once to decode and once to widen for +Arrow. An MSP at the canonical width and arrays with lower parts show no change. The +`dbp_assemble_kernel*` benches call slice kernels and do not change. `dbp_assemble_narrowed` is not +in CodSpeed (`dbp_assemble.rs:105`). + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings` +- `cargo nextest run -p vortex-decimal-byte-parts` and + `cargo test --doc -p vortex-decimal-byte-parts` +- `cargo +nightly-2026-09-10 fmt -p vortex-decimal-byte-parts -p vortex-cuda` +- `cargo nextest run -p vortex-btrblocks --all-features` and `cargo nextest run -p vortex-file`, + whose round trips decode DecimalByteParts. +- On a CUDA host: `cargo clippy -p vortex-cuda --all-targets --all-features -- -D warnings`, + `cargo nextest run -p vortex-cuda`, and + `clang-format --style=file -i vortex-cuda/kernels/src/decimal_cast.cu`. +- `cargo run -p vortex-compat --release -- check --dir --mode exact` +- Decoding gives `values_type() == decimal_dtype.values_type()` for every MSP width and part count. +- The reuse path copies nothing. A narrowing copy writes zeros in null slots and fails on a valid + value outside the precision. The constructor and the v2 reader reject a lower part that is not + `u64`. + +Review focus: + +- Sign extension from an `i8` MSP into `i256`, and one lower part at precision 39 or more. +- Null slots in the narrowing copy, and no scan on the reuse path. +- The place of the `TODO(connor)`, and the CUDA kernel names (`decimal_cast_i8_i16`). + +Size: medium. One crate changes, plus a small CUDA change, with many test updates. + +Open questions: None. The constructor bounds the lower-part count (D29). On the GPU, an MSP wider +than the canonical width returns an error, because the btrblocks scheme never writes one. + +### CD6: The `vortex.decimal` read plugin converts old nodes + +- Goal: An old `vortex.decimal` node has a stored width that differs from the canonical width of its + dtype. The reader converts it into the canonical array or a decimal `Narrow`. After this PR no + read produces a decimal leaf with a non-canonical width. +- Depends on: FX1, CD3, CD4, and the published CD2 fixtures (D31). Soft: CD5. Blocks: CD8a. +- Decisions: D7, D9, D21, D14, D10, D24, D31. +- Branch: `ct/decimal-read-plugin`, based on `develop` after FX1, CD3 and CD4 merge, or stacked on + CD4. +- Scope: + - A new `DecimalPlugin` for `vortex.decimal` with the canonical, wider-node and narrower-node + paths. + - Its `is_supported_encoding` override and its registration. + - A `Decimal` VTable `deserialize` that rejects a non-canonical width. + - Unit tests with hand-built old nodes. +- Out of scope: + - Stored statistics in plugins (rejected, D9). + - DecimalByteParts (CD5) and strict construction (CD8a). + - The `vortex.narrow` wire format and a serializer for `Narrow` (WR1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:151-186` | Move the body of `deserialize` into a module-private function that returns the stored width, the buffer and the validity. `Decimal::deserialize` accepts only the canonical width, and otherwise fails with an error that names `DecimalPlugin` | +| `vortex-array/src/arrays/decimal/vtable/plugin/mod.rs` (new) | `DecimalPlugin` | +| `vortex-array/src/arrays/decimal/vtable/plugin/tests.rs` (new) | Tests | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:31-33` | `mod plugin` and `pub use plugin::DecimalPlugin` | +| `vortex-array/src/arrays/decimal/mod.rs:13-16` | `pub use vtable::DecimalPlugin` | +| `vortex-array/src/session/mod.rs:24`, `:92` | Register `DecimalPlugin` in place of `Decimal` | + +Design notes: + +```rust +/// Reads every `vortex.decimal` wire form into a canonical decimal array or a decimal `Narrow`. +pub struct DecimalPlugin + +impl ArrayPlugin for DecimalPlugin { + fn id(&self) -> ArrayId // `vortex.decimal` + fn serialize(&self, array: &ArrayRef, session: &VortexSession) + -> VortexResult> // delegates to `Decimal` + fn deserialize(&self, parts: ArrayDeserialization<'_>, session: &VortexSession) + -> VortexResult + fn is_supported_encoding(&self, id: &ArrayId) -> bool // `Decimal` or `Narrow` +} +``` + +The ALP plugin is the model (`encodings/alp/src/alp/plugin.rs:33-94`). The deserializer asserts +`is_supported_encoding` (`vortex-array/src/serde.rs:412-417`), and so does the Python CUDA path +(`vortex-python-cuda/src/lib.rs:328-333`). Statistics attach after the plugin returns +(`serde.rs:419-424`), so the plugin scans the buffer (D9). + +`deserialize` with the stored width `W_s` and the canonical width +`W_c = decimal_dtype.values_type()`: + +1. Decode the metadata with FX1. Check that the buffer has `len * W_s.byte_width()` bytes and the + alignment of `W_s`. A malformed node returns an error and never panics (D14). +2. If `W_s == W_c`, return today's leaf. No copy and no scan. +3. Otherwise, copy a device buffer to the host with `BufferHandle::try_to_host_sync` + (`vortex-array/src/buffer.rs:295-312`). CUDA reads decode device segments through plugins + (`vortex-cuda/src/layout.rs:241-247`). Then execute the validity mask once with + `session.create_execution_ctx()`, as the ALP plugin does (`plugin.rs:84`). +4. If `W_s > W_c` (a wider node), `copy_to_canonical_width` into `W_c` and return the canonical + array. This path needs no max scan. +5. If `W_s < W_c` (a narrower node): + 1. Make one pass over the valid values for their min and max. Use a fold for an all-true mask and + `for_each_set_index` otherwise (`STYLE.md:108-133`). Do not use the `min_max` aggregate: it + builds decimal scalars and panics on a value outside the precision (see CD7). + 2. Pass the range, or no range when no value is valid, to the best-child-dtype function of CD4 + (see Shared helpers). It returns `Decimal(p*, s)`, with the smallest precision that holds the + range and the scale. + 3. `p*` never exceeds `p`, because the largest magnitude of a width below `W_c` is below + `10^p - 1`. + 4. Let `W* = values_type(p*)`. If `W* == W_c`, return the canonical array, a widening copy from + `W_s`. + 5. Otherwise build the child `Decimal(p*, s)` at `W*`. Reuse the buffer if `W* == W_s`, make a + widening copy if `W* > W_s`, and `copy_to_canonical_width` if `W* < W_s`. Return + `NarrowArray::try_new(child, parent_dtype)`. +6. Build every decimal array with the unchecked constructors. Reuse trusts the precision (D14). A + narrowing copy checks it, and a widening copy from a narrower width cannot leave it. + +The brief picks the best precision exactly, not a larger one. D9 allows a larger one. But D21 makes +a narrowing copy when the best precision has a smaller width than the stored width. A tight child +label also serves the compare shortcut of CD4. + +| Case | Result | +| --- | --- | +| All values null | `p*` comes from the scale alone. A child at `values_type(p*)`, a copy when that width differs from `W_s` | +| Empty array | The same rule, with an empty child | +| Negative values | The min counts: `i8` -128 needs precision 3, so the child is `i16` | +| Scale 0 or negative | No lower bound from the scale | +| `Decimal(18, 10)` at `i8` with values up to 99 | `p* = 10`, `W* = i64 = W_c`, so the canonical array | +| `Decimal(4, 2)` at `i8` with values up to 127 | `p* = 3`, `W* = i16 = W_c`, so the canonical array at `i16` | +| `Decimal(18, 2)` at `i16` with values up to 9,999 | `p* = 4`, child `Decimal(4, 2)` at `i16`, buffer reused | +| `Decimal(18, 2)` at `i32` with values up to 99 | `p* = 2`, child `Decimal(2, 2)` at `i8`, narrowing copy | +| `Decimal(18, 2)` at `i16` with 32,767 | `p* = 5`, child `Decimal(5, 2)` at `i32`, widening copy | +| `Decimal(40, 2)` at `i128` (old v1 compressor output) | A child at `i128` with reuse, or the canonical `i256` array when the values need 39 digits | +| `Decimal(4, 2)` at `i32` with the valid value 20,000 | A wider node. The narrowing copy fails | +| Nonzero values in null slots | The scan ignores them. Reuse and widening keep them, and a narrowing copy writes zeros | +| Unknown stored width | An error (FX1) | + +Steps: + +1. Add `DecimalPlugin` with the canonical path, the `is_supported_encoding` override and the + registration. Make `Decimal::deserialize` reject a non-canonical width. +2. Add the wider-node path. +3. Add the narrower-node path. +4. Add the tests. + +Tests: + +- New in `vtable/plugin/tests.rs`. They build nodes with `ArrayDeserialization::new` and + hand-written `DecimalMetadata`, as + `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs:317-328` does. They build no + non-canonical `DecimalArray`, so they survive CD8a. + - `reads_old_node` (`rstest`) with every row of the table above. It asserts the encoding + (`Decimal` or `Narrow`), the child dtype and width, buffer reuse by pointer, and the values. + - `wider_node_rejects_value_outside_precision`, `narrower_node_ignores_null_slots`, + `unknown_values_type_fails` and `short_buffer_fails`. + - `decode_through_serialized_array`: serialize a canonical `Decimal(2, 0)` array and decode it as + `Decimal(10, 0)`, which is an old narrow node. Serialize a canonical `Decimal(38, 0)` array and + decode it as `Decimal(5, 0)`, which is a wider node. This covers the assert at + `serde.rs:412-417` without a non-canonical array. +- Unchanged: `vtable/mod.rs:241-273` `test_array_serde` builds `i128` at `Decimal(10, 2)`. It now + reads through the narrowing copy and passes. CD8a rewrites it. +- The compat check reads CD2's fixtures through this plugin. + +Benchmarks: None. Old narrow nodes are rare, and the speed of this path matters little (D9). + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo nextest run -p vortex-array` and `cargo test --doc -p vortex-array` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-file` +- `cargo run -p vortex-compat --release -- check --dir --mode exact` +- A node at the canonical width reads with no copy and no scan. Every old node reads into the + canonical array or a `Narrow` with a true child label. A malformed node returns an error. + +Review focus: + +- The child label, where both the min and the max count, and the scale bound. +- The choice between reuse and copy. +- The `is_supported_encoding` override, no panic on malformed input, and the host copy of device + buffers. + +Size: medium. One new module and one registration change. + +Open questions: None. A read can now return an in-memory `Narrow`. Until WR1, the serializer writes +its canonical array (D26). + +### CD7: One DecimalByteParts scheme on the canonical array + +- Goal: The btrblocks decimal scheme takes the canonical array and chooses the number of lower parts + from the values. It narrows the MSP as the compressor narrows other internal children. + `narrowed_decimal` goes. File contents stay the same, except for precision 39 to 76 with values + that need more than 64 bits in v1 mode. +- Depends on: NR1 (`encode_values`). Soft: CD3, CD5. Blocks: CD8a, WR3. +- Decisions: D17, D23, D22, D19, D11, D28. +- Branch: `ct/dbp-scheme-canonical`, based on `develop` after NR1 merges. +- Scope: `lower_part_count`, `split_decimal` with an explicit count, `DecimalByteParts::encode`, + `DecimalScheme::compress`, the removal of `narrowed_decimal`, docs, golden corpus entries and + tests. +- Out of scope: + - The decimal `Narrow` scheme (WR3) and DecimalByteParts v2 in an edition (WR2). + - Narrowing the MSP when lower parts exist. It changes the v2 output, so WR2 or WR3 decides. + - A shape with two lower parts, and changes to the scheme estimate. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-btrblocks/src/schemes/decimal.rs:43-66`, `:121`, `:125-165` | Docs, the estimate comment, and the new `compress` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:30-142` | `lower_part_count`, `split_decimal` with a count, a split of `i256` into one lower part, and `DecimalByteParts::encode` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:38-47` | Export `lower_part_count` | +| `encodings/decimal-byte-parts/src/lib.rs:7-14` | Crate doc: the MSP narrows, the decimal does not | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/testing.rs:23-38` | `i128_parts` and `i256_parts` pass the counts 1 and 3 | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:257`, `:303`, `:318`, `:376`, `split.rs:271`, `:295` | Callers of `split_decimal` pass a count | +| `encodings/decimal-byte-parts/benches/dbp_split.rs:130-138`, `dbp_assemble.rs:145-156` | Callers pass the natural count of the width | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_v2.rs:26-37` | `encode_byte_parts` calls `DecimalByteParts::encode` | +| `vortex-array/src/arrays/decimal/utils.rs:4-5`, `:91-144` | Remove `narrowed_decimal`, `try_downcast!` and their imports | +| `vortex-btrblocks/tests/golden.rs:142-165`, `:369-374` | New decimal corpus entries | +| `vortex-btrblocks/tests/snapshots/` | New snapshots | +| `vortex-btrblocks/tests/decimal_config.rs` | New scheme tests | + +Design notes: + +```rust +/// The number of lower parts that the valid values need: 0 when every valid value fits `i64`, 1 +/// when every valid value fits `i128`, and 3 otherwise. +pub fn lower_part_count(decimal: &DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult + +/// Split a decimal array into an MSP and `lower_part_count` lower parts. +pub fn split_decimal( + decimal: &DecimalArray, + lower_part_count: usize, + ctx: &mut ExecutionCtx, +) -> VortexResult +``` + +- `lower_part_count` returns 0 without a scan for a width up to `i64`. For `i128` and `i256` it + executes the mask once and scans the valid values. It must not use the `min_max` aggregate: that + aggregate builds decimal scalars with `Scalar::decimal` + (`vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:61-80`), which panics on a value outside + the precision (`vortex-array/src/scalar/constructor.rs:68-78`). Decoded data trusts the precision, + and `prop_tests.rs:104-132` draws such values. +- `split_decimal` accepts these shapes: + - A width up to `i64` with count 0. The MSP reuses the buffer, as today (`split.rs:108-111`). + - `i128` with count 0, a copy into `i64` with zeros in null slots, or count 1 with `split_wide`. + - `i256` with count 0, count 1 with a new `i256_to_i128_parts`, or count 3, as today. + - A count below the need of a valid value returns an error, because the copy reads every value. + Other counts return an error. + - No valid row gives constant parts with the requested count, as today (`split.rs:123-142`). +- `DecimalByteParts::encode` is `split_decimal(decimal, lower_part_count(decimal, ctx)?, ctx)`. +- `DecimalScheme::compress`: + 1. Execute the input to the canonical `DecimalArray`. + 2. Compute `lower_part_count`. + 3. In v1 mode with a count above 0, return the input array. The cascade keeps its input when the + output is not smaller (`vortex-compressor/src/compressor/cascade.rs:338-351`), so the file + holds the canonical leaf. + 4. Call `split_decimal` with the count. + 5. With count 0, narrow the MSP with `NarrowArray::encode_values`. The call carries + `#[expect(deprecated)]` with the `TODO(connor)` reason of NR6 (D28). + 6. Compress the children and build the array, as today (`decimal.rs:140-164`). +- The MSP narrowing uses `NarrowArray::encode_values` of NR1, not `PrimitiveArrayExt::narrow`. + `narrow` picks an unsigned type for non-negative values + (`vortex-array/src/arrays/primitive/array/mod.rs:209-230`). An unsigned MSP fails + `DecimalBytePartsData::validate` (`array.rs:97-99`). The widths also change: values from 0 to 200 + get `U8`, where `narrowed_decimal` gives `I16`. `encode_values` keeps the signedness, so it gives + the width of `narrowed_decimal`: the smallest signed type that holds the min and the max + (`utils.rs:91-121`). So CD7 lands after NR1. CD7 changes no golden snapshot of other internal + children. + +File contents, compared with today: + +| Input | Today | After CD7 | +| --- | --- | --- | +| Width up to `i64` | MSP at the width of `narrowed_decimal` | The same width from `encode_values` | +| `i128` or `i256` with values that fit `i64` | Narrowed decimal, then the MSP | A copy into `i64`, then `encode_values`: the same width | +| Precision 19 to 38, values above 64 bits | v1: canonical `i128` leaf. v2: `i64` MSP and one lower part | The same | +| Precision 39 to 76, values from 65 to 128 bits | v1: `vortex.decimal` at `i128`. v2: `i64` MSP and one lower part | v1: canonical `i256` leaf (the accepted change, D17). v2: the same | +| Precision 39 to 76, values above 128 bits | v1: canonical `i256`. v2: `i64` MSP and three lower parts | The same | +| Null slots that hold values outside the range of the valid values | `narrowed_decimal` counts them (`utils.rs:96`) | `encode_values` ignores them, so the MSP can be narrower. The maintainer accepts this | +| One-row arrays | `narrowed_decimal` does not narrow them (`utils.rs:98-99`) | Unreachable: the cascade handles empty, all-null and constant arrays first (`cascade.rs:281-298`) | + +- Compact mode uses the same scheme. Only the excluded schemes differ + (`vortex-btrblocks/src/builder.rs:46-63`), so the MSP can use Pco. The MSP is the same array, so + the output is the same. +- The CUDA mode excludes the v2 ID (`builder.rs:94-101`), so it takes the v1 path. + +Steps: + +1. Add the decimal golden corpus entries on the old code, in their own commit. +2. Add `lower_part_count` and the explicit count of `split_decimal`. Update `encode`, the test + helpers, the benches and the compat fixture. +3. Rewrite `DecimalScheme::compress`. Delete `narrowed_decimal`. Fix the docs. +4. Accept the snapshot change of the precision-40 entry only. + +Tests: + +- New golden entries in `vortex-btrblocks/tests/golden.rs`, added in step 1 on the old code: + - `decimal_wide_dtype_small_values`: `Decimal(38, 2)` with `i128` values below 10,000,000. + - `decimal_nullable`: `Decimal(12, 2)` with a tenth of the rows null and zeros in null slots. + - `decimal_negative`: `Decimal(10, 2)` with values from -100,000 to 100,000. + - `decimal_wide_p40`: `Decimal(40, 2)` with values from 2^64 to 2^100. +- After step 3, only `regular__decimal_wide_p40` and `compact__decimal_wide_p40` change, from an + `i128` leaf to an `i256` leaf. `regular__decimal_prices` and `compact__decimal_prices` do not + change. +- New in `vortex-btrblocks/tests/decimal_config.rs`: + - `decimal_scheme_part_shapes` (`rstest`) for v1 and v2. It asserts the encoding, the lower-part + count and the MSP ptype, and it round-trips through serialization. Its cases: + - `Decimal(12, 2)` with small values. + - `Decimal(38, 2)` with values that fit `i64`, and with values above 64 bits. + - `Decimal(40, 2)` with values that fit `i64`, and with values from 65 to 128 bits. + - `Decimal(76, 2)` with values above 128 bits. + - A nullable case. + - `msp_keeps_signedness`: values from 0 to 200 at `Decimal(10, 0)` give an `I16` MSP. +- New in `split.rs`: + - `lower_part_count` cases for each width, a null slot that holds `1 << 100`, all nulls, and an + empty array. + - `lower_part_count` on values outside the precision does not panic. + - `split_decimal` with a count that is too small fails. + - `i256` values that fit `i128` split into one lower part and round-trip. +- Unchanged: `decimal_scheme_follows_allowed_ids` and `wide_decimal_parts_roundtrip` in + `decimal_config.rs`, `refine_picks_v2_when_v2_id_is_allowed` in `decimal.rs`, + `decimal_writer_uses_default_or_session_restrictions` in `vortex-file/src/tests.rs`, and both + property tests in `prop_tests.rs`. +- The golden tests are `golden_regular`, `golden_onpair` and `golden_compact` + (`vortex-btrblocks/tests/golden.rs:469-499`). `golden_onpair` holds no decimal entry. + `golden_compact` needs the `zstd` and `pco` features. + +Benchmarks: No CodSpeed bench covers decimal compression. A search of `vortex-btrblocks/benches` and +`vortex-compressor/benches` for "decimal" found none. File contents and decoding cost do not change. +Compression of a wide canonical array with small values makes more passes than today. It scans, +copies into `i64`, computes a min and max, and casts. + +Acceptance checks: + +- `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings`, and the same for + `-p vortex-decimal-byte-parts` and `-p vortex-array` +- `cargo nextest run -p vortex-btrblocks --all-features` +- `cargo nextest run -p vortex-decimal-byte-parts`, `cargo nextest run -p vortex-file` +- `cargo +nightly-2026-09-10 fmt -p vortex-btrblocks -p vortex-decimal-byte-parts`, and the same for + `-p vortex-array` and `-p vortex-compat` +- `cargo clippy -p vortex-compat --all-targets -- -D warnings` (the v2 fixture compiles) +- The snapshot diff of step 3 touches only the two `decimal_wide_p40` snapshots. + +Review focus: + +- Equal file contents, and the null-slot difference. +- The signedness of the MSP, and the v1 early return. +- No `min_max` aggregate on decimal arrays. + +Size: medium. Two crates change, with new tests and snapshots. + +Open questions: None. The maintainer accepts the null-slot difference in the file contents. + +### CD8a: Every `DecimalArray` has the canonical width + +- Goal: Every decimal array has the canonical width of its dtype. Typed constructors reject another + width, the iterator constructors and the builder convert, and the last non-canonical producers go. +- Depends on: CD3, CD5, CD6, CD7. Soft: CD4. Blocks: CD8b, CD8c. +- Decisions: D3, D12, D14 (the width half), X8. +- Branch: `ct/decimal-canonical-width`, based on `develop` after CD3, CD5, CD6 and CD7 merge. +- Scope: + - The width check in `VTable::validate` and in every constructor. + - `from_iter` and `from_option_iter` that convert, and `DecimalBuilder` without a width parameter. + - Constant canonicalization, and the removal of `upcast_decimal_values` and the fill-null upcast + branch. + - Docs, tests, and the compat fixture builders of CD2. +- Out of scope: + - The value check (CD8b), and branches that this PR makes unreachable (CD8c). + - Scalar normalization at construction (future work), and export rewrites (CD3 or CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:123-149` | `validate` checks `values_type == decimal_dtype.values_type()` | +| `vortex-array/src/arrays/decimal/array.rs:188-343` | `DecimalData` constructors check the width and become crate-private. No caller outside the decimal module uses them | +| `vortex-array/src/arrays/decimal/array.rs:404-531` | `Array` constructors: the width rule, and conversion in `from_iter` and `from_option_iter` | +| `vortex-array/src/arrays/decimal/array.rs:53-109` | Type doc: one width for each precision | +| `vortex-array/src/builders/decimal.rs:33-145` | The builder takes its width from the dtype | +| `vortex-array/src/builders/mod.rs:412-418`, `vortex-array/src/arrays/arbitrary.rs:146-147` | Builder calls without the type parameter | +| `vortex-array/src/arrays/constant/vtable/canonical.rs:113-121` | The buffer uses the canonical type, not the type of the scalar value | +| `vortex-array/src/arrays/decimal/compute/cast.rs:439-489` | Remove `upcast_decimal_values` and its tests at `:1065-1170` | +| `vortex-array/src/arrays/decimal/compute/fill_null.rs:54-72` | `cast::().vortex_expect(...)` in place of the upcast branch | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/` | Delete the `build_written` overrides of CD2, because they can no longer build narrow leaves. The published files keep the old forms | +| Tests across the workspace | See Tests | + +Design notes: + +- The width rule is `values_type == decimal_dtype.values_type()`: + - `VTable::validate` checks it. This covers `Array::try_from_parts`, deserialization and + `with_buffers`. + - `try_new::` and `try_new_handle` return an error that names the dtype, the canonical width + and the given width. + - `new::` and `new_handle` panic through `vortex_expect`, as today. + - `new_unchecked::`, `new_unchecked_handle` and `new_unchecked_from_byte_buffer` keep the check + with `vortex_panic!`. The check costs nothing for each row, and a mismatch is a kernel bug + (`STYLE.md:54`). +- `DecimalArray::new::` with a `T` other than the canonical type panics, and `try_new::` + fails. The constructors do not convert silently, because a silent copy hides a wrong `T`. +- `from_iter` and `from_option_iter` keep their generic `T` and collect into the canonical type with + `BigCast`. They panic when a value does not fit the canonical width. CD8b adds the precision + check. +- `DecimalBuilder::new`, `new_in`, `with_capacity` and `with_capacity_in` lose their type parameter. + The buffer variant comes from `decimal.values_type()`. `append_value::` keeps converting. +- Constant canonicalization builds `Buffer::full(value.cast::(), len)` with `T` the canonical + type. A scalar keeps any width that fits its precision (`scalar/validate.rs:58-60`), so a + `DecimalValue::I128` at precision 5 gives an `i128` buffer today. Scalars from DataFusion + (`vortex-datafusion/src/convert/scalars.rs`), DuckDB, Python, FFI and JNI reach arrays through + this path. If CD3 already changed it, skip this item. +- `upcast_decimal_values` builds a decimal array wider than the canonical width by design. Its only + non-test caller is the fill-null upcast branch, which is unreachable: a fill value fits its + precision, so it fits the canonical type. If CD4 started to use the function, replace that use + with the decimal cast. + +Producers of a non-canonical width on `develop`: + +| Producer | Location | Owner | +| --- | --- | --- | +| Arrow import | `vortex-arrow/src/convert.rs:179-226` | CD3 | +| DuckDB import, including the `i16` to `i8` copy of PR #10155 | `vortex-duckdb/src/convert/vector.rs:303-338` | CD3 | +| DecimalByteParts decoding on the CPU | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-160` | CD5 | +| DecimalByteParts decoding on the GPU | `vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:49-65` | CD5 | +| `vortex.decimal` reads | `vortex-array/src/arrays/decimal/vtable/mod.rs:151-186` | CD6 | +| `narrowed_decimal` in the btrblocks scheme | `vortex-array/src/arrays/decimal/utils.rs:91-144`, `vortex-btrblocks/src/schemes/decimal.rs:133-137` | CD7 | +| Typed and iterator constructors | `vortex-array/src/arrays/decimal/array.rs:188-531` | CD8a | +| The builder width parameter | `vortex-array/src/builders/decimal.rs:97-145` | CD8a | +| Constant canonicalization | `vortex-array/src/arrays/constant/vtable/canonical.rs:113-121` | CD8a, unless CD3 | +| `upcast_decimal_values` | `vortex-array/src/arrays/decimal/compute/cast.rs:439-489` | CD8a | + +Checked, and already canonical when the input is canonical: + +- Arithmetic writes the canonical width of the result + (`scalar_fn/fns/binary/numeric/decimal.rs:410-429`). +- The integer to decimal cast targets the canonical width (`arrays/primitive/compute/cast.rs:144`). + Its buffer reuse at `:147-158` applies only when the source width equals it. +- The decimal to decimal cast copies into the canonical target width + (`arrays/decimal/compute/cast.rs:148-169`). The reuse branch at `:128-142` keeps the width only + when it is already the target width. +- Take, filter, mask, slice and masked validity keep the input width + (`arrays/decimal/compute/fixed_width.rs:14-35`, `arrays/filter/execute/take/fixed_width.rs:78`, + `arrays/decimal/compute/mask.rs:20`, `arrays/decimal/compute/rules.rs:47-82`, + `arrays/masked/execute.rs:89-99`). CUDA filter and dictionary kernels keep the input width + (`vortex-cuda/src/kernel/filter/decimal.rs:21-36`, + `vortex-cuda/src/kernel/arrays/dict.rs:220-305`). +- Fill null and patch keep the input width (`arrays/decimal/compute/fill_null.rs:76-85`, + `arrays/decimal/array.rs:537-579`). +- Sparse, run-end and constant-list canonicalization use the canonical type + (`encodings/sparse/src/canonical.rs:314-327`, `encodings/runend/src/compress.rs:386`, + `arrays/constant/vtable/canonical.rs:354-372`). So do the CUDA constant kernel + (`vortex-cuda/src/kernel/arrays/constant.rs:70-75`), the builder factory + (`builders/mod.rs:412-418`) and `arbitrary` (`arrays/arbitrary.rs:146`). +- Python, FFI, JNI and DataFusion build no decimal arrays directly. A search for `DecimalArray::` + and `DecimalBuilder` in `vortex-python/src`, `vortex-ffi/src`, `vortex-jni/src` and + `vortex-datafusion/src` finds only scalar code and the Python subclass. + +Steps: + +1. Add the width rule to `validate` and to every constructor. +2. Convert in `from_iter` and `from_option_iter`. Remove the width parameter of `DecimalBuilder`. +3. Fix constant canonicalization. Remove `upcast_decimal_values` and the fill-null upcast branch. +4. Update the docs, the tests and the compat fixture builders. + +Tests: + +- New: + - `try_new` rejects each non-canonical `T`, with the canonical width in the error. + - `new_unchecked` panics on a wrong width. + - `from_iter` converts a narrower and a wider `T`. + - The builder writes the canonical width. + - Constant canonicalization of a `DecimalValue::I128` at precision 5 gives `i32`. + - `VTable::validate` rejects a wrong width through `Array::try_from_parts`. +- Changed, because they build a non-canonical array with a typed constructor or the builder. This + list comes from a search. The test run finds the rest: + - `vortex-array/src/arrays/decimal/compute/mod.rs:30-118`, including `decimal_narrow_storage` at + `:83-87` (the report cites `:82-86` at `5fc5cb8`). + - `vortex-array/src/arrays/decimal/compute/cast.rs` tests (`:742-1000`), `fill_null.rs:171-195` + (`fill_null_overflow_upcasts` goes), `vtable/operations.rs:32-90`, `vtable/mod.rs:241-273`. + - `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:113-290`, `mean/mod.rs:340-400`, + `all_non_distinct/tests.rs:283-292`, `uncompressed_size_in_bytes/mod.rs:672`, `:782`, + `min_max/mod.rs:945`. + - `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs:421`, `:542`, `:667`, `:818`, + `vortex-array/src/scalar_fn/fns/between/mod.rs:690-740`, + `vortex-array/src/arrays/fixed_width/take/tests.rs:178`. + - `vortex-array/src/builders/decimal.rs:340-440`, `vortex-array/src/builders/tests.rs:96-98`. + - `vortex-arrow/src/executor/decimal.rs:276-400`: the cases over `T` at precision 2 collapse, so + they iterate over precisions instead. + - `vortex-row/src/tests.rs:834`, `encodings/runend/src/array.rs:910-918`, + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:81-89`, + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs:174-179`. + - `vortex-cuda/src/arrow/canonical.rs:2190-2330`. `test_export_decimal_narrowing_errors` and + `test_export_decimal_narrowing_from_arrow_import` go, because `from_iter` now converts their + inputs and the export no longer fails. + - `vortex-cuda/src/kernel/filter/decimal.rs:62-90`: the cases use one precision for every `T`, so + after the conversion they lose their width coverage. Use the largest precision of each width. + - `fuzz/src/array/fill_null.rs:415-508`. +- Unchanged but weaker: tests that use `from_iter` with a non-canonical `T` keep passing on values. + `vortex-array/src/scalar_fn/fns/binary/compare/tests.rs:577-616` no longer mixes widths. CD8c + deletes those tests with their code. + +Benchmarks: `vortex-arrow/benches/to_arrow.rs:117` builds `i64` at `Decimal(19, 2)` with +`from_iter`. The input becomes canonical `i128`, so the export skips its widening copy. CodSpeed +then shows a change that comes from the input, not from the export code. The other decimal benches +already build canonical inputs (`vortex-array/benches/cast_decimal.rs:66-82`, `compare.rs:167-170`, +`binary_ops.rs:395-403`, `filter_fixed_width.rs:145-157`, `expr/case_when_bench.rs:70-80`). + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings` for each changed crate: `vortex-array`, + `vortex-arrow`, `vortex-btrblocks`, `vortex-decimal-byte-parts`, `vortex-runend`, `vortex-row`, + `vortex-duckdb`, `vortex-fuzz`, `vortex-compat`, and `vortex-cuda` on a CUDA host +- `cargo nextest run` for the same crates, and `cargo test --doc -p vortex-array` +- `cargo +nightly-2026-09-10 fmt` for the same crates +- `cargo run -p vortex-compat --release -- check --dir --mode exact` +- No code path builds a decimal array with a width other than the canonical width. + +Review focus: completeness of the producer audit, the panic in the unchecked constructors, the test +rewrites that keep their intent, and the compat fixture builders. + +Size: large. The API change touches many tests in many crates. + +Open questions: None. + +### CD8b: `DecimalArray::try_new` checks every valid value + +- Goal: `try_new` checks that every valid value fits the precision (D14). Kernels that keep values + legal use `new_unchecked`. The builder and the iterator constructors check values. +- Depends on: CD8a. Soft: CD1. Blocks: None. +- Decisions: D14, D13, D30, XQ3, XQ4. +- Branch: `ct/decimal-value-check`, based on `develop` after CD8a merges, or stacked on CD8a. +- Scope: + - A `ctx` parameter on the checked constructors, and the value check. + - Kernel call sites that move to the unchecked constructors. + - The precision check in the builder and the iterator constructors. + - Docs and tests. +- Out of scope: the `ExtensionArray` check (X7, another track), file reads (they keep trusting the + precision), unreachable code (CD8c). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/array.rs:404-531` | `ctx` on `new`, `try_new`, `new_handle`, `try_new_handle`, the value check, and the `new_unchecked*` docs | +| `vortex-array/src/arrays/decimal/utils.rs` | Reuse `check_precision` from CD3 | +| `vortex-array/src/builders/decimal.rs:147-190` | Precision check in `append_value` and `append_n_values`, `new_unchecked` in `finish_into_decimal` | +| Kernel call sites (table below) | Unchecked constructors | +| Callers of the checked constructors | `ctx`, or a move to `from_iter` and `from_option_iter` | + +Design notes: + +```rust +pub fn try_new( + buffer: Buffer, + decimal_dtype: DecimalDType, + validity: Validity, + ctx: &mut ExecutionCtx, +) -> VortexResult + +pub fn try_new_handle( + values: BufferHandle, + values_type: DecimalType, + decimal_dtype: DecimalDType, + validity: Validity, + ctx: &mut ExecutionCtx, +) -> VortexResult +``` + +- `new` and `new_handle` take the same `ctx` and panic on the same errors. +- The check executes the validity mask once (`vortex-array/src/validity.rs:256`). It tests + `MIN_BY_PRECISION[p] <= v <= MAX_BY_PRECISION[p]` for every valid value of the canonical type. A + branchless fold runs first, and a second pass finds the index only on failure, as + `vortex-array/src/arrays/decimal/utils.rs:50-80` does. The error names the index, the value and + the dtype. +- The `ctx` follows `VarBinViewArray::try_new` + (`vortex-array/src/arrays/varbinview/array.rs:230-237`), which executes its validity for its view + check. +- A device buffer skips the value check, as VarBin and ListView skip their host-only checks + (`vortex-array/src/arrays/varbin/array.rs:271-280`, + `vortex-array/src/arrays/listview/array.rs:265-266`). +- `check_precision` serves `try_new`, the imports of CD3 and the integer cast + (`vortex-array/src/arrays/primitive/compute/cast.rs:176-226` has its own check today). +- `from_iter` and `from_option_iter` check each valid value while they convert, and panic on + failure. +- `append_value` and `append_n_values` check the precision and panic, as they already panic on a + value that does not fit the width (`builders/decimal.rs:264-288`). The builder computes the bounds + once. `append_scalar`, `append_decimal_array`, zeros and nulls need no check, because their values + are legal. +- The `new_unchecked*` docs say: every valid value must fit the precision. A violation gives wrong + results, not memory unsafety. File reads use these constructors and trust the precision (D14). + +Kernel call sites that move to the unchecked constructors: + +| Location | Why the values are legal | +| --- | --- | +| `vortex-array/src/arrays/decimal/compute/fixed_width.rs:29` | Take and filter copy legal values | +| `vortex-array/src/arrays/decimal/compute/fill_null.rs:84` | Legal values and a fill value that fits the precision | +| `vortex-array/src/arrays/decimal/compute/cast.rs:274` | `cast_decimal_values` checks the target bounds | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:418`, `:427` | The kernel checks the precision bound | +| `vortex-array/src/arrays/primitive/compute/cast.rs:252` | The cast checks the precision | +| `vortex-array/src/builders/decimal.rs:187` | `append_value` checks | +| `vortex-array/src/arrays/constant/vtable/canonical.rs:369` | Scalars fit their precision | +| `encodings/runend/src/compress.rs:386` | Run-end decoding copies legal values | +| `vortex-cuda/src/kernel/filter/decimal.rs:31`, `vortex-cuda/src/kernel/arrays/dict.rs:294`, `vortex-cuda/src/kernel/arrays/constant.rs:146`, `:182` | Device kernels copy legal values | +| The imports of CD3 | They check their values themselves (D13) | + +Steps: + +1. Reuse `check_precision` from CD3. +2. Add `ctx` and the value check to the checked constructors. +3. Move the kernels in the table to the unchecked constructors. +4. Add the check to the builder and the iterator constructors. +5. Update the callers and the tests. + +Tests: + +- New: + - `try_new` rejects a valid value outside the precision with `NonNullable`, `AllValid` and + `Validity::Array`. + - `try_new` accepts such a value in a null slot, and anything with `AllInvalid`. + - `try_new` accepts `MIN_BY_PRECISION[p]` and `MAX_BY_PRECISION[p]`. + - `from_iter` and `append_value` panic on a value outside the precision. +- Changed: every caller of `new` and `try_new` passes `ctx` or moves to `from_iter` and + `from_option_iter`. A search finds 219 lines that call `DecimalArray::new(` or + `DecimalArray::try_new(`, most of them in tests. +- Changed: tests that build values outside the precision on purpose. + `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:328-367` uses `i128::MAX` at + precision 38 and `i256::MAX` at precision 76. `split.rs:263-278` uses `i256::MIN` at precision + 76. Test these extremes on the slice kernels `split_wide` and `assemble_wide_decimal`. Keep the + round trips through `DecimalArray` inside the precision. Other candidate files mix decimals with + integer extremes. Not verified line by line: + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs`, `plugin/tests.rs`, + `encodings/runend/src/compute/sum/tests.rs`, `fuzz/src/array/sum/tests.rs`, + `vortex-array/src/aggregate_fn/fns/all_non_distinct/tests.rs`, `sum_v2/tests.rs`, + `sum/decimal.rs`, `vortex-array/src/arrays/decimal/compute/cast.rs`, + `vortex-array/src/arrays/primitive/compute/cast.rs`, + `vortex-array/src/scalar_fn/fns/between/mod.rs`, + `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs`, + `vortex-array/src/scalar_fn/fns/list_contains/prepared/tests.rs`, + `vortex-cuda/src/arrow/canonical.rs`, `vortex-duckdb/src/convert/vector.rs`, + `vortex-file/src/tests.rs`, `vortex-row/src/tests.rs`. +- The property tests in `prop_tests.rs` need only the `ctx` argument. + `decoded_survives_encode_then_decode` draws values inside the precision (`:25-30`). + `encoded_survives_decode_then_encode` draws arbitrary parts (`:104-132`), and decoding uses the + unchecked constructors. + +Benchmarks: The expected CodSpeed result is no change in +`vortex-array/benches/filter_fixed_width.rs`, `cast_decimal.rs`, `binary_ops.rs`, `compare.rs`, +`expr/case_when_bench.rs`, `vortex-arrow/benches/to_arrow.rs` and the CD1 benches. A regression +points at a kernel that still calls a checked constructor. + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings`, `cargo nextest run` and + `cargo +nightly-2026-09-10 fmt` for each changed crate, as in CD8a +- `cargo test --doc -p vortex-array` +- Kernels run no value scan. A checked constructor fails on a valid value outside the precision and + accepts any value in a null slot. + +Review focus: + +- Every kernel in the table uses an unchecked constructor. +- The host-only rule and the null-slot rule. +- No per-element hidden cost in the check (`STYLE.md:108-133`). + +Size: large. The signature change touches many call sites. + +Open questions: None. `try_new` takes an `ExecutionCtx`, as `VarBinViewArray::try_new` does (D30). + +### CD8c: Remove the code for mixed decimal widths + +- Goal: Delete the code that handles a decimal array whose width differs from its dtype. CD8a makes + this code unreachable. +- Depends on: CD8a. Soft: CD8b. Blocks: None. +- Decisions: D3, D14. +- Branch: `ct/decimal-remove-width-workarounds`, based on `develop` after CD8a merges. +- Scope: the items in the table, and their tests. +- Out of scope: the narrowing arms of the Arrow export + (`vortex-arrow/src/executor/decimal.rs:52-221`), which a caller still reaches with an Arrow type + narrower than the precision. The MSP checks of DecimalByteParts compare and between + (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs:111-150`, + `between.rs:59-83`), because the MSP is a narrow integer by design (D16). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/array.rs:537-609` | `patch`: one dispatch over the canonical type in place of two, and no panic at `:594-600` | +| `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:6-8`, `:114-153` | No widening of mismatched operands, and no "constant does not fit" branch at `:137-150` | +| `vortex-array/src/arrays/decimal/compute/between.rs:63-105` | The bounds cast to the canonical type always succeed, so `Option` becomes `T` | +| `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:167-176` | No common width | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:431-462` | No widening of a mismatched pair at `:445-449` | +| `vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:12-45` | Only the same-width path | +| `vortex-array/src/arrays/decimal/utils.rs:19-89` | Remove `widened_buffer` and `converted_buffer` | +| `vortex-array/src/array/haystack/mod.rs:118-126`, `:216`, `vortex-row/src/codec.rs:958` | `buffer::()` in place of `converted_buffer` | +| `vortex-array/src/builders/decimal.rs:160-179`, `:298-307` | `append_decimal_array` copies the same width. `DecimalBuffer::extend` stops converting | +| `vortex-cuda/src/arrow/canonical.rs:461-473` | The export target comes from the precision (`vortex-cuda/src/arrow/mod.rs:841-849`) and is never narrower than the canonical width, so the rejection goes | +| `vortex-array/src/dtype/decimal/types.rs:71-74` | Remove `is_compatible_decimal_value_type` if no caller remains | +| Tests | `vortex-array/src/scalar_fn/fns/binary/compare/tests.rs:575-616`, `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs:421`, `:818`, `vortex-array/src/builders/decimal.rs:347` (`test_mixed_extend`), `vortex-array/src/aggregate_fn/fns/all_non_distinct/tests.rs:283-292` | + +Design notes: Each change replaces a width dispatch with one dispatch over +`decimal_dtype.values_type()`. Where a `None` becomes impossible, the code uses `vortex_expect` with +a message that names the invariant. Two operands of one decimal dtype share one width after CD8a. +Scalars of a dtype fit its precision (`scalar/validate.rs:58-60`), so they fit the canonical type. + +Steps: + +1. Remove the branches in the kernels, one commit for each file group. +2. Remove `widened_buffer`, `converted_buffer` and `is_compatible_decimal_value_type`, with their + callers updated. +3. Delete the tests of the removed behavior. + +Tests: The deleted tests are in the table. The conformance suites (`test_array_consistency`, +`test_binary_numeric_array`, `test_cast_conformance`, `test_filter_conformance`) cover the kept +paths. + +Benchmarks: `vortex-array/benches/compare.rs`, `binary_ops.rs` and `filter_fixed_width.rs` stay the +same or improve a little, because a dispatch level goes. Not verified. + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings`, `cargo nextest run` and + `cargo +nightly-2026-09-10 fmt` for `vortex-array`, `vortex-row` and `vortex-cuda` (on a CUDA + host) +- No function in `vortex-array` dispatches on two decimal widths of one dtype. + +Review focus: no removed branch is reachable with a canonical array, and the kept paths keep their +coverage. + +Size: medium. Many files, each with a small deletion. + +Open questions: None. + +## Notes from planning + +These facts came up while the briefs were drafted. Each one is either handled by a brief or +tracked elsewhere. + +1. The compat harness rebuilds the expected arrays with the current code (`check.rs:69-71`). One + broken fixture aborts the whole check (`check.rs:70`, `:195`), and no workflow runs + `compat.py check --mode all`. The README also says that `--git-ref` builds from that ref, but the + code does not (`scripts/compat.py:922-925`). A follow-up issue tracks these gaps. +2. `vortex-arrow/benches/to_arrow.rs:117` builds an `i64` buffer at `Decimal(19, 2)`, whose + canonical width is `i128`, so CD8a changes that input. Its `schema()` says `Decimal(19, 10)` + (`:94`), which does not match. +3. `dbp_assemble_narrowed` (`encodings/decimal-byte-parts/benches/dbp_assemble.rs:158-184`) uses + `u8`, `u16` and `u32` lower parts, which D16 forbids. CD5 changes it. +4. PR #10155 (`659c7f76e6`) changed only `vortex-duckdb/src/convert/vector.rs`, where it added the + lossy `i16` to `i8` narrowing. The builder accepted wider buffers before it. +5. `DecimalDType` derives `serde::Deserialize` (`vortex-array/src/dtype/decimal/mod.rs:36`), which + bypasses `try_new`. The wire readers use `try_new`. +6. `vortex-arrow/src/convert.rs:184`, `:195`, `:206` and `:217` call `DecimalDType::new`, which + panics on an invalid precision or scale from FFI. CD3 switches them to `try_new`. +7. The decimal `min_max` aggregate panics on a valid value outside the precision + (`vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:61-80`, through `Scalar::decimal`). File + reads trust the precision, so a file can trigger it. CD6 and CD7 avoid the aggregate. A + follow-up issue tracks the general case. +8. The CUDA decoder of DecimalByteParts also produces narrow leaves + (`vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:63-65`). CD5 changes it. +9. Constant canonicalization uses the width of the scalar value + (`vortex-array/src/arrays/constant/vtable/canonical.rs:113-121`), and `upcast_decimal_values` + (`vortex-array/src/arrays/decimal/compute/cast.rs:439-489`) builds arrays wider than canonical. + CD8a fixes both. Scalars still keep any width that fits their precision. +10. After CD6, Python sees `Narrow` arrays for old narrow nodes. The binding keys its + `DecimalArray` subclass on the encoding ID (`vortex-python/src/arrays/native.rs:174-175`), so + such arrays get no decimal subclass until the binding maps `Narrow`. +11. `docs/specs/versioning/design.md:264-267` says that the decimal scheme writes only single-child + arrays, but the scheme has a v2 mode. A follow-up issue tracks the stale versioning docs. +12. CD6 copies device buffers to the host with `BufferHandle::try_to_host_sync`, because CUDA reads + hand device segments to plugins (`vortex-cuda/src/layout.rs:241-247`). Whether this works for + every device buffer that reaches the plugin is Not verified. +13. Research line references that moved on `develop`: `decimal_narrow_storage` is at + `vortex-array/src/arrays/decimal/compute/mod.rs:83-87`, and the `DecimalBuffer` enum is at + `vortex-array/src/builders/decimal.rs:48-55`. diff --git a/research/decimal/agents/briefs/04-wire.md b/research/decimal/agents/briefs/04-wire.md new file mode 100644 index 00000000000..5702c0878b8 --- /dev/null +++ b/research/decimal/agents/briefs/04-wire.md @@ -0,0 +1,487 @@ + + +# Wire change: briefs WR1 to WR3 + +These briefs assume that NR1 to NR5 and CD3 to CD8a are on `develop`. Integer and decimal `Narrow` +then exist in memory with no wire form, and one DecimalByteParts (DBP) scheme takes the canonical +decimal array. + +Paths under `vortex-array/src/arrays/narrow/` do not exist at `b70206818f`. NR1 creates them. The +briefs cite the local rewrite (`f7408481ce`) or the closed #10333 (`adc1e83b9d`) for them. Every +other `file:line` is checked against the `develop` checkout at `b70206818f`. + +The wire change lands as a stack of three PRs that merge in order, in one release, right after +CD8a. +The section [Lockstep](#lockstep) gives the reasons. + +### WR1: `vortex.narrow` wire format for integers and decimals + +- Goal: Give `Narrow` the wire form of D8. The metadata is a prost message with a typed `oneof` + (D32): the child `PType` for an integer parent, or the child precision for a decimal parent. The + reader builds and checks the child dtype before it decodes the child, so invalid metadata is an + error and never a panic. When the selected editions do not permit `vortex.narrow`, the writer + still writes the canonical array (D26). +- Depends on: NR1 (encoding, `validate_dtypes`, the lowering hook, registration), CD3 + (`DecimalDType::values_type()`), CD4 (decimal rules in `validate_dtypes`, decimal `Narrow`). + Soft: FX1, NR3, NR4, NR5, CD6. Blocks: WR2, WR3. +- Decisions: D8, D10, D14, D15 (one `Narrow` over a `Narrow`), D22, D26, D32. Invariants 1, 2 and 6 + of `docs/specs/versioning/design.md:219-235`. +- Branch: `ct/narrow-wire-format`, based on `develop` after CD8a. +- Scope: The serializer and deserializer of `vortex.narrow`, the documented wire contract, the + lowering when the context does not permit the ID, and tests for round trips and for every + rejected input. +- Out of scope: Edition membership (WR2). Compressor output (WR3). Compat fixtures (LT1). Any + change to the child rules of `validate_dtypes` (NR1 and CD4 own them). A second wire ID: + `vortex.narrow` has one wire form, so the blanket `ArrayPlugin` impl for `VTable` + (`vortex-array/src/array/plugin.rs:165-205`) is enough. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/wire.rs` | New. `NarrowMetadata`, `metadata` and `child_dtype`, with the contract in the module doc | +| `vortex-array/src/arrays/narrow/vtable.rs` | Created by NR1. Replace the `deserialize` stub (`vortex_bail!`), and make `serialize` return the `vortex.narrow` form | +| `vortex-array/src/arrays/narrow/mod.rs` | Created by NR1. Declare `mod wire;`. Remove "cannot be serialized" from the module doc | +| `vortex-array/src/serde.rs:161-256` | The tree builder also lowers an array when the context does not permit its serialized ID | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | Created by NR1. Remove `test_deserialization_unsupported`. Change `test_serialization_writes_canonical` to use a context that leaves out `vortex.narrow` | +| `vortex-array/src/arrays/narrow/tests/wire.rs` | New. Round-trip and rejection tests. Add `mod wire;` to `tests/mod.rs` | +| `vortex-file/src/tests.rs` | One file round-trip test with integer and decimal `Narrow` columns | + +Design notes: + +The wire contract of `vortex.narrow`: + +| Part | Rule | +| --- | --- | +| Dtype | An integer `DType::Primitive` or a `DType::Decimal`. Any other dtype is an error | +| Buffers | None | +| Children | Exactly one, the values child | +| Metadata | `NarrowMetadata`, a prost message with one `oneof`: the child `PType` for an integer parent, or the child precision for a decimal parent | +| Child dtype | The reader builds it from the metadata. It takes the nullability of the parent, and the scale of the parent for a decimal. It must pass `validate_dtypes`: D22 for integers, D10 for decimals | +| Child encoding | Any encoding whose dtype is the child dtype, including a `Narrow` produced by another plugin | +| Values | No scan. A read trusts the child dtype (D14). A legal child value is a legal parent value | + +```rust +// vortex-array/src/arrays/narrow/wire.rs + +#[derive(Clone, prost::Message)] +pub(super) struct NarrowMetadata { + #[prost(oneof = "NarrowChild", tags = "1, 2")] + pub child: Option, +} + +#[derive(Clone, Copy, prost::Oneof)] +pub(super) enum NarrowChild { + /// The child `PType` of an integer parent. + #[prost(enumeration = "PType", tag = "1")] + Integer(i32), + /// The child precision of a decimal parent. + #[prost(uint32, tag = "2")] + Decimal(u32), +} + +/// Returns the metadata of a `vortex.narrow` node whose values child has `child` dtype. +pub(super) fn metadata(child: &DType) -> VortexResult + +/// Returns the child dtype that `metadata` names for a parent of dtype `parent`. +/// +/// Fails on any metadata that does not name a valid child. Never panics. +pub(super) fn child_dtype(parent: &DType, metadata: &[u8]) -> VortexResult +``` + +- A missing `oneof` is an error, so an empty metadata buffer can never decode silently as `U8`. + The integer `PType` decodes with `PType::try_from`, not the prost getter, as FX1 does for other + metadata. An unknown value is an error. +- `serialize` returns the encoded `NarrowMetadata` with the `vortex.narrow` ID. A valid in-memory + `Narrow` always serializes. +- The lowering of D26 also covers a context that does not permit `vortex.narrow`. Today the tree + builder has no context, and `try_write_node` rejects a non-permitted ID with "not permitted by + ctx" (`vortex-array/src/serde.rs:251-256`). So the tree builder gets the permitted IDs, and it + asks the NR1 hook when an array's serialized ID is not permitted. The doc keeps the workaround + note and its end at LT1. + +`deserialize` runs these checks in this order. Every check returns an error: + +1. `buffers` is empty, and `children.len() == 1`. `SerializedArray::child` panics on a missing + child index (`vortex-array/src/serde.rs:480-493`), so this check comes before any `get`. +2. `child_dtype(dtype, metadata)`: + - The metadata decodes, and it holds a `oneof` variant. + - The variant matches the parent: `Integer` for an integer parent, `Decimal` for a decimal + parent. + - Integer parent: `PType::try_from` of the value. An unknown value is an error. + - Decimal parent: `DecimalDType::try_new(precision, parent.scale())`. This rejects 0, a value + above 76, and a precision below a positive scale + (`vortex-array/src/dtype/decimal/mod.rs:48-78`). + - `validate_dtypes(&child, parent)`. This rejects a float child, other signedness, a width that + is not smaller, and a decimal child whose canonical width is not narrower (D10, D22). +3. `children.get(0, &child, len)`. The child dtype is valid at this point. The slice impl of + `ArrayChildren` asserts on a dtype mismatch (`vortex-array/src/serde.rs:314-318`), so the tests + catch any check that runs too late. +4. Flatten a `Narrow` child into its values, as `NarrowArray::try_new` does. A child can be a + `Narrow` without a nested `vortex.narrow` node: the CD6 read plugin returns a `Narrow` for an old + narrow `vortex.decimal` node. +5. Return `ArrayParts` with the parent dtype. The generic adapter runs `validate` again. + +Edge cases: `I8`, `U8` and `Decimal(1..=2, s)` parents have no narrower child, so every metadata +fails. A negative scale puts no lower bound on the child precision. Nullability is never read from +the file. + +Steps: + +1. Add `wire.rs` with `NarrowMetadata`, `metadata`, `child_dtype` and the module doc of the + contract. Add unit tests for both functions. +2. Replace the `deserialize` stub and change `serialize` in `vtable.rs`. Add `tests/wire.rs`. +3. Pass the permitted IDs to the tree builder, and lower a non-permitted `Narrow` with the NR1 + hook. +4. Add the `vortex-file` round-trip test. + +Tests: + +- `tests/wire.rs`, `round_trip_integer`, `rstest` over parent and child pairs: `I64` to `I8`, + `I16` and `I32`, `I32` to `I8`, `I16` to `I8`, `U64` to `U8`, `U32` to `U16`, each nullable and + non-nullable. Serialize with a context that permits `vortex.narrow`, decode, and check that the + result is a `Narrow` with the same child dtype. Compare values with `assert_arrays_eq!`. +- `round_trip_decimal`, `rstest`: `Decimal(12, 2)` to `Decimal(7, 2)`, `Decimal(38, 2)` to + `Decimal(3, 2)`, `Decimal(76, 0)` to `Decimal(31, 0)`, `Decimal(10, -2)` to `Decimal(4, -2)`, and + a nullable case. +- `metadata_variant`: an `I16` child gives `Integer` with the `I16` value, and a `Decimal(7, 2)` + child gives `Decimal(7)`. The serialized ID is `vortex.narrow`. +- `reject_metadata`, `rstest`: an empty buffer, bytes that do not decode, an unknown `PType`, a + `Decimal` variant under an integer parent, an `Integer` variant under a decimal parent, precision + 0, precision 77, a precision below the scale, a precision whose canonical width is not narrower, + `F32`, `U8` under `I64` (signedness), and `I64` under `I64` (same width). +- `reject_parent_dtype`, `rstest`: `F64`, `Utf8`, `Bool`, `I8`, `Decimal(2, 1)`. +- `reject_child_and_buffer_counts`: zero children, two children, one buffer. +- `flattens_narrow_child`: a child that is itself a `Narrow` decodes to one `Narrow`. +- `non_permitted_context_writes_canonical`: serialization with a context whose allowed IDs leave + out `vortex.narrow` writes the canonical array, and it reads back with equal values. Build the + context with `ArrayContext::new(..).with_allowed_ids(..)`, as the private `restricted_context` + helper does (`vortex-array/src/serde.rs:907-909`). +- All rejection tests assert `is_err()`. A panic fails them, which pins the check order. +- `vortex-file/src/tests.rs`, `narrow_round_trips_through_a_file`: write a struct with an `I64` + column and a `Decimal(12, 2)` column of `Narrow` arrays. The test session permits every + registered array (`vortex-file/src/tests.rs:120-129`). Read back, find `Narrow` nodes with + `depth_first_traversal`, and compare values. + +Benchmarks: None for the wire format. WR3 needs a decode case for `Narrow` over FoR and BitPacked +and for `Narrow` over DBP. If NR2 and CD1 do not have them, add them to +`vortex/benches/common_encoding_tree_throughput.rs` as the first commit of this PR (D19). + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo clippy -p vortex-file --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-file` +- `cargo nextest run -p vortex-array`, `cargo nextest run -p vortex-file` +- `cargo test --doc -p vortex-array` +- An integer or decimal `Narrow` round-trips through serialization with its child dtype. +- Every invalid metadata, parent dtype, child count or buffer count returns an error. +- A write with the default `core` edition writes the canonical array instead of `vortex.narrow`. + +Review focus: The check order before `children.get`. No `unwrap` or `vortex_expect` on metadata. A +missing `oneof` and a variant that does not match the parent both fail. The decimal child takes the +parent scale. The module doc states the full contract, because it freezes when an edition first +permits the ID. After that, relaxing D10 or D22 is a reader-visible change and needs a new wire ID +(invariant 1). + +Size: Medium. One small module, two vtable methods, one serializer change, and many small test +cases. + +Open questions: None. + +### WR2: One preview edition with `vortex.narrow` and DecimalByteParts v2 + +- Goal: Declare one new draft `preview` edition that permits `vortex.narrow` and + `vortex.decimal_byte_parts.v2` (D18). Writers that select it get DBP with lower parts at once, + and `Narrow` after WR3. +- Depends on: WR1 (D8 and invariant 6: the release that first permits the ID reads both forms), + CD5 (the v2 contract with `u64` lower parts, D16), CD7 (the one DBP scheme on the canonical + array, whose v2 mode this edition turns on). Soft: None. Blocks: WR3. +- Decisions: D8, D16, D17, D18. +- Branch: `ct/narrow-preview-edition`, based on WR1. +- Scope: The edition declaration, its export, its generated record and the default preview + constant. Edition tests. The versioning docs that the new edition makes false. +- Out of scope: Compressor changes (WR3). Any `core` edition (LT1). Compat fixtures, which can hold + only `core` formats (LT1). Rewriting the "Planned scheme configuration" text (a follow-up issue). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-edition/src/declarations/preview/v2026_10.rs` | New. `PREVIEW_2026_10_0` and its `DECLARATION` | +| `vortex-edition/src/declarations/preview/mod.rs:17-19` | Add `pub mod v2026_10;` and `pub use v2026_10::PREVIEW_2026_10_0;` | +| `vortex-edition/src/declarations/mod.rs:24-33` | Add `&preview::v2026_10::DECLARATION` after line 32 | +| `vortex/src/editions/mod.rs:44-54` | Re-export `PREVIEW_2026_10_0`. Set `DEFAULT_PREVIEW_EDITION` to it | +| `vortex/editions/preview/preview2026.10.0.toml` | New. Generated by `cargo run -p xtask -- generate-editions` | +| `vortex/src/editions/tests.rs` | New tests (see below). `preview_starts_empty` (`:148-159`) stays: it pins `preview2026.08.0` | +| `docs/specs/versioning/editions.md:63-72` | Add a `preview2026.10.0` entry under "Draft editions" | +| `docs/specs/versioning/editions.md:125-126` | Remove "The current `preview` edition is empty, so its first component must go into a new edition." | +| `docs/specs/versioning/design.md:259-267` | Say that the decimal scheme writes v2 when an enabled edition permits it, as `DecimalScheme::refine` does (`vortex-btrblocks/src/schemes/decimal.rs:99-105`) | +| `docs/specs/versioning/design.md:276-278` | The footnote says no edition permits v2. Name the draft `preview2026.10.0` instead | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:15-18`, `:46-47` | Change "joins an edition" to "joins a `core` edition". Fixtures are written with the default session (`vortex-test/compat-gen/src/adapter.rs:66-67`) | + +Design notes: + +```rust +/// The October 2026 draft edition of the `preview` family. +pub const PREVIEW_2026_10_0: EditionId = EditionId::new("preview", 2026, 10, 0); + +/// The declaration of [`PREVIEW_2026_10_0`] and the components that join the family at it. +pub static DECLARATION: EditionDeclaration = EditionDeclaration { + edition: Edition { id: PREVIEW_2026_10_0, min_library_version: None }, + added: &[ + EditionMember::array(&"vortex.decimal_byte_parts.v2"), + EditionMember::array(&"vortex.narrow"), + ], +}; +``` + +- The name uses the month in which the PR merges. `preview2026.10.0` is the name of #10333 + (`adc1e83b9d:vortex-edition/src/declarations/preview/v2026_10.rs`). Rename all files and + constants if the PR merges in a later month. `check-editions` requires a new record to follow + its family's chronology (`xtask/src/check_editions.rs:160-180`). +- The edition is a draft: `min_library_version` is `None`. Members are sorted by ID, as in the + `core` declarations. +- The default session registers the edition but does not enable it + (`vortex/src/editions/mod.rs:53-83`). Nothing changes for default writers. +- A writer that enables it gets DBP v2 for wide decimal values, because + `BtrBlocksCompressorBuilder::allowed_schemes` calls `refine` + (`vortex-btrblocks/src/builder.rs:234-245`). No compressor produces `Narrow` until WR3. +- The process in `docs/specs/versioning/editions.md:116-126` and `:147-150` starts a new component + in its own family. D18 puts both IDs straight into `preview`, and this brief follows D18. + +The edition adds only `vortex.narrow` and `vortex.decimal_byte_parts.v2`. Whether other IDs +join it is a decision for the owners of those encodings, not part of this plan. + +Steps: + +1. Add the declaration, the module, the re-exports and `DEFAULT_PREVIEW_EDITION`. Regenerate the + record. +2. Add the edition tests. +3. Update the docs and the compat-gen comments. + +Tests (all in `vortex/src/editions/tests.rs`): + +- `preview_2026_10_0_adds_narrow_and_decimal_byte_parts_v2`: the array set of the new edition is + exactly the two IDs. Its other kinds are empty. +- `preview_edition_ids_are_registered_array_encodings`: the same check as + `core_edition_ids_are_registered_array_encodings` (`:232-250`), for `DEFAULT_PREVIEW_EDITION`. +- `preview_edition_writes_decimal_byte_parts_v2`: a default session with the preview edition + enabled writes `Decimal(38, 2)` values at or above 2^70 as DBP with lower parts. With `core` + only, the read-back tree has no lower parts. These values need precision 22, whose canonical + width is the column width, so WR3 adds no `Narrow` and the test stays valid. +- `every_declared_edition_validates` (`:70-77`) covers the new declaration with no change. + +Benchmarks: None. Default writers do not change. + +Acceptance checks: + +- `cargo run -p xtask -- generate-editions`, then `git status --porcelain` is empty. +- `cargo run -p xtask -- check-editions --base origin/develop` +- `cargo clippy -p vortex-edition -p vortex --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-edition -p vortex` +- `cargo nextest run -p vortex-edition`, `cargo nextest run -p vortex` +- The default session enables `core2026.08.3` only (`vortex/src/editions/tests.rs:216-230`). +- Docs build as `CONTRIBUTING.md#documentation` describes, if the user wants a docs check. + +Review focus: No frozen record changes. The edition is a draft. The docs no longer say that no +edition permits v2. WR1 is in the same release or an earlier one. + +Size: Small. One declaration, generated records, three tests and doc edits. + +Open questions: The edition name follows the merge month. Rename all files and constants if the +PR merges after October 2026. + +### WR3: A `Narrow` scheme in the compressor + +This brief needs a design pass before a coding agent starts. Two questions are still open: which +arrays the scheme may narrow (deferred, D27), and how the scheme fits the `AlwaysUse` contract and +the cascade budget (D33). The facts below are the input for that pass. + +- Goal: When the selected editions permit `vortex.narrow`, the compressor can store an integer or + decimal array at a narrower width under a `Narrow`. For decimals, the scheme narrows the decimal + type to the best precision first, and the DecimalByteParts scheme then compresses the child + (D17). Integers narrow by width with the same signedness (D22). This replaces part 2 of #10333. +- Depends on: WR1 (serializer), WR2 (the edition that tests and snapshots select), NR1 + (`NarrowArray::encode`), NR5 (the rules that move structural encodings above `Narrow`), CD3 + (`values_type()` and the narrowing copy), CD4 (`NarrowArray::encode_decimal` and the best child + dtype), CD7 (the DBP scheme on the canonical array). Soft: NR2, CD1, CD8a. Blocks: LT1. +- Decisions: D4, D10, D11, D17, D18, D19, D22, D23, D25, D27, D33. +- Branch: `ct/narrow-compressor`, based on WR2. +- Scope: A `Narrow` scheme for integers and decimals, its gate in the BtrBlocks builder, the CUDA + exclusion, tests and a `preview` golden variant. Test expectations that change because + permissive sessions now permit `vortex.narrow`. +- Out of scope: `Narrow` on internal children: offsets, codes, run ends, indices, the DBP MSP + (D23). The `encode_values` switch of the existing call sites (NR6). Any `core` edition (LT1). + Fuzzing under the preview edition. + +Decided: + +- It is a real scheme (D33). If it later turns out that the compressor always wants it, it can + become a hard-coded step. +- The scheme runs only when `vortex.narrow` passes the same edition test as the IDs that a scheme + produces. `BtrBlocksCompressorBuilder::build` already applies that test + (`vortex-btrblocks/src/builder.rs:228-245`). `empty()` and the CUDA preset leave it out, because + CUDA has no `Narrow` decoder (`builder.rs:93-102`). +- Placement follows D25. If the child compresses to `Dict`, `RunEnd` or `Sparse`, the NR5 rules + move that encoding above the `Narrow`. The compressor output must pass through those rules before + it is written or probed. Then the dict-layout probe, which checks `is::()` on its output + (`vortex-layout/src/layouts/dict/writer.rs:157-166`), still sees `Dict` at the root. The Arrow and + DuckDB exporters keep their fast paths (`vortex-duckdb/src/exporter/mod.rs:250`, + `vortex-arrow/src/executor/dictionary.rs:36`, `run_end.rs:39`). +- For decimals, the scheme writes no `Narrow` when the best precision has the canonical width of + the array. The array then goes to the DBP scheme unchanged (D17). +- The decimal child dtype comes from `NarrowArray::encode_decimal` of CD4, so one function owns the + child label. A false label breaks the compare shortcut. + +To design: + +1. Which arrays the scheme may narrow (D27). Every array whose dtype comes from outside enters + `CascadingCompressor::compress` (`vortex-compressor/src/compressor/cascade.rs:72-88`): columns, + struct fields, list and fixed-size-list elements, extension storage, union and variant children + (`cascade.rs:143-245`, `vortex-compressor/src/compressor/structural.rs:45`, `:78`, `:144`). + Arrays that layouts create also enter there: dict-layout codes, which are `U16` by default and + fixed by layout metadata (`vortex-layout/src/layouts/dict/writer.rs:69-73`), and zone-map + statistics tables (`vortex-file/src/strategy.rs:202-206`). Scheme children enter through + `compress_child` (`cascade.rs:99-119`). +2. Precedence with the DBP scheme. The DBP scheme returns `AlwaysUse` + (`vortex-btrblocks/src/schemes/decimal.rs:115-123`). The contract says that `AlwaysUse` schemes + must be mutually exclusive for each canonical type + (`vortex-compressor/src/scheme/estimate.rs:63-72`), and pass 1 returns the first match in + registration order (`vortex-compressor/src/compressor/select.rs:76-96`). A decimal `Narrow` + scheme must run before the DBP scheme when the array narrows, and not otherwise. The design + chooses how the contract changes, for example an explicit order between `AlwaysUse` schemes. +3. The cascade level. A scheme uses one level of `MAX_CASCADE = 3` + (`vortex-compressor/src/scheme/ctx.rs:14-16`). So the MSP under a `Narrow` gets one level less + than in a file without `Narrow`. The design chooses whether that is acceptable, or whether the + `Narrow` scheme does not count against the budget. +4. Integer estimates. An integer `Narrow` scheme competes with BitPacking and FoR, which already + store the narrow bits. The design decides when `Narrow` wins: for example only when it improves + the estimate, or always for in-memory compute. + +Facts and tests to keep from the draft: + +- Expected shapes under the preview edition, from the value ranges in + `vortex-btrblocks/tests/golden.rs` (Not verified by a run): + + | Column | Values | Result | + | --- | --- | --- | + | `I64` | 0 to 200 | `Narrow(I64)` over a compressed `I16` child | + | `Decimal(12, 2)` | 0 to 9,999,999 | `Narrow(Decimal(12, 2))` over DBP of `Decimal(7, 2)` | + | `Decimal(10, 2)` (gap) | 10,000 to 32,767 | `Narrow` over DBP of `Decimal(5, 2)`, whose canonical width is `i32` (D19) | + | `Decimal(18, 2)` | up to 2,000,000,000 | No `Narrow`: precision 10 has the column width `i64`. DBP with a narrowed MSP | + | `Decimal(76, 2)` | about 2^100 | `Narrow` over DBP v2 of `Decimal(31, 2)` with one lower part | + +- Golden snapshots. `regular`, `onpair` and `compact` select `core2026.08.3`, plus `zstd2026.02.0` + for `compact` (`vortex-btrblocks/tests/golden.rs:469-497`). `vortex.narrow` is not permitted + there, so no existing snapshot changes. A change there means that the gate leaks. A new + `golden_preview` variant selects `core` and the preview edition, and writes one snapshot for each + corpus entry. +- Permissive sessions. `disable_editions()` and `unrestricted()` allow every registered ID + (`vortex-btrblocks/src/builder.rs:212-226`, `vortex-file/src/writer.rs:252-256`). NR1 registers + `Narrow` in the default `ArraySession`, so from WR3 on these settings can write `Narrow`. Many + `vortex-file` tests use a test edition with every registered array (`vortex-file/src/lib.rs:200`), + and two compression benchmarks use `unrestricted()`. `test_stock_ticker_file_size` + (`vortex-file/src/tests.rs:150-191`) is one known change. Review every changed expectation + against D23. Do not accept a change only because it is new output. +- `test_array_stream_no_double_dict_encode` (`vortex-file/src/tests.rs:1598-1622`) stays unpinned. + The closed #10333 pinned it to an explicit compressor (`07fafc2a5b`), which hid the probe + problem. With the NR5 rules, the test is the regression test for placement. +- Tests to port from `adc1e83b9d:vortex-btrblocks/tests/narrow.rs`, with the module `SESSION`: + round trips under `core` and preview selections, an edition without `vortex.narrow`, the CUDA + preset, and internal children that never narrow. +- The closed #10333 showed `take_map` regressions with no known cause. Check `take_map` in this + CodSpeed run, and compare it with the NR6 run. + +Acceptance checks, once the design is settled: + +- For each `` in `vortex-compressor`, `vortex-btrblocks`, `vortex-layout`, `vortex-file` + and `vortex`: `cargo clippy -p --all-targets --all-features -- -D warnings`, + `cargo +nightly-2026-09-10 fmt -p `, and `cargo nextest run -p --all-features`. +- With `core` only, compressor output is the same as before WR3. +- With the preview edition, no `Narrow` sits directly above a structural encoding, no internal + child is a `Narrow`, and decoding gives the input. + +Size: Large. + +Open questions: The four items under "To design". + +## Lockstep + +Recommendation: a stack of three PRs, WR1 then WR2 then WR3, each based on the previous one. The +stack merges in order in one release, right after CD8a, so the wire change ships with the in-memory +change (D11). + +Reasons: + +- No PR needs temporary code (D11). After WR1, readers read a format that no edition permits. + After WR2, preview writers write DBP v2 and no `Narrow`. After WR3, preview writers write + `Narrow`. Each state is consistent, so a release cut between two PRs does no harm. It only + delays the D19 benefit for preview writers. +- The review topics differ: a frozen-to-be wire contract, edition records and docs, and + compressor behavior with many expectation updates. One PR would bury the wire review under + snapshot churn. +- The one hard order is WR1 before WR2. D8 and invariant 6 need the reader for both forms in + the release that first permits `vortex.narrow`. +- WR2 adds no golden variant. WR3 adds it, so preview snapshots change once. + +One PR is acceptable if the maintainer wants one revert unit. If the release cut cannot wait, only +WR3 can slip. WR3 also waits for its design pass. + +Hard dependencies on the NR and CD PRs: + +| PR | Needs | For | +| --- | --- | --- | +| WR1 | NR1 | `Narrow`, `validate_dtypes`, the lowering hook, registration in the default `ArraySession` (`f7408481ce:vortex-array/src/session/mod.rs`) | +| WR1 | CD3 | `DecimalDType::values_type()` for the width check | +| WR1 | CD4 | Decimal rules in `validate_dtypes` and decimal `Narrow` arrays | +| WR2 | CD5 | The v2 contract with `u64` lower parts and canonical decoding (D16) | +| WR2 | CD7 | The DBP scheme on the canonical array, which v2 mode runs once the edition permits v2 | +| WR3 | NR1 | `NarrowArray::encode` | +| WR3 | NR5 | The rules that move structural encodings above `Narrow` (D25) | +| WR3 | CD3 | The narrowing copy behind the precision-drop cast | +| WR3 | CD4 | `NarrowArray::encode_decimal` and the best child dtype | +| WR3 | CD7 | The DBP scheme that compresses the narrowed child | + +## Later work (pointers only) + +- LT1, both formats enter `core`: a new `core` edition with `vortex.decimal_byte_parts.v2` and + `vortex.narrow` (D18). Other wire improvements from WR2 can join with them. Bump + `DEFAULT_CORE_EDITION` (`vortex/src/editions/mod.rs:51`). Re-enable the DBP v2 compat fixture + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`) and add integer + and decimal `Narrow` fixtures. The `regular` and `compact` golden snapshots change then. + `vortex-duckdb/src/lib.rs:52` and `vortex-jni/src/session.rs:25` pin `CORE_2026_08_3` and need a + decision. Default writers stop writing plain `i256` for wide values (D17). Decide then whether + the serialization workaround of D26 stays for writers that select an older `core` edition. +- LT2, the max pass in DBP decoding (D16): resolve the `TODO(connor)` that CD5 adds in + `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs`. Decoding returns a `Narrow` + that reuses a narrow MSP buffer. No wire change. It waits for the CD1 benchmarks to show a large + regression. +- LT3, sum, min and max pushdown into the MSP (D19): kernels under + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/` for gap columns. No wire change. + It waits for benchmarks. + +## Notes from planning + +1. The documented process starts a new component in its own family draft before `preview` + (`docs/specs/versioning/editions.md:116-126`, `:147-150`). D18 puts both IDs straight into + `preview`, and WR2 follows D18. +2. Some versioning docs are stale on `develop`. `docs/specs/versioning/design.md:264-267` says that + the decimal scheme writes only single-child arrays, but `DecimalScheme::refine` selects v2 when + it is allowed (`vortex-btrblocks/src/schemes/decimal.rs:99-105`). `design.md:269-274` and + `compatibility.md:78-83` say that per-edition scheme configuration does not exist, but + `Scheme::refine` implements it (`vortex-compressor/src/scheme/mod.rs:138-145`). WR2 fixes only + the lines that its edition makes false. A follow-up issue tracks the rest. +3. Compat fixtures are written with the default session, which enables only `core` + (`vortex-test/compat-gen/src/adapter.rs:66-67`). So no fixture can pin `vortex.narrow` or DBP v2 + while they are preview-only. The compat harness covers them from LT1 on. +4. D10 says that the child rule can be relaxed later. Once an edition permits `vortex.narrow`, older + readers reject a relaxed child. So a relaxation must happen before WR2, or it needs a new wire + ID (invariant 1 in `docs/specs/versioning/design.md:221-222`). +5. Before WR1, a `vortex.narrow` node fails with "Narrow deserialization is not supported", not + "Unknown encoding", because NR1 registers `Narrow`. This is harmless. +6. CUDA has no `Narrow` decoder, so WR3 leaves `vortex.narrow` out of the CUDA preset. What a CUDA + reader does with a preview file that holds `Narrow` is Not verified. diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index c7db6028581..5f5cfcabbe9 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -519,15 +519,14 @@ the same signedness and nullability as its parent, and a strictly smaller width the rule of `validate_dtypes` in the local rewrite. `vortex.narrow` stays in memory only until the maintainer decides to write it to files (D18). -`vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level -integer columns (D23). Internal integer children keep changing their own dtype with -`PrimitiveArrayExt::narrow`, as they do today. The btrblocks schemes call it for internal children -of the patch, sparse, RLE, temporal, VarBin, FSST and OnPair schemes -(`vortex-btrblocks/src/schemes/{patches,temporal}.rs`, `integer/{sparse,rle}.rs`, +`vortex.narrow` is for arrays with a fixed logical dtype (D23). Which arrays the compressor wraps is +deferred (D27). Internal integer children keep changing their own dtype. Today they use +`PrimitiveArrayExt::narrow`, and NR6 moves them to the deprecated `encode_values` (D28). The +btrblocks schemes call it for internal children of the patch, sparse, RLE, temporal, VarBin, FSST +and OnPair schemes (`vortex-btrblocks/src/schemes/{patches,temporal}.rs`, `integer/{sparse,rle}.rs`, `float/sparse.rs`, `binary/varbin.rs`, `string/{fsst,onpair,sparse}.rs`). The structural compressor calls it for list offsets (`vortex-compressor/src/compressor/structural.rs:54`). No call narrows a -DBP part. -Issue #10733 keeps open whether `List` and `ListView` offsets get a fixed `I64` dtype with +DBP part. Issue #10733 keeps open whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow`. `Narrow` kernels pass work to the child, and the fallback is to canonicalize (D15): @@ -636,9 +635,9 @@ the private `validate_dtypes`, so decimal support stays additive. When that chec decimals, the compare kernel and the sum pushdown need decimal code in the same change. The closed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one -metadata byte that holds the child `PType`. Under D8, the metadata stores the child dtype: the -child `PType` for an integer parent, or the child precision for a decimal parent. The parent dtype -tells the reader which of the two it holds. Every release that permits an ID must read every valid +metadata byte that holds the child `PType`. Under D8 and D32, the metadata is a prost message with +a typed `oneof`: the child `PType` for an integer parent, or the child precision for a decimal +parent. A missing variant is an error. Every release that permits an ID must read every valid instance of it (invariant 6 in `docs/specs/versioning/design.md`). So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The second part of #10333 comes back as a new PR in the wire change, which puts `vortex.narrow` and DBP v2 into one preview @@ -738,7 +737,8 @@ execution is not verified. Rule X5 pulls generic structural arrays out from under the extension wrapper. Recommendation XQ1 names them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. So generic structural encodings sit above the type layer. The decimal storage encodings, DBP and -`Narrow`, sit below them. +`Narrow`, sit below them. For `Narrow`, reduce rules move these seven encodings above it (D25). +The rewrite's chunked and interleave merge rules, which put `Narrow` on top, are dropped. For `Decimal`, the wire forms of today already follow this rule: @@ -1661,6 +1661,23 @@ The `Narrow` metadata stores only the child precision (D8). Each in-memory step (D11). The DBP constructor and module doc require `u64` lower parts (D16). One DBP scheme serves every edition, and default writers pay the `i256` cost until both formats enter `core` (D17). +A planning review on 2026-10-23 split the work into 22 PRs, with one brief for each PR in +`agents/briefs/` (part 4 of the Decimal notes). It added D25 to D35: + +- Generic structural encodings always sit above `Narrow` (D25). +- A writer that cannot write `vortex.narrow` writes the canonical array, as a documented + workaround until `core` (D26). +- Which arrays the compressor narrows is deferred (D27). +- `encode` builds a `Narrow`, and the deprecated `encode_values` returns only the child. The + existing narrowing call sites move to `encode_values` (D28). +- The DBP constructor rejects lower parts that assemble wider than the canonical width (D29). +- `try_new` takes an `ExecutionCtx`, and imports check their own values (D30). +- Only the PRs that change how files are read wait for the compat fixtures (D31). +- The `vortex.narrow` metadata is a prost message with a typed `oneof` (D32). +- The compressor uses a `Narrow` scheme at first (D33). +- Strict decoding covers eleven more `PType` fields (D34). +- The integer `Narrow` lands as five stacked PRs, then the call-site PR (D35). + These notes add the integer view of section 3.2, so that decimal kernels delegate to integer kernels. D20 makes it future work. Section 9 lists the open questions with the recommendations of these notes. @@ -1698,6 +1715,9 @@ of D20, and they are not reviewed: ### 8.3 Sequence +[Part 4](../04-implementation.md) of the Decimal notes replaces this sequence with PR IDs, waves and +briefs. The list below keeps the reasons for the order. + PR #10230 gave decimals an integer child, which D2 replaces. It closed on 2026-10-19. Its fix to the DBP cast rule becomes P2 below. The plan below replaces its other steps. diff --git a/research/decisions.md b/research/decisions.md index 7414de56c5a..993a2b65816 100644 --- a/research/decisions.md +++ b/research/decisions.md @@ -55,30 +55,42 @@ decimal encoding. | D5 | Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types | Wide decimals need an integer view, and wide integers are useful as columns | | D6 | FixedSizeBinary becomes a core type for opaque bytes of a fixed length, such as UUIDs and hashes | Its defining operations (equality, hashing, byte order, cheap scalars) are core operations. Arrow and Parquet have it natively | | D7 | `vortex.decimal` keeps its wire ID. A reader finds an old node when its stored width differs from the canonical width of its dtype | D3 removes forms and adds none. The versioning spec asks for a new ID only when readers must understand something new | -| D8 | One `Narrow` encoding, with one `vortex.narrow` wire ID, covers integers and decimals. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. The metadata stores only the child `PType` of an integer child, or only the child precision of a decimal child. It stores no scale and no nullability. This settles DQ2 and DQ3 | A release that permits an ID must read every valid instance of it. So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The child must have the scale and nullability of the parent, so metadata that stored them would only let a file hold an invalid child. The format is in no edition yet, so it can still change | +| D8 | One `Narrow` encoding, with one `vortex.narrow` wire ID, covers integers and decimals. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. The metadata stores only the child `PType` of an integer child, or only the child precision of a decimal child, in a prost message with a typed `oneof` (D32). It stores no scale and no nullability. This settles DQ2 and DQ3 | A release that permits an ID must read every valid instance of it. So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The child must have the scale and nullability of the parent, so metadata that stored them would only let a file hold an invalid child. The format is in no edition yet, so it can still change | | D9 | A reader converts an old `vortex.decimal` node. For a narrower node, it scans the non-null values for their max, and it chooses the best precision that the values and the scale allow, or a larger one. The result is a `Narrow` over a child at the width of that precision, or the canonical array when that width is the canonical width of the column. The result copies the buffer when its width differs from the stored width. For a wider node, the reader makes a narrowing copy (D21) | A width does not give a precision. The read plugin cannot use stored statistics, because the deserializer attaches them after the plugin returns. Old narrow nodes are rare, so the speed of this path matters little | | D10 | A decimal `Narrow` child has the same scale and nullability as its parent, and a canonical width strictly narrower than the canonical width of the parent. The producer chooses the child precision: the best precision that the values and the scale allow, or a larger one | A `Narrow` that does not reduce the width has no use. The integer `Narrow` has the same rule. The rule can be relaxed later | | D11 | The canonical-width work lands as an in-memory change, and then as a wire change (D18). Each step of the in-memory change is its own PR, because none of the steps needs temporary code. The wire change lands in lockstep, if possible in the same release as the in-memory change. PRs [#10305](https://github.com/vortex-data/vortex/pull/10305) and [#10314](https://github.com/vortex-data/vortex/pull/10314) stay open, and the work ignores them for now | Small PRs are easier to review, but not when a split needs temporary code or changes file contents twice. In one release, writers that select the preview edition never pay the `i256` cost of D17 | | D12 | Each precision has the smallest signed width that holds its values: `i8` for precision 1 to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, `i128` for 19 to 38 and `i256` for 39 to 76. This settles DQ1 | It is the existing `smallest_decimal_value_type`. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision | | D13 | Imports copy into the canonical width. They write zeros in null slots, and they fail on a value outside the precision. The Arrow import logic can be configured to handle such values in another way. To widen an array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it | arrow-rs does not check decimal values when it builds an array. An error is better than a silent change of the data | -| D14 | `DecimalArray::try_new` checks the width and every valid value. `new_unchecked` skips the value check, for kernels that keep values legal. File reads trust the precision and never panic | This is the rule of X7 for decimals, and the recommendation XQ3 for file reads | -| D15 | `Narrow` kernels pass work to the child, with canonicalization as the fallback. A `Narrow` over a `Narrow` becomes one `Narrow`. A cast of a `Narrow` passes to the child with the same target dtype. Another decimal cast to a precision with the same scale returns a `Narrow` when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops | A `Narrow` holds exactly the values of its child, so work on the child gives the same result on less data | -| D16 | In DecimalByteParts, the MSP is any signed integer up to 64 bits, and every lower part has the dtype `u64`. The constructor and the module doc become stricter. This settles DQ4. Operations run on the parts wherever they can. Decoding produces the canonical array of the DecimalByteParts dtype. It widens a narrower MSP, and it narrows a wider MSP with the copy of D21. A `TODO(connor)` marks a later optimization: find the max of a narrow MSP, and return a `Narrow` that reuses its buffer | A narrow MSP needs no conversion at read time. The stricter constructor breaks no file, because no file has lower parts: no edition permits v2, and its compat fixture is disabled. The compressor never narrows the dtype of a part. Simple decoding is correct, and the optimization waits until benchmarks show large regressions | -| D17 | The compressor has one DecimalByteParts scheme. The scheme takes the canonical array, uses the value range to choose the number of lower parts, and narrows the MSP dtype as the compressor narrows every internal child. This replaces `narrowed_decimal` and keeps file contents. The scheme stays after the wire change, for writers whose edition does not permit `vortex.narrow`. When an edition permits `vortex.narrow`, a decimal `Narrow` scheme runs before it and narrows the decimal type first. In an edition without the lower-parts format, precision 39 to 76 with values that need more than 64 bits is written as a plain `i256` array. The `core` family holds the formats of the default writer, so default writers pay this cost until both formats enter `core` | The canonical fix makes a narrow decimal array illegal in memory, so the scheme cannot call `narrowed_decimal`. File contents do not change before the wire change. The `i256` cost affects only very wide decimals with values above 64 bits. The only way around it writes the old narrow form into new files, and it needs a special serializer | -| D18 | `vortex.narrow` stays in memory only until the maintainer decides to write it to files. Every new wire format enters a preview edition before `core`. `vortex.narrow` and DecimalByteParts with lower parts enter one preview edition together, with the other wire improvements, and later `core` together. This settles DQ9 | In-memory changes land first, so the edition change stays small | +| D14 | `DecimalArray::try_new` checks the width and every valid value, with an `ExecutionCtx` (D30). `new_unchecked` skips the value check, for kernels that keep values legal. File reads trust the precision and never panic | This is the rule of X7 for decimals, and the recommendation XQ3 for file reads | +| D15 | `Narrow` kernels pass work to the child, with canonicalization as the fallback. A `Narrow` over a `Narrow` becomes one `Narrow`. A cast of a `Narrow` passes to the child with the same target dtype, except that a wider integer target returns a `Narrow` over the child. Another decimal cast to a precision with the same scale returns a `Narrow` when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops | A `Narrow` holds exactly the values of its child, so work on the child gives the same result on less data | +| D16 | In DecimalByteParts, the MSP is any signed integer up to 64 bits, and every lower part has the dtype `u64`. The constructor and the module doc become stricter, and the constructor rejects lower parts that assemble wider than the canonical width (D29). This settles DQ4. Operations run on the parts wherever they can. Decoding produces the canonical array of the DecimalByteParts dtype. It widens a narrower MSP, and it narrows a wider MSP with the copy of D21. A `TODO(connor)` marks a later optimization: find the max of a narrow MSP, and return a `Narrow` that reuses its buffer | A narrow MSP needs no conversion at read time. The stricter constructor breaks no file, because no file has lower parts: no edition permits v2, and its compat fixture is disabled. The compressor never narrows the dtype of a part. Simple decoding is correct, and the optimization waits until benchmarks show large regressions | +| D17 | The compressor has one DecimalByteParts scheme. The scheme takes the canonical array, uses the value range to choose the number of lower parts, and narrows the MSP dtype as the compressor narrows every internal child. This replaces `narrowed_decimal` and keeps file contents, except that large values in null slots can give a narrower MSP. The scheme stays after the wire change, for writers whose edition does not permit `vortex.narrow`. When an edition permits `vortex.narrow`, a `Narrow` scheme runs before it and narrows the decimal type first (D33). In an edition without the lower-parts format, precision 39 to 76 with values that need more than 64 bits is written as a plain `i256` array. The `core` family holds the formats of the default writer, so default writers pay this cost until both formats enter `core` | The canonical fix makes a narrow decimal array illegal in memory, so the scheme cannot call `narrowed_decimal`. File contents do not change before the wire change. The `i256` cost affects only very wide decimals with values above 64 bits. The only way around it writes the old narrow form into new files, and it needs a special serializer | +| D18 | `vortex.narrow` stays in memory only until the maintainer decides to write it to files. Every new wire format enters a preview edition before `core`. `vortex.narrow` and DecimalByteParts with lower parts enter one preview edition together, and later `core` together. Other IDs can join that edition, but this plan adds only these two. This settles DQ9 | In-memory changes land first, so the edition change stays small | | D19 | The bar of no compute regression applies to the final state, not to the steps in between. Missing benchmarks land before the changes that they cover. A gap column holds values that fit a width but need the precision of the next width. In the final state, such a column uses the wider width in memory, and that cost is accepted. Pushdown of sum, min and max into the MSP can come later | A true child label needs the next width for such values. Disk size and exports do not change | | D20 | The canonical decimal work does not include the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and scales, casts that lose information, or the lower bound of a negative scale. Their recommendations are unreviewed future work. Epic [#10730](https://github.com/vortex-data/vortex/issues/10730) tracks wide integers and FixedSizeBinary. Issue [#9902](https://github.com/vortex-data/vortex/issues/9902) is its sub-issue for FixedSizeBinary, and wide integers get a tracking issue when that work is scheduled. The recommendation to always store min and max on the MSP (DQ8) is dropped | The canonical fix needs none of them. Stored min and max exist only on a BitPacked MSP, and a slice drops them | | D21 | Every narrowing copy writes zeros in null slots, and it fails on a valid value outside the precision of the target dtype. Three paths make such a copy: an old node wider than its canonical width, an old narrow node whose best precision has a smaller width than the stored width, and DecimalByteParts decoding with an MSP wider than the canonical width. A read without a copy trusts the precision and does not scan | The copy reads every value anyway, so the precision check costs nothing extra, and it matches the import rule of D13. A check without a copy needs a scan that the read does not otherwise make | | D22 | An integer `Narrow` child has the same signedness and nullability as its parent, and a strictly smaller width. So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` child | Mixed signedness needs more code in casts, in the compare shortcut and in the arithmetic bounds, and it saves at most one width. Part 1 of PR [#10333](https://github.com/vortex-data/vortex/pull/10333) also keeps the signedness of internal buffers | -| D23 | `vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level integer columns. Internal integer children keep changing their own dtype, as they do today. These include list offsets, dictionary codes, run ends, patch and sparse indices, `DateTimeParts`, VarBin and FSST offsets, and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend on it | The parent chooses the dtype of an internal child, so the child can narrow its dtype without a `Narrow`. A column has a dtype that the user chose | -| D24 | The compat fixtures land before the in-memory change, and they are published from a release before it, because only the old writer can make narrow leaves. They cover only reads that succeed: narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP. Error cases, such as a value outside the precision in a narrowing copy, are unit tests | The compat harness (`vortex-test/compat-gen`) expects the current reader to read every published fixture | - -D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D24 from 2026-10-23. On +| D23 | `vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level integer columns. Which arrays the compressor wraps in a `Narrow` is deferred (D27). Internal integer children keep changing their own dtype, through the deprecated `encode_values` (D28). These include list offsets, dictionary codes, run ends, patch and sparse indices, `DateTimeParts`, VarBin and FSST offsets, and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend on it | The parent chooses the dtype of an internal child, so the child can narrow its dtype without a `Narrow`. A column has a dtype that the user chose | +| D24 | The compat fixtures are published from a release before the PRs that change how files are read (D31), because only the old writer can make narrow leaves. They cover only reads that succeed: narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP. Error cases, such as a value outside the precision in a narrowing copy, are unit tests | The compat harness (`vortex-test/compat-gen`) expects the current reader to read every published fixture | +| D25 | Generic structural encodings always sit above `Narrow`. Reduce rules move `Dict`, `RunEnd`, `Sparse`, `Constant`, `Masked`, `Chunked` and `Interleave` above a `Narrow`. The chunked and interleave merge rules of the local rewrite, which put `Narrow` on top, are dropped. A follow-up issue covers the same rule for every encoding | The Arrow and DuckDB exporters and the dict layout look for these encodings at the root. One direction for every rule cannot loop | +| D26 | When a writer cannot write `vortex.narrow`, because no wire form exists yet or the selected editions do not permit it, the serializer writes the executed canonical array. The code documents this as a workaround. It stays until `vortex.narrow` is in `core` | Decimal `Narrow` appears in ordinary pipelines through casts and old files, and IPC or a writer without compression would fail on it. A plugin cannot return the wire form of another encoding, so the fallback lives in the serializer | +| D27 | Which arrays the compressor narrows is deferred to the design pass of WR3 | The candidates include struct fields, list elements, extension storage, and arrays that layouts create, such as dict-layout codes and zone maps | +| D28 | `Narrow` has `encode`, which builds a `Narrow`, and `encode_values`, which returns only the narrowed child with the same signedness. `encode_values` is `#[deprecated]`. It replaces `PrimitiveArrayExt::narrow` at every call site, and each call carries `#[expect(deprecated)]` with a `TODO(connor)` reason. The PR that moves the call sites checks that the latest release reads the files that it writes | Every narrowing that changes a dtype is then marked for a later decision. Readers on `develop` accept signed internal children | +| D29 | The DecimalByteParts constructor rejects a lower-part count whose assembled width is wider than the canonical width. Precision up to 18 allows no lower parts, 19 to 38 at most one, and 39 to 76 any count | Decoding then never needs a narrowing copy after assembly. No file breaks, because no edition permits lower parts | +| D30 | `DecimalArray::try_new` takes an `ExecutionCtx`, because the value check reads the validity, which can be lazy. Imports check their own values against the Arrow null buffer and then use the unchecked constructors, so a zero-copy import scans once | This follows `VarBinViewArray::try_new`, and it keeps the context out of the import API | +| D31 | Library changes land first. Only the PRs that change how files are read (DecimalByteParts decoding and the read plugin) wait for the compat fixtures, which the next regular release publishes. This changes D24 | The fixtures must be in the store when CI first runs the changed reader. Nothing else needs them | +| D32 | The `vortex.narrow` metadata is a prost message with a typed `oneof`: the child `PType` of an integer parent, or the child precision of a decimal parent. A missing variant is an error, and the `PType` decodes strictly | The typed `oneof` keeps the two kinds apart, and a missing variant cannot decode silently as `U8` | +| D33 | The compressor uses a real `Narrow` scheme at first. If the compressor turns out to always want it, it can become a hard-coded step. The design pass of WR3 settles its precedence with the DecimalByteParts scheme and the cascade level that it uses | A scheme fits the existing framework. The `AlwaysUse` contract and the cascade budget are the two costs to settle | +| D34 | Strict decoding covers the decimal width, the protobuf `PType` and the eleven `PType` fields of array and layout metadata, in one PR with three commits | It is the same bug with the same fix | +| D35 | The integer `Narrow` lands as five stacked PRs: the encoding, benchmarks, aggregates, arithmetic and casts, and the structural rules. The PR that moves the call sites to `encode_values` comes after them. The dictionary fill-code fix lands before the fifth PR | The benchmarks exist before the kernels change, and the fill-code fix guards the dictionary-codes rule | + +D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D35 from 2026-10-23. On 2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the largest precision whose canonical width is `W`, as the child precision of old data, and they planned the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it -changed D8, D9, D11, D16, D17 and D20. [Part 2](decimal/02-design.md) of the Decimal notes explains -D7 to D19 and D21 to D24. +changed D8, D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D35, and it +changed D8, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal +notes explains D7 to D35, and [part 4](decimal/04-implementation.md) splits the work into PRs. ### Open @@ -130,12 +142,14 @@ This review read the draft stack on 2026-10-19 and again on 2026-10-22. The stat - The CodSpeed reports on #10223 and #10332 compare against `3c87da0`, because `cc018ee` had no run. Their gains, such as those of the FSST benchmarks, do not come from `Narrow`. The new PRs need new reports. -- The work of PR #10333 comes back as two new PRs: - 1. The first PR replaces `PrimitiveArrayExt::narrow` with `NarrowArray::encode_values`. - Internal buffers then keep their signedness in every preset. Golden snapshots and file - contents change, but no wire format changes. This PR lands after the rewrite. - 2. The second PR adds the serialization of D8, the edition entry, and `Narrow` children in the - compressor. It is part of the wire change (D11 and D18). +- The work of PR #10333 comes back in new PRs: + 1. NR1 adds `NarrowArray::encode_values`, and NR6 replaces `PrimitiveArrayExt::narrow` with it at + every call site (D28). Internal buffers then keep their signedness in every preset. Golden + snapshots and file contents change, but no wire format changes. + 2. The serialization of D8, the edition entry and the `Narrow` scheme in the compressor are part + of the wire change, WR1 to WR3 (D11, D18). +- [Part 4](decimal/04-implementation.md) of the Decimal notes lists every new PR, and the briefs in + `decimal/agents/briefs/` describe each one. - The CodSpeed report on #10333 shows `take_map` regressions of 22% and 11% in Simulation mode. The `take_map` benchmark builds `u64` patch indices directly and does not call the narrowing code. The cause is not known. The new PRs get fresh CodSpeed runs, and the investigation From 4c2670210d20bd0442e1c6ba78d296aee368530d Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 15:21:16 -0400 Subject: [PATCH 10/12] Apply the plan review to the Narrow and decimal briefs Fix the dependency data, mark the PRs that share files, and explain how agents reach the briefs and rebase stacks. Record D36 and D37, rename the canonical width function, and let agents run any command the work needs. Signed-off-by: Connor Tsui --- research/decimal/02-design.md | 2 +- research/decimal/03-roadmap.md | 6 +- research/decimal/04-implementation.md | 64 ++++-- research/decimal/agents/briefs/01-fixes.md | 5 +- .../agents/briefs/02-integer-narrow.md | 47 +++- .../agents/briefs/03-canonical-decimal.md | 215 ++++++++++-------- research/decimal/agents/briefs/04-wire.md | 28 ++- research/decimal/agents/report.md | 16 +- research/decisions.md | 12 +- 9 files changed, 240 insertions(+), 155 deletions(-) diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index c2cfaf9c61f..902576f65d5 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -29,7 +29,7 @@ value of the precision: ```rust impl DecimalDType { /// The width of the values buffer for this precision. - pub const fn values_type(&self) -> DecimalType { + pub const fn canonical_values_type(&self) -> DecimalType { match self.precision.get() { 1..=2 => DecimalType::I8, 3..=4 => DecimalType::I16, diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 82858b3e25f..90df6172a53 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -178,9 +178,9 @@ Benchmarks and fixtures come first: The in-memory change has these steps, in order. Each step is its own PR, because none of them needs temporary code (decided, D11): -1. Give the width function one owner. `DecimalDType::values_type()` replaces the 25 direct calls - to `smallest_decimal_value_type`. Producers and imports take the width from the precision, and - every import checks its values (decided, D12, D13 and D30). +1. Give the width function one owner. `DecimalDType::canonical_values_type()` replaces the 25 direct + calls to `smallest_decimal_value_type`. Producers and imports take the width from the precision, + and every import checks its values (decided, D12, D13 and D30). 2. Add `Narrow` over decimals (N28). This step holds the child rules of D10, the kernels and casts of D15, and the rule that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for decimals, the compare kernel and the sum pushdown get decimal code in the same change. diff --git a/research/decimal/04-implementation.md b/research/decimal/04-implementation.md index de905fb6d93..9b67fac7bce 100644 --- a/research/decimal/04-implementation.md +++ b/research/decimal/04-implementation.md @@ -16,11 +16,28 @@ planning review on 2026-10-23. - Give a coding agent one brief, `decisions.md`, and the repository rules in `AGENTS.md` and `STYLE.md`. The brief names the branch, the files, the steps, the tests and the acceptance checks. +- The briefs and `decisions.md` exist only on the `ct/ext-types-research` branch. The agent's own + worktree branches from `develop` and has no `research/` directory, so the relative links in the + briefs do not work there. Give the agent absolute paths into the research worktree + (`/Users/connor/spiral/vortex-data/vortex1/research/`), or tell it to read the files with + `git show ct/ext-types-research:research/`. Do not change a brief while an agent works from + it. If a brief must change, tell that agent. - The briefs cite `develop` at `b70206818f`. Line numbers move as PRs land, so the agent reads the cited code before it changes it. Code from the local integer `Narrow` rewrite is cited with its commit, for example `f7408481ce:vortex-array/src/arrays/narrow/mod.rs`. -- The acceptance checks are commands for the maintainer to run. An agent does not run them unless - the maintainer asks for it. +- The rewrite commits (`f7408481ce`, `8d548646ee`, `74aa17644e`, `af4b663059` and `7e2e5e4d7a`) are + not on `origin`. Only the local branches `ct/narrow-pushdown` and `ct/narrow-encoding` hold them, + and those names also exist on `origin` with the older commits of the closed PRs. Agents in local + worktrees of this repository can read them. A remote agent or another clone cannot. Push them to a + new backup branch before such an agent starts, and do not delete or reset the two local branches. +- Agents can run any command that the work needs: builds, tests, the acceptance checks, and the + generators of committed files, such as golden snapshots (NR6, CD7, WR3) and the edition record + (WR2). CI fails when a committed file is stale. Use the narrowest command, as `AGENTS.md` + describes. +- Several PRs form stacks: NR1 to NR5, WR1 to WR3, and optionally CD3 and CD5, CD4 and CD6, and CD8a + and CD8b. `develop` squash-merges PRs. So after the base PR merges, a plain `git rebase develop` + replays the base commits again and conflicts. Rebase the next PR with + `git rebase --onto develop `, or with `gh stack`. - Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author line. - WR3 needs a design pass before an agent starts it. Its brief lists the open questions. @@ -44,18 +61,18 @@ planning review on 2026-10-23. | NR3 | Aggregates run on the child | NR2 | Small | | NR4 | Arithmetic, `between`, `zip` and casts run on the child | NR3 | Large | | NR5 | Structural encodings move above `Narrow`, and dictionary codes use the child | NR4, FX3 | Medium | -| NR6 | The existing narrowing call sites move to `encode_values` | NR1 | Medium | +| NR6 | The existing narrowing call sites move to `encode_values` | NR1. It merges after NR5 (D35) | Medium | | CD1 | Decimal benchmarks | None | Medium | | CD2 | Compat fixtures for old decimal forms | None | Medium | | CD3 | One width function, imports that check and copy, and the shared helpers | None | Medium | | CD4 | Decimal `Narrow` in memory | NR1, NR3, NR4, CD1, CD3 | Large | | CD5 | DecimalByteParts decodes to the canonical width | CD1, CD3, released CD2 fixtures | Medium | | CD6 | The `vortex.decimal` read plugin converts old nodes | FX1, CD3, CD4, released CD2 fixtures | Medium | -| CD7 | One DecimalByteParts scheme on the canonical array | NR1 | Medium | +| CD7 | One DecimalByteParts scheme on the canonical array | NR1, CD1 | Medium | | CD8a | Every decimal array has the canonical width | CD3, CD5, CD6, CD7 | Large | | CD8b | `DecimalArray::try_new` checks every valid value, with an `ExecutionCtx` | CD8a | Large | | CD8c | Remove the code for mixed decimal widths | CD8a | Medium | -| WR1 | `vortex.narrow` wire format for integers and decimals | NR1, CD3, CD4, after CD8a | Medium | +| WR1 | `vortex.narrow` wire format for integers and decimals | NR1, CD3, CD4, CD8a | Medium | | WR2 | One preview edition with `vortex.narrow` and DecimalByteParts v2 | WR1, CD5, CD7 | Small | | WR3 | A `Narrow` scheme in the compressor (needs a design pass) | WR1, WR2, NR1, NR5, CD3, CD4, CD7 | Large | @@ -65,8 +82,9 @@ min and max into the MSP (D19). LT2 and LT3 wait until benchmarks show that they ## Graph -Each arrow is a hard dependency. The graph leaves out dependencies that follow from other arrows. -The dashed arrows are the release gate of the compat fixtures (D31). +Each solid arrow is a hard dependency. Some arrows also follow from other arrows. The graph keeps +them, so that the direct needs of each PR show. The dashed arrows are order rules, not code +dependencies: the release gate of the compat fixtures (D31), and NR6 merging after NR5 (D35). ```mermaid flowchart TD @@ -105,6 +123,7 @@ flowchart TD NR1 --> NR2 --> NR3 --> NR4 --> NR5 FX3 --> NR5 NR1 --> NR6 + NR5 -.-> NR6 CD2 --> REL REL -.-> CD5 REL -.-> CD6 @@ -119,6 +138,7 @@ flowchart TD CD3 --> CD6 CD4 --> CD6 NR1 --> CD7 + CD1 --> CD7 CD3 --> CD8a CD5 --> CD8a CD6 --> CD8a @@ -138,18 +158,20 @@ FX2 has no edge. It can land at any time. ## Order of work -Several agents can work at once. A wave starts when its dependencies have merged. +A PR can start when its dependencies have merged. The waves group the PRs whose dependencies merge +at about the same time. Two PRs of one wave can run at the same time only when they touch different +files. The column "Shared files" names the pairs that do not, so run those one after the other. -| Wave | PRs | Note | -| --- | --- | --- | -| 1 | FX1, FX2, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | -| 2 | NR2, NR6, CD7, CD5 | NR6 merges after NR5 (D35). CD5 waits for the release with CD2 | -| 3 | NR3, then NR4 | One stack. Merge NR1 and NR2 before reading the CodSpeed reports of NR3 and NR4 | -| 4 | NR5, CD4 | NR5 waits for FX3 | -| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | -| 6 | CD8a | The last in-memory step that the wire change needs | -| 7 | CD8b, CD8c, WR1 | Independent of each other | -| 8 | WR2, then WR3 | WR3 after its design pass. If possible, the wire stack ships in the same release as the in-memory change (D11) | +| Wave | PRs | Note | Shared files | +| --- | --- | --- | --- | +| 1 | FX1, FX2, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts | +| 2 | NR2, NR6, CD7, CD5 | NR6 merges after NR5 (D35). CD7 waits for CD1. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. NR6 and CD7: the golden snapshots of `vortex-btrblocks` | +| 3 | NR3, then NR4 | One stack. Merge NR1 and NR2 before reading the CodSpeed reports of NR3 and NR4 | None | +| 4 | NR5, CD4 | NR5 waits for FX3 | Both edit `vortex-array/src/arrays/narrow/rules.rs` | +| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | None | +| 6 | CD8a | The last in-memory step that the wire change needs | None | +| 7 | CD8b, CD8c, WR1 | Each needs only CD8a | CD8b adds a `ctx` parameter to `DecimalArray::new` and `try_new`, which CD8c and WR1 call. CD8b and CD8c both edit `vortex-array/src/builders/decimal.rs` and the same tests. Land CD8b first, or rebase the others on it | +| 8 | WR2, then WR3 | WR3 after its design pass. If possible, the wire stack ships in the same release as the in-memory change (D11) | None | The critical path runs NR1, NR2, NR3, NR4, CD4, CD6, CD8a, WR1, WR2 and WR3. The release that publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wave 1. @@ -159,7 +181,7 @@ publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wa - No PR adds temporary code (D11). The serialization workaround of D26 is the one documented exception. It stays until `vortex.narrow` is in `core`. - Benchmarks land before the changes that they cover (D19): NR2 before NR3 to NR5, and CD1 before - CD4, CD5 and CD7. + CD4, CD5 and CD7. The dependency table lists these as hard dependencies. - Every call of the deprecated `encode_values` carries `#[expect(deprecated)]` with a `TODO(connor)` reason (D28). - No `Narrow` sits directly above a generic structural encoding after optimization (D25). @@ -195,7 +217,9 @@ issue. | Risk | Mitigation | | --- | --- | | CD5 changes the CUDA decoder, which few reviewers know | A minimal change in its own commit, with a CUDA reviewer. It becomes its own PR if it grows | +| After CD6, a CUDA read of an old narrow decimal node copies the buffer to the host and returns a `Narrow`, and CUDA has no `Narrow` kernel | Accepted (D37): the host converts old nodes, which are rare. A CUDA reviewer confirms in CD6 that such a scan still works | +| The rewrite commits exist only on local branches | Push them to a backup branch before a remote agent needs them (see "How to use the briefs") | | NR6 writes signed internal children that earlier releases have not read | Readers on `develop` accept any integer type there. NR6 runs the latest release on its files | | The `take_map` regressions on #10333 have no known cause | NR6 compares CodSpeed runs on NR1, NR3, NR4 and NR6, and reads the flame graph diff | | WR3 has open design questions | A design pass before any code. The other wire PRs do not depend on it | -| The decimal critical path is long | Wave 1 starts seven PRs at once, and CD5, CD7 and NR6 need only early PRs | +| The decimal critical path is long | Wave 1 holds seven PRs with no dependencies, and CD5, CD7 and NR6 need only early PRs | diff --git a/research/decimal/agents/briefs/01-fixes.md b/research/decimal/agents/briefs/01-fixes.md index 8dd20d60896..47283a34124 100644 --- a/research/decimal/agents/briefs/01-fixes.md +++ b/research/decimal/agents/briefs/01-fixes.md @@ -13,8 +13,9 @@ Rules for all three PRs: - Each commit ends with `Signed-off-by: Connor Tsui `. Add no AI co-author line. -- The user chooses which checks to run. The acceptance checks list the commands, scoped to the - affected crates. The pinned nightly is `nightly-2026-09-10` (`.github/workflows/ci.yml:25`). +- Agents can run any command that the work needs, including the acceptance checks. The checks are + scoped to the affected crates. The pinned nightly is `nightly-2026-09-10` + (`.github/workflows/ci.yml:25`). - Tests return `VortexResult<()>` where they can and use named `rstest` cases. Tests that need a session share the module `static SESSION: LazyLock` that PR #10681 introduced. - Every "before the fix" result in these briefs comes from reading the code, not from a run. diff --git a/research/decimal/agents/briefs/02-integer-narrow.md b/research/decimal/agents/briefs/02-integer-narrow.md index 85168337d7c..5920af1409d 100644 --- a/research/decimal/agents/briefs/02-integer-narrow.md +++ b/research/decimal/agents/briefs/02-integer-narrow.md @@ -10,8 +10,13 @@ These briefs cover the integer `Narrow` track. NR1, NR3 and NR4 land the local r `b70206818f`. NR1 also gets `encode_values`, the serialization workaround and a flatten rule. NR2 is new. NR5 keeps only the dictionary-codes rule of the rewrite, and it adds the rules that move structural encodings above `Narrow` (D25). NR6 moves the existing narrowing call sites to -`encode_values` (D28). All rewrite commits and `adc1e83b9d` are reachable from every worktree, -because the worktrees share one bare repository. +`encode_values` (D28). All rewrite commits and `adc1e83b9d` are reachable from every local worktree, +because the worktrees share one bare repository. The rewrite commits are not on `origin`. Only the +local branches `ct/narrow-pushdown` (checked out in `vortex3`) and `ct/narrow-encoding` hold them, +and both names also exist on `origin` with the older commits of the closed PRs. Do not push, reset +or delete these two local branches. A remote agent cannot reach the commits until they are pushed +to a new backup branch. `adc1e83b9d`, `0d94e44a91` and `07fafc2a5b` are on +`origin/ct/narrow-compression`. No build, test or benchmark ran for these briefs. Every statement that a commit builds or passes comes from reading the code, and it is marked "Not verified" where it matters. @@ -60,7 +65,7 @@ These changes on `develop` since `61ff28d970` touch the APIs that the rewrite us | `fill_null` on an empty input | `633c52deab` (#10627) | `FillNull` returns an empty array for a non-constant fill on an empty input (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:145`). The `Narrow` rule receives a `Scalar`, so it is not affected | | Kernel for constant dictionary codes | `a169338930` (#10643) | A new execute-parent kernel handles `Constant` codes (`vortex-array/src/arrays/dict/take.rs:181-205`). The NR5 rule for `Narrow` codes is a reduce rule, so the two do not overlap | | Deprecated APIs and `legacy_session` removed | `85ae34d5f0` (#10595) | The rewrite uses none of the removed APIs | -| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064` | `ListViewArray::try_new` and `Patches` take a context. The rewrite uses neither | +| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064`, `b549eb5a97` | `ListViewArray::try_new`, `Patches::new`, `MaskedArray::try_new`, `RunEndArray::try_new` and `SparseArray::try_new` take a context. The rewrite uses none of them. The new NR5 rules rebuild `Masked`, `RunEnd` and `Sparse`, so the NR5 brief names constructors without a context | `VTable`, `OperationsVTable`, `ValidityChild`, the kernel registry, `CompareKernel`, `execute_compare`, `execute_numeric`, the reduce adaptors, `ChunkedSlots::CHUNKS_OFFSET`, @@ -127,6 +132,10 @@ to `develop` (`.github/workflows/codspeed.yml:13-17`). How CodSpeed picks a base branch is another PR branch is not verified. The safe order is to merge NR1 and NR2, then rebase NR3 to NR5 on `develop` before their CodSpeed reports are read. +`develop` squash-merges PRs. After a PR of the stack merges, a plain `git rebase develop` on the +next branch replays the commits of the merged PR again and conflicts. Rebase with +`git rebase --onto develop `, or with `gh stack`. + ### Integer-only sites in the rewrite (input for CD4) The private `validate_dtypes` is the only check that rejects a decimal dtype. Many other sites @@ -444,7 +453,8 @@ Open questions: None. - Goal: Register one fallback aggregate kernel for `Narrow`. It runs order-preserving and sum-like aggregates on the child, and it casts the partial state to the state dtype of the parent. -- Depends on: NR1. Soft: NR2. Blocks: NR4 (stack), CD4. +- Depends on: NR1, NR2 (stack, D35: the benchmarks exist before the kernels change). Soft: None. + Blocks: NR4 (stack), CD4. - Decisions: D15. - Branch: `ct/narrow-aggregate-kernels`, based on NR2. - Scope: Cherry-pick `8d548646ee`. Replace `array_session()` in its tests. @@ -646,8 +656,23 @@ Design notes: | `Narrow(Chunked(c1, ..., cn))` | `Chunked(Narrow(c1), ..., Narrow(cn))` | | `Narrow(Interleave(selectors, v1, ..., vn))` | `Interleave(selectors, Narrow(v1), ..., Narrow(vn))` | -- Every output is valid. Each new `Narrow` wraps an array of the child dtype, so `validate_dtypes` - holds. A cast of a scalar from the child dtype to the parent dtype is exact. +- Every output is valid if each new `Narrow` gets the right dtype. The array that it wraps can have + another nullability than the structural array above it. `Masked` has the nullable dtype of its + child (`vortex-array/src/arrays/masked/array.rs:84`), so its child is often non-nullable. `Dict` + values can be non-nullable under nullable codes, and `Sparse` values can be non-nullable under a + null fill. So each new `Narrow` takes the parent dtype with the nullability of the array that it + wraps: `parent.with_nullability(inner.dtype().nullability())`. Then `validate_dtypes` holds. A + cast of a scalar from the child dtype to the parent dtype is exact. +- Reduce rules have no `ExecutionCtx`, but `MaskedArray::try_new`, `RunEndArray::try_new`, + `RunEndArray::new`, `SparseArray::try_new` and `Patches::new` need one. Use the constructors + without a context: + - `Masked`: `MaskedArray::new_unchecked`. Its safety needs a child without nulls. The new + `Narrow` has the validity of the old child, so it holds. + - `RunEnd`: `RunEnd::new_unchecked(ends, values, offset, length)`, with the ends, offset and + length of the input. + - `Sparse`: `patches.map_values(..)` (`vortex-array/src/patches.rs:972`), then + `SparseArray::try_new_from_patches`. + - `Dict`, `Chunked` and `Interleave`: their `try_new` takes no context. - Where the rules live. `Dict`, `Constant`, `Masked`, `Chunked` and `Interleave` are in `vortex-array`, so their rules are `Narrow` reduce rules that match the child encoding. `RunEnd` and `Sparse` are in their own crates, which depend on `vortex-array`. So their rules are @@ -677,7 +702,9 @@ Tests: - `test_pull_up_structural`, an `rstest` over the five `vortex-array` shapes. Each result has the structural encoding at the root and a `Narrow` below it. Its values equal the input, checked with `assert_arrays_eq!`. -- `test_pull_up_nullable`: nullable children, a null constant, and `Masked` with nulls. +- `test_pull_up_nullable`: nullable children, a null constant, `Masked` with nulls over a + non-nullable child, `Dict` with nullable codes over non-nullable values, and `Sparse` with a null + fill over non-nullable values. - `test_pull_up_fixed_point`: `optimize()` on a `Chunked` of `Narrow` chunks returns the same tree. So no rule pair loops. - In `encodings/runend` and `encodings/sparse`: `Narrow` over the encoding optimizes to the encoding @@ -698,7 +725,11 @@ Acceptance checks: - After `optimize()`, no `Narrow` sits directly above one of the seven encodings. Review focus: The fill and constant casts to the parent dtype. Nullability of each new `Narrow`. -No rule pair loops. The dictionary rule copies `DictData`. +No rule pair loops. The dictionary rule copies `DictData`. Each `unsafe` constructor has a SAFETY +comment that names why its precondition holds. + +Shares files with CD4: both edit `vortex-array/src/arrays/narrow/rules.rs`. Do not run NR5 and CD4 +at the same time. Size: Medium. Seven small rules in three crates, and their tests. diff --git a/research/decimal/agents/briefs/03-canonical-decimal.md b/research/decimal/agents/briefs/03-canonical-decimal.md index 71499d2256b..0f309aa6970 100644 --- a/research/decimal/agents/briefs/03-canonical-decimal.md +++ b/research/decimal/agents/briefs/03-canonical-decimal.md @@ -18,11 +18,13 @@ These briefs cover issue #10701: one canonical width for each decimal dtype, and fill-null and cast rules (NR4). Commit `af4b663059` holds merges and indices (NR5). - Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author line. -- Verification commands are the user's choice. The acceptance checks list them for the user. +- Agents can run any command that the work needs, including the acceptance checks. Use the + narrowest command, as `AGENTS.md` describes. -Two terms recur. The canonical width of a dtype is `DecimalDType::values_type()`, which CD3 adds -(D12). The stored width of an array is `DecimalArray::values_type()`. The two are equal for every -array only after CD8a. +Two terms recur. The canonical width of a dtype is `DecimalDType::canonical_values_type()`, which +CD3 adds (D12). The name differs from the stored-width accessor on purpose, so that code cannot mix +the two up. The stored width of an array is `DecimalArray::values_type()`. The two are equal for +every array only after CD8a. ## Shared helpers @@ -30,7 +32,7 @@ Each helper has one owner. Later PRs use it and do not add their own. | Helper | Owner | Users | Contract | | --- | --- | --- | --- | -| `DecimalDType::values_type()` | CD3 | Every later PR | The canonical width of the precision (D12) | +| `DecimalDType::canonical_values_type()` | CD3 | Every later PR | The canonical width of the precision (D12) | | `copy_to_canonical_width` | CD3 | CD3 imports, CD4 execution, CD5 decoding, CD6 old nodes | Public in `vortex_array::arrays::decimal`. A narrowing copy writes zeros in null slots and fails on a valid value outside the target precision (D13, D21). A widening copy checks nothing | | `check_precision` | CD3 | The zero-copy imports of CD3, `try_new` in CD8b | Public. Fails on a valid value outside the precision. It takes a `Mask`, so it needs no `ExecutionCtx` | | The best child dtype for a range | CD4 | `NarrowArray::encode_decimal`, CD6, the `Narrow` scheme of WR3 | Public. From the min and max of the valid values, or no range, and a parent `DecimalDType`, it returns the dtype with the same scale and the smallest precision that holds the range and the scale (D10) | @@ -364,7 +366,9 @@ Design notes: `uv run vortex-test/compat-gen/scripts/compat.py publish --yes --git-ref `, which uploads every fixture to `v/arrays/`. So: 1. Merge CD2. - 2. Cut a release from a `develop` commit that has CD2 and not CD3. + 2. Cut a release from a `develop` commit that has CD2. CD3 can be in it too: CD3 changes no form + that the CD2 fixtures write, because `build_written` builds its arrays directly and not + through an import. CD5 and CD6 must not be in it. 3. Confirm that the upload job passed, and that `uv run vortex-test/compat-gen/scripts/compat.py list --version ` lists the seven new names. @@ -419,16 +423,17 @@ keeps `msp_i64_p4_s2_wide`, because it is a read that succeeds. ### CD3: One width owner, and imports copy into the canonical width -- Goal: `DecimalDType::values_type()` becomes the one owner of the width of each precision (D12). - The Arrow and DuckDB imports copy into that width, with zeros in null slots and an error on a - valid value outside the precision (D13). One shared helper performs every such copy (D21). +- Goal: `DecimalDType::canonical_values_type()` becomes the one owner of the width of each precision + (D12). The Arrow and DuckDB imports copy into that width, with zeros in null slots and an error on + a valid value outside the precision (D13). One shared helper performs every such copy (D21). - Depends on: None. Soft: CD1, CD2. Blocks: CD4, CD5, CD6, CD8a. - Decisions: D12, D13, D21 (the helper), D30 (imports check their own values), D3 (as far as imports go). - Branch: `ct/decimal-width-owner`, based on `develop`. -- Scope: Add `DecimalDType::values_type()`. Remove `DecimalType::smallest_decimal_value_type` and - migrate all its calls. Add the shared helpers `copy_to_canonical_width` and `check_precision`. - Change the Arrow and DuckDB imports, so that every import checks its values (D30). +- Scope: Add `DecimalDType::canonical_values_type()`. Remove + `DecimalType::smallest_decimal_value_type` and migrate all its calls. Add the shared helpers + `copy_to_canonical_width` and `check_precision`. Change the Arrow and DuckDB imports, so that + every import checks its values (D30). - Out of scope: Builders, including `DecimalBuilder::extend` and `append_decimal_array` (`vortex-array/src/builders/decimal.rs:298-306`, `:160-178`), and a strict `try_new` (CD8a, CD8b). DBP decoding (CD5). The read plugin (CD6). `narrowed_decimal` and the compressor (CD7). The @@ -441,30 +446,30 @@ too. Run `rg -n smallest_decimal_value_type` at the base of CD3. | # | Call | Use | New code | | --- | --- | --- | --- | -| 1 | `vortex-array/src/dtype/decimal/types.rs:73` | `is_compatible_decimal_value_type` | `self >= dtype.values_type()` | -| 2 | `vortex-array/src/dtype/dtype_impl.rs:320` | element byte width | `decimal.values_type().byte_width()` | -| 3 | `vortex-array/src/array/haystack/mod.rs:119` | IN-list width | `decimal.values_type()` | -| 4 | `vortex-array/src/builders/mod.rs:416` | builder factory width | `decimal_type.values_type()` | -| 5 | `vortex-array/src/scalar/arbitrary.rs:164` | arbitrary scalar | `decimal_type.values_type()` | -| 6 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186` | `DecimalValue::zero` | `decimal_type.values_type()` | -| 7 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209` | `DecimalValue::normalize` | `decimal_type.values_type()` | -| 8 | `vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112` | scalar work width | `work.values_type()` | -| 9 | `vortex-array/src/arrays/constant/vtable/canonical.rs:97` | constant canonicalization | `decimal_type.values_type()` | -| 10 | `vortex-array/src/arrays/constant/vtable/canonical.rs:356` | constant canonicalization | `decimal.values_type()` | -| 11 | `vortex-array/src/arrays/decimal/compute/cast.rs:155` | cast target width | `to_decimal_dtype.values_type()` | -| 12 | `vortex-array/src/arrays/arbitrary.rs:146` | arbitrary array | `decimal.values_type()` | -| 13 | `vortex-array/src/arrays/primitive/compute/cast.rs:144` | integer to decimal cast | `decimal_dtype.values_type()` | -| 14 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98` | arithmetic work width | `work_dtype.values_type()` | -| 15 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416` | arithmetic result width | `decimal_dtype.values_type()` | -| 16 | `vortex-array/src/canonical.rs:188` | empty canonical array | `decimal_type.values_type()` | -| 17 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | size statistic | `array.decimal_dtype().values_type()` | -| 18 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365` | size statistic | `decimal_type.values_type()` | -| 19 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103` | size statistic | `decimal.values_type()` | -| 20 | `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49` | sum accumulation width | `dtype.values_type()` | -| 21 | `vortex-cuda/src/kernel/arrays/constant.rs:73` | CUDA constant | `decimal_dtype.values_type()` | -| 22 | `fuzz/src/array/sum/mod.rs:54` | fuzz oracle | `output_dtype.values_type()` | -| 23 | `vortex-row/src/codec.rs:583` | row key width in `decimal_key_type` | `dt.values_type()`, keep the function and its doc | -| 24 | `encodings/sparse/src/canonical.rs:316` | sparse canonicalization | `decimal_dtype.values_type()` | +| 1 | `vortex-array/src/dtype/decimal/types.rs:73` | `is_compatible_decimal_value_type` | `self >= dtype.canonical_values_type()` | +| 2 | `vortex-array/src/dtype/dtype_impl.rs:320` | element byte width | `decimal.canonical_values_type().byte_width()` | +| 3 | `vortex-array/src/array/haystack/mod.rs:119` | IN-list width | `decimal.canonical_values_type()` | +| 4 | `vortex-array/src/builders/mod.rs:416` | builder factory width | `decimal_type.canonical_values_type()` | +| 5 | `vortex-array/src/scalar/arbitrary.rs:164` | arbitrary scalar | `decimal_type.canonical_values_type()` | +| 6 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186` | `DecimalValue::zero` | `decimal_type.canonical_values_type()` | +| 7 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209` | `DecimalValue::normalize` | `decimal_type.canonical_values_type()` | +| 8 | `vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112` | scalar work width | `work.canonical_values_type()` | +| 9 | `vortex-array/src/arrays/constant/vtable/canonical.rs:97` | constant canonicalization | `decimal_type.canonical_values_type()` | +| 10 | `vortex-array/src/arrays/constant/vtable/canonical.rs:356` | constant canonicalization | `decimal.canonical_values_type()` | +| 11 | `vortex-array/src/arrays/decimal/compute/cast.rs:155` | cast target width | `to_decimal_dtype.canonical_values_type()` | +| 12 | `vortex-array/src/arrays/arbitrary.rs:146` | arbitrary array | `decimal.canonical_values_type()` | +| 13 | `vortex-array/src/arrays/primitive/compute/cast.rs:144` | integer to decimal cast | `decimal_dtype.canonical_values_type()` | +| 14 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98` | arithmetic work width | `work_dtype.canonical_values_type()` | +| 15 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416` | arithmetic result width | `decimal_dtype.canonical_values_type()` | +| 16 | `vortex-array/src/canonical.rs:188` | empty canonical array | `decimal_type.canonical_values_type()` | +| 17 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | size statistic | `array.decimal_dtype().canonical_values_type()` | +| 18 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365` | size statistic | `decimal_type.canonical_values_type()` | +| 19 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103` | size statistic | `decimal.canonical_values_type()` | +| 20 | `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49` | sum accumulation width | `dtype.canonical_values_type()` | +| 21 | `vortex-cuda/src/kernel/arrays/constant.rs:73` | CUDA constant | `decimal_dtype.canonical_values_type()` | +| 22 | `fuzz/src/array/sum/mod.rs:54` | fuzz oracle | `output_dtype.canonical_values_type()` | +| 23 | `vortex-row/src/codec.rs:583` | row key width in `decimal_key_type` | `dt.canonical_values_type()`, keep the function and its doc | +| 24 | `encodings/sparse/src/canonical.rs:316` | sparse canonicalization | `decimal_dtype.canonical_values_type()` | | 25 | `vortex-duckdb/src/convert/vector.rs:314` | DuckDB import narrowing | Removed. The import uses the helper | Producers and imports that create a width other than the canonical width today: @@ -486,7 +491,7 @@ Files (checked at `b70206818f`): | Path | Change | | --- | --- | -| `vortex-array/src/dtype/decimal/mod.rs` | Add `DecimalDType::values_type()` in the impl block (`:43-110`). Tests in the module test block | +| `vortex-array/src/dtype/decimal/mod.rs` | Add `DecimalDType::canonical_values_type()` in the impl block (`:43-110`). Tests in the module test block | | `vortex-array/src/dtype/decimal/types.rs` | Remove `smallest_decimal_value_type` (`:45-57`). Update `is_compatible_decimal_value_type` (`:71-74`) | | The 23 other files of the table above | Mechanical migration | | `vortex-array/src/arrays/decimal/utils.rs` | Add `copy_to_canonical_width` and `check_precision`, with tests | @@ -500,7 +505,7 @@ Design notes: ```rust impl DecimalDType { /// The canonical width of the values buffer for this precision. - pub const fn values_type(&self) -> DecimalType { + pub const fn canonical_values_type(&self) -> DecimalType { match self.precision.get() { 1..=2 => DecimalType::I8, 3..=4 => DecimalType::I16, @@ -529,16 +534,16 @@ Design notes: ) -> VortexResult ``` - - The result holds `values.len()` values of `target.values_type()`. + - The result holds `values.len()` values of `target.canonical_values_type()`. - It returns an error when `validity.len()` differs from `values.len()`. - - Narrowing copy, when `target.values_type()` is narrower than `F`: every valid value must be - within `F::MIN_BY_PRECISION[p]` and `F::MAX_BY_PRECISION[p]` for the precision `p` of `target` - (`vortex-array/src/dtype/decimal/types.rs:102-108`). Otherwise the copy fails, and the error - names the row, the value and `target`. Null slots of the result hold zeros. - - Widening copy, when `target.values_type()` is wider than `F`: each value is widened, and no - value is checked. The caller must know that every valid value fits `target`. This holds when - `target` has at least the precision of the source dtype, or when the caller chose `target` from - the values (CD6). + - Narrowing copy, when `target.canonical_values_type()` is narrower than `F`: every valid value + must be within `F::MIN_BY_PRECISION[p]` and `F::MAX_BY_PRECISION[p]` for the precision `p` of + `target` (`vortex-array/src/dtype/decimal/types.rs:102-108`). Otherwise the copy fails, and the + error names the row, the value and `target`. Null slots of the result hold zeros. + - Widening copy, when `target.canonical_values_type()` is wider than `F`: each value is widened, + and no value is checked. The caller must know that every valid value fits `target`. This holds + when `target` has at least the precision of the source dtype, or when the caller chose `target` + from the values (CD6). - Same width: a plain copy without a check. A caller that owns a `Buffer` reuses it instead. - The helper never rescales. The caller keeps the scale. - Use the lane kernels for the checked loop (`vortex_compute::lane_kernels::IndexedSourceExt`, as @@ -555,10 +560,10 @@ Design notes: - It builds the dtype with `DecimalDType::try_new`, not `new` (`:184`, `:195`, `:206`, `:217`). An invalid precision or scale from FFI then returns an error, not a panic. - It builds a `Mask` from the Arrow `NullBuffer`, which is all valid when absent. - - When the Arrow native width equals `decimal_dtype.values_type()`, it keeps the zero-copy - buffer, calls `check_precision`, and builds the array with the unchecked constructor. This - covers `Decimal32` at precision 5 to 9, `Decimal64` at 10 to 18, `Decimal128` at 19 to 38, and - `Decimal256` at 39 to 76. + - When the Arrow native width equals `decimal_dtype.canonical_values_type()`, it keeps the + zero-copy buffer, calls `check_precision`, and builds the array with the unchecked constructor. + This covers `Decimal32` at precision 5 to 9, `Decimal64` at 10 to 18, `Decimal128` at 19 to 38, + and `Decimal256` at 39 to 76. - Otherwise it calls `copy_to_canonical_width` and builds the array with the unchecked constructor. `Decimal256` keeps the transmute of `:219-222` before the copy. - The error adds context with `.with_context()`: the import rejects values outside the precision, @@ -587,7 +592,7 @@ Design notes: Steps: -1. Add `DecimalDType::values_type()` with tests. +1. Add `DecimalDType::canonical_values_type()` with tests. 2. Migrate the 25 calls, remove `smallest_decimal_value_type`, and update `is_compatible_decimal_value_type`. 3. Add `copy_to_canonical_width` with tests. @@ -613,8 +618,8 @@ Tests: - `vortex-arrow/src/convert.rs` tests: - `test_decimal_import_canonical_width`: `rstest` over `Decimal32(4, 2)`, `Decimal64(9, 2)`, `Decimal128(5, 2)`, `Decimal128(18, 2)`, `Decimal128(38, 2)`, `Decimal256(38, 2)` and - `Decimal256(40, 2)`. The result has `values_type() == decimal_dtype.values_type()` and equal - values. The equal-width cases keep the Arrow buffer pointer. + `Decimal256(40, 2)`. The result has `values_type() == decimal_dtype.canonical_values_type()` and + equal values. The equal-width cases keep the Arrow buffer pointer. - `test_decimal_import_zeros_null_slots`: an Arrow `Decimal128(5, 2)` array with 10^30 in a null slot imports, and the null slot holds zero. - `test_decimal_import_rejects_value_outside_precision`: `Decimal128(5, 2)` with a valid @@ -672,7 +677,7 @@ their values in CD3 (D30). - Depends on: NR1, NR3, NR4, CD3, CD1. Soft: NR2 (bench file layout), NR5 (its structural rules also hold for decimals). Blocks: CD6, CD8a, WR1, WR3. - Decisions: D10, D15, D4, D8 (one encoding for integers and decimals, in memory only), D19, D25, - D26. + D26, D36. - Branch: `ct/decimal-narrow`, based on `develop` after NR1, NR3, NR4, CD1 and CD3 merge. - Scope: Open `validate_dtypes` for decimals, and add decimal execution and a decimal encoder. Add decimal branches to the compare, aggregate, cast and fill-null kernels. Change the decimal cast @@ -700,10 +705,13 @@ Places in the rewrite that assume an integer child. CD4 handles each one: | 11 | `af4b663059:.../narrow/rules.rs:240-279` `zip_values` | `PType::try_from`, declines | Keep the decline. Test the fallback | | 12 | `af4b663059:.../narrow/rules.rs:316-346` `BetweenReduce` | None, a scalar cast and `two_compares` | Test only. Relies on 6 and 7 | | 13 | `af4b663059:.../narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | None. A decimal cannot be a code | -| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | Decimal `Sum` and `SumV2` use the result dtype of the parent (design notes) | +| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | The kernel builds the decimal sum accumulator from the parent dtype (D36) | | 15 | `af4b663059:.../narrow/numeric.rs:69-72` | `PType::try_from`, declines | Keep the decline. Test the fallback | -| 16 | `af4b663059:.../narrow/merge.rs:112-148` `stored_inputs` | `PType::try_from`, declines | Keep the decline. Test the fallback | -| 17 | `af4b663059:.../narrow/merge.rs:77-94` interleave indices | Indices are integers | None | + +The rows cite the tip `af4b663059`, which also holds NR5 code. CD4 does not need NR5, so on its base +some of these lines are elsewhere, and `TakeIndicesReduce` (row 13) exists only after NR5. The +rewrite file `narrow/merge.rs` never lands: NR5 drops its chunked and interleave merge rules (D25). +So CD4 has no merge code to change. Files. The `narrow` module comes from NR1 to NR4, so its lines are not on `b70206818f`: @@ -714,9 +722,7 @@ Files. The `narrow` module comes from NR1 to NR4, so its lines are not on `b7020 | `vortex-array/src/arrays/narrow/encoding.rs` | `NarrowArray::encode_decimal` and the precision helper | | `vortex-array/src/arrays/narrow/compare.rs` | Decimal branches | | `vortex-array/src/arrays/narrow/rules.rs` | Decimal `CastReduce` and `FillNullReduce` | -| `vortex-array/src/arrays/narrow/aggregates.rs` | Decline decimal `Sum` and `SumV2` | -| `vortex-array/src/aggregate_fn/fns/sum/mod.rs` | `try_accumulate` (`:218-254`) sums a decimal `Narrow` child with the result dtype of the parent | -| `vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs` | The same in `try_accumulate` (`:221-244`) | +| `vortex-array/src/arrays/narrow/aggregates.rs` | The decimal sum accumulator comes from the parent dtype, and it reads the child batch (D36) | | `vortex-array/src/arrays/decimal/compute/cast.rs` | The D15 rule in `CastReduce` (`:45-82`) and `CastKernel` (`:84-170`) | | `vortex-array/src/arrays/narrow/tests/decimal.rs` | New tests, registered in `narrow/tests/mod.rs` | | The `Narrow` bench file of NR2, or `vortex-array/benches/` | Decimal `Narrow` benches | @@ -726,7 +732,7 @@ Design notes: - `validate_dtypes(child, parent)` under D10: - Two primitive dtypes follow the integer rule of D22, unchanged. - Two decimal dtypes need the same scale, the same nullability, and - `child.values_type() < parent.values_type()`. `DecimalType` orders by width + `child.canonical_values_type() < parent.canonical_values_type()`. `DecimalType` orders by width (`vortex-array/src/dtype/decimal/types.rs:25-42`). - Any other pair fails, with an error that names both dtypes. - A strictly narrower canonical width implies a lower precision. `DecimalDType::try_new` already @@ -777,21 +783,19 @@ Design notes: - Compare two decimal `Narrow` arrays: cast both children to the higher child precision at the common scale, then compare. The narrower child becomes a `Narrow` or a relabeled leaf under the cast rule. The comparison then falls back to its canonical form at the child width. -- Sum. The result dtype of a decimal sum depends on the input precision (`sum_decimal_dtype`, - `vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`). A child accumulator uses - `sum_decimal_dtype(child)`, which overflows where the parent result does not. So: - - The NR3 kernel declines decimal `Sum` and `SumV2` (`Ok(None)`). - - `Sum::try_accumulate` and `SumV2::try_accumulate` handle a batch that is a decimal `Narrow`. - They execute the child to a columnar form (`execute::`, as - `vortex-array/src/aggregate_fn/accumulator.rs:169` does) and pass it to their own `accumulate` - with the arguments of the parent. The accumulator dispatches kernels before `try_accumulate` - (`accumulator.rs:107-138`), so the decline comes first. +- Sum (decided, D36). The result dtype of a decimal sum depends on the input precision + (`sum_decimal_dtype`, `vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`). An accumulator for + the child dtype uses `sum_decimal_dtype(child)`, which overflows where the parent result does not. + A `Narrow` must never fail where its parent would not (D36). So: + - The NR3 kernel builds the sum accumulator from the dtype of the parent, and it feeds that + accumulator the child batch. `Sum` and `SumV2` need no `Narrow` code. - `accumulate_decimal` reads any input width into the width of the result dtype (`vortex-array/src/aggregate_fn/fns/sum/decimal.rs:27-61`). `multiply_constant` handles a constant child of any precision (`vortex-array/src/aggregate_fn/fns/sum/constant.rs:69-89`). So the sum never widens the child buffer. - - Alternative: keep the NR3 path, and decline when the child partial overflows. It is simpler, - but its fallback widens a child of any length. + - First, confirm that the accumulator accepts a batch whose decimal dtype differs from its input + dtype. Not verified. If it does not, stop and report, because the fix then touches the shared + accumulator code. - Min, max, first and last: the NR3 path stands. The child partial casts to the parent partial dtype, so each value becomes a value of the parent dtype. `IsSorted`, `IsConstant` and the counts need no change. Whether the struct partial of `MinMax` casts field by field is Not verified, so @@ -822,7 +826,7 @@ Steps: 1. Open `validate_dtypes` for decimals and update the docs. 2. Add decimal execution and `encode_decimal`. 3. Add the decimal compare branches. -4. Route decimal `Sum` and `SumV2` through the result dtype of the parent. +4. Build the decimal sum accumulator from the parent dtype in the NR3 kernel (D36). 5. Add the D15 cast rules on `Narrow` and on the decimal leaf, and the fill-null branch. 6. Add the conformance and fallback tests. 7. Add the benchmarks. @@ -914,9 +918,9 @@ Acceptance checks: Review focus: The child label (D10) comes only from a dtype or from `encode_decimal`. Execution never goes through the cast rule. The sum overflows only against the precision of the parent. The -compare shortcut gets the sign right for negative constants and keeps nulls. Casts that wrap set -the nullability first. Inputs at a non-canonical stored width before CD8a still give correct -results. `DecimalArray::values_type()` (stored) and `DecimalDType::values_type()` (canonical) have +compare shortcut gets the sign right for negative constants and keeps nulls. Casts that wrap set the +nullability first. Inputs at a non-canonical stored width before CD8a still give correct results. +`DecimalArray::values_type()` (stored) and `DecimalDType::canonical_values_type()` (canonical) have the same name. Size: large. One encoding opens for a second type, with four kernels, the cast kernel and broad @@ -931,6 +935,7 @@ canonical width wraps it in a `Narrow` without decoding is left to the design pa part has the dtype `u64`. Today decoding reuses the MSP buffer at the MSP width, a form that D3 removes. - Depends on: CD1, CD3, and the published CD2 fixtures (D31). Soft: FX2. Blocks: CD8a. +- Shares files with CD7 (see the CD7 brief). Do not run CD5 and CD7 at the same time. - Decisions: D16, D21, D29, D3, D12, D14. - Branch: `ct/dbp-canonical-decode`, based on `develop` after CD3 merges, or stacked on CD3. - Scope: @@ -967,8 +972,8 @@ Files (checked at `b70206818f`): Design notes: `assemble_decimal(msp, lower_parts, decimal_dtype, ctx)` keeps its signature. It returns a decimal -array whose width is `T = decimal_dtype.values_type()`. The assembled width `A` is the MSP width -with no lower parts, `i128` with one lower part, and `i256` with two or three. +array whose width is `T = decimal_dtype.canonical_values_type()`. The assembled width `A` is the MSP +width with no lower parts, `i128` with one lower part, and `i256` with two or three. | Lower parts | Case | Result | | --- | --- | --- | @@ -1024,8 +1029,8 @@ Tests: - New in `assemble.rs`: - `test_assemble_returns_canonical_width` (`rstest`): MSP ptype and precision `(I8, 2)` reuse, `(I8, 4)`, `(I32, 18)`, `(I64, 38)`, `(I16, 76)` widen, and `(I64, 9)`, `(I32, 2)` narrow, each - with and without nulls. It asserts `values_type()` against `decimal_dtype.values_type()` and the - values. + with and without nulls. It asserts `values_type()` against + `decimal_dtype.canonical_values_type()` and the values. - `test_assemble_reuses_msp_buffer`: pointer equality when the MSP width is the canonical width. - `test_assemble_narrowing_copy_zeroes_null_slots`: an `i64` MSP at `Decimal(2, 0)` with 1000 in a null slot gives 0 in that slot. @@ -1084,8 +1089,11 @@ Acceptance checks: - On a CUDA host: `cargo clippy -p vortex-cuda --all-targets --all-features -- -D warnings`, `cargo nextest run -p vortex-cuda`, and `clang-format --style=file -i vortex-cuda/kernels/src/decimal_cast.cu`. -- `cargo run -p vortex-compat --release -- check --dir --mode exact` -- Decoding gives `values_type() == decimal_dtype.values_type()` for every MSP width and part count. +- `cargo run -p vortex-compat --release -- check --dir --mode exact`, where + `` holds the files that the release with CD2 published, or files generated at a + commit before CD5. Files generated with the CD5 code only test the new code against itself. +- Decoding gives `values_type() == decimal_dtype.canonical_values_type()` for every MSP width and + part count. - The reuse path copies nothing. A narrowing copy writes zeros in null slots and fails on a valid value outside the precision. The constructor and the v2 reader reject a lower part that is not `u64`. @@ -1107,7 +1115,7 @@ than the canonical width returns an error, because the btrblocks scheme never wr dtype. The reader converts it into the canonical array or a decimal `Narrow`. After this PR no read produces a decimal leaf with a non-canonical width. - Depends on: FX1, CD3, CD4, and the published CD2 fixtures (D31). Soft: CD5. Blocks: CD8a. -- Decisions: D7, D9, D21, D14, D10, D24, D31. +- Decisions: D7, D9, D21, D14, D10, D24, D31, D37. - Branch: `ct/decimal-read-plugin`, based on `develop` after FX1, CD3 and CD4 merge, or stacked on CD4. - Scope: @@ -1154,7 +1162,7 @@ The ALP plugin is the model (`encodings/alp/src/alp/plugin.rs:33-94`). The deser (`serde.rs:419-424`), so the plugin scans the buffer (D9). `deserialize` with the stored width `W_s` and the canonical width -`W_c = decimal_dtype.values_type()`: +`W_c = decimal_dtype.canonical_values_type()`: 1. Decode the metadata with FX1. Check that the buffer has `len * W_s.byte_width()` bytes and the alignment of `W_s`. A malformed node returns an error and never panics (D14). @@ -1174,8 +1182,8 @@ The ALP plugin is the model (`encodings/alp/src/alp/plugin.rs:33-94`). The deser range and the scale. 3. `p*` never exceeds `p`, because the largest magnitude of a width below `W_c` is below `10^p - 1`. - 4. Let `W* = values_type(p*)`. If `W* == W_c`, return the canonical array, a widening copy from - `W_s`. + 4. Let `W* = canonical_values_type(p*)`. If `W* == W_c`, return the canonical array, a widening + copy from `W_s`. 5. Otherwise build the child `Decimal(p*, s)` at `W*`. Reuse the buffer if `W* == W_s`, make a widening copy if `W* > W_s`, and `copy_to_canonical_width` if `W* < W_s`. Return `NarrowArray::try_new(child, parent_dtype)`. @@ -1188,7 +1196,7 @@ label also serves the compare shortcut of CD4. | Case | Result | | --- | --- | -| All values null | `p*` comes from the scale alone. A child at `values_type(p*)`, a copy when that width differs from `W_s` | +| All values null | `p*` comes from the scale alone. A child at `canonical_values_type(p*)`, a copy when that width differs from `W_s` | | Empty array | The same rule, with an empty child | | Negative values | The min counts: `i8` -128 needs precision 3, so the child is `i16` | | Scale 0 or negative | No lower bound from the scale | @@ -1246,6 +1254,9 @@ Review focus: - The choice between reuse and copy. - The `is_supported_encoding` override, no panic on malformed input, and the host copy of device buffers. +- The CUDA read of an old narrow node (decided, D37). The host converts old narrow nodes, also on + CUDA read paths. A GPU scan of such a node copies the buffer to the host and gets a host-side + `Narrow`. State this in the PR, and ask a CUDA reviewer to confirm that such a scan still works. Size: medium. One new module and one registration change. @@ -1258,9 +1269,13 @@ its canonical array (D26). from the values. It narrows the MSP as the compressor narrows other internal children. `narrowed_decimal` goes. File contents stay the same, except for precision 39 to 76 with values that need more than 64 bits in v1 mode. -- Depends on: NR1 (`encode_values`). Soft: CD3, CD5. Blocks: CD8a, WR3. +- Depends on: NR1 (`encode_values`), CD1 (benchmarks first, D19). Soft: CD3, CD5. Blocks: CD8a, + WR3. +- Shares files with CD5: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in + `assemble.rs` and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`, + which CD5's tests call. Do not run CD5 and CD7 at the same time. - Decisions: D17, D23, D22, D19, D11, D28. -- Branch: `ct/dbp-scheme-canonical`, based on `develop` after NR1 merges. +- Branch: `ct/dbp-scheme-canonical`, based on `develop` after NR1 and CD1 merge. - Scope: `lower_part_count`, `split_decimal` with an explicit count, `DecimalByteParts::encode`, `DecimalScheme::compress`, the removal of `narrowed_decimal`, docs, golden corpus entries and tests. @@ -1439,7 +1454,7 @@ Files (checked at `b70206818f`): | Path | Change | | --- | --- | -| `vortex-array/src/arrays/decimal/vtable/mod.rs:123-149` | `validate` checks `values_type == decimal_dtype.values_type()` | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:123-149` | `validate` checks `values_type == decimal_dtype.canonical_values_type()` | | `vortex-array/src/arrays/decimal/array.rs:188-343` | `DecimalData` constructors check the width and become crate-private. No caller outside the decimal module uses them | | `vortex-array/src/arrays/decimal/array.rs:404-531` | `Array` constructors: the width rule, and conversion in `from_iter` and `from_option_iter` | | `vortex-array/src/arrays/decimal/array.rs:53-109` | Type doc: one width for each precision | @@ -1453,7 +1468,7 @@ Files (checked at `b70206818f`): Design notes: -- The width rule is `values_type == decimal_dtype.values_type()`: +- The width rule is `values_type == decimal_dtype.canonical_values_type()`: - `VTable::validate` checks it. This covers `Array::try_from_parts`, deserialization and `with_buffers`. - `try_new::` and `try_new_handle` return an error that names the dtype, the canonical width @@ -1468,7 +1483,8 @@ Design notes: `BigCast`. They panic when a value does not fit the canonical width. CD8b adds the precision check. - `DecimalBuilder::new`, `new_in`, `with_capacity` and `with_capacity_in` lose their type parameter. - The buffer variant comes from `decimal.values_type()`. `append_value::` keeps converting. + The buffer variant comes from `decimal.canonical_values_type()`. `append_value::` keeps + converting. - Constant canonicalization builds `Buffer::full(value.cast::(), len)` with `T` the canonical type. A scalar keeps any width that fits its precision (`scalar/validate.rs:58-60`), so a `DecimalValue::I128` at precision 5 gives an `i128` buffer today. Scalars from DataFusion @@ -1592,6 +1608,9 @@ Open questions: None. - Goal: `try_new` checks that every valid value fits the precision (D14). Kernels that keep values legal use `new_unchecked`. The builder and the iterator constructors check values. - Depends on: CD8a. Soft: CD1. Blocks: None. +- Shares files with CD8c and WR1. This PR adds `ctx` to `DecimalArray::new` and `try_new`, which + CD8c and WR1 call, and it edits the same builder code and tests as CD8c. Land this PR first, or + run the three one after the other. - Decisions: D14, D13, D30, XQ3, XQ4. - Branch: `ct/decimal-value-check`, based on `develop` after CD8a merges, or stacked on CD8a. - Scope: @@ -1737,6 +1756,9 @@ Open questions: None. `try_new` takes an `ExecutionCtx`, as `VarBinViewArray::tr - Goal: Delete the code that handles a decimal array whose width differs from its dtype. CD8a makes this code unreachable. - Depends on: CD8a. Soft: CD8b. Blocks: None. +- Shares files with CD8b: `vortex-array/src/builders/decimal.rs`, + `vortex-array/src/arrays/decimal/utils.rs` and several tests. Start after CD8b merges, or rebase + on it. - Decisions: D3, D14. - Branch: `ct/decimal-remove-width-workarounds`, based on `develop` after CD8a merges. - Scope: the items in the table, and their tests. @@ -1764,9 +1786,10 @@ Files (checked at `b70206818f`): | Tests | `vortex-array/src/scalar_fn/fns/binary/compare/tests.rs:575-616`, `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs:421`, `:818`, `vortex-array/src/builders/decimal.rs:347` (`test_mixed_extend`), `vortex-array/src/aggregate_fn/fns/all_non_distinct/tests.rs:283-292` | Design notes: Each change replaces a width dispatch with one dispatch over -`decimal_dtype.values_type()`. Where a `None` becomes impossible, the code uses `vortex_expect` with -a message that names the invariant. Two operands of one decimal dtype share one width after CD8a. -Scalars of a dtype fit its precision (`scalar/validate.rs:58-60`), so they fit the canonical type. +`decimal_dtype.canonical_values_type()`. Where a `None` becomes impossible, the code uses +`vortex_expect` with a message that names the invariant. Two operands of one decimal dtype share one +width after CD8a. Scalars of a dtype fit its precision (`scalar/validate.rs:58-60`), so they fit the +canonical type. Steps: diff --git a/research/decimal/agents/briefs/04-wire.md b/research/decimal/agents/briefs/04-wire.md index 5702c0878b8..5a31cb7e42d 100644 --- a/research/decimal/agents/briefs/04-wire.md +++ b/research/decimal/agents/briefs/04-wire.md @@ -5,9 +5,9 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # Wire change: briefs WR1 to WR3 -These briefs assume that NR1 to NR5 and CD3 to CD8a are on `develop`. Integer and decimal `Narrow` -then exist in memory with no wire form, and one DecimalByteParts (DBP) scheme takes the canonical -decimal array. +WR1 needs NR1, CD3, CD4 and CD8a on `develop`. WR3 also needs NR5 and CD7. In practice NR1 to NR5 +and CD3 to CD8a are all on `develop` by then. Integer and decimal `Narrow` then exist in memory with +no wire form, and one DecimalByteParts (DBP) scheme takes the canonical decimal array. Paths under `vortex-array/src/arrays/narrow/` do not exist at `b70206818f`. NR1 creates them. The briefs cite the local rewrite (`f7408481ce`) or the closed #10333 (`adc1e83b9d`) for them. Every @@ -25,11 +25,13 @@ The section [Lockstep](#lockstep) gives the reasons. error and never a panic. When the selected editions do not permit `vortex.narrow`, the writer still writes the canonical array (D26). - Depends on: NR1 (encoding, `validate_dtypes`, the lowering hook, registration), CD3 - (`DecimalDType::values_type()`), CD4 (decimal rules in `validate_dtypes`, decimal `Narrow`). - Soft: FX1, NR3, NR4, NR5, CD6. Blocks: WR2, WR3. + (`DecimalDType::canonical_values_type()`), CD4 (decimal rules in `validate_dtypes`, decimal + `Narrow`). Soft: FX1, NR3, NR4, NR5, CD6. Blocks: WR2, WR3. - Decisions: D8, D10, D14, D15 (one `Narrow` over a `Narrow`), D22, D26, D32. Invariants 1, 2 and 6 of `docs/specs/versioning/design.md:219-235`. - Branch: `ct/narrow-wire-format`, based on `develop` after CD8a. +- Shares an API with CD8b: CD8b adds `ctx` to `DecimalArray::new` and `try_new`, which the decimal + tests of this PR call. Start after CD8b merges, or rebase on it. - Scope: The serializer and deserializer of `vortex.narrow`, the documented wire contract, the lowering when the context does not permit the ID, and tests for round trips and for every rejected input. @@ -112,9 +114,10 @@ pub(super) fn child_dtype(parent: &DType, metadata: &[u8]) -> VortexResult DecimalType { + pub const fn canonical_values_type(&self) -> DecimalType { match self.precision.get() { 1..=2 => DecimalType::I8, 3..=4 => DecimalType::I16, @@ -411,7 +411,7 @@ impl DecimalDType { } impl Array { - /// Fails unless `T::DECIMAL_TYPE` equals `decimal_dtype.values_type()`. + /// Fails unless `T::DECIMAL_TYPE` equals `decimal_dtype.canonical_values_type()`. pub fn try_new( buffer: Buffer, decimal_dtype: DecimalDType, @@ -430,8 +430,8 @@ one-canonical-form fix (node N11) does not wait for the wide integers (node N9). Invariants: -- **I1, width.** The buffer width equals `decimal_dtype.values_type()`. `DecimalData::validate` - checks it next to the length and alignment checks of today +- **I1, width.** The buffer width equals `decimal_dtype.canonical_values_type()`. + `DecimalData::validate` checks it next to the length and alignment checks of today (`vortex-array/src/arrays/decimal/array.rs:301-319`). The check costs nothing for each row. - **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `try_new` checks every valid value, and `new_unchecked` skips that check for kernels that keep values legal @@ -787,7 +787,7 @@ pair of dtypes exists (C1). `Decimal` ships built-in rules for the common pairs | Cast to an integer | For a positive scale, `check_range` on the unscaled values with the inclusive bounds `MIN · 10^s` and `MAX · 10^s` of the target, then divide by `10^s` toward zero. A bound that does not fit the storage width saturates. For a negative scale, multiply by `10^-s` with a checked multiply at a width that holds the result. Then do a checked narrowing to the target. A zero value needs no factor, so it succeeds even when `10^-s` fits no integer width | Replaces `DecimalToIntegerCast` and keeps its rule: a value outside the range fails before truncation, so `127.1` fails for `i8` (`scalar/typed_view/decimal/integer_cast.rs:14-53`, test `cast_decimal_to_integer_bounds` in `arrays/decimal/compute/cast.rs:651`). The current code computes in `i256` and rejects a factor `10^s` that does not fit (`integer_cast.rs:37-39`). Under recommendation CQ1, truncation becomes a separate `trunc` step | | Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays. CQ1 also decides whether this rounding cast stays a default cast | | Cast from an integer | Widen, multiply by `10^s`, precision check | Replaces `cast_to_decimal` | -| Sum | Integer sum with accumulator `values_type(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | +| Sum | Integer sum with accumulator `canonical_values_type(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | | Min, max, min_max, bounded_min, bounded_max | Integer aggregate on the view, then a decimal of the input dtype for each result value | Matches `min_max/extension.rs:17-35`. `min_max` returns a struct of two values. Truncation applies only to `Utf8` and `Binary`, so the bounded variants return exact values for decimals | | is_sorted, is_constant | Integer aggregate on the view. The `Bool` result stays as it is | Valid within one `(p, s)` | | Hash, group by, distinct, join keys | Integer hash and equality on the view | Valid within one `(p, s)` | @@ -1749,9 +1749,9 @@ Benchmarks and fixtures, before the producers change: The in-memory change (D11). Each step is its own PR, because none of them needs temporary code: -1. Add `DecimalDType::values_type()` and route the 25 `smallest_decimal_value_type` calls through - it, so that the width function has one owner (D12). Producers and imports take the width from - the precision. Imports copy, write zeros in null slots, and fail on a value outside the +1. Add `DecimalDType::canonical_values_type()` and route the 25 `smallest_decimal_value_type` calls + through it, so that the width function has one owner (D12). Producers and imports take the width + from the precision. Imports copy, write zeros in null slots, and fail on a value outside the precision (D13). 2. `Narrow` over decimals (N28): the child rules of D10, the kernels and casts of D15, and the rule that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for decimals, the compare diff --git a/research/decisions.md b/research/decisions.md index 993a2b65816..f7b167a8219 100644 --- a/research/decisions.md +++ b/research/decisions.md @@ -59,7 +59,7 @@ decimal encoding. | D9 | A reader converts an old `vortex.decimal` node. For a narrower node, it scans the non-null values for their max, and it chooses the best precision that the values and the scale allow, or a larger one. The result is a `Narrow` over a child at the width of that precision, or the canonical array when that width is the canonical width of the column. The result copies the buffer when its width differs from the stored width. For a wider node, the reader makes a narrowing copy (D21) | A width does not give a precision. The read plugin cannot use stored statistics, because the deserializer attaches them after the plugin returns. Old narrow nodes are rare, so the speed of this path matters little | | D10 | A decimal `Narrow` child has the same scale and nullability as its parent, and a canonical width strictly narrower than the canonical width of the parent. The producer chooses the child precision: the best precision that the values and the scale allow, or a larger one | A `Narrow` that does not reduce the width has no use. The integer `Narrow` has the same rule. The rule can be relaxed later | | D11 | The canonical-width work lands as an in-memory change, and then as a wire change (D18). Each step of the in-memory change is its own PR, because none of the steps needs temporary code. The wire change lands in lockstep, if possible in the same release as the in-memory change. PRs [#10305](https://github.com/vortex-data/vortex/pull/10305) and [#10314](https://github.com/vortex-data/vortex/pull/10314) stay open, and the work ignores them for now | Small PRs are easier to review, but not when a split needs temporary code or changes file contents twice. In one release, writers that select the preview edition never pay the `i256` cost of D17 | -| D12 | Each precision has the smallest signed width that holds its values: `i8` for precision 1 to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, `i128` for 19 to 38 and `i256` for 39 to 76. This settles DQ1 | It is the existing `smallest_decimal_value_type`. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision | +| D12 | Each precision has the smallest signed width that holds its values: `i8` for precision 1 to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, `i128` for 19 to 38 and `i256` for 39 to 76. `DecimalDType::canonical_values_type()` owns the mapping. Its name differs from the stored-width accessor `DecimalArray::values_type()`, so code cannot mix the two up. This settles DQ1 | It is the existing `smallest_decimal_value_type`. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision | | D13 | Imports copy into the canonical width. They write zeros in null slots, and they fail on a value outside the precision. The Arrow import logic can be configured to handle such values in another way. To widen an array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it | arrow-rs does not check decimal values when it builds an array. An error is better than a silent change of the data | | D14 | `DecimalArray::try_new` checks the width and every valid value, with an `ExecutionCtx` (D30). `new_unchecked` skips the value check, for kernels that keep values legal. File reads trust the precision and never panic | This is the rule of X7 for decimals, and the recommendation XQ3 for file reads | | D15 | `Narrow` kernels pass work to the child, with canonicalization as the fallback. A `Narrow` over a `Narrow` becomes one `Narrow`. A cast of a `Narrow` passes to the child with the same target dtype, except that a wider integer target returns a `Narrow` over the child. Another decimal cast to a precision with the same scale returns a `Narrow` when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops | A `Narrow` holds exactly the values of its child, so work on the child gives the same result on less data | @@ -83,14 +83,16 @@ decimal encoding. | D33 | The compressor uses a real `Narrow` scheme at first. If the compressor turns out to always want it, it can become a hard-coded step. The design pass of WR3 settles its precedence with the DecimalByteParts scheme and the cascade level that it uses | A scheme fits the existing framework. The `AlwaysUse` contract and the cascade budget are the two costs to settle | | D34 | Strict decoding covers the decimal width, the protobuf `PType` and the eleven `PType` fields of array and layout metadata, in one PR with three commits | It is the same bug with the same fix | | D35 | The integer `Narrow` lands as five stacked PRs: the encoding, benchmarks, aggregates, arithmetic and casts, and the structural rules. The PR that moves the call sites to `encode_values` comes after them. The dictionary fill-code fix lands before the fifth PR | The benchmarks exist before the kernels change, and the fill-code fix guards the dictionary-codes rule | +| D36 | An operation on a `Narrow` fails only where the same operation on the parent dtype fails, and it gives the same result. Aggregates on a decimal `Narrow` use the accumulator of the parent dtype and feed it the child, so `Sum` and `SumV2` need no `Narrow` code | `Narrow` is a physical encoding, so it must not change behavior. A child accumulator overflows at a lower precision than the parent | +| D37 | The `vortex.decimal` read plugin converts old narrow nodes on the host, also on CUDA read paths. A GPU scan of such a node copies its buffer to the host | Old narrow nodes are rare, and CUDA has no `Narrow` kernel | -D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D35 from 2026-10-23. On +D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D37 from 2026-10-23. On 2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the largest precision whose canonical width is `W`, as the child precision of old data, and they planned the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it -changed D8, D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D35, and it -changed D8, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal -notes explains D7 to D35, and [part 4](decimal/04-implementation.md) splits the work into PRs. +changed D8, D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D37, and it +changed D8, D12, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal +notes explains D7 to D37, and [part 4](decimal/04-implementation.md) splits the work into PRs. ### Open From 6f261db2bed82d6c7560922fc24865b5253904d9 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 15:34:30 -0400 Subject: [PATCH 11/12] Write the integer Narrow PRs from scratch in the plan Treat the local rewrite as a reference for tests and edge cases, not as a base (D38). Build every Narrow aggregate accumulator from the parent dtype (D36), and explain D36 and D37 in the design notes. Signed-off-by: Connor Tsui --- research/decimal/02-design.md | 10 +- research/decimal/03-roadmap.md | 19 +- research/decimal/04-implementation.md | 11 +- research/decimal/README.md | 3 +- .../agents/briefs/02-integer-narrow.md | 261 +++++++++--------- .../agents/briefs/03-canonical-decimal.md | 13 +- research/decimal/agents/report.md | 18 +- research/decisions.md | 27 +- 8 files changed, 185 insertions(+), 177 deletions(-) diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md index 902576f65d5..5ec04121d72 100644 --- a/research/decimal/02-design.md +++ b/research/decimal/02-design.md @@ -193,8 +193,10 @@ on it. | Filter, take and slice | Run on the child | | Cast | Pass the cast to the child, with the same target dtype | -A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the same -scale follows three rules: +An operation on a `Narrow` fails only where the same operation on the parent fails, and it gives the +same result (decided, D36). So aggregates use the accumulator of the parent dtype and feed it the +child. A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the +same scale follows three rules: - When the canonical width grows, the cast returns a `Narrow` over its input. Nothing is copied until execution. @@ -269,7 +271,9 @@ buffer and values up to 1.27 (unscaled 127). The values need precision 3, and pr precision 4 both use `i16`. No legal array keeps the `i8` buffer, so the result is a plain `Decimal(4, 2)` array at `i16`. -Old narrow `vortex.decimal` nodes are rare, so the speed of this conversion matters little. +Old narrow `vortex.decimal` nodes are rare, so the speed of this conversion matters little. The +host converts them, also on CUDA read paths, where the buffer is first copied to the host (decided, +D37). ## DecimalByteParts diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md index 90df6172a53..c5cc2ac7dd4 100644 --- a/research/decimal/03-roadmap.md +++ b/research/decimal/03-roadmap.md @@ -9,7 +9,7 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors Many planned type changes depend on each other. The IDs below match the full graph in section 7 of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. -The decision IDs (D1 to D35, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) +The decision IDs (D1 to D38, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) come from [`decisions.md`](../decisions.md). | ID | Work | Kind | @@ -142,14 +142,15 @@ into PRs, with one brief for each PR. 4. **The integer `Narrow` in memory.** The draft PRs [#10223](https://github.com/vortex-data/vortex/pull/10223), [#10332](https://github.com/vortex-data/vortex/pull/10332) and - [#10333](https://github.com/vortex-data/vortex/pull/10333) closed on 2026-10-23. A rewrite of - #10223 and #10332 goes up as new PRs after a rebase. The rewrite makes `Narrow` in-memory only. - It lands as five stacked PRs (decided, D35). The first adds `encode` and the deprecated - `encode_values`, and a later PR moves the existing narrowing call sites to `encode_values` - (decided, D28). The last PR of the stack adds the rules that move structural encodings above - `Narrow` (decided, D25). The fix for dictionary fill codes is its own PR, before that last PR. - Decimal support needs real kernel changes, because some kernels assume an integer child. - [`decisions.md`](../decisions.md#the-narrow-pull-requests) lists the state of these PRs. + [#10333](https://github.com/vortex-data/vortex/pull/10333) closed on 2026-10-23. The new PRs are + written from scratch, with a local rewrite of #10223 and #10332 as a reference for tests and edge + cases (decided, D38). `Narrow` is in-memory only. It lands as five stacked PRs (decided, D35). + The first adds `encode` and the deprecated `encode_values`, and a later PR moves the existing + narrowing call sites to `encode_values` (decided, D28). The last PR of the stack adds the rules + that move structural encodings above `Narrow` (decided, D25). The fix for dictionary fill codes + is its own PR, before that last PR. Decimal support needs real kernel changes, because some + kernels assume an integer child. [`decisions.md`](../decisions.md#the-narrow-pull-requests) lists + the state of these PRs. ## One canonical form diff --git a/research/decimal/04-implementation.md b/research/decimal/04-implementation.md index 9b67fac7bce..35d08765b75 100644 --- a/research/decimal/04-implementation.md +++ b/research/decimal/04-implementation.md @@ -24,12 +24,15 @@ planning review on 2026-10-23. it. If a brief must change, tell that agent. - The briefs cite `develop` at `b70206818f`. Line numbers move as PRs land, so the agent reads the cited code before it changes it. Code from the local integer `Narrow` rewrite is cited with its - commit, for example `f7408481ce:vortex-array/src/arrays/narrow/mod.rs`. + commit, for example `f7408481ce:vortex-array/src/arrays/narrow/mod.rs`. That rewrite is a + reference for tests and edge cases. The PRs are written from scratch, and only FX3 ports a + rewrite commit as it is (D38). - The rewrite commits (`f7408481ce`, `8d548646ee`, `74aa17644e`, `af4b663059` and `7e2e5e4d7a`) are not on `origin`. Only the local branches `ct/narrow-pushdown` and `ct/narrow-encoding` hold them, and those names also exist on `origin` with the older commits of the closed PRs. Agents in local - worktrees of this repository can read them. A remote agent or another clone cannot. Push them to a - new backup branch before such an agent starts, and do not delete or reset the two local branches. + worktrees of this repository can read them. A remote agent or another clone cannot. Before such + an agent needs the reference, push them to a new backup branch. Do not delete or reset the two + local branches. - Agents can run any command that the work needs: builds, tests, the acceptance checks, and the generators of committed files, such as golden snapshots (NR6, CD7, WR3) and the edition record (WR2). CI fails when a committed file is stale. Use the narrowest command, as `AGENTS.md` @@ -218,7 +221,7 @@ issue. | --- | --- | | CD5 changes the CUDA decoder, which few reviewers know | A minimal change in its own commit, with a CUDA reviewer. It becomes its own PR if it grows | | After CD6, a CUDA read of an old narrow decimal node copies the buffer to the host and returns a `Narrow`, and CUDA has no `Narrow` kernel | Accepted (D37): the host converts old nodes, which are rare. A CUDA reviewer confirms in CD6 that such a scan still works | -| The rewrite commits exist only on local branches | Push them to a backup branch before a remote agent needs them (see "How to use the briefs") | +| The reference commits of the rewrite exist only on local branches | Keep the local branches. Push a backup branch before a remote agent needs the reference (see "How to use the briefs") | | NR6 writes signed internal children that earlier releases have not read | Readers on `develop` accept any integer type there. NR6 runs the latest release on its files | | The `take_map` regressions on #10333 have no known cause | NR6 compares CodSpeed runs on NR1, NR3, NR4 and NR6, and reads the flame graph diff | | WR3 has open design questions | A design pass before any code. The other wire PRs do not depend on it | diff --git a/research/decimal/README.md b/research/decimal/README.md index e74dbc06f75..a1ac0978b1f 100644 --- a/research/decimal/README.md +++ b/research/decimal/README.md @@ -45,7 +45,8 @@ canonical decimal encoding. - Decided: the bar of no compute regression applies to the final state. Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full width in memory. - Decided: the work lands as 22 PRs in four tracks. Each PR has a brief that a coding agent can - implement alone. Generic structural encodings always sit above `Narrow`. A writer that cannot + implement alone. The integer `Narrow` PRs are written from scratch, with a local rewrite as a + reference. Generic structural encodings always sit above `Narrow`. A writer that cannot write `vortex.narrow` writes the canonical array, as a documented workaround. Which arrays the compressor narrows is deferred. - Arrow has the same decimal model and the same gap at the top of each width. Its width is part diff --git a/research/decimal/agents/briefs/02-integer-narrow.md b/research/decimal/agents/briefs/02-integer-narrow.md index 5920af1409d..79dbc4b8465 100644 --- a/research/decimal/agents/briefs/02-integer-narrow.md +++ b/research/decimal/agents/briefs/02-integer-narrow.md @@ -5,67 +5,49 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # Integer `Narrow` PR briefs: NR1 to NR6 -These briefs cover the integer `Narrow` track. NR1, NR3 and NR4 land the local rewrite in -`/Users/connor/spiral/vortex-data/vortex3` (base `61ff28d970`) after a rebase onto `develop` at -`b70206818f`. NR1 also gets `encode_values`, the serialization workaround and a flatten rule. NR2 -is new. NR5 keeps only the dictionary-codes rule of the rewrite, and it adds the rules that move -structural encodings above `Narrow` (D25). NR6 moves the existing narrowing call sites to -`encode_values` (D28). All rewrite commits and `adc1e83b9d` are reachable from every local worktree, -because the worktrees share one bare repository. The rewrite commits are not on `origin`. Only the -local branches `ct/narrow-pushdown` (checked out in `vortex3`) and `ct/narrow-encoding` hold them, -and both names also exist on `origin` with the older commits of the closed PRs. Do not push, reset -or delete these two local branches. A remote agent cannot reach the commits until they are pushed -to a new backup branch. `adc1e83b9d`, `0d94e44a91` and `07fafc2a5b` are on -`origin/ct/narrow-compression`. - -No build, test or benchmark ran for these briefs. Every statement that a commit builds or passes -comes from reading the code, and it is marked "Not verified" where it matters. +These briefs cover the integer `Narrow` track. Each PR is written from scratch from its brief +(D38). A local rewrite of the closed PRs #10223 and #10332 exists in +`/Users/connor/spiral/vortex-data/vortex3`. It is a reference, not a base: read its code and tests +for edge cases, reuse its test cases, and do not cherry-pick it. The rewrite predates D25, D26, D28 +and D36, so its structure differs from these briefs. The one exception is FX3, which ports the +dictionary fill fix of the rewrite as it is. + +No build, test or benchmark ran for these briefs. Every claim about code comes from reading it, +and it is marked "Not verified" where it matters. ## Shared context for the track -### Commit map +### The reference rewrite -| Local commit | Subject | PR | Files | +| Commit | Subject | Use it for | Do not copy | | --- | --- | --- | --- | -| `f7408481ce` | feat(array): add in-memory Narrow integer encoding | NR1 | `vortex-array/src/arrays/narrow/{mod,vtable,encoding,compare,rules}.rs`, `narrow/tests/{mod,encoding,compute}.rs`, `vortex-array/src/arrays/mod.rs`, `vortex-array/src/session/mod.rs` | -| (none) | new benchmarks | NR2 | `vortex-array/benches/narrow.rs`, `vortex-array/Cargo.toml` | -| `8d548646ee` | feat(array): forward Narrow integer aggregates | NR3 | `narrow/aggregates.rs`, `narrow/mod.rs`, `narrow/tests/{mod,aggregates}.rs`, `vortex-array/src/aggregate_fn/session.rs`, `encodings/fastlanes/tests/narrow.rs` | -| `74aa17644e` | feat(array): push arithmetic and scalar operations through Narrow | NR4 | `narrow/numeric.rs`, `narrow/{rules,encoding,mod}.rs`, `narrow/tests/{mod,numeric,pushdown}.rs` | -| `af4b663059` | feat(array): retain Narrow storage through merges and indices | NR5, in part | Only `TakeIndicesReduce` in `narrow/rules.rs` and `test_narrowed_take_indices`. The chunked and interleave merge rules in `narrow/merge.rs` are dropped, because they put `Narrow` above `Chunked` and `Interleave` (D25) | -| `7e2e5e4d7a` | fix(array): avoid truncating dictionary fill codes | FX3, plus one test for NR5 | `vortex-array/src/arrays/dict/compute/fill_null.rs` (FX3), `narrow/tests/merge.rs` test `test_fill_null_after_narrowing_take_indices` (NR5, in `narrow/tests/structural.rs`) | -| `adc1e83b9d` | feat(compress): retain narrow integer children through compression | `encode_values` to NR1, the call-site switch to NR6, the rest to WR1 to WR3 | See the NR1 and NR6 briefs | - -Each commit builds on its predecessor only, by reading: - -- `f7408481ce` needs nothing but `develop`. Its tests use only NR1 code. -- `8d548646ee` needs NR1. It does not need NR2. -- `74aa17644e` needs NR1 code only. Its diff context in `narrow/mod.rs` and `narrow/tests/mod.rs` - names `mod aggregates`, so a cherry-pick without NR3 conflicts textually. NR3 is its base. -- The part of `af4b663059` that NR5 keeps needs NR4, because `TakeIndicesReduce` sits in the NR4 - version of `rules.rs`. -- NR4 rewrites the cast and fill-null rules of NR1. The NR1 versions are correct subsets, so the - churn is acceptable. +| `f7408481ce` | feat(array): add in-memory Narrow integer encoding | NR1: the width selection of `encode`, `validate_dtypes`, the compare constant shortcut, the selection rules, and the cases of `narrow/tests/{encoding,compute}.rs` | The integer-only parts of the compare kernel (see below). The serialization stub | +| `8d548646ee` | feat(array): forward Narrow integer aggregates | NR3: the list of supported aggregates, and the cases of `narrow/tests/aggregates.rs` and `encodings/fastlanes/tests/narrow.rs` | The accumulator for the child dtype. NR3 uses the accumulator of the parent dtype (D36) | +| `74aa17644e` | feat(array): push arithmetic and scalar operations through Narrow | NR4: the interval arithmetic, the cast, fill-null, zip and between rules, and the cases of `narrow/tests/{numeric,pushdown}.rs` | The replacement of the NR1 cast and fill-null rules. NR1 writes those rules so that NR4 extends them | +| `af4b663059` | feat(array): retain Narrow storage through merges and indices | NR5: `TakeIndicesReduce` and `test_narrowed_take_indices` | The chunked and interleave merge rules in `narrow/merge.rs`, which put `Narrow` above `Chunked` and `Interleave` (D25) | +| `7e2e5e4d7a` | fix(array): avoid truncating dictionary fill codes | FX3 ports the fix as it is. NR5 reuses `test_fill_null_after_narrowing_take_indices` | Nothing | +| `adc1e83b9d` | feat(compress): retain narrow integer children through compression | NR1: `encode_values`. NR6: the list of call sites. WR1 to WR3: background | The `Narrow` wrappers for internal children, and the preview edition with only `vortex.narrow` | -None of these statements is verified by a build. +All these commits are reachable from every local worktree, because the worktrees share one bare +repository. The five rewrite commits are not on `origin`. Only the local branches +`ct/narrow-pushdown` (checked out in `vortex3`) and `ct/narrow-encoding` hold them, and both names +also exist on `origin` with the older commits of the closed PRs. Do not push, reset or delete +these two local branches. A remote agent cannot read the reference until it is pushed to a new +backup branch. `adc1e83b9d`, `0d94e44a91` and `07fafc2a5b` are on `origin/ct/narrow-compression`. -### Rebase onto `develop` +### Changes on `develop` since the rewrite -`git merge-tree --write-tree --merge-base=61ff28d970 b70206818f ` reports no textual -conflict for `f7408481ce`, `8d548646ee`, `74aa17644e` and `af4b663059`. It reports one conflict -for `7e2e5e4d7a`, in `vortex-array/src/arrays/dict/compute/fill_null.rs`. That commit is FX3, not -part of this track. +The rewrite is based on `61ff28d970`. These changes on `develop` matter when an agent reads it: -These changes on `develop` since `61ff28d970` touch the APIs that the rewrite uses: - -| Change on `develop` | Commit | Effect on the rewrite | +| Change on `develop` | Commit | What new code does | | --- | --- | --- | -| `vortex_array::array_session()` removed | `d834a11faa` (#10681) | NR1 to NR5 have 39 calls and 6 `use crate::array_session` lines. FX3 adds 2 more calls. Together these are the 47 mentions that the notes count. Every one must change, with the pattern below | -| Interleave uses named slots | `96ac9f2cc5` (#10688) | `InterleaveArray::try_new(values, array_indices, row_indices)` and `InterleaveArrayExt` keep their signatures. The slot order is the same: selectors at 0 and 1, values from 2. `InterleaveSlots` is now `pub(super)` (`vortex-array/src/arrays/interleave/array.rs:24`). NR5 drops `narrow/merge.rs`, which used the literal `2`. The NR5 pull-up rule for `Interleave` builds a new array with `InterleaveArray::try_new` | -| Storage pushdown in the aggregate session | `d7562aa0e6` (#10566), `3bf14ba647` (#10616) | `AggregateFnSession::default` gained `register_storage_ordered` calls (`vortex-array/src/aggregate_fn/session.rs:127-129`). The NR3 insertion merges cleanly. `register_aggregate_kernel` and the fallback lookup are unchanged (`:199-224`) | +| `vortex_array::array_session()` removed | `d834a11faa` (#10681) | Every test module uses a module `static SESSION`, with the pattern below. The tests of the rewrite call `array_session()`, so port their cases, not their setup | +| Interleave uses named slots | `96ac9f2cc5` (#10688) | `InterleaveArray::try_new(values, array_indices, row_indices)` keeps its signature. The slot order is the same: selectors at 0 and 1, values from 2. `InterleaveSlots` is `pub(super)` (`vortex-array/src/arrays/interleave/array.rs:24`). The NR5 pull-up rule builds a new array with `InterleaveArray::try_new` | +| Storage pushdown in the aggregate session | `d7562aa0e6` (#10566), `3bf14ba647` (#10616) | `AggregateFnSession::default` has `register_storage_ordered` calls (`vortex-array/src/aggregate_fn/session.rs:127-129`). Register the NR3 kernel after the `Dict` kernels. `register_aggregate_kernel` and the fallback lookup are unchanged (`:199-224`) | | `fill_null` on an empty input | `633c52deab` (#10627) | `FillNull` returns an empty array for a non-constant fill on an empty input (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:145`). The `Narrow` rule receives a `Scalar`, so it is not affected | -| Kernel for constant dictionary codes | `a169338930` (#10643) | A new execute-parent kernel handles `Constant` codes (`vortex-array/src/arrays/dict/take.rs:181-205`). The NR5 rule for `Narrow` codes is a reduce rule, so the two do not overlap | -| Deprecated APIs and `legacy_session` removed | `85ae34d5f0` (#10595) | The rewrite uses none of the removed APIs | -| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064`, `b549eb5a97` | `ListViewArray::try_new`, `Patches::new`, `MaskedArray::try_new`, `RunEndArray::try_new` and `SparseArray::try_new` take a context. The rewrite uses none of them. The new NR5 rules rebuild `Masked`, `RunEnd` and `Sparse`, so the NR5 brief names constructors without a context | +| Kernel for constant dictionary codes | `a169338930` (#10643) | An execute-parent kernel handles `Constant` codes (`vortex-array/src/arrays/dict/take.rs:181-205`). The NR5 rule for `Narrow` codes is a reduce rule, so the two do not overlap | +| Deprecated APIs and `legacy_session` removed | `85ae34d5f0` (#10595) | None of the removed APIs is needed | +| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064`, `b549eb5a97` | `ListViewArray::try_new`, `Patches::new`, `MaskedArray::try_new`, `RunEndArray::try_new` and `SparseArray::try_new` take a context. Reduce rules have none, so the NR5 brief names constructors without a context | `VTable`, `OperationsVTable`, `ValidityChild`, the kernel registry, `CompareKernel`, `execute_compare`, `execute_numeric`, the reduce adaptors, `ChunkedSlots::CHUNKS_OFFSET`, @@ -74,7 +56,7 @@ The execute-parent registry keeps a list for each key (`vortex-array/src/optimizer/kernels.rs:263-276`). So the compare kernel of NR1 and the numeric kernel of NR4 can both register for `Binary` on `Narrow`. -#### Session replacement pattern +#### Test session pattern PR #10681 gave each test module one session. `VortexSession::empty()` creates the vortex-array session variables on first use, and the default `KernelSession` runs `arrays::initialize` @@ -89,16 +71,6 @@ use vortex_session::VortexSession; static SESSION: LazyLock = LazyLock::new(VortexSession::empty); ``` -| Before | After | -| --- | --- | -| `use crate::array_session;` | Delete | -| `let mut ctx = array_session().create_execution_ctx();` | `let mut ctx = SESSION.create_execution_ctx();` | -| `let session = array_session();` | Delete, and use `SESSION` | -| `session.aggregate_fns()` | `SESSION.aggregate_fns()` | -| `session.array_serialize(&array)?` | `SESSION.array_serialize(&array)?` | -| `array.serialize(&ctx, &session, &opts)` | `array.serialize(&ctx, &SESSION, &opts)` | -| `Narrow.deserialize(.., &session)` | `Narrow.deserialize(.., &SESSION)` | - A test outside vortex-array that needs another crate registers it in the closure, as `encodings/fastlanes/src/bitpacking/array/tests.rs:43-47` does: @@ -110,8 +82,6 @@ static SESSION: LazyLock = LazyLock::new(|| { }); ``` -No `Narrow` test changes its session after setup, so every test can share the static. - ### Stack and branches The closed PRs used `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and @@ -136,12 +106,25 @@ to NR5 on `develop` before their CodSpeed reports are read. next branch replays the commits of the merged PR again and conflicts. Rebase with `git rebase --onto develop `, or with `gh stack`. -### Integer-only sites in the rewrite (input for CD4) +### Write the kernels so that decimal support is additive + +CD4 opens `Narrow` for decimals. The rewrite has integer-only assumptions that break then (table +below). The new NR code avoids them from the start where that costs little: + +- No `as_ptype()` and no `DType::Primitive` on a path that a decimal child can reach once + `validate_dtypes` accepts decimals. For example, the compare of two `Narrow` operands picks the + wider child dtype without `as_ptype()`. +- The out-of-range compare decides the direction by comparing the constant with the bounds of the + child dtype, not through `i64::try_from` and `is_signed_int()`. +- Aggregates use the accumulator of the parent dtype (D36), so a decimal sum needs no special case. +- `execute` can still yield a cast for integers, so that the child fuses decoding and widening. + CD4 adds the decimal path, because the decimal cast of D15 returns a `Narrow`. +- Kernels that do not support decimals decline them with `Ok(None)`, so canonicalization handles + them. -The private `validate_dtypes` is the only check that rejects a decimal dtype. Many other sites -assume integers. Most of them decline a decimal with `Ok(None)` and fall back to -canonicalization, which is safe. Five sites give a panic, a wrong result or a likely execution loop -once `validate_dtypes` accepts decimals. CD4 must change those five in the same PR. +These sites of the rewrite assume integers. Most of them decline a decimal with `Ok(None)`, which +is safe. The others give a panic, a wrong result or a likely execution loop once `validate_dtypes` +accepts decimals: | Site | Assumption | Effect with a decimal `Narrow` | Safe for decimals | | --- | --- | --- | --- | @@ -168,11 +151,13 @@ once `validate_dtypes` accepts decimals. CD4 must change those five in the same null pass to the child. `Narrow` has no wire form yet, so the serializer writes its canonical array instead (D26). - Depends on: None. Soft: None. Blocks: NR2, NR3, NR4, NR5, NR6, CD4, CD7. -- Decisions: D8 (one encoding with ID `vortex.narrow`), D15, D18, D22, D23, D26, D28. +- Decisions: D8 (one encoding with ID `vortex.narrow`), D15, D18, D22, D23, D26, D28, D38. - Branch: `ct/narrow-in-memory`, based on `develop` (`b70206818f` or later). - Scope: - - Cherry-pick `f7408481ce`, and replace `array_session()` in its tests. - - Add `NarrowArray::encode_values`, marked `#[deprecated]` (D28), from `adc1e83b9d`. + - Write the encoding, `validate_dtypes`, the compare kernel, and the selection, cast and + fill-null rules. `f7408481ce` is the reference (D38). + - Add `NarrowArray::encode_values`, marked `#[deprecated]` (D28). `adc1e83b9d` has a reference + version. - Add the serialization workaround of D26: a plugin hook that lowers an array before it is serialized, which `Narrow` implements by executing itself to its canonical array. - Add a reduce rule that flattens a `Narrow` child, so D15 holds after physical rewrites. @@ -186,14 +171,14 @@ Files (checked at `b70206818f`): | Path | Change | | --- | --- | -| `vortex-array/src/arrays/narrow/mod.rs` | New, from `f7408481ce`. `Narrow`, `NarrowArray`, `NarrowSlots`, `try_new`, `validate_dtypes`, `initialize` | -| `vortex-array/src/arrays/narrow/vtable.rs` | New, from `f7408481ce`. Add `VTable::reduce` that flattens a `Narrow` child, and the lowering hook | -| `vortex-array/src/arrays/narrow/encoding.rs` | New, from `f7408481ce`. `NarrowArray::encode`, plus `encode_values` from `adc1e83b9d:vortex-array/src/arrays/narrow/encoding.rs:45-90` | -| `vortex-array/src/arrays/narrow/compare.rs` | New, from `f7408481ce`. Compare kernel | -| `vortex-array/src/arrays/narrow/rules.rs` | New, from `f7408481ce`. Slice, filter, take, mask, cast and fill-null rules | +| `vortex-array/src/arrays/narrow/mod.rs` | New. Reference: `f7408481ce`. `Narrow`, `NarrowArray`, `NarrowSlots`, `try_new`, `validate_dtypes`, `initialize` | +| `vortex-array/src/arrays/narrow/vtable.rs` | New. Reference: `f7408481ce`. With `VTable::reduce` that flattens a `Narrow` child, and the lowering hook | +| `vortex-array/src/arrays/narrow/encoding.rs` | New. `NarrowArray::encode` and `encode_values`. References: `f7408481ce` and `adc1e83b9d:vortex-array/src/arrays/narrow/encoding.rs:45-90` | +| `vortex-array/src/arrays/narrow/compare.rs` | New. Reference: `f7408481ce`. Compare kernel | +| `vortex-array/src/arrays/narrow/rules.rs` | New. Reference: `f7408481ce`. Slice, filter, take, mask, cast and fill-null rules | | `vortex-array/src/arrays/narrow/tests/mod.rs` | New | -| `vortex-array/src/arrays/narrow/tests/encoding.rs` | New. 9 `array_session()` calls and 1 import to replace. Add the flatten, `encode_values` and serialization tests | -| `vortex-array/src/arrays/narrow/tests/compute.rs` | New. 5 calls and 1 import to replace | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | New. Port the cases of the reference tests, and add the flatten, `encode_values` and serialization tests | +| `vortex-array/src/arrays/narrow/tests/compute.rs` | New. Port the cases of the reference tests | | `vortex-array/src/array/plugin.rs:113-158` | Add the lowering hook to `ArrayPlugin`, with a default of `Ok(None)`. The blanket impl for `VTable` (`:165-205`) forwards it | | `vortex-array/src/serde.rs:169-190` | `ArraySerializationTree::try_new` asks the hook when `array_serialize` returns `None` | | `vortex-array/src/arrays/mod.rs:85-91` | Add `pub mod narrow;` with `pub use narrow::Narrow;` and `pub use narrow::NarrowArray;` between `masked` and `null` | @@ -263,11 +248,12 @@ pub(super) fn validate_dtypes(storage: &DType, logical: &DType) -> VortexResult< - Compare kernel: two `Narrow` operands compare at the wider child width. A constant that casts to the child dtype compares against the child. A constant outside the child range gives a constant answer with the child validity. `Eq` is false and `NotEq` is true. `Lt`, `Lte`, `Gt` and `Gte` - follow from whether the constant is below the range. A null constant gives all nulls. + follow from whether the constant is below the range. A null constant gives all nulls. Follow + "Write the kernels so that decimal support is additive" above. - Cast rule in this PR: take an integer target of the same signedness. If it is no wider than the child, the cast goes to the child with the same target. If it is wider, the rule returns a `Narrow` over the child cast to the target nullability (D15 as amended). Other targets fall - back. NR4 widens this rule. + back. Write the cast and fill-null rules so that NR4 can add target kinds without replacing them. - Fill null declines when the fill value does not fit the child dtype, so canonical execution handles it. @@ -275,19 +261,20 @@ Steps: 1. Create `ct/narrow-in-memory` from `develop` in a new worktree. Do not reuse the `vortex3` worktree or its branch. -2. Run `git cherry-pick f7408481ce`. It applies without conflicts. -3. Replace `array_session()` in `narrow/tests/encoding.rs` and `narrow/tests/compute.rs` with the - module `SESSION` pattern of the shared section. -4. Add `encode_values` and the shared private function, with their tests. -5. Add the lowering hook in `plugin.rs` and `serde.rs`, and the `Narrow` impl, with its tests. -6. Add `VTable::reduce` to `vortex-array/src/arrays/narrow/vtable.rs` and its test. -7. Keep the cherry-picked commit `feat(array): add in-memory Narrow integer encoding`, and add steps - 3 to 6 as new commits. Each commit has the line +2. Read `f7408481ce` and its tests with `git show f7408481ce`. +3. Write the module: the encoding, `validate_dtypes`, the vtable, the compare kernel and the rules. + Register it in `arrays/mod.rs` and `session/mod.rs`. +4. Add `encode_values` and the shared private function. +5. Add the lowering hook in `plugin.rs` and `serde.rs`, and the `Narrow` impl. +6. Add `VTable::reduce` and the flatten rule. +7. Add the tests, with the module `SESSION`. Port the cases of the reference tests. +8. Commit in reviewable steps. Each commit has the line `Signed-off-by: Connor Tsui ` and no co-author line. Tests: -- From `f7408481ce`, in `narrow/tests/encoding.rs`: `test_signed_encoding` and +- Port the cases of these tests in `f7408481ce:vortex-array/src/arrays/narrow/tests/encoding.rs`: + `test_signed_encoding` and `test_unsigned_encoding` (`rstest` width boundaries), `test_null_payloads_and_empty`, `test_reject_invalid_dtypes`, `test_validate_slots`, `test_flatten_and_scalar_dtype`, `test_encoded_child_execution`, `test_smallest_width_unchanged`, `test_reject_float_encoding`, @@ -305,7 +292,8 @@ Tests: the same values. - In `vortex-array/src/serde.rs` tests: an encoding without a wire form and without the hook still fails with "does not support serialization". -- From `f7408481ce`, in `narrow/tests/compute.rs`: `test_selection_and_fill_null`, +- Port the cases of these tests in `f7408481ce:vortex-array/src/arrays/narrow/tests/compute.rs`: + `test_selection_and_fill_null`, `test_cast_preserves_checked_nullability`, `test_fill_null_preserves_result_dtype`, `test_arithmetic_uses_logical_width`, and `test_comparison`. The last one is an `rstest` over six operators. It covers constants on both sides of the child range, both operand orders, and a @@ -319,7 +307,6 @@ Acceptance checks: - `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` - `cargo +nightly-2026-09-10 fmt -p vortex-array` - `cargo nextest run -p vortex-array narrow` and `cargo nextest run -p vortex-array serde` -- `git grep -n array_session -- vortex-array/src/arrays/narrow` prints nothing. - Serializing a `Narrow` writes its canonical array, and the read-back array has equal values. - No file writer or compressor produces `vortex.narrow`. - `encode_values` carries `#[deprecated]` and has no caller yet. @@ -330,8 +317,8 @@ follow the child. The fill-null fallback. The new `reduce` rule. The lowering ho when an array has no wire form, and only `Narrow` implements it. Registration of the compare kernel through `arrays::initialize`. -Size: Large. The rewrite commit has about 550 lines of source and 435 lines of tests, mostly a -new module. `encode_values`, the hook and their tests come on top. +Size: Large. The reference commit has about 550 lines of source and 435 lines of tests. +`encode_values`, the hook and their tests come on top. Open questions: None. @@ -450,27 +437,27 @@ Open questions: None. ### NR3: Aggregates forward to the child -- Goal: Register one fallback aggregate kernel for `Narrow`. It runs order-preserving and - sum-like aggregates on the child, and it casts the partial state to the state dtype of the - parent. +- Goal: Register one fallback aggregate kernel for `Narrow`. It feeds the child to an accumulator + of the parent dtype (D36), so every result and every overflow is the same as on the parent. - Depends on: NR1, NR2 (stack, D35: the benchmarks exist before the kernels change). Soft: None. Blocks: NR4 (stack), CD4. -- Decisions: D15. +- Decisions: D15, D36, D38. - Branch: `ct/narrow-aggregate-kernels`, based on NR2. -- Scope: Cherry-pick `8d548646ee`. Replace `array_session()` in its tests. +- Scope: Write the kernel. `8d548646ee` is the reference for the list of aggregates and for test + cases. - Out of scope: Decimal aggregates (CD4). Pushdown of sum, min and max into the MSP (LT3). Files (checked at `b70206818f`): | Path | Change | | --- | --- | -| `vortex-array/src/arrays/narrow/aggregates.rs` | New, from `8d548646ee` | +| `vortex-array/src/arrays/narrow/aggregates.rs` | New. Reference: `8d548646ee` | | `vortex-array/src/arrays/narrow/mod.rs` | `mod aggregates;` and `pub(crate) use aggregates::register_aggregate_kernels;`, module doc names aggregates | | `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod aggregates;` | -| `vortex-array/src/arrays/narrow/tests/aggregates.rs` | New. 2 calls and 1 import to replace | +| `vortex-array/src/arrays/narrow/tests/aggregates.rs` | New. Port the cases of the reference tests | | `vortex-array/src/aggregate_fn/session.rs:51-52` | `use crate::arrays::narrow;` | | `vortex-array/src/aggregate_fn/session.rs:135-136` | `narrow::register_aggregate_kernels(&this);` after the `Dict` kernels | -| `encodings/fastlanes/tests/narrow.rs` | New integration test with a `BitPacked` child. Replace `vortex_array::array_session()` with the closure pattern | +| `encodings/fastlanes/tests/narrow.rs` | New integration test with a `BitPacked` child, with the closure session pattern | Design notes: @@ -485,24 +472,25 @@ struct NarrowAggregateKernel; // impl DynAggregateKernel `Count`, `NullCount`, `AllNull`, `AllNonNull`. Every other aggregate returns `None` and uses the default path. `UncompressedSizeInBytes` must stay on the default path, because it depends on the logical width. -- Algorithm: get the state dtype of the parent. Accumulate the child with an accumulator for the - child dtype. Flush, and cast the partial to the parent state dtype. +- Algorithm (D36): build the accumulator for the parent dtype, and feed it the child batch. First + confirm that each supported accumulator accepts a batch of the narrower child dtype. Not + verified. An aggregate whose accumulator rejects it leaves the supported list and uses the + default path. - Integer correctness: exact widening keeps values and order. The integer `Sum` state is `I64` for every signed input and `U64` for every unsigned input - (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:114-122`). So the child sum overflows exactly - when the parent sum does. -- Integer-only assumption: a decimal sum state depends on the input precision - (`sum/mod.rs:81-86`). Add a comment at the accumulator line that says this kernel is correct for - integers only, and that decimals need the parent state precision. CD4 owns that change. + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:114-122`). With the parent accumulator, overflow + matches the parent by construction. +- A decimal sum state depends on the input precision (`sum/mod.rs:81-86`). Because the + accumulator comes from the parent dtype, a decimal sum is correct once CD4 opens + `validate_dtypes`. CD4 only adds the decimal tests. Steps: -1. Run `git cherry-pick 8d548646ee` on the NR2 branch. -2. Replace `array_session()` in `narrow/tests/aggregates.rs` with the module `SESSION`. -3. Replace the session in `encodings/fastlanes/tests/narrow.rs` with the closure pattern. Remove - the redundant `#![cfg(test)]` at the top of that integration test. -4. Add the integer-only comment in `aggregates.rs`. -5. Amend the commit with the sign-off line. +1. Create `ct/narrow-aggregate-kernels` on the NR2 branch. +2. Write `narrow/aggregates.rs` and its registration in `aggregate_fn/session.rs`. +3. Port the test cases of `8d548646ee`, including the fastlanes integration test, with the module + sessions. That integration test needs no `#![cfg(test)]`. +4. Commit with the sign-off line. Tests: @@ -528,7 +516,7 @@ Review focus: The supported list. Each listed aggregate must give the same resul on the widened values. The cast of partial states that hold extrema or boundary values. The fallback registration does not shadow kernels for exact aggregates. -Size: Small. About 100 lines of source and 235 lines of tests. +Size: Small. The reference has about 100 lines of source and 235 lines of tests. Open questions: None. @@ -538,9 +526,11 @@ Open questions: None. operands and the result interval. Push `between`, `zip`, `fill_null` and more casts through `Narrow`. - Depends on: NR3 (stack). Soft: NR2. Blocks: NR5. -- Decisions: D15, D22. +- Decisions: D15, D22, D38. - Branch: `ct/narrow-arithmetic-kernels`, based on NR3. -- Scope: Cherry-pick `74aa17644e`. Replace `array_session()` in its tests. Fix one module doc. +- Scope: Write the numeric kernel and the rules. `74aa17644e` is the reference for the interval + arithmetic and for test cases. Extend the cast and fill-null rules of NR1 instead of replacing + them. - Out of scope: Structural rules and the dictionary-codes rule (NR5). Decimal arithmetic (D20). Decimal casts (CD4). @@ -548,13 +538,13 @@ Files (checked at `b70206818f`): | Path | Change | | --- | --- | -| `vortex-array/src/arrays/narrow/numeric.rs` | New, from `74aa17644e`. Execute-parent kernel for `Binary` | -| `vortex-array/src/arrays/narrow/rules.rs` | New cast rule, fill null through `integer_storage_type`, `ZipReduce` and `ZipFalseReduce`, `BetweenReduce` | +| `vortex-array/src/arrays/narrow/numeric.rs` | New. Reference: `74aa17644e`. Execute-parent kernel for `Binary` | +| `vortex-array/src/arrays/narrow/rules.rs` | Extend the cast and fill-null rules. Add `ZipReduce`, `ZipFalseReduce` and `BetweenReduce` | | `vortex-array/src/arrays/narrow/encoding.rs` | `integer_types` helper, used by `encode` and `numeric.rs` | | `vortex-array/src/arrays/narrow/mod.rs` | `mod numeric;`, `initialize` registers the numeric kernel | | `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod numeric;` and `mod pushdown;` | -| `vortex-array/src/arrays/narrow/tests/numeric.rs` | New. 9 calls and 1 import to replace | -| `vortex-array/src/arrays/narrow/tests/pushdown.rs` | New. 7 calls and 1 import to replace. Its module doc names concatenation and interleave, which these tests do not cover. Remove those words | +| `vortex-array/src/arrays/narrow/tests/numeric.rs` | New. Port the cases of the reference tests | +| `vortex-array/src/arrays/narrow/tests/pushdown.rs` | New. Port the cases of the reference tests | Design notes: @@ -579,10 +569,11 @@ Design notes: Steps: -1. Run `git cherry-pick 74aa17644e` on the NR3 branch. -2. Replace `array_session()` in `narrow/tests/numeric.rs` and `narrow/tests/pushdown.rs`. -3. Fix the module doc of `narrow/tests/pushdown.rs`. -4. Amend the commit with the sign-off line. +1. Create `ct/narrow-arithmetic-kernels` on the NR3 branch. +2. Write the numeric kernel, extend the cast and fill-null rules, and add the zip and between + rules. +3. Port the test cases of `74aa17644e`, with the module `SESSION`. +4. Commit in reviewable steps, with the sign-off line. Tests: @@ -616,7 +607,7 @@ Review focus: Interval arithmetic for division across zero and for `i64::MIN / - checks stay with the checked kernel at the logical width. The cast rule for unsigned children and signed targets. The `min_max` pass on a wide operand. Null payloads outside the chosen width. -Size: Large. About 450 lines of source and 425 lines of tests. +Size: Large. The reference has about 450 lines of source and 425 lines of tests. Open questions: None. @@ -626,10 +617,10 @@ Open questions: None. `Dict`, `RunEnd`, `Sparse`, `Constant`, `Masked`, `Chunked` or `Interleave` into that encoding over `Narrow` children. Dictionary codes that are a `Narrow` use its child directly. - Depends on: NR4 and FX3. Soft: NR2. Blocks: WR3. -- Decisions: D25, D15, D23. +- Decisions: D25, D15, D23, D38. - Branch: `ct/narrow-structural-rules`, based on NR4, after FX3 is on `develop`. -- Scope: Seven pull-up rules, the dictionary-codes rule `TakeIndicesReduce` from `af4b663059`, and - the `Narrow` regression test from `7e2e5e4d7a`. +- Scope: Seven pull-up rules, the dictionary-codes rule `TakeIndicesReduce` (reference: + `af4b663059`), and the case of the `Narrow` regression test from `7e2e5e4d7a`. - Out of scope: The chunked and interleave merge rules of `af4b663059` (`narrow/merge.rs`). They put `Narrow` above `Chunked` and `Interleave`, so they would loop against the new rules. The same pull-up rule for every other encoding, which a follow-up issue tracks. Decimal tests (CD4). @@ -694,8 +685,9 @@ Steps: 1. Confirm that FX3 is on `develop`. Rebase NR1 to NR4 if needed. 2. Add the five rules in `vortex-array`, with their tests. 3. Add the `RunEnd` and `Sparse` rules in their crates, with one test each. -4. Port `TakeIndicesReduce` and `test_narrowed_take_indices` from `af4b663059`. -5. Port `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`. +4. Write `TakeIndicesReduce`, with `af4b663059` as the reference, and port the cases of + `test_narrowed_take_indices`. +5. Port the case of `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`. Tests: @@ -744,7 +736,8 @@ Open questions: None. - Depends on: NR1. Soft: NR5 (merge after the stack, D35). Blocks: None. - Decisions: D28, D22, D23, D11. - Branch: `ct/narrow-encode-values`, based on `develop` after NR1 merges. -- Scope: The call sites of part 1 of `adc1e83b9d`, plus the matching lines of `0d94e44a91`. The +- Scope: The call sites that part 1 of `adc1e83b9d` changed, with that commit and `0d94e44a91` as + references. The removal of `PrimitiveArrayExt::narrow`, with its tests ported. Regenerated golden snapshots. A check that the latest release reads files that this PR writes. - Out of scope: `encode_values` itself (NR1). Part 2 of `adc1e83b9d` (WR1 to WR3). Which arrays a @@ -924,8 +917,8 @@ Open questions: None. 4. The rewrite flattens nested `Narrow` arrays only in `try_new`. Physical rewrites check only the slot dtype (`vortex-array/src/array/erased.rs:595-620`), so NR1 adds a `VTable::reduce` flatten rule. -5. The commits hold 41 `array_session()` calls and 6 `use` lines. NR1 to NR4 and the kept part of - NR5 hold 39 calls and 6 `use` lines, and FX3 holds 2 calls. +5. The tests of the rewrite call `array_session()`, which `develop` removed. New tests use the + module session, so port their cases, not their setup. 6. The report cites `narrow/mod.rs:81-84` for the width rule. That matches the tip `af4b663059`. In `f7408481ce` the same lines are `:76-79`. 7. `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and `ct/narrow-compression` diff --git a/research/decimal/agents/briefs/03-canonical-decimal.md b/research/decimal/agents/briefs/03-canonical-decimal.md index 0f309aa6970..f176958135e 100644 --- a/research/decimal/agents/briefs/03-canonical-decimal.md +++ b/research/decimal/agents/briefs/03-canonical-decimal.md @@ -11,7 +11,8 @@ These briefs cover issue #10701: one canonical width for each decimal dtype, and - A `file:line` without a commit points at the `develop` checkout at `b70206818f`. - A reference with a commit, for example `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73`, points at the local integer `Narrow` rewrite in `/Users/connor/spiral/vortex-data/vortex3`. The - NR PRs rebase that code, so those line numbers will move. + NR PRs are written from scratch with that rewrite as a reference (D38), so the landed code can + differ. Check each cited site against the NR code on `develop`. - In the rewrite, commit `f7408481ce` holds the encoding, `validate_dtypes`, the compare kernel and the cast, fill-null, filter, mask, slice and take rules (NR1). Commit `8d548646ee` holds the aggregate kernel (NR3). Commit `74aa17644e` holds arithmetic, between, zip and the wider @@ -705,7 +706,7 @@ Places in the rewrite that assume an integer child. CD4 handles each one: | 11 | `af4b663059:.../narrow/rules.rs:240-279` `zip_values` | `PType::try_from`, declines | Keep the decline. Test the fallback | | 12 | `af4b663059:.../narrow/rules.rs:316-346` `BetweenReduce` | None, a scalar cast and `two_compares` | Test only. Relies on 6 and 7 | | 13 | `af4b663059:.../narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | None. A decimal cannot be a code | -| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | The kernel builds the decimal sum accumulator from the parent dtype (D36) | +| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | NR3 builds every accumulator from the parent dtype (D36). CD4 adds the decimal tests | | 15 | `af4b663059:.../narrow/numeric.rs:69-72` | `PType::try_from`, declines | Keep the decline. Test the fallback | The rows cite the tip `af4b663059`, which also holds NR5 code. CD4 does not need NR5, so on its base @@ -722,7 +723,7 @@ Files. The `narrow` module comes from NR1 to NR4, so its lines are not on `b7020 | `vortex-array/src/arrays/narrow/encoding.rs` | `NarrowArray::encode_decimal` and the precision helper | | `vortex-array/src/arrays/narrow/compare.rs` | Decimal branches | | `vortex-array/src/arrays/narrow/rules.rs` | Decimal `CastReduce` and `FillNullReduce` | -| `vortex-array/src/arrays/narrow/aggregates.rs` | The decimal sum accumulator comes from the parent dtype, and it reads the child batch (D36) | +| `vortex-array/src/arrays/narrow/aggregates.rs` | No change expected: NR3 builds every accumulator from the parent dtype (D36). Change it only if a decimal accumulator rejects the child batch | | `vortex-array/src/arrays/decimal/compute/cast.rs` | The D15 rule in `CastReduce` (`:45-82`) and `CastKernel` (`:84-170`) | | `vortex-array/src/arrays/narrow/tests/decimal.rs` | New tests, registered in `narrow/tests/mod.rs` | | The `Narrow` bench file of NR2, or `vortex-array/benches/` | Decimal `Narrow` benches | @@ -787,8 +788,8 @@ Design notes: (`sum_decimal_dtype`, `vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`). An accumulator for the child dtype uses `sum_decimal_dtype(child)`, which overflows where the parent result does not. A `Narrow` must never fail where its parent would not (D36). So: - - The NR3 kernel builds the sum accumulator from the dtype of the parent, and it feeds that - accumulator the child batch. `Sum` and `SumV2` need no `Narrow` code. + - NR3 already builds every accumulator from the dtype of the parent, and it feeds that + accumulator the child batch. `Sum` and `SumV2` need no `Narrow` code, and CD4 adds only tests. - `accumulate_decimal` reads any input width into the width of the result dtype (`vortex-array/src/aggregate_fn/fns/sum/decimal.rs:27-61`). `multiply_constant` handles a constant child of any precision (`vortex-array/src/aggregate_fn/fns/sum/constant.rs:69-89`). @@ -826,7 +827,7 @@ Steps: 1. Open `validate_dtypes` for decimals and update the docs. 2. Add decimal execution and `encode_decimal`. 3. Add the decimal compare branches. -4. Build the decimal sum accumulator from the parent dtype in the NR3 kernel (D36). +4. Confirm with tests that decimal sums through the NR3 kernel use the parent result dtype (D36). 5. Add the D15 cast rules on `Narrow` and on the decimal leaf, and the fill-null branch. 6. Add the conformance and fallback tests. 7. Add the benchmarks. diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md index c94d2647813..10f59fa7378 100644 --- a/research/decimal/agents/report.md +++ b/research/decimal/agents/report.md @@ -629,10 +629,10 @@ at precision 39 to 76, or it comes from a write that skipped the decimal scheme. The integer `Narrow` was a draft stack: PRs #10223, #10332 and #10333. The maintainer closed all three on 2026-10-23, and new PRs replace them. Section 8.3 and the decisions file give the state on -2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory only. It goes up as new -PRs after a rebase over PR #10681, which removed `array_session()`. Its only integer-only check is -the private `validate_dtypes`, so decimal support stays additive. When that check opens for -decimals, the compare kernel and the sum pushdown need decimal code in the same change. +2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory only. The new PRs are +written from scratch, with that rewrite as a reference for tests and edge cases (D38). The rewrite +assumes integers in more places than `validate_dtypes`, so the new code avoids those assumptions +where that costs little, and CD4 handles the rest. The closed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one metadata byte that holds the child `PType`. Under D8 and D32, the metadata is a prost message with @@ -1677,6 +1677,10 @@ A planning review on 2026-10-23 split the work into 22 PRs, with one brief for e - The compressor uses a `Narrow` scheme at first (D33). - Strict decoding covers eleven more `PType` fields (D34). - The integer `Narrow` lands as five stacked PRs, then the call-site PR (D35). +- An operation on a `Narrow` fails only where the parent would fail, and aggregates use the + accumulator of the parent dtype (D36). +- The host converts old narrow nodes, also on CUDA read paths (D37). +- The integer PRs are written from scratch, with the local rewrite as a reference (D38). These notes add the integer view of section 3.2, so that decimal kernels delegate to integer kernels. D20 makes it future work. Section 9 lists the open questions with the recommendations of @@ -1733,9 +1737,9 @@ Independent changes, in any order. On 2026-10-23, none of these is on `develop`: nulls, and with only nulls. It is its own PR, with no issue. - **P3.** N1: the urgent extension changes U1 to U8. - **P4.** The integer `Narrow` in memory. #10223, #10332 and #10333 closed on 2026-10-23. The - rewrite of #10223 and #10332 goes up as new PRs after a rebase. A new PR with the first part of - #10333 (`NarrowArray::encode_values`) lands after it. The fix for dictionary fill codes is its - own PR. + new PRs are written from scratch, with the local rewrite of #10223 and #10332 as a reference + (D38). NR1 adds `NarrowArray::encode_values`, and NR6 moves the call sites to it (D28). The fix + for dictionary fill codes is its own PR, and it ports the rewrite's fix as it is. Benchmarks and fixtures, before the producers change: diff --git a/research/decisions.md b/research/decisions.md index f7b167a8219..af4741fb7c3 100644 --- a/research/decisions.md +++ b/research/decisions.md @@ -83,16 +83,18 @@ decimal encoding. | D33 | The compressor uses a real `Narrow` scheme at first. If the compressor turns out to always want it, it can become a hard-coded step. The design pass of WR3 settles its precedence with the DecimalByteParts scheme and the cascade level that it uses | A scheme fits the existing framework. The `AlwaysUse` contract and the cascade budget are the two costs to settle | | D34 | Strict decoding covers the decimal width, the protobuf `PType` and the eleven `PType` fields of array and layout metadata, in one PR with three commits | It is the same bug with the same fix | | D35 | The integer `Narrow` lands as five stacked PRs: the encoding, benchmarks, aggregates, arithmetic and casts, and the structural rules. The PR that moves the call sites to `encode_values` comes after them. The dictionary fill-code fix lands before the fifth PR | The benchmarks exist before the kernels change, and the fill-code fix guards the dictionary-codes rule | -| D36 | An operation on a `Narrow` fails only where the same operation on the parent dtype fails, and it gives the same result. Aggregates on a decimal `Narrow` use the accumulator of the parent dtype and feed it the child, so `Sum` and `SumV2` need no `Narrow` code | `Narrow` is a physical encoding, so it must not change behavior. A child accumulator overflows at a lower precision than the parent | +| D36 | An operation on a `Narrow` fails only where the same operation on the parent dtype fails, and it gives the same result. Aggregates on a `Narrow` use the accumulator of the parent dtype and feed it the child, for integers and decimals alike, so `Sum` and `SumV2` need no `Narrow` code | `Narrow` is a physical encoding, so it must not change behavior. A child accumulator overflows at a lower precision than the parent | | D37 | The `vortex.decimal` read plugin converts old narrow nodes on the host, also on CUDA read paths. A GPU scan of such a node copies its buffer to the host | Old narrow nodes are rare, and CUDA has no `Narrow` kernel | +| D38 | The integer `Narrow` PRs are written from scratch from their briefs. The local rewrite of #10223 and #10332 is a reference for tests and edge cases, not a base. FX3 ports the rewrite's dictionary fill fix as it is | The rewrite predates D25, D26, D28 and D36, so a cherry-pick would carry its old structure into the new PRs | -D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D37 from 2026-10-23. On +D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D38 from 2026-10-23. On 2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the -largest precision whose canonical width is `W`, as the child precision of old data, and they -planned the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it -changed D8, D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D37, and it -changed D8, D12, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal -notes explains D7 to D37, and [part 4](decimal/04-implementation.md) splits the work into PRs. +largest precision whose canonical width is `W`, as the child precision of old data, and they planned +the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it changed D8, +D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D38, and it changed D8, +D12, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal notes +explains the design decisions from D7 on, and [part 4](decimal/04-implementation.md) splits the work +into PRs. ### Open @@ -130,12 +132,11 @@ This review read the draft stack on 2026-10-19 and again on 2026-10-22. The stat closed all three on 2026-10-23, and new PRs replace them. Issue [#10733](https://github.com/vortex-data/vortex/issues/10733) tracks the new PRs. - A rewrite of #10223 and #10332 exists in five local commits, but it is not pushed. It makes - `Narrow` an in-memory encoding with no serialized form. Since PR - [#10681](https://github.com/vortex-data/vortex/pull/10681) merged on 2026-10-23, the rewrite - needs a rebase. That PR removed `array_session()`, which the rewrite calls 47 times, and it - conflicts with the rewrite in the test module of `dict/compute/fill_null.rs`. The local branch - has the name `ct/narrow-pushdown`, which is the head branch of the closed #10332, so the new PRs - use new branch names. + `Narrow` an in-memory encoding with no serialized form. The new PRs are written from scratch, and + the rewrite is only a reference for tests and edge cases (D38). It predates + [#10681](https://github.com/vortex-data/vortex/pull/10681), which removed `array_session()`. The + local branch has the name `ct/narrow-pushdown`, which is the head branch of the closed #10332, so + the new PRs use new branch names. - The rewrite also fixes a fill code that the dictionary `fill_null` kernel truncates. That bug exists on `develop` without `Narrow`, so the fix becomes its own PR. - The only integer-only check of #10223 and #10332 is the private `validate_dtypes`, so decimal From fe6586e553a79a14595620c5aea53cc7397007f8 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 23 Oct 2026 16:41:21 -0400 Subject: [PATCH 12/12] Record the first wave of Narrow and decimal PRs in the research Lists the draft PRs and follow-up issues opened on 2026-10-23, and notes in the CD6 brief that it must update the round-trip check that CD2 adds. Signed-off-by: Connor Tsui --- research/decimal/04-implementation.md | 37 +++++++++++++------ .../agents/briefs/03-canonical-decimal.md | 5 +++ 2 files changed, 31 insertions(+), 11 deletions(-) diff --git a/research/decimal/04-implementation.md b/research/decimal/04-implementation.md index 35d08765b75..73359c2ed25 100644 --- a/research/decimal/04-implementation.md +++ b/research/decimal/04-implementation.md @@ -83,6 +83,19 @@ Later work: LT1 puts both formats into `core` and decides the end of the seriali workaround. LT2 is the max pass in DecimalByteParts decoding (D16). LT3 is the pushdown of sum, min and max into the MSP (D19). LT2 and LT3 wait until benchmarks show that they are needed. +## Status + +The first wave opened on 2026-10-23 as draft PRs, all based on `develop` at `3974ee069c`: + +| ID | PR | Note | +| --- | --- | --- | +| FX1 | #10755 | Three commits, as planned. Its tests use a new `test_harness::assert_serde_error` helper | +| FX2 | #10746 | As planned | +| FX3 | #10745 | Fixes #10623. Non-nullable codes keep the fast path, because they need no fill code | +| CD1 | #10752 | The compress bench enables the default `core` edition, so the decimal scheme uses its v1 variant | +| CD2 | #10753 | Adds `fixtures_build_canonical_decimal_widths`, and two columns at precisions 1 and 3 | +| None | #10747 | The versioning docs describe the scheme variants that #10076 added | + ## Graph Each solid arrow is a hard dependency. Some arrows also follow from other arrows. The graph keeps @@ -203,17 +216,19 @@ publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wa ## Follow-up issues -These problems came up during planning and are outside the planned PRs. Each one gets its own -issue. - -1. Generic structural encodings always sit above other encodings. NR5 does this for `Narrow` only. -2. The decimal `min_max` aggregate panics on a valid value outside the precision, which a file read - can produce. -3. The cast rules of RunEnd, ZigZag, FoR and RLE push nullability casts down without the - `trivially_cast_nullability` guard that FX2 adds to DecimalByteParts. -4. The compat harness: no CI job runs `--mode all`, one failing fixture aborts the whole check, and - the README describes `--git-ref` wrongly. -5. Stale versioning docs about the decimal scheme and per-edition scheme configuration. +These problems came up during planning and the first wave of PRs. They are outside the planned +PRs, and each one has its own issue. + +| Issue | Problem | +| --- | --- | +| #10748 | Generic structural encodings always sit above other encodings. NR5 does this for `Narrow` only | +| #10749 | Decimal `scalar_at` and `min_max` panic on a stored value outside the precision, which a file read can produce | +| #10750 | Seven cast rules (RunEnd, ZigZag, FoR, RLE, ALP, ALP-RD, DateTimeParts) push nullability casts without the guard that FX2 adds to DecimalByteParts | +| #10751 | The FoR scheme bit-packs its child outside the allowed schemes, so its child skips the edition filter | +| #10754 | The compat harness: no CI job runs `--mode all`, one failing fixture stops the whole check, and the README describes `--git-ref` wrongly | +| #10756 | Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read | + +The stale versioning docs needed no issue, because #10747 fixes them. ## Risks diff --git a/research/decimal/agents/briefs/03-canonical-decimal.md b/research/decimal/agents/briefs/03-canonical-decimal.md index f176958135e..5bf4e09ac1b 100644 --- a/research/decimal/agents/briefs/03-canonical-decimal.md +++ b/research/decimal/agents/briefs/03-canonical-decimal.md @@ -1221,6 +1221,11 @@ Steps: Tests: +- Change `legacy_decimal_fixtures_keep_stored_widths` in + `vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs`, which CD2 (#10753) adds. Its + file round-trip check asserts that a read keeps the stored width of each old column. After CD6 + the read converts old nodes, so the check must assert the read results of the CD2 tables + instead. Keep its in-memory width check. - New in `vtable/plugin/tests.rs`. They build nodes with `ArrayDeserialization::new` and hand-written `DecimalMetadata`, as `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs:317-328` does. They build no