diff --git a/research/README.md b/research/README.md new file mode 100644 index 00000000000..1b4d7cb090b --- /dev/null +++ b/research/README.md @@ -0,0 +1,22 @@ +# Type system research + +This directory holds research notes about the Vortex type system. The notes are based on `develop` +at `5fc5cb8` (2026-10-17). They live on the `ct/ext-types-research` branch and are not meant to be +merged. + +Reviews on 2026-10-22 and 2026-10-23 updated the Decimal notes and [`decisions.md`](decisions.md). +Part 4 of the Decimal notes holds the implementation plan for `Narrow` and the canonical decimal +fix. +The decimal, DecimalByteParts, serialization and versioning code that the notes cite is the same on +`develop` at `dbaf5f8` (2026-10-22). + +| Topic | Summary for people | Detail for agents | +| --- | --- | --- | +| Extension types | [`extension-types/`](extension-types/README.md) | [`extension-types/agents/`](extension-types/agents/) | +| Decimal and the type roadmap | [`decimal/`](decimal/README.md) | [`decimal/agents/`](decimal/agents/) | + +Each summary document links to the agent documents. The agent documents hold file and line +references, reproductions, fix sketches and file lists. + +[`decisions.md`](decisions.md) records the decisions that the maintainer made while reviewing these +notes, and the questions that stay open. The other documents follow it. diff --git a/research/decimal/01-current-state.md b/research/decimal/01-current-state.md new file mode 100644 index 00000000000..853ab43f142 --- /dev/null +++ b/research/decimal/01-current-state.md @@ -0,0 +1,76 @@ + + +# 1. Current state of `Decimal` + +All statements are for `develop` at `5fc5cb8`. Section 2 of [`agents/report.md`](agents/report.md) +gives file and line references for each one. + +## How it works + +- The dtype is `DType::Decimal(DecimalDType { precision, scale }, nullability)`. The precision is + 1 to 76. +- A decimal value is stored as an unscaled integer. The value of a row is + `unscaled * 10^(-scale)`. +- `DecimalArray` stores the unscaled integers in one buffer. The metadata field `values_type` + (`I8` to `I256`) gives the width of each value in the buffer. +- The width does not depend on the precision. The array docs allow precision 39 in an `i8` buffer, + and precision 2 in an `i256` buffer. +- The usual width is the smallest integer type that holds every value of the precision. It is `I8` + for precision 1 and 2, and `I256` for precision 39 to 76. These notes call it the canonical + width. + +## What is wrong + +| Problem | Effect | +| --- | --- | +| One dtype has up to six physical forms | Kernels must handle mixed widths, widen on the fly, or test whether a constant fits the storage. `DecimalArray::patch` panics when the width cannot hold the dtype. Issue [#5820](https://github.com/vortex-data/vortex/issues/5820) collects the fuzzer crashes | +| The compressor chooses the width from the values | `narrowed_decimal` picks the smallest integer type that holds the values, not the width of a lower precision. So an `i32` buffer of a `Decimal(18, 2)` column can hold 2,000,000,000, which needs precision 10 | +| DecimalByteParts decodes to narrow leaves | Since 2025-05, the compressors put narrow values of 64 bits or less into DecimalByteParts. Its decode reuses the buffer of its most significant part, so the canonical result often has a narrow width. The only narrow `vortex.decimal` node that the compressors write is `I128` at precision 39 to 76 | +| The canonical width is a convention, not a contract | 25 call sites compute it. The row codec uses it so that keys "from differently compressed chunks stay memcmp-comparable" | +| Arrays never check the precision | Scalars check it, but arrays do not. So out-of-range values can reach kernels, and a comment in the arithmetic kernel says so | +| The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel". A throwaway test confirmed the `f32` case | +| `Decimal` duplicates `Primitive` | It has its own canonical variant (51 dispatch sites), builder, compare, arithmetic and aggregates. Decimal-specific files hold about 7,600 non-test lines | +| DecimalByteParts carries the decimal dtype | It splits wide integers into 64-bit parts, but only decimals can use it | +| The DecimalByteParts cast rule skips a nullability check | For a cast that changes only the nullability, the rule casts the most significant part. It does not first make sure that the change is safe without execution, as other cast rules do with `Validity::trivially_cast_nullability` | +| Unknown enum values decode silently | An unknown `values_type` decodes as `I8`. An unknown protobuf `PType` decodes as `U8`, and Variant metadata stores protobuf dtypes in files | +| Shared code has decimal special cases | Bloom filters, `IN` lists, size statistics, sparse arrays, run-end decoding and the row codec each have a decimal path | +| No 128-bit integer columns | DuckDB `HUGEINT` cannot be imported, although decimal storage already holds 128-bit values | + +The silent decoding causes no wrong reads today, because no writer emits unknown values. But it +blocks safe additions to both enums, so a fix is part of the independent changes in +[part 3](03-roadmap.md#independent-changes). + +## `Decimal` as a test of the extension design + +Each decimal operation is an integer operation plus bookkeeping for the scale. So `Decimal` has the +shape of an extension type over integer storage, and it is a good test of the +[extension design](../extension-types/03-design.md). It needs both kinds of behavior from that +design: operations that derive from the storage, and operations that the type must rewrite. + +| Operation | Same precision and scale | Different scales | +| --- | --- | --- | +| `=`, `<`, hash | The same operation on the unscaled integers | Rescale to a common scale first | +| `+`, `-` | An integer operation with overflow and precision checks | Rescale to a common scale first | +| `*`, `/` | A new result type, for example scale `s1 + s2` for `*` | A new result type | +| `min`, `max`, sort | The same operation on the unscaled integers | Not applicable | +| `sum` | An integer sum with a wider result precision | Not applicable | +| Casts | Owned by the type | Owned by the type | + +One precision and scale is one dtype. In the terms of the extension design, `=`, `<`, hash, `min`, +`max` and sort within one precision and scale are derives for one dtype. The other operations are +rewrites that the type owns. + +Equality and hashing need care. On `develop`, one value can have several physical widths, so the +storage bytes do not identify a value. For this reason, `DecimalValue` converts to `i256` before it +compares or hashes. With one storage width for each precision, a value has exactly one storage form +within one precision and scale. Then storage equality and storage hashing are value equality and +value hashing. + +Other pairs are different dtypes. `1.0` at scale 1 is stored as `10`, and `1.00` at scale 2 is +stored as `100`. So a comparison across scales must rescale one side first. Two precisions with +the same scale can have different storage widths, so their comparison must widen one side first. +Under decision X3 in [`decisions.md`](../decisions.md), the engine does this step with a cast to a +common dtype. Vortex then compares only equal dtypes. diff --git a/research/decimal/02-design.md b/research/decimal/02-design.md new file mode 100644 index 00000000000..5ec04121d72 --- /dev/null +++ b/research/decimal/02-design.md @@ -0,0 +1,595 @@ + + +# 2. Design + +This part follows [`decisions.md`](../decisions.md), which records the decisions of the maintainer +from 2026-10-19, 2026-10-22 and 2026-10-23. "Decided" marks a settled choice. "Recommendation" +marks a proposal of these notes that is still open. + +## Goal + +`Decimal` stays a built-in type that behaves like a hardcoded extension type (decided, D1). Each +decimal dtype has exactly one canonical encoding (decided, D3), and a `Narrow` encoding keeps +narrow storage possible (decided, D4). Issue +[#10701](https://github.com/vortex-data/vortex/issues/10701) tracks this fix. The move of decimal +kernels onto the integer kernels is future work (decided, D20). + +## The leaf + +A canonical decimal array is a leaf: one values buffer and validity, with no data child (decided, +D2). `DecimalArray` and the `vortex.decimal` wire node already have this shape. + +The precision fixes the width of the buffer (decided, D12). The mapping is the existing +`DecimalType::smallest_decimal_value_type`, which gives the smallest signed width that holds every +value of the precision: + +```rust +impl DecimalDType { + /// The width of the values buffer for this precision. + pub const fn canonical_values_type(&self) -> DecimalType { + match self.precision.get() { + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, + } + } +} +``` + +The leaf keeps three invariants: + +1. **Width.** The buffer has the width of the precision. Construction checks it, and the check + costs nothing for each row. +2. **Value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `DecimalArray::try_new` + checks every valid value, and `new_unchecked` skips that check for kernels that keep values + legal (decided, D14). The check reads the validity, so `try_new` takes an `ExecutionCtx` + (decided, D30). Imports check their own values and then use the unchecked constructors. File + reads trust the precision and never panic. A read that makes a narrowing copy checks the + precision, because it reads every value anyway (decided, D21). +3. **Leaf.** The array has one buffer and validity, and no data child. + +So a kernel never receives a width that it does not expect. The `patch` panic and the "does the +constant fit" special cases go away. + +`DecimalType` already has `I128` and `I256`. So the leaf needs no new `PType`, and this fix can +land before the wide integers. + +## Imports, exports and Arrow + +An import copies into the canonical width when the source has another width (decided, D13). It +writes zeros in null slots, and it fails on a value outside the precision. An import at the +canonical width stays zero-copy, but it scans every valid value once for the precision check +(decided, D30). Two common imports copy: + +- Arrow `Decimal128` data at precision 18 or below. This import is zero-copy today. +- DuckDB data at precision 1 to 2, which DuckDB stores as `i16`. + +The Arrow import logic can be configured to handle values outside the precision in another way. +No import options exist yet. Until they do, the import fails, and the error names arrow-rs +`null_if_overflow_precision`. + +Exports widen as they do today. The Arrow export uses `Decimal128` for every precision up to 38, +because DataFusion does not fully support `Decimal32` and `Decimal64` yet. The DuckDB export uses +`i16` for precision 1 to 4. To widen an array, Vortex executes a `Narrow` with a wider dtype over +it. + +Arrow has the same model as Vortex. A decimal type has a precision and a scale, and each value is +an unscaled two's complement integer. arrow-rs also rejects a positive scale larger than the +precision. Arrow has one decimal type for each width: + +| Arrow type | Largest precision | Vortex width for that precision | +| --- | --- | --- | +| `Decimal32` | 9 | `i32` | +| `Decimal64` | 18 | `i64` | +| `Decimal128` | 38 | `i128` | +| `Decimal256` | 76 | `i256` | + +In Arrow, the width is part of the type, so `Decimal32(5, 2)` and `Decimal128(5, 2)` are two +types. A Vortex dtype has no width, so Vortex chooses one. From precision 5, the Vortex width is +the width of the smallest Arrow type that holds the precision. Arrow has no 8-bit or 16-bit +decimal, so precision 1 to 4 has no exact Arrow match. When engines support the small Arrow types, +exports for precision 5 to 18 can stop copying. + +Arrow has the same gap as Vortex at the top of each width (see +[Width and precision](#width-and-precision)). Arrow treats a value above the precision as invalid, +but arrow-rs does not check values when it builds an array. A separate call checks them, and +another call turns them into nulls. So an Arrow array can hold invalid values, and the Vortex +import returns an error for them. + +## The integer view + +This section is future work (decided, D20). Its recommendations are not reviewed. + +The integer view is a `PrimitiveArray` over the same buffer and validity. It needs no copy. +Decimal kernels can delegate to integer kernels through it. The `Narrow` work does not need the +view, because `Narrow` kernels pass work to the decimal kernels of the child (decided, D15). + +For precision 1 to 18, the view uses `I8` to `I64`, which exist today. For precision 19 to 76, the +view needs `I128` and `I256` (decided, D5). Until those types exist, the kernels for wide decimals +stay decimal-specific. + +For extension types, `ext_storage` is the explicit way to reach the storage. For `Decimal`, the +integer view plays that role. + +## `Narrow` + +Without a narrow form, D3 forces the full width on every column. A `Narrow` encoding solves this +(decided, D4). It stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)`. The child +follows three rules (decided, D10): + +- It has the same scale and the same nullability as the parent. +- Its canonical width is strictly narrower than the canonical width of `p`. So a `Narrow` from + `Decimal(76, s)` to `Decimal(75, s)` is not allowed, because both use `i256`. The integer + `Narrow` has the same rule, and the rule can be relaxed later. +- The producer chooses `p'`: the best precision that the values and the scale allow, or a larger + one. + +`DecimalDType::try_new` rejects a positive scale larger than the precision +(`vortex-array/src/dtype/decimal/mod.rs:68`). So `p'` is at least `s`, and the scale limits how far +a column can narrow: + +| Scale `s` | Narrowest child width | +| --- | --- | +| Up to 2 | `i8` | +| 3 to 4 | `i16` | +| 5 to 9 | `i32` | +| 10 to 18 | `i64` | +| 19 to 38 | `i128` | + +This limit matters only for a column whose values all have fewer unscaled digits than the scale. +Such values have an absolute value below 1. + +A `Narrow` has three properties: + +- It has no buffers. Validity comes from the child. +- It decodes by widening the child buffer to the width of `p`. +- It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. + +One `Narrow` encoding serves integers and decimals, with one wire ID `vortex.narrow` (decided, +D8). Integers narrow by width (issue +[#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. +An integer child has the same signedness and nullability as its parent, and a strictly smaller +width (decided, D22). So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` +child. The metadata stores only the child `PType` or the child precision, in a prost message with +a typed `oneof` (decided, D32). The plugin of the parent builds the child dtype from it. The child +must have the scale and nullability of the parent, so metadata that stored them would only let a +file hold an invalid child. `vortex.narrow` stays in memory only until the maintainer decides to +write it to files (decided, D18). + +Until then, a writer that meets a `Narrow` writes its executed canonical array (decided, D26). This +also holds later for writers whose editions do not permit `vortex.narrow`. Decimal `Narrow` appears +in ordinary pipelines, through widening casts and old files, so IPC or a writer without +compression would fail without it. A plugin cannot return the wire form of another encoding, so +the fallback lives in the serializer. The code documents it as a workaround, which stays until +`vortex.narrow` is in `core`. + +`Narrow` has two producer functions (decided, D28). `encode` builds a `Narrow` over the narrowest +child of the same signedness. `encode_values` returns only that child, so it changes the dtype. +`encode_values` is `#[deprecated]`, and it replaces `PrimitiveArrayExt::narrow` at every call site +with `#[expect(deprecated)]`. So every narrowing that changes a dtype is marked for a later +decision. + +`vortex.narrow` is for arrays with a fixed logical dtype (decided, D23). Exactly which arrays the +compressor wraps is deferred to the design of the compressor scheme (decided, D27). Internal integer +children keep changing their own dtype, through `encode_values`. Examples are list offsets, +dictionary codes, run ends and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a +fixed `I64` dtype with `Narrow` is an open question on issue +[#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend +on it. + +`Narrow` kernels pass work to the child, and the fallback is to canonicalize (decided, D15): + +| Operation | Kernel on a decimal `Narrow` | +| --- | --- | +| Compare with a constant | If the constant is outside the range of the child dtype, the answer needs no values. Otherwise, compare the child with the constant at the child precision | +| Min and max | The result of the child, as a value of the parent dtype | +| Sum | The sum of the child, with the result dtype of the parent | +| Filter, take and slice | Run on the child | +| Cast | Pass the cast to the child, with the same target dtype | + +An operation on a `Narrow` fails only where the same operation on the parent fails, and it gives the +same result (decided, D36). So aggregates use the accumulator of the parent dtype and feed it the +child. A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the +same scale follows three rules: + +- When the canonical width grows, the cast returns a `Narrow` over its input. Nothing is copied + until execution. +- When the canonical width stays the same, for example from `Decimal(5, 2)` to `Decimal(9, 2)`, the + cast changes only the dtype. +- When the precision drops, the cast checks every value, and it fails on a value that does not + fit. + +### Width and precision + +Precision limits are powers of ten, and integer widths are powers of two, so they do not line up. +The table shows the largest precision for each width: + +| Width | Largest precision for the width | Largest unscaled value at that precision | Largest value of the width | +| --- | --- | --- | --- | +| `i8` | 2 | 99 | 127 | +| `i16` | 4 | 9,999 | 32,767 | +| `i32` | 9 | 999,999,999 | 2,147,483,647 | +| `i64` | 18 | 10^18 - 1 | about 9.2 * 10^18 | +| `i128` | 38 | 10^38 - 1 | about 1.7 * 10^38 | + +A value between the two limits fits the width, but it needs a precision of the next width. For +example, 2,000,000,000 fits `i32`, but it needs precision 10, whose canonical width is `i64`. So a +width alone does not give `p'`, and each producer finds `p'` from the values. A false child label +gives wrong results, because the `Narrow` rules trust the child dtype. Examples are the compare +shortcut and the cast that passes to the child. + +A gap column holds values in that range. In the final state, it uses the next width in memory +(decided, D19): + +| Column | Values | Width in memory today | Width in the final state | +| --- | --- | --- | --- | +| `Decimal(10, 2)` | 100.00 to 327.67 | `i16` | `i32`, at precision 5 | +| `Decimal(18, 2)` | 10,000,000.00 to 21,474,836.47 | `i32` | `i64`, at precision 10 | + +Disk size does not change, because integer encodings such as BitPacking store the same bits. +Exports do not change, because they already widen. Pushdown of sum, min and max into the parts of +DecimalByteParts can remove this cost for those operations later. + +### Reading old narrow data + +Old writers chose the width from the values (`narrowed_decimal`), not from a precision. A reader +finds an old node because its stored width differs from the canonical width of its dtype (decided, +D7). It then converts the node (decided, D9). For a narrower node: + +1. The reader scans the non-null values to find their max. The read plugin cannot use stored + statistics, because the deserializer attaches them after the plugin returns. +2. It chooses the best precision that the values and the scale allow, or a larger one. +3. If the width of that precision is narrower than the canonical width of the column, the result + is a `Narrow` over a child at that width. Otherwise, the result is the canonical array. +4. The result reuses the buffer when its width is the stored width. Otherwise, the result is a + copy. + +For a wider node, the reader makes a narrowing copy. Null slots can hold any value (decided, D14), +so the max ignores them. + +Every narrowing copy follows one rule (decided, D21). It writes zeros in null slots, and it fails +on a valid value outside the precision of the target dtype. Three paths make such a copy: + +- An old node that is wider than its canonical width. +- An old narrow node whose best precision has a smaller width than the stored width. +- DecimalByteParts decoding with an MSP wider than the canonical width. + +The copy reads every value anyway, so the precision check costs nothing extra. For example, an old +`Decimal(4, 2)` node at `i32` with the unscaled value 20,000 fails, because 20,000 is outside +precision 4. A read without a copy trusts the precision and does not scan. Values in a buffer +narrower than the canonical width always fit the precision, so only wider nodes can hold values +above it. + +In the gaps, the result is a copy. For example, take an old `Decimal(4, 2)` node with an `i8` +buffer and values up to 1.27 (unscaled 127). The values need precision 3, and precision 3 and +precision 4 both use `i16`. No legal array keeps the `i8` buffer, so the result is a plain +`Decimal(4, 2)` array at `i16`. + +Old narrow `vortex.decimal` nodes are rare, so the speed of this conversion matters little. The +host converts them, also on CUDA read paths, where the buffer is first copied to the host (decided, +D37). + +## DecimalByteParts + +DecimalByteParts (DBP) stays the compressed decimal form: + +- Each unscaled value is cut into 64-bit words of its two's complement bits. The most significant + part (MSP) holds the top word, and up to three lower parts hold the rest. With no lower parts, + the value is the MSP. +- The MSP is any signed integer up to 64 bits, with the nullability of the decimal, and it holds + the validity (decided, D16). Each lower part is a non-nullable `u64`, with zeros in null rows. + Today the constructor accepts a lower part of any unsigned width, and the module doc says that + each part can use a narrower dtype. But the split makes only `u64` parts, and the compressor + never narrows the dtype of a part. D16 tightens the constructor and the doc. The constructor + also rejects lower parts that assemble wider than the canonical width (decided, D29): precision + up to 18 allows none, 19 to 38 at most one, and 39 to 76 any count. This breaks no file, because + no file has lower parts. +- Each part can use any integer encoding. The parts are integers, so they promise only a width, + never a number of digits. +- One in-memory array has two wire forms. `vortex.decimal_byte_parts` (v1) has only the MSP. + `vortex.decimal_byte_parts.v2` adds lower parts, and no edition permits it yet. The writer picks + v1 when there are no lower parts. + +Operations run on the parts wherever they can (decided, D16). Today compare, between, filter, +take, slice, mask and the nullability cast do this. Every other operation decodes DBP first. + +Decoding produces the canonical array of the DBP dtype (decided, D16). Today it reuses the MSP +buffer at the MSP width. So a `Decimal(18, 2)` array with an `i32` MSP decodes to a decimal array +with an `i32` buffer. D3 removes that form, so decoding widens a narrower MSP. The MSP rule also +allows an MSP wider than the canonical width for precision up to 18, for example an `i64` MSP for +`Decimal(4, 2)`. Decoding narrows such an MSP with the copy of D21. A `TODO(connor)` in the code +marks a later optimization: find the max of the MSP, and return a `Narrow` that reuses its buffer. +It waits until benchmarks show large regressions. + +The compressor has one DBP scheme (decided, D17): + +- Today the scheme calls `narrowed_decimal`, which narrows the decimal array to the smallest + integer type that holds the values, and the split then copies its buffer into the MSP. The + canonical fix makes that narrow array illegal. Instead, the scheme takes the canonical array. + It uses the value range to choose the number of lower parts, because v1 has none. It also + narrows the MSP dtype, as the compressor narrows every internal child. The parts get the same + widths as today, so file contents do not change. +- The scheme is not temporary. Writers whose edition does not permit `vortex.narrow` need it. These + include default writers until `vortex.narrow` enters `core`, and writers that select an older + edition. +- When an edition permits `vortex.narrow`, a `Narrow` scheme runs first and narrows the decimal type + (decided, D33). It is a real scheme at first. Its design settles its precedence with the DBP + scheme, which returns `AlwaysUse`, and the cascade level that it uses. The DBP scheme then + compresses the child. So the compressor writes a `Narrow` of dtype `Decimal(p, s)` over a DBP + child of dtype `Decimal(p', s)`. The MSP of that child usually has the canonical width of `p'`, so + decoding reuses its buffer. In the gaps, decoding widens it. If the best precision has the + canonical width of the column, the compressor writes DBP without `Narrow`. +- For precision 39 to 76 with values that need more than 64 bits, the compressor uses the + lower-parts form when an edition permits it. Otherwise, it writes a plain `i256` array. Today it + writes a plain `i128` array, which D3 forbids. + +The `core` family holds the formats of the default writer +(`docs/specs/versioning/using-editions.md`). So default writers write these wide columns as plain +`i256` arrays until `vortex.narrow` and DBP v2 enter `core`. The maintainer accepts this cost. If +possible, the wire change ships in the same release as the in-memory change, so that writers that +select the preview edition never pay it (decided, D11). + +Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full +width in memory. This applies to existing files and to files that are written in that period. The +maintainer accepts this cost, because the bar of no compute regression applies to the final state +(decided, D19). + +The DBP cast rule has a bug that is independent of these changes. For a cast that changes only the +nullability, the rule casts the MSP without the check that other cast rules make with +`Validity::trivially_cast_nullability`. The fix lands as its own PR +([part 3](03-roadmap.md#independent-changes)). + +## Where structural encodings sit + +Generic structural encodings sit above the type layer (decided, X5). These notes recommend this +list for them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave` +(XQ1). Decimal storage encodings, DBP and `Narrow`, sit below them. + +For `Narrow`, reduce rules enforce this (decided, D25). They move `Dict`, `RunEnd`, `Sparse`, +`Constant`, `Masked`, `Chunked` and `Interleave` above a `Narrow`, for example +`Narrow(Dict(codes, values))` becomes `Dict(codes, Narrow(values))`. No rule moves `Narrow` above +them, so the rules cannot loop. The Arrow and DuckDB exporters and the dict layout then find these +encodings at the root. A follow-up issue covers the same rule for every encoding. + +The decimal wire forms already follow this rule. A constant decimal column is a `vortex.constant` +node with a decimal scalar, or `vortex.masked` over that node when some rows are null. Run-end and +dictionary arrays already hold decimal values. So `Decimal` needs no plugin that translates an +in-memory form back to the legacy wire forms. + +## Decimal operations as rules over the view + +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + +`Decimal` implements its semantics as rewrite rules on arrays that the type owns (decided, X4). +Each rule runs integer kernels on the view and builds a decimal result. There is no `ext_wrap` +expression. + +| Operation | Rule over the integer view | +| --- | --- | +| `=`, `<`, hash, with the same precision and scale | The same integer operation | +| `=`, `<` across precisions or scales | None. The engine casts both sides to one dtype first (decided, X3) | +| `+`, `-`, `*`, `/` with the same dtype | Widen, do a checked integer operation, then check the precision | +| `+`, `-`, `*`, `/` across precisions or scales | Open (DQ6). Recommendation: accept them, with the Arrow rules for the result type | +| Cast to `Decimal(p2, s2)` | Rescale with an exactness check, then check the precision. A wider precision with the same scale needs no copy | +| Cast to an integer | Check the target range before truncation, then divide by `10^s` toward zero. So `127.1` fails for `i8`, as today | +| Cast to a float | Convert to `f64`, divide by `10^s`, then narrow. Scalars and arrays use one path | +| `sum` | Sum the integers at a wider width, then check the precision | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max` | Run the integer aggregate, then make a decimal of each result value | +| `is_sorted`, `is_constant` | Run the integer aggregate. The `Bool` result stays as it is | + +A cast exists only where a rule for the pair of dtypes exists (decided, C1). `Decimal` ships rules +for the common pairs above. Recommendation (CQ1): the default cast fails on a value that it cannot +represent exactly, and lossy steps such as `trunc` are separate expressions. Under CQ1, the integer +cast above becomes `trunc` and then a cast. The table keeps the current behavior until CQ1 is +settled. + +These rules need two integer kernels that do not exist yet. `rescale` multiplies or divides by a +power of ten with a rounding mode, and `check_range` fails on values outside a range. The unit +casts of `Timestamp`, `Time` and `Duration` need the same kernels, because their units are powers +of ten apart (except days). The current decimal arithmetic kernel does the operation and the +precision check in one pass. A fused integer kernel keeps that speed. + +## Integer lanes or FixedSizeBinary + +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + +D2 and D3 fix the shape of the leaf. The remaining choice is the type of the values in the buffer, +and of the view over it. + +### Options + +| Option | Values in the buffer | +| --- | --- | +| (a) Integer lanes | Little-endian signed integers, as today, with an integer view up to `I256` | +| (b) FixedSizeBinary | `FixedSizeBinary(n)` values, a new core type | +| (c) Hybrid | Integers up to 64 bits, FixedSizeBinary above | +| (d) Separate wide-integer family | A new dtype and canonical array for 128 and 256 bits | +| (e) Integer lanes without a view | The buffer of (a), with decimal-specific kernels | + +### Byte order is not integer order + +FixedSizeBinary values order by byte comparison (memcmp). Arrow, DuckDB and Vortex store decimal +values as little-endian two's complement integers. Byte comparison of those bytes does not give +integer order: + +- `1` and `256` as 16-byte little-endian values: byte comparison says `256 < 1`. +- `-1` and `0`: byte comparison says `-1 > 0`. +- On 19,999 random pairs of 128-bit values, little-endian byte comparison agreed with integer order + 10,060 times. That is chance. + +Byte comparison gives integer order only for big-endian bytes with the sign bit flipped. The row +codec already uses that form. But that layout breaks zero-copy exchange with Arrow and DuckDB +decimals, and it needs a conversion before each arithmetic operation. With either layout, +FixedSizeBinary values keep every decimal kernel and add a reinterpretation step. + +### Comparison + +| | (a) Integer lanes | (b) FixedSizeBinary | (e) No view | +| --- | --- | --- | --- | +| Compare, arithmetic, aggregates | Generic integer kernels | Decimal-specific kernels | Decimal-specific kernels | +| Compression | `Narrow`, DecimalByteParts, and all integer schemes on its parts | Dictionaries and general-purpose schemes only | As (a) | +| Min and max statistics | Correct | Wrong for little-endian bytes | Correct | +| DuckDB `HUGEINT` | Possible | Not possible | Not possible | +| Decimal file format | No change | Needs FixedSizeBinary first | No change | +| Size of the change | Largest, because of the `PType` migration. The compiler finds most sites | A new core type | Smallest | +| Duplicated logic | Lowest | High | High, about 7,600 lines stay | + +### Verdict: integer lanes + +Option (a) is the proposal. The leaf keeps integer lanes, and the integer view reaches the integer +kernels. Option (e) is the state before the wide `PType` values land. It fixes the crashes, but +the kernels for precision 19 to 76 stay decimal-specific until then. + +Wide integers do not need FixedSizeBinary: + +- Rust stores `i128` natively, and Vortex already has an `i256` type with checked arithmetic. +- The shared fixed-width layer (`FixedWidthArray`) already moves records of 16 and 32 bytes for + take and filter. +- Compression of wide integers is a split into 64-bit parts, which DecimalByteParts already does. +- Arrow export of decimals uses the Arrow decimal types. Arrow has no 128-bit integer type, so only + a top-level wide-integer column needs an export rule. `FixedSizeBinary(16)` is one option for + that rule, not a prerequisite. + +## Wide integers + +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + +- `PType` gets `I128`, `I256`, `U128` and `U256` (decided, D5), for example as the values 11 to 14. +- Two typed sub-enums split the integers. `LaneIntPType` holds the eight integer types up to 64 + bits, and `WideIntPType` holds the four wide types. Offsets, codes, take indices and lane + encodings such as FastLanes accept only lane integers. +- `is_int()` goes away, so each of its 53 call sites must choose between "lane integer" and "any + integer". The dispatch macros that end in `other => panic!` retire, and typed macros replace + them. +- The compiler drives the migration. The four new variants come last, so every exhaustive match + fails to compile until it handles them. +- The flatbuffer `PType` enum gets the four values, and old readers fail loudly on them. The + protobuf `PType` enum stays as it is, because old readers decode an unknown value as `U8` without + an error. A new protobuf `oneof` variant carries wide integers instead. An edition gates + top-level wide columns. +- All wide types use 16-byte alignment. Two places assume that the alignment equals the byte + width, and they need a fix for the 256-bit types. +- Vortex has an `i256` type but no `u256` type. `U256` needs a new one. +- Wide integers get no FastLanes encodings, never act as offsets or codes, and get no CUDA dynamic + dispatch at first. + +Recommendation (DQ5): wide integers are user-visible columns behind an edition. They get a small +kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations. + +Top-level wide columns need a rule in each engine: + +| Engine | `I128` | `U128` | `I256`, `U256` | +| --- | --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `Decimal256(39, 0)` or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)` | +| DuckDB | `HUGEINT` | `UHUGEINT` | Error | +| DataFusion | Error | Error | Error | +| Python | Python `int` scalars | Python `int` scalars | Python `int` scalars | + +## FixedSizeBinary + +This section is future work outside the canonical decimal fix (decided, D20). Its +recommendations are not reviewed. + +FixedSizeBinary is a core type for opaque bytes of a fixed length (decided, D6). It refines +`Binary` in the same way that `FixedSizeList` refines `List`. + +- **Layout.** One buffer of `len * n` bytes, validity for whole values, and an explicit length so + that `n = 0` works. Take and filter come from `FixedWidthArray`. +- **Operations.** Equality and hashing by bytes, and byte order like `Binary` and Arrow. Explicit + casts to and from `Binary` and `FixedSizeList`. No arithmetic and no integer semantics. +- **UUID.** `Uuid` storage becomes `FixedSizeBinary(16)` in memory. Files keep the current + `FixedSizeList` storage until an edition allows the new form. A 128-bit lane can implement UUID + equality and hashing. Whether `Uuid` orders by its bytes is open (XQ6), with no recommendation + yet. + +Recommendation (DQ7): FixedSizeBinary appears on the wire as a new core dtype tag behind an +edition. + +`Decimal`, wide integers and FixedSizeBinary share only the fixed-width structural layer. That +layer is the right place for shared code. PR +[#8864](https://github.com/vortex-data/vortex/pull/8864) started it. + +## The file format + +The leaf is the `vortex.decimal` wire node, so the decimal file format does not change. The node +keeps its wire ID (decided, D7). The versioning spec asks for a new ID only when readers must +understand something new (`docs/specs/versioning/design.md`). D3 removes forms and adds none, so +old readers read every new file. The read plugin of `vortex.decimal` finds an old node by its +stored width, and it converts the node. Readers handle every existing decimal wire form: + +| Wire node | In memory | +| --- | --- | +| `vortex.decimal` at the canonical width | The leaf, zero-copy | +| `vortex.decimal` at a narrower width | The result of [Reading old narrow data](#reading-old-narrow-data): a `Narrow`, or the canonical array | +| `vortex.decimal` at a wider width, for example Arrow `Decimal128` data at a low precision | The leaf, after the narrowing copy of D21 with zeros in null slots. A value outside the precision is an error | +| `vortex.decimal` with an unknown width | An error, not a silent `I8` | +| DecimalByteParts v1 and v2 | DecimalByteParts, unchanged. Decoding widens a narrower MSP and narrows a wider MSP | +| `vortex.narrow`, after the wire change | A `Narrow` over its child | + +Writers produce forms that old readers understand. The leaf becomes `vortex.decimal` at the +canonical width. DBP becomes DecimalByteParts v1, which the first core edition already contains, +or v2 where an edition permits it. `Narrow` is written only where an edition permits +`vortex.narrow`, and the compressor produces it only then. Elsewhere the serializer writes the +canonical array of a `Narrow` (decided, D26). `vortex.narrow` and DBP v2 enter one +preview edition together, and later `core` together (decided, D18). Constant, dictionary and +run-end arrays of decimals keep their current wire forms. + +So wide `PType` values never reach the wire for decimal columns, and decimal statistics keep the +`Decimal` dtype. + +The compat harness (`vortex-test/compat-gen`) publishes fixtures from a released version, and the +current reader must read every published fixture. Only the old writer can make narrow leaves. So +the new fixtures are published from a release before the PRs that change how files are read: +DecimalByteParts decoding and the read plugin (decided, D24 and D31). Library changes do not wait. +The fixtures cover these cases: + +- Narrow, wide and Arrow-style widths. +- Narrow buffers with gap values. +- High scales at narrow widths. +- Nonzero values in null slots. +- DBP with a narrow MSP. + +The fixtures cover only reads that succeed. The error cases are unit tests, because the harness +expects every read to succeed. Examples are a value outside the precision in a narrowing copy, and +an unknown width. Today the decimal fixture covers only canonical `i32` and `i64` widths. + +## `Decimal` stays built in + +`Decimal` stays a built-in type (decided, D1). It behaves like a hardcoded extension type: + +- It is opaque by default. Integer semantics apply only where a rule of the type says so. +- It derives `eq` and `ord` for one precision and scale. With one width for each precision, the + integers on the view then give value equality and value order. +- Its other operations are rewrite rules that the type owns. + +A move to an extension type adds no capability. If the built-in type and an extension type become +identical, a later change can remove the built-in type. That step is optional. + +RFC 0062 proposes `Decimal` as a well-known extension type. If the optional move happens, it waits +for the extension fixes. Today, as an extension type, a cast from `Decimal(10, 2)` to `i64` returns +the unscaled integers, and `price < 5` compares `500` with `5`. + +## Open questions + +The decisions file lists these questions as open. They belong to future work (decided, D20), and +their recommendations are not reviewed. + +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ5 | Are wide integers user-visible, and which kernels come first? | User-visible behind an edition, with a small kernel set first | +| DQ6 | Does decimal arithmetic accept different precisions and scales? | Yes, with the Arrow rules. Comparisons keep the rule of equal dtypes | +| DQ7 | How does FixedSizeBinary appear on the wire? | A new core dtype tag behind an edition | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error | + +One smaller question is also future work. What is the lower bound of a negative scale? The docs +name a `MIN_SCALE` that does not exist, and tests use -128. DuckDB rejects negative scales, and +Arrow allows them. diff --git a/research/decimal/03-roadmap.md b/research/decimal/03-roadmap.md new file mode 100644 index 00000000000..c5cc2ac7dd4 --- /dev/null +++ b/research/decimal/03-roadmap.md @@ -0,0 +1,274 @@ + + +# 3. Roadmap + +## The type work + +Many planned type changes depend on each other. The IDs below match the full graph in section 7 +of [`agents/report.md`](agents/report.md), which gives a reason for each edge. There is no N25. +The decision IDs (D1 to D38, X1 to X9, C1) and the open questions (DQ5 to DQ7, XQ1 to XQ6, CQ1) +come from [`decisions.md`](../decisions.md). + +| ID | Work | Kind | +| --- | --- | --- | +| N1 | Extension opacity fixes ([part 4](../extension-types/04-urgent-changes.md) of the extension notes) | Urgent | +| N2 | Rewrite rules that the type owns, registered with the type (X4) | Research, then implementation | +| N3 | Derives for one dtype (X2) | Implementation | +| N4 | Cast rules for pairs of dtypes (C1), with cast hooks for extension types ([#8570](https://github.com/vortex-data/vortex/issues/8570)) | Implementation | +| N5 | Value checks in `try_new`, with `new_unchecked` for kernels (X7) | Implementation | +| N6 | Nullability on the extension dtype | Research, optional | +| N7 | Shared fixed-width structural kernels | Implementation | +| N8 | Scaled-integer kernels: `rescale` and `check_range` | Implementation | +| N9 | Wide integers: `I128`, `I256`, `U128` and `U256` (D5) | Implementation | +| N10 | Wire safety: strict enum decoding, and edition gates for core dtypes | Implementation | +| N11 | One canonical width for each decimal dtype (D3) | Urgent | +| N12 | Zero-copy integer view of the decimal leaf | Implementation | +| N13 | Decimal operations as rules over the view | Implementation | +| N14 | Limb split for wide-integer columns | Implementation, optional | +| N15 | `Decimal` as an extension type (D1 keeps it built in) | Optional, later | +| N16 | Wire translation for well-known types | Research, then implementation | +| N17 | FixedSizeBinary core type (D6) | Implementation | +| N18 | `Uuid` on FixedSizeBinary | Implementation | +| N19 | `Utf8` as a well-known extension type | Research, low priority | +| N20 | `FixedSizeList` and `Map` as well-known extension types | Research, low priority | +| N21 | `Duration` and `Interval` extension types ([#10467](https://github.com/vortex-data/vortex/pull/10467), [#10468](https://github.com/vortex-data/vortex/pull/10468)) | In progress | +| N22 | Execution layouts (RFC 0062) | Research | +| N23 | Engine type plugins for DuckDB and DataFusion | Implementation | +| N24 | Refinement types (RFC 0055) | Research | +| N26 | Scalar value model (RFC 0056) | Research, then implementation | +| N27 | DuckDB `HUGEINT` and `UHUGEINT` import and export | Implementation | +| N28 | One `Narrow` encoding for integers and decimals (D4) | In progress | + +For decimals, N9, N12, N13, N14, N17, N18 and N27 are future work outside the canonical decimal fix +(D20). The decimal part of N5, the value check in `DecimalArray::try_new`, moves into N11 (D14). + +## Graph + +A solid line is a hard dependency: the later work must not land before the earlier work. A dashed +line is a soft dependency: the earlier work makes the later work cheaper. This graph shows all hard +dependencies and the five most useful soft ones. The report lists every edge. + +```mermaid +flowchart TD + subgraph EXT["Extension types"] + N1["N1 Opacity fixes"] + N2["N2 Rules owned by the type"] + N3["N3 Derives for one dtype"] + N4["N4 Cast rules"] + end + subgraph DEC["Decimal"] + N28["N28 One Narrow encoding"] + N11["N11 One canonical width"] + N12["N12 Integer view"] + N7["N7 Fixed-width kernels"] + N8["N8 Scaled-integer kernels"] + N9["N9 Wide integers"] + N13["N13 Rules over the view"] + N14["N14 Limb split, optional"] + end + subgraph BYTES["Bytes"] + N17["N17 FixedSizeBinary"] + N18["N18 Uuid on FixedSizeBinary"] + end + subgraph WIRE["Wire and engines"] + N10["N10 Wire safety"] + N16["N16 Wire translation"] + N27["N27 DuckDB HUGEINT and UHUGEINT"] + end + subgraph LATER["Later or optional"] + N15["N15 Decimal as extension type, optional"] + N19["N19 Utf8 as extension type"] + N20["N20 FixedSizeList and Map as extension types"] + N22["N22 Execution layouts"] + end + N21["N21 Duration and Interval"] + + N2 --> N3 + N1 --> N21 + N4 --> N21 + N28 --> N11 + N9 --> N12 + N12 --> N13 + N8 --> N13 + N9 --> N14 + N9 --> N27 + N10 --> N27 + N17 --> N18 + N1 --> N15 + N3 --> N15 + N4 --> N15 + N13 --> N15 + N16 --> N15 + N16 --> N19 + N16 --> N20 + N22 --> N19 + N22 --> N20 + N11 -.-> N12 + N28 -.-> N13 + N7 -.-> N9 + N7 -.-> N17 + N8 -.-> N4 +``` + +The nodes N5, N6, N23, N24 and N26 have only soft edges, so the graph leaves them out. + +Two edges need a note. N9 to N12 applies only to precision 19 to 76. The view for precision 1 to +18 uses existing types, so it can land at any point. N11 does not depend on N9, because the leaf +uses `DecimalType`, which already has 128-bit and 256-bit widths. + +## Independent changes + +These changes are small and independent, and they can land in any order. On 2026-10-23, none of +them is on `develop`. [Part 4](04-implementation.md) splits these changes and the canonical form +into PRs, with one brief for each PR. + +1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and + the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a + silent `I8` or `U8`. Readers must have this change before any writer emits a new value. The + decimal part must also land before the read plugin converts old nodes, because the detection + of old nodes reads this field. The same PR fixes eleven `PType` fields of array and layout + metadata with the same fallback (decided, D34). +2. **The DecimalByteParts nullability fix.** For a cast that changes only the nullability, the + cast rule casts the MSP without the check that `Validity::trivially_cast_nullability` makes in + other cast rules. The fix adds that check, as PR + [#10230](https://github.com/vortex-data/vortex/pull/10230) did. It also adds a cast kernel for + the case that needs execution, so that a declined cast still runs on the parts. It lands as its + own PR with tests, and it needs no issue. +3. **Extension opacity fixes (N1).** These are the urgent changes in + [part 4 of the extension notes](../extension-types/04-urgent-changes.md). +4. **The integer `Narrow` in memory.** The draft PRs + [#10223](https://github.com/vortex-data/vortex/pull/10223), + [#10332](https://github.com/vortex-data/vortex/pull/10332) and + [#10333](https://github.com/vortex-data/vortex/pull/10333) closed on 2026-10-23. The new PRs are + written from scratch, with a local rewrite of #10223 and #10332 as a reference for tests and edge + cases (decided, D38). `Narrow` is in-memory only. It lands as five stacked PRs (decided, D35). + The first adds `encode` and the deprecated `encode_values`, and a later PR moves the existing + narrowing call sites to `encode_values` (decided, D28). The last PR of the stack adds the rules + that move structural encodings above `Narrow` (decided, D25). The fix for dictionary fill codes + is its own PR, before that last PR. Decimal support needs real kernel changes, because some + kernels assume an integer child. [`decisions.md`](../decisions.md#the-narrow-pull-requests) lists + the state of these PRs. + +## One canonical form + +N11 fixes the crashes of issue [#5820](https://github.com/vortex-data/vortex/issues/5820), and +issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks it. It lands as two +changes: an in-memory change, then a wire change (decided, D11). A part of a change lands as its +own PR only if it needs no temporary code. Otherwise, it lands together with the parts that it +depends on. + +Benchmarks and fixtures come first: + +- Benchmarks for decimal compare, filter, sum, min, max and the Arrow and DuckDB exports, on + DecimalByteParts and on `Narrow`. They include a gap column and the decoding of DecimalByteParts + with a narrow MSP. They land on `develop` before the producers change, so that CodSpeed has a + base. The existing ones are + `vortex-array/benches/{compare,binary_ops,cast_decimal,filter_fixed_width}.rs`, + `encodings/decimal-byte-parts/benches/dbp_assemble.rs` and `vortex-arrow/benches/to_arrow.rs`. +- Compatibility fixtures for narrow, wide and Arrow-style widths, gap values, high scales at + narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP + ([part 2](02-design.md#the-file-format)). A release publishes them before the PRs that change + how files are read, because only the old writer can make narrow leaves (decided, D24 and D31). + Library changes do not wait for it. The error + cases, such as a value outside the precision in a narrowing copy, are unit tests, because the + compat harness expects every read to succeed. + +The in-memory change has these steps, in order. Each step is its own PR, because none of them +needs temporary code (decided, D11): + +1. Give the width function one owner. `DecimalDType::canonical_values_type()` replaces the 25 direct + calls to `smallest_decimal_value_type`. Producers and imports take the width from the precision, + and every import checks its values (decided, D12, D13 and D30). +2. Add `Narrow` over decimals (N28). This step holds the child rules of D10, the kernels and casts + of D15, and the rule that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for + decimals, the compare kernel and the sum pushdown get decimal code in the same change. +3. Make DecimalByteParts decoding produce the canonical array of its dtype, with the + `TODO(connor)` of D16. Decoding widens a narrower MSP, and it narrows a wider MSP with the copy + of D21. Require `u64` lower parts, and reject lower parts that assemble wider than the + canonical width (decided, D29). Correct the module doc, which allows narrower parts. +4. Make the `vortex.decimal` read plugin convert old nodes (decided, D9 and D21). The plugin scans + the values, because it cannot use stored statistics. +5. Make the DecimalByteParts scheme take the canonical array, and delete `narrowed_decimal` + (decided, D17). The MSP narrows with the deprecated `encode_values` (decided, D28). The scheme + stays after the wire change, for writers whose edition does not permit `vortex.narrow`. File + contents stay the same. The exception is precision 39 to 76 with + values that need more than 64 bits, which becomes a plain `i256` array. +6. Make `DecimalArray::try_new` accept only the canonical width and check every valid value, with + `new_unchecked` for kernels (decided, D14). `try_new` takes an `ExecutionCtx` (decided, D30). + Builders follow, including `DecimalBuilder::extend`, which accepts wider buffers. This step is + three PRs: the width rule, the value check, and the removal of dead code. + +Then the `patch` panic and the "does the constant fit" special cases are unreachable. + +The wire change lands in lockstep (decided, D18). If possible, it ships in the same release as the +in-memory change (decided, D11). It holds three parts: + +- The `vortex.narrow` wire format for integers and decimals (decided, D8). This is the second part + of PR #10333. +- One new preview edition that permits `vortex.narrow` and DecimalByteParts with lower parts. +- A `Narrow` scheme that narrows the decimal type first, for writers that select that edition. It + runs before the DecimalByteParts scheme, which then compresses the child (decided, D33). Its + design pass settles which arrays it narrows (decided, D27), its precedence with the + DecimalByteParts scheme, and its cascade level. + +Later, both formats enter `core` together. Two optimizations wait until benchmarks show that they +are needed. One is the max pass in DecimalByteParts decoding (D16). The other is the pushdown of +sum, min and max into the MSP for gap columns (D19). + +After the in-memory change, decoding a compressed decimal runs at the full width in memory until +the compressor narrows the decimal type first. This applies to existing files, and to files from +writers that do not select an edition with `vortex.narrow`. The maintainer accepts this, because +the bar of no compute regression applies to the final state (decided, D19). + +## Order of work + +| Wave | Work | +| --- | --- | +| 0, independent | Strict enum decoding, the DecimalByteParts nullability fix, N1, the integer `Narrow` in memory followed by the `encode_values` PR, and the decimal benchmarks and fixtures | +| 1, one canonical form | N11 as two changes (D11). The in-memory change holds the width owner, N28 for decimals, DecimalByteParts decoding, the read plugin, the DecimalByteParts scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds the `vortex.narrow` wire format, the preview edition with DecimalByteParts lower parts, and the decimal `Narrow` scheme | +| 2, foundations | N7 fixed-width kernels. N8 scaled-integer kernels, used first by `Timestamp` unit casts. N4 cast rules, then N21 `Duration` and `Interval`. N2 and then N3, with the decimal rules as acceptance tests. N23 engine plugins | +| 3, new types | N9 wide integers, user-visible only behind an edition (recommendation DQ5). N10 edition gates and the protobuf `oneof`. N17 FixedSizeBinary, after DQ7. N5 array validation | +| 4, decimal kernels | N12 for precision 19 to 76. N13 rules over the view, which delete most decimal kernels. N14, which is optional. N18 and N27 | +| 5, optional | N16, then N15 if the built-in type and an extension type become identical. N19 and N20 after N22 | + +[Part 4](04-implementation.md) gives the order of the PRs inside waves 0 and 1. Waves 1 and 2 can +run in parallel. The decimal fix (N28 and N11) does not wait for the extension +work, because the built-in type owns its rules. Only the optional move to an extension type (N15) +waits for N1 to N4. For decimals, waves 3 and 4 are future work (D20), and their recommendations +are not reviewed. + +## Shared work between the tracks + +- The scaled-integer kernels (N8) serve decimal rescales and the unit casts of `Timestamp`, `Time` + and `Duration`. +- The decimal rules are acceptance tests for the rules and derives of extension types (N2 and + N3). Phase 1 of the extension notes adds a decimal-like extension type over `i64`, and that type + can run the same tests against the built-in `Decimal`. +- The fixed-width structural kernels (N7) serve wide integers and FixedSizeBinary. +- One `Narrow` encoding (N28) with one wire ID serves integers and decimals (D8). +- The placement rule (X5, recommendation XQ1) serves `Decimal` and every extension type. + Structural encodings sit above, and storage encodings sit below. `DateTimeParts` under the + `Timestamp` type is the temporal twin of DecimalByteParts under `Decimal`. + +## Risks + +| Risk | Mitigation | +| --- | --- | +| The `PType` migration touches 36 crates and several languages | Typed sub-enums, compiler-driven steps, and one crate in each PR | +| A leftover `other => panic!` arm panics on a wide type | Retire those macros, and fuzz decimals and wide integers at every width | +| An old reader reads a protobuf wide `PType` as `U8` | A new protobuf `oneof` variant, and strict readers early | +| Rewritten arithmetic makes several passes over the data | Fused scaled-integer kernels, checked with the existing decimal and compare benchmarks | +| D3 without `Narrow` forces the full width on every column | `Narrow` for decimals lands in the in-memory change, before `try_new` rejects other widths | +| An in-memory `Narrow` reaches a writer that cannot write `vortex.narrow` | The serializer writes the canonical array (D26). The code documents this as a workaround until `vortex.narrow` is in `core` | +| Moving the narrowing call sites to `encode_values` changes the signedness of internal children in new files | Readers on `develop` accept any integer type there, and the PR checks that the latest release reads its files (D28) | +| A narrow buffer does not give a precision | Every producer finds the precision from the values. Readers convert old nodes by their max (D9). DecimalByteParts decoding widens a narrow MSP (D16) | +| Compute on DecimalByteParts regresses when its decoding stops returning narrow leaves | In the final state, the compressor narrows the decimal type first, so decoding reuses the buffer. Until then, the maintainer accepts the cost (D19). The max pass of D16 can come later, and new benchmarks measure the cost | +| Gap columns use the next width in memory in the final state | Accepted (D19). Pushdown of sum, min and max into the MSP can come later | +| The conversion of old `vortex.decimal` nodes scans their buffer, because the plugin cannot use stored statistics | Old narrow nodes are rare, and the speed of this path matters little (D9) | +| Default writers use `core`, so they write precision 39 to 76 with values above 64 bits as plain `i256` arrays until `vortex.narrow` and DecimalByteParts with lower parts enter `core` | Accepted (D17). The wire change puts both into one preview edition. If it ships in the same release as the in-memory change, writers that select that edition never pay the cost (D11) | +| Arrow import at another width copies the buffer | Accepted (D12 and D13) | +| The public API breaks: `DecimalArray::new::` with a `T` other than the canonical type | Constructors that convert, and errors that name the canonical width. The in-memory ID stays `vortex.decimal`, which the Python bindings use | +| Wide integers reach users before engines can export them | Edition gates, and an explicit export rule for each engine | diff --git a/research/decimal/04-implementation.md b/research/decimal/04-implementation.md new file mode 100644 index 00000000000..73359c2ed25 --- /dev/null +++ b/research/decimal/04-implementation.md @@ -0,0 +1,243 @@ + + +# 4. Implementation plan + +This part splits the `Narrow` encoding (issue +[#10733](https://github.com/vortex-data/vortex/issues/10733)) and the canonical decimal fix (issue +[#10701](https://github.com/vortex-data/vortex/issues/10701)) into PRs. Each PR has a brief in +[`agents/briefs/`](agents/briefs/), written so that a coding agent can implement the PR from the +brief alone. The briefs follow [`decisions.md`](../decisions.md), including D25 to D35 from the +planning review on 2026-10-23. + +## How to use the briefs + +- Give a coding agent one brief, `decisions.md`, and the repository rules in `AGENTS.md` and + `STYLE.md`. The brief names the branch, the files, the steps, the tests and the acceptance checks. +- The briefs and `decisions.md` exist only on the `ct/ext-types-research` branch. The agent's own + worktree branches from `develop` and has no `research/` directory, so the relative links in the + briefs do not work there. Give the agent absolute paths into the research worktree + (`/Users/connor/spiral/vortex-data/vortex1/research/`), or tell it to read the files with + `git show ct/ext-types-research:research/`. Do not change a brief while an agent works from + it. If a brief must change, tell that agent. +- The briefs cite `develop` at `b70206818f`. Line numbers move as PRs land, so the agent reads the + cited code before it changes it. Code from the local integer `Narrow` rewrite is cited with its + commit, for example `f7408481ce:vortex-array/src/arrays/narrow/mod.rs`. That rewrite is a + reference for tests and edge cases. The PRs are written from scratch, and only FX3 ports a + rewrite commit as it is (D38). +- The rewrite commits (`f7408481ce`, `8d548646ee`, `74aa17644e`, `af4b663059` and `7e2e5e4d7a`) are + not on `origin`. Only the local branches `ct/narrow-pushdown` and `ct/narrow-encoding` hold them, + and those names also exist on `origin` with the older commits of the closed PRs. Agents in local + worktrees of this repository can read them. A remote agent or another clone cannot. Before such + an agent needs the reference, push them to a new backup branch. Do not delete or reset the two + local branches. +- Agents can run any command that the work needs: builds, tests, the acceptance checks, and the + generators of committed files, such as golden snapshots (NR6, CD7, WR3) and the edition record + (WR2). CI fails when a committed file is stale. Use the narrowest command, as `AGENTS.md` + describes. +- Several PRs form stacks: NR1 to NR5, WR1 to WR3, and optionally CD3 and CD5, CD4 and CD6, and CD8a + and CD8b. `develop` squash-merges PRs. So after the base PR merges, a plain `git rebase develop` + replays the base commits again and conflicts. Rebase the next PR with + `git rebase --onto develop `, or with `gh stack`. +- Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author + line. +- WR3 needs a design pass before an agent starts it. Its brief lists the open questions. + +| Brief | PRs | +| --- | --- | +| [`01-fixes.md`](agents/briefs/01-fixes.md) | FX1 to FX3, independent fixes | +| [`02-integer-narrow.md`](agents/briefs/02-integer-narrow.md) | NR1 to NR6, the integer `Narrow` | +| [`03-canonical-decimal.md`](agents/briefs/03-canonical-decimal.md) | CD1 to CD8c, the canonical decimal fix and decimal `Narrow` | +| [`04-wire.md`](agents/briefs/04-wire.md) | WR1 to WR3, the wire change, and LT1 to LT3 | + +## The PRs + +| ID | PR | Hard dependencies | Size | +| --- | --- | --- | --- | +| FX1 | Strict decoding of stored `DecimalType` and `PType` values, in three commits | None | Medium | +| FX2 | DecimalByteParts nullability cast follows the cast-rule contract | None | Small | +| FX3 | Dictionary `fill_null` no longer truncates the fill code | None | Small | +| NR1 | In-memory integer `Narrow`, with `encode`, the deprecated `encode_values`, the serialization workaround and a flatten rule | None | Large | +| NR2 | Benchmarks for compute on integer `Narrow` | NR1 | Medium | +| NR3 | Aggregates run on the child | NR2 | Small | +| NR4 | Arithmetic, `between`, `zip` and casts run on the child | NR3 | Large | +| NR5 | Structural encodings move above `Narrow`, and dictionary codes use the child | NR4, FX3 | Medium | +| NR6 | The existing narrowing call sites move to `encode_values` | NR1. It merges after NR5 (D35) | Medium | +| CD1 | Decimal benchmarks | None | Medium | +| CD2 | Compat fixtures for old decimal forms | None | Medium | +| CD3 | One width function, imports that check and copy, and the shared helpers | None | Medium | +| CD4 | Decimal `Narrow` in memory | NR1, NR3, NR4, CD1, CD3 | Large | +| CD5 | DecimalByteParts decodes to the canonical width | CD1, CD3, released CD2 fixtures | Medium | +| CD6 | The `vortex.decimal` read plugin converts old nodes | FX1, CD3, CD4, released CD2 fixtures | Medium | +| CD7 | One DecimalByteParts scheme on the canonical array | NR1, CD1 | Medium | +| CD8a | Every decimal array has the canonical width | CD3, CD5, CD6, CD7 | Large | +| CD8b | `DecimalArray::try_new` checks every valid value, with an `ExecutionCtx` | CD8a | Large | +| CD8c | Remove the code for mixed decimal widths | CD8a | Medium | +| WR1 | `vortex.narrow` wire format for integers and decimals | NR1, CD3, CD4, CD8a | Medium | +| WR2 | One preview edition with `vortex.narrow` and DecimalByteParts v2 | WR1, CD5, CD7 | Small | +| WR3 | A `Narrow` scheme in the compressor (needs a design pass) | WR1, WR2, NR1, NR5, CD3, CD4, CD7 | Large | + +Later work: LT1 puts both formats into `core` and decides the end of the serialization +workaround. LT2 is the max pass in DecimalByteParts decoding (D16). LT3 is the pushdown of sum, +min and max into the MSP (D19). LT2 and LT3 wait until benchmarks show that they are needed. + +## Status + +The first wave opened on 2026-10-23 as draft PRs, all based on `develop` at `3974ee069c`: + +| ID | PR | Note | +| --- | --- | --- | +| FX1 | #10755 | Three commits, as planned. Its tests use a new `test_harness::assert_serde_error` helper | +| FX2 | #10746 | As planned | +| FX3 | #10745 | Fixes #10623. Non-nullable codes keep the fast path, because they need no fill code | +| CD1 | #10752 | The compress bench enables the default `core` edition, so the decimal scheme uses its v1 variant | +| CD2 | #10753 | Adds `fixtures_build_canonical_decimal_widths`, and two columns at precisions 1 and 3 | +| None | #10747 | The versioning docs describe the scheme variants that #10076 added | + +## Graph + +Each solid arrow is a hard dependency. Some arrows also follow from other arrows. The graph keeps +them, so that the direct needs of each PR show. The dashed arrows are order rules, not code +dependencies: the release gate of the compat fixtures (D31), and NR6 merging after NR5 (D35). + +```mermaid +flowchart TD + subgraph FIX["Independent fixes"] + FX1["FX1 Strict decoding"] + FX2["FX2 DBP nullability cast"] + FX3["FX3 Dict fill code"] + end + subgraph NR["Integer Narrow"] + NR1["NR1 Encoding"] + NR2["NR2 Benchmarks"] + NR3["NR3 Aggregates"] + NR4["NR4 Arithmetic and casts"] + NR5["NR5 Structural rules"] + NR6["NR6 Call sites"] + end + subgraph CD["Canonical decimal"] + CD1["CD1 Benchmarks"] + CD2["CD2 Compat fixtures"] + REL(["Release with CD2"]) + CD3["CD3 Width and imports"] + CD4["CD4 Decimal Narrow"] + CD5["CD5 DBP decoding"] + CD6["CD6 Read plugin"] + CD7["CD7 DBP scheme"] + CD8a["CD8a Canonical width"] + CD8b["CD8b Value check"] + CD8c["CD8c Dead code"] + end + subgraph WR["Wire change"] + WR1["WR1 Wire format"] + WR2["WR2 Preview edition"] + WR3["WR3 Narrow scheme"] + end + + NR1 --> NR2 --> NR3 --> NR4 --> NR5 + FX3 --> NR5 + NR1 --> NR6 + NR5 -.-> NR6 + CD2 --> REL + REL -.-> CD5 + REL -.-> CD6 + CD1 --> CD4 + CD3 --> CD4 + NR1 --> CD4 + NR3 --> CD4 + NR4 --> CD4 + CD1 --> CD5 + CD3 --> CD5 + FX1 --> CD6 + CD3 --> CD6 + CD4 --> CD6 + NR1 --> CD7 + CD1 --> CD7 + CD3 --> CD8a + CD5 --> CD8a + CD6 --> CD8a + CD7 --> CD8a + CD8a --> CD8b + CD8a --> CD8c + CD8a --> WR1 + CD4 --> WR1 + WR1 --> WR2 + CD5 --> WR2 + CD7 --> WR2 + WR2 --> WR3 + NR5 --> WR3 +``` + +FX2 has no edge. It can land at any time. + +## Order of work + +A PR can start when its dependencies have merged. The waves group the PRs whose dependencies merge +at about the same time. Two PRs of one wave can run at the same time only when they touch different +files. The column "Shared files" names the pairs that do not, so run those one after the other. + +| Wave | PRs | Note | Shared files | +| --- | --- | --- | --- | +| 1 | FX1, FX2, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts | +| 2 | NR2, NR6, CD7, CD5 | NR6 merges after NR5 (D35). CD7 waits for CD1. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. NR6 and CD7: the golden snapshots of `vortex-btrblocks` | +| 3 | NR3, then NR4 | One stack. Merge NR1 and NR2 before reading the CodSpeed reports of NR3 and NR4 | None | +| 4 | NR5, CD4 | NR5 waits for FX3 | Both edit `vortex-array/src/arrays/narrow/rules.rs` | +| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | None | +| 6 | CD8a | The last in-memory step that the wire change needs | None | +| 7 | CD8b, CD8c, WR1 | Each needs only CD8a | CD8b adds a `ctx` parameter to `DecimalArray::new` and `try_new`, which CD8c and WR1 call. CD8b and CD8c both edit `vortex-array/src/builders/decimal.rs` and the same tests. Land CD8b first, or rebase the others on it | +| 8 | WR2, then WR3 | WR3 after its design pass. If possible, the wire stack ships in the same release as the in-memory change (D11) | None | + +The critical path runs NR1, NR2, NR3, NR4, CD4, CD6, CD8a, WR1, WR2 and WR3. The release that +publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wave 1. + +## Rules for every PR + +- No PR adds temporary code (D11). The serialization workaround of D26 is the one documented + exception. It stays until `vortex.narrow` is in `core`. +- Benchmarks land before the changes that they cover (D19): NR2 before NR3 to NR5, and CD1 before + CD4, CD5 and CD7. The dependency table lists these as hard dependencies. +- Every call of the deprecated `encode_values` carries `#[expect(deprecated)]` with a + `TODO(connor)` reason (D28). +- No `Narrow` sits directly above a generic structural encoding after optimization (D25). +- A narrowing copy writes zeros in null slots and fails on a valid value outside the precision + (D21). Reads without a copy trust the precision and never panic (D14). + +## Deferred + +- Which arrays the compressor narrows (D27). The WR3 design pass decides it, together with the + precedence between the `Narrow` scheme and the DecimalByteParts scheme and the cascade level + (D33). +- Other members of the preview edition. WR2 adds only `vortex.narrow` and + `vortex.decimal_byte_parts.v2`. +- Arrow import options for invalid decimal values (D13). The import fails, and the error names + arrow-rs `null_if_overflow_precision`. + +## Follow-up issues + +These problems came up during planning and the first wave of PRs. They are outside the planned +PRs, and each one has its own issue. + +| Issue | Problem | +| --- | --- | +| #10748 | Generic structural encodings always sit above other encodings. NR5 does this for `Narrow` only | +| #10749 | Decimal `scalar_at` and `min_max` panic on a stored value outside the precision, which a file read can produce | +| #10750 | Seven cast rules (RunEnd, ZigZag, FoR, RLE, ALP, ALP-RD, DateTimeParts) push nullability casts without the guard that FX2 adds to DecimalByteParts | +| #10751 | The FoR scheme bit-packs its child outside the allowed schemes, so its child skips the edition filter | +| #10754 | The compat harness: no CI job runs `--mode all`, one failing fixture stops the whole check, and the README describes `--git-ref` wrongly | +| #10756 | Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read | + +The stale versioning docs needed no issue, because #10747 fixes them. + +## Risks + +| Risk | Mitigation | +| --- | --- | +| CD5 changes the CUDA decoder, which few reviewers know | A minimal change in its own commit, with a CUDA reviewer. It becomes its own PR if it grows | +| After CD6, a CUDA read of an old narrow decimal node copies the buffer to the host and returns a `Narrow`, and CUDA has no `Narrow` kernel | Accepted (D37): the host converts old nodes, which are rare. A CUDA reviewer confirms in CD6 that such a scan still works | +| The reference commits of the rewrite exist only on local branches | Keep the local branches. Push a backup branch before a remote agent needs the reference (see "How to use the briefs") | +| NR6 writes signed internal children that earlier releases have not read | Readers on `develop` accept any integer type there. NR6 runs the latest release on its files | +| The `take_map` regressions on #10333 have no known cause | NR6 compares CodSpeed runs on NR1, NR3, NR4 and NR6, and reads the flame graph diff | +| WR3 has open design questions | A design pass before any code. The other wire PRs do not depend on it | +| The decimal critical path is long | Wave 1 holds seven PRs with no dependencies, and CD5, CD7 and NR6 need only early PRs | diff --git a/research/decimal/README.md b/research/decimal/README.md new file mode 100644 index 00000000000..a1ac0978b1f --- /dev/null +++ b/research/decimal/README.md @@ -0,0 +1,88 @@ +# Decimal and the type roadmap + +`Decimal` is a built-in dtype with the shape of an extension type: an integer storage value plus a +scale. On `develop`, the decimal array does not keep to that shape. One decimal dtype can have six +physical forms, and `Decimal` duplicates much of `Primitive`. These notes describe the target +design that the maintainer settled on 2026-10-19, 2026-10-22 and 2026-10-23, and they put the type +work that it needs in order. [`decisions.md`](../decisions.md) records the decisions and the open +questions. Issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks the fix of the +canonical decimal encoding. + +| Part | Content | +| --- | --- | +| [1. Current state](01-current-state.md) | How `Decimal` works on `develop`, and what is wrong | +| [2. Design](02-design.md) | The leaf, imports and Arrow, `Narrow`, DecimalByteParts, the file format, and the future work | +| [3. Roadmap](03-roadmap.md) | The dependency graph of the type work, the independent changes, and the order of work | +| [4. Implementation plan](04-implementation.md) | The PRs of the `Narrow` and canonical decimal work, their dependencies and waves, with one brief for each PR | + +## Summary + +- The width of the decimal buffer does not depend on the precision. So one dtype has up to six + physical forms, every kernel must handle mixed widths, and some paths panic. +- Decided: a decimal array stays a leaf with one buffer and validity. Each decimal dtype has + exactly one canonical encoding, because the precision fixes the width of the buffer. Each + precision gets the smallest signed width that holds it. +- Decided: `DecimalArray::try_new` checks the width and every valid value. Imports copy into the + canonical width, and they fail on a value outside the precision. File reads trust the precision. +- Decided: a `Narrow` encoding stores a column as a child with a lower precision and the same + scale. The child must have a narrower canonical width. One `Narrow` encoding serves integers and + decimals, and its kernels pass work to the child. An integer child keeps the signedness of its + parent. The wire form is only for columns with a fixed logical dtype, and internal integer + children keep changing their own dtype. +- Precision limits are powers of ten, and widths are powers of two. So a width does not give a + precision: an `i32` holds 2,000,000,000, which needs precision 10. Each producer finds the child + precision from the values. +- Decided: a reader finds an old narrow `vortex.decimal` node by its stored width. It scans the + values for their max, and it returns a `Narrow` or a copy. Every narrowing copy writes zeros in + null slots and fails on a value outside the precision. +- Decided: DecimalByteParts stays the compressed form. Operations run on its parts where they can, + and it decodes to the canonical array of its own dtype. One compressor scheme builds it from the + canonical array, so file contents stay the same. +- Decided: `vortex.narrow` stays in memory until the maintainer decides to write it. Then it + enters one preview edition together with DecimalByteParts with lower parts. Only then does a + decimal `Narrow` scheme narrow the decimal type first. Default writers use `core`, so until both + formats enter `core`, they write wide decimals with values above 64 bits as plain `i256` arrays. +- Decided: the bar of no compute regression applies to the final state. Until the compressor + narrows the decimal type first, decoding a compressed decimal runs at the full width in memory. +- Decided: the work lands as 22 PRs in four tracks. Each PR has a brief that a coding agent can + implement alone. The integer `Narrow` PRs are written from scratch, with a local rewrite as a + reference. Generic structural encodings always sit above `Narrow`. A writer that cannot + write `vortex.narrow` writes the canonical array, as a documented workaround. Which arrays the + compressor narrows is deferred. +- Arrow has the same decimal model and the same gap at the top of each width. Its width is part + of the type, and from precision 5 the Vortex widths match the smallest Arrow types. +- Decided: `vortex.decimal` keeps its wire ID, and the decimal file format does not change. +- Decided: `Decimal` stays a built-in type that behaves like a hardcoded extension type. A move to + an extension type is an optional later step. +- Decided: Vortex adds `I128`, `I256`, `U128` and `U256`, and FixedSizeBinary becomes a core type. + Their designs are future work, outside the canonical decimal fix. So are the integer view, + decimal arithmetic across precisions, and casts that lose information. The recommendations for + this future work are not reviewed. + +## Agent documents + +- [`agents/briefs/`](agents/briefs/): one brief for each PR of part 4, in four files. +- [`agents/report.md`](agents/report.md): the full report. It has file and line references, API + sketches, the wire plan, the complete dependency graph with a reason for each edge, and the + evidence log. +- [`agents/decimal-call-sites.md`](agents/decimal-call-sites.md): every decimal-specific file with + line counts, every use of the width function, and every canonical dispatch site. +- [`agents/ptype-match-inventory.md`](agents/ptype-match-inventory.md): every `PType` dispatch + macro use, every exhaustive `PType` match, and every copy of `PType` in other languages and + formats. + +## Evidence + +The notes come from reading `develop` at `5fc5cb8`, RFC drafts, issues and pull requests. On +2026-10-19 the review also read the draft `Narrow` PRs and PR #10230. On 2026-10-22 it read the +versioning spec, the array plugin and serialization code, and the DecimalByteParts code. It also +read the history of the decimal compression scheme, and PRs #10305 and #10314. On 2026-10-23 it +traced which statistics the compressors store on the DecimalByteParts parts, read the decimal +types of arrow-rs 59.3, and read the cast change of PR #10230. A second review on the same day +read when the reader attaches stored statistics, the DecimalByteParts constructor and compressor +scheme, the compat harness, and the edition families. A planning review on the same day had five +agents draft the PR briefs from the code at `b70206818f`, and the review checked their main claims +in the code. One throwaway Rust test confirmed +the `f32` cast disagreement and the `ext_storage` pushdown rules (probes R1 and R6 in the extension +notes). One short Python script checked byte order. Section 10 of the report lists the claims that +are not verified, which include every performance claim. diff --git a/research/decimal/agents/briefs/01-fixes.md b/research/decimal/agents/briefs/01-fixes.md new file mode 100644 index 00000000000..47283a34124 --- /dev/null +++ b/research/decimal/agents/briefs/01-fixes.md @@ -0,0 +1,578 @@ + + +# Independent fixes: FX1, FX2 and FX3 + +These three PRs depend on no other PR. FX1 blocks CD6, and FX3 blocks NR5. Each one starts from +`develop` at `b70206818f`. Every `file:line` below points at that checkout, unless a commit is +named. + +Rules for all three PRs: + +- Each commit ends with `Signed-off-by: Connor Tsui `. Add no AI + co-author line. +- Agents can run any command that the work needs, including the acceptance checks. The checks are + scoped to the affected crates. The pinned nightly is `nightly-2026-09-10` + (`.github/workflows/ci.yml:25`). +- Tests return `VortexResult<()>` where they can and use named `rstest` cases. Tests that need a + session share the module `static SESSION: LazyLock` that PR #10681 introduced. +- Every "before the fix" result in these briefs comes from reading the code, not from a run. + +### FX1: Strict decoding of stored `DecimalType` and `PType` values + +- Goal: A reader returns an error for an unknown stored `DecimalType` or `PType` value. Today the + prost getters decode such a value silently as `I8` or `U8`. CD6 detects old decimal nodes by the + stored width, so that field must be strict first. +- Depends on: None. Soft: None. Blocks: CD6. +- Decisions: D34 (one PR for all three commits). FX1 is the prerequisite of the D7 detection in + CD6. It follows the read rule of D14: a file read never panics. It is the first half of node N10 + in the roadmap. +- Branch: `ct/strict-enum-decoding`, based on `develop`. +- Scope: Commit 1 makes the decimal `values_type` strict. Commit 2 makes the protobuf dtype `PType` + strict, and it adds a test for the flatbuffer path, which is already strict. Commit 3 makes the + eleven `PType` fields of array and layout metadata strict. +- Out of scope: New `PType` values and the protobuf `oneof` for wide integers (D5, D20). Edition + gates for core dtypes (the second half of N10). The detection of old decimal nodes (CD6). Paths + that are already strict (listed below). The renaming of the test modules named `test`. The + unused `DictLayoutMetadata` in `vortex-layout/src/layouts/dict/writer.rs:537-550` (see Notes). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs` | `DecimalMetadata::get_values_type`, the strict decode in `deserialize` (`:151-186`), tests | +| `vortex-array/src/dtype/serde/proto.rs` | Strict `PType` decode at `:36`, tests | +| `vortex-array/src/dtype/serde/flatbuffers.rs` | The raw value in the error at `:559`, a test | +| `vortex-array/src/arrays/dict/array.rs` | `DictMetadata::get_codes_ptype` beside the struct (`:29-43`), a test module | +| `vortex-array/src/arrays/dict/vtable/mod.rs` | Call `get_codes_ptype` instead of the getter at `:180` | +| `vortex-array/src/arrays/varbin/vtable/mod.rs` | `VarBinMetadata::get_offsets_ptype` (`:60-64`), call at `:180`, a test module | +| `vortex-array/src/arrays/list/vtable/mod.rs` | `ListMetadata::get_offset_ptype` (`:48-54`), call at `:178`, a test module | +| `vortex-array/src/arrays/listview/vtable/mod.rs` | `ListViewMetadata::get_offset_ptype` and `get_size_ptype` (`:56-64`), calls at `:198` and `:205`, a test module | +| `encodings/runend/src/array.rs` | `RunEndMetadata::get_ends_ptype` (`:62-70`), call at `:155`, a test in `mod tests` (`:574`) | +| `encodings/fastlanes/src/rle/vtable/mod.rs` | `RLEMetadata::get_indices_ptype` and `get_values_idx_offsets_ptype` (`:48-62`), calls at `:178` and `:185`, tests in `mod tests` (`:329`) | +| `encodings/alp/src/alp_rd/array.rs` | `ALPRDMetadata::get_left_parts_ptype` (`:61-73`), call at `:182`, a test in `mod test` (`:563`) | +| `vortex-layout/src/layouts/dict/mod.rs` | `DictLayoutMetadata::get_codes_ptype` (`:163-178`), call at `:83`, a test module | +| `vortex-layout/src/layouts/list/mod.rs` | `ListLayoutMetadata::get_offsets_ptype` (`:228-240`), one call that serves `:87` and `:105`, a test module | + +Design notes: + +The cause. prost-derive 0.14.4, the version in `Cargo.lock`, generates each enumeration getter as +`TryFrom::try_from(raw).unwrap_or(default)` (`prost-derive-0.14.4/src/field/scalar.rs:291-300` in +the cargo registry). The default is the first variant (`src/lib.rs:308`). `DecimalType` and `PType` +both derive `prost::Enumeration` (`vortex-array/src/dtype/decimal/types.rs:25`, +`vortex-array/src/dtype/ptype.rs:33`). So an unknown value reads as `I8` or `U8`. + +Silent fallbacks on `develop`, all fixed by FX1: + +| Read site | Field | Commit | +| --- | --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:170`, `:177` | `DecimalMetadata.values_type` | 1 | +| `vortex-array/src/dtype/serde/proto.rs:36` | `pb::Primitive.r#type` | 2 | +| `vortex-array/src/arrays/dict/vtable/mod.rs:180` | `DictMetadata.codes_ptype` | 3 | +| `vortex-array/src/arrays/varbin/vtable/mod.rs:180` | `VarBinMetadata.offsets_ptype` | 3 | +| `vortex-array/src/arrays/list/vtable/mod.rs:178` | `ListMetadata.offset_ptype` | 3 | +| `vortex-array/src/arrays/listview/vtable/mod.rs:198`, `:205` | `ListViewMetadata.offset_ptype`, `size_ptype` | 3 | +| `encodings/runend/src/array.rs:155` | `RunEndMetadata.ends_ptype` | 3 | +| `encodings/fastlanes/src/rle/vtable/mod.rs:178`, `:185` | `RLEMetadata.indices_ptype`, `values_idx_offsets_ptype` | 3 | +| `encodings/alp/src/alp_rd/array.rs:182` | `ALPRDMetadata.left_parts_ptype` | 3 | +| `vortex-layout/src/layouts/dict/mod.rs:83` | `DictLayoutMetadata.codes_ptype` | 3 | +| `vortex-layout/src/layouts/list/mod.rs:87`, `:105` | `ListLayoutMetadata.offsets_ptype` | 3 | + +Already strict, no change: + +| Path | Item | +| --- | --- | +| `vortex-array/src/dtype/serde/flatbuffers.rs:543-562` | Flatbuffer `PType` | +| `vortex-array/src/patches.rs:103-124` | `PatchesMetadata` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v1.rs:71` | DecimalByteParts v1 | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs:83`, `:93` | DecimalByteParts v2 | +| `encodings/fastlanes/src/bitpacking/plugin/v2.rs:98` | BitPacked v2 | +| `encodings/datetime-parts/src/array.rs:74-88` | DateTimeParts | +| `encodings/fsst/src/array.rs:85-88`, `:269` | FSST | +| `encodings/onpair/src/array.rs:554-563` | OnPair | +| `vortex-array/src/scalar_fn/fns/operators.rs:84` | Expression `BinaryOp` | + +Variant metadata. `VariantMetadataProto.shredded_dtype` +(`vortex-array/src/arrays/variant/vtable/mod.rs:51-55`, decoded at `:154-159`) and +`ParquetVariantMetadataProto.typed_value_dtype` (`encodings/parquet-variant/src/vtable.rs:66-77`, +decoded at `:212-216`) store protobuf dtypes on disk. Neither one holds an enum field directly. Both +call `DType::from_proto`, so commit 2 covers them. Commit 2 also covers the other callers of +`DType::from_proto`: scalars (`vortex-array/src/scalar/proto.rs:198`), cast options +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:91`), `variant_get` options +(`vortex-array/src/scalar_fn/fns/variant_get/mod.rs:88`), the indexed layout +(`vortex-layout/src/layouts/indexed/mod.rs:393`), zstd buffers +(`encodings/zstd/src/zstd_buffers.rs:578`), `vortex-tensor` and `vortex-json`. + +Rules for the change: + +- Each metadata struct gets one `get_()` accessor per enum field. It returns + `VortexResult` or `VortexResult`. This follows + `DateTimePartsMetadata::get_days_ptype` and `FSSTMetadata::get_uncompressed_lengths_ptype`. The + plain field name is taken by the prost getter. +- Put each accessor in an `impl` block right below its struct. Give it the smallest visibility that + its caller needs. +- Each `deserialize` calls the accessor right after it decodes the metadata. The call comes before + any other check and before any buffer or child read. So an unknown value always fails first. +- Keep the `enumeration` attributes. They document the wire type, and the setters such as + `set_codes_ptype` use them. The generated `pb::Primitive::r#type()` getter also stays, because + generated files are never edited by hand. +- Decode the protobuf field with `pb::PType::try_from(p.r#type)`, then convert with the existing + `From for PType` (`proto.rs:229-246`). Do not use `PType::try_from(i32)` here. It ties + the wire numbers to the discriminants of the Rust enum. The numbers match today + (`vortex-array/proto/dtype.proto:11-23`, `ptype.rs:37-60`), but the two enums are separate. +- proto3 omits a field that holds 0, and a missing field decodes as 0. Zero is `U8` or `I8`, which + are valid values. So a file from a current writer reads the same way after FX1. +- Every error uses the `Serde` category. It states the requirement first and ends with the raw + value. + +| Site | Message | +| --- | --- | +| Decimal metadata | `DecimalMetadata values_type must be a known DecimalType, got {raw}` | +| Protobuf dtype | `protobuf Primitive type must be a known PType, got {raw}` | +| Flatbuffer dtype | `flatbuffer Primitive ptype must be a known PType, got {raw}` | +| Array and layout metadata | `{Struct} {field} must be a known PType, got {raw}`, for example `DictMetadata codes_ptype must be a known PType, got 11` | + +```rust +impl DecimalMetadata { + fn get_values_type(&self) -> VortexResult { + DecimalType::try_from(self.values_type).map_err(|_| { + vortex_err!( + Serde: "DecimalMetadata values_type must be a known DecimalType, got {}", + self.values_type + ) + }) + } +} + +// In `DType::from_proto`. +DtypeType::Primitive(p) => { + let ptype = pb::PType::try_from(p.r#type).map_err(|_| { + vortex_err!(Serde: "protobuf Primitive type must be a known PType, got {}", p.r#type) + })?; + Ok(Self::Primitive(ptype.into(), p.nullable.into())) +} +``` + +Steps: + +1. `fix(array): decode decimal values_type strictly`. Add `DecimalMetadata::get_values_type`. In + `Decimal::deserialize`, call it right after `DecimalMetadata::decode` at `:160`. Use the result + at `:170` and `:177`. Add the decimal tests. +2. `fix(array): decode protobuf PType strictly`. Change `proto.rs:36` as sketched. Put the raw value + into the error at `flatbuffers.rs:559`. Add the protobuf and flatbuffer tests. +3. `fix: decode PType metadata fields strictly`. Add the accessors of commit 3 in the table above. + Call each one at the top of its `deserialize`. In `vortex-layout/src/layouts/list/mod.rs`, bind + the result once and use it at `:87` and `:105`. Add the accessor tests. + +Tests: + +- `vortex-array/src/arrays/decimal/vtable/mod.rs`, `mod tests` (`:220`): + - `test_deserialize_rejects_unknown_values_type`, with named cases `first_unused` (6), + `negative` (-1) and `max` (`i32::MAX`). Encode `DecimalMetadata { values_type: raw }`. Call + `Decimal.deserialize` with one host buffer of 4 bytes, the dtype `Decimal(2, 0)` non-nullable, + length 4 and no children. Assert an error that contains `must be a known DecimalType`. Before + the fix, case `first_unused` returns `Ok` with an `I8` array. + - `test_serde_each_values_type`, with one named case for each width at a precision with that + canonical width: `i8` at `Decimal(2, 0)`, `i16` at `(4, 0)`, `i32` at `(9, 2)`, `i64` at + `(18, 2)`, `i128` at `(38, 2)` and `i256` at `(76, 2)`. Round-trip as `test_array_serde` does + (`:241-273`). Assert the decoded `values_type()` and use `assert_arrays_eq!`. Every case uses + the canonical width, so CD8a keeps all of them valid. +- `vortex-array/src/dtype/serde/proto.rs`, `mod tests` (`:286`): + - `test_from_proto_rejects_unknown_ptype`, with named cases `first_unused` (11), `negative` (-1) + and `max` (`i32::MAX`). Assert an error that contains `must be a known PType`. Before the fix, + every case returns `Primitive(U8)`. + - `test_from_proto_rejects_unknown_ptype_in_struct_field`. A struct dtype with one field whose + `PType` is 11. This is the shape of a shredded Variant dtype. Assert an error. +- `vortex-array/src/dtype/serde/flatbuffers.rs`, `mod test` (`:565`): + `test_unknown_ptype_from_flatbuffer`. Assert that `PType::try_from(fb::PType(11))` returns an + error that contains `got 11`. The path is strict today, so this test covers the message only. +- Commit 3: one test per struct, named `test_get__rejects_unknown_ptype`. Use named cases + where a struct has two fields (`ListViewMetadata`, `RLEMetadata`). Each test builds the struct + with the field set to 11 and asserts an error that names the struct and the field. One more case + with a valid value asserts `Ok`. Put the test in the file that defines the struct. Add a + `#[cfg(test)] mod tests` at the bottom of `dict/array.rs`, `varbin/vtable/mod.rs`, + `list/vtable/mod.rs`, `listview/vtable/mod.rs`, `vortex-layout/src/layouts/dict/mod.rs` and + `vortex-layout/src/layouts/list/mod.rs`. +- No existing test changes. + +Benchmarks: None. Each read adds one integer match per enum field. CodSpeed is expected to show +no change. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp` +- `cargo nextest run -p vortex-array -p vortex-layout -p vortex-runend -p vortex-fastlanes -p vortex-alp -p vortex-parquet-variant` +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode all`. It reads every published + fixture, so it shows that no published file holds a value outside the enums. +- An unknown stored value returns a `Serde` error. No read path panics on it. +- No read path in the table above calls a prost enumeration getter. + +Review focus: + +- Every read site in the table calls its accessor, and the call comes before any other check. +- `rg -n 'metadata\.\w+_ptype\(\)|metadata\.values_type\(\)' | rg -v '\.get_'` finds no read + path. +- The error messages follow the table, with the raw value last. +- The wire format and the setters do not change. +- An old file with a value outside the enum now fails. Before FX1 it read as `U8` or `I8`. Whether + such a file exists is Not verified. The compat check covers the published fixtures. +- CD6 keeps `get_values_type()` as the first step of its detection. + +Size: medium. Thirteen fields in five crates, each with a small test. + +Open questions: None. CD6 needs only commit 1, but the maintainer chose one PR for all three +commits (D34). + +### FX2: DecimalByteParts nullability cast follows the cast-rule contract + +- Goal: The DecimalByteParts (DBP) cast rule rewrites a nullability cast only when + `Validity::trivially_cast_nullability` proves it safe. A new cast kernel handles the declined + case on the parts. Today the rule casts the most significant part (MSP) for every target + nullability, and DBP has no cast kernel. +- Depends on: None. Soft: CD5 (same crate, other files). Blocks: None. +- Decisions: None by ID. `decisions.md`, section "The `Narrow` pull requests", settles this PR: the + guard of #10230, a cast kernel for the case that needs execution, tests, and no issue. +- Branch: `ct/dbp-nullability-cast`, based on `develop`. +- Scope: The guard in `CastReduce`, a `CastKernel` for casts that change only the nullability, its + registration, and tests. +- Out of scope: Casts that change the precision or the scale stay unsupported (the `TODO(mk)` at + `compute/cast.rs:19`). DBP decoding and `u64` lower parts (CD5). The same push-down pattern in + other encodings (see Notes). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs` | The guard in `CastReduce` (`:16-35`), a new `CastKernel`, one shared private helper, tests, and a test session with `crate::initialize` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/kernel.rs` | Register `CastExecuteAdaptor(DecimalByteParts)` under `Cast.id()` (`:14-26`) | + +`rules.rs:14` keeps the cast rule as it is. + +Design notes: + +The contract (`vortex-array/src/validity.rs:430-494`). A cast rule runs in the optimizer and cannot +execute. It calls `trivially_cast_nullability` and returns `Ok(None)` when that call returns +`Ok(None)`. A cast kernel runs at execution and calls `cast_nullability`, which can read the +validity array. FSST (`encodings/fsst/src/compute/cast.rs:48-91`, registered at +`encodings/fsst/src/kernel.rs:25`) and BitPacked +(`encodings/fastlanes/src/bitpacking/compute/cast.rs:56`, `:74`) show the pair. + +DBP has no validity of its own. Its validity is the validity of the MSP +(`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:200`, `:404-409`). So a cast of the +MSP is the only way to change it. `with_msp` keeps the lower parts (`array.rs:346-352`). + +Outcomes for a cast that changes only the nullability: + +| MSP validity | Target | Rule | Kernel | +| --- | --- | --- | --- | +| Any | Nullable | DBP over the MSP cast to nullable | Same result | +| `NonNullable`, `AllValid`, or length 0 | Non-nullable | DBP over the MSP cast to non-nullable | Same result | +| `AllInvalid`, length above 0 | Non-nullable | Error `Cannot cast AllInvalid to NonNullable` | Error, if an unoptimized cast reaches it | +| `Array` | Non-nullable | `Ok(None)` | Error when a null exists, else DBP over the MSP cast to non-nullable | + +```rust +/// Rebuilds `array` over its MSP cast to `nullability`. +/// +/// The MSP carries the validity of a DBP array, so the cast of the MSP sets the validity of the +/// result. The lower parts are non-nullable and stay as they are. +fn with_msp_nullability( + array: ArrayView<'_, DecimalByteParts>, + nullability: Nullability, +) -> VortexResult; + +impl CastReduce for DecimalByteParts { + fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { + // TODO(mk): Support non-nullability changes as well, e.g. precision. + if !dtype.eq_ignore_nullability(array.dtype()) { + return Ok(None); + } + let nullability = dtype.nullability(); + if array + .validity()? + .trivially_cast_nullability(nullability, array.len())? + .is_none() + { + return Ok(None); + } + + with_msp_nullability(array, nullability).map(Some) + } +} + +impl CastKernel for DecimalByteParts { + fn cast( + array: ArrayView<'_, Self>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !dtype.eq_ignore_nullability(array.dtype()) { + return Ok(None); + } + let nullability = dtype.nullability(); + array.validity()?.cast_nullability(nullability, array.len(), ctx)?; + + with_msp_nullability(array, nullability).map(Some) + } +} +``` + +- The guard is the change of #10230 to this file, as `gh pr diff 10230 --repo vortex-data/vortex` + shows. FX2 keeps it as it is. +- The kernel returns `Ok(None)` for every other cast. The generic `Cast::execute` then decodes the + DBP, as it does today (`vortex-array/src/scalar_fn/fns/cast/mod.rs:126-159`). +- The kernel discards the `Validity` that `cast_nullability` returns, because the MSP stores the + validity. A comment beside that call must say this. For a Primitive MSP with a validity array, + the MSP cast stays lazy, because the Primitive rule declines + (`vortex-array/src/arrays/primitive/compute/cast.rs:62-67`). The Primitive kernel checks again + when the DBP executes. It reads the minimum of the same validity array, which is a cached + statistic. That this second check costs only a cache lookup is Not verified. +- The existing check `let DType::Decimal(_, target_nullability) = dtype` at `cast.rs:24-26` goes + away. `eq_ignore_nullability` against a decimal dtype already implies it. + +Is a wrong validity observable before execution today? It is not shown from the code: + +- The current rule returns a DBP over `Cast(msp)` with a non-nullable dtype (`cast.rs:28-33`). +- `validity()` of an array with a non-nullable dtype returns `Validity::NonNullable` without asking + the encoding (`vortex-array/src/array/mod.rs:304-319`). +- The lazy `Cast(dbp)` that the rule replaces has the same dtype, so it reports the same validity. + `Cast::validity` also reports all rows valid for a non-nullable target + (`vortex-array/src/scalar_fn/fns/cast/mod.rs:183-195`). +- The null check still runs when the MSP cast executes. The existing test + `test_cast_decimal_byte_parts_nullable_to_non_nullable` (`cast.rs:92-108`) expects that error. +- So the defect is a contract violation. The rule rewrites a cast that it cannot prove safe, and it + depends on the cast of each MSP encoding to keep the null check. +- These MSP cast rules were read: Primitive, BitPacked, FoR, ZigZag, RunEnd, Delta, Sparse, + Sequence, Constant, Dict and RLE. Each one keeps the check, pushes it to a child, or declines. + Whether another MSP encoding drops the check is Not verified. + +After FX2, `cast` of a DBP whose MSP has a validity array returns a lazy `Cast` over the DBP, not a +DBP. Execution gives the same values or the same error as today. No caller in the workspace +downcasts a cast result to DBP (checked with a search for `is::` and +`as_opt::`). + +Steps: + +1. `fix(decimal-byte-parts): guard the nullability cast rule`. Add `with_msp_nullability`, the + guard, the `CastKernel`, and the registration in `compute/kernel.rs`. Add the tests. + +Tests, all in the `mod tests` of `compute/cast.rs` (`:37-135`): + +- Change the module session to the one in `compute/mod.rs:35-39`, which calls + `crate::initialize(&session)`. Without it, the execution context does not hold the new kernel. +- Add one fixture helper at the top. It takes a `Validity` and builds a DBP at + `DecimalDType::new(10, 2)` over `PrimitiveArray::new(buffer![100i32, 200, 300], validity)`. +- Build each validity array with `Validity::Array(BoolArray::from_iter([..]).into_array())`. + `Validity::from_iter` and `PrimitiveArray::from_option_iter` turn all-true input into `AllValid` + and all-false input into `AllInvalid` (`vortex-array/src/validity.rs:506-532`). So they cannot + build these cases. Each validity-array case asserts that the fixture reports `Validity::Array`, + so that it reaches the guard that it claims to test. +- `test_cast_rule_declines_validity_array`, with named cases `with_nulls` (`[true, false, true]`), + `without_nulls` (`[true, true, true]`) and `only_nulls` (`[false, false, false]`). Assert that + `::cast` returns `None` for the non-nullable target. The direct + call follows `encodings/fastlanes/src/bitpacking/array/tests.rs:191-192`. +- `test_cast_kernel_rejects_nulls`, with named cases `with_nulls` and `only_nulls` as above. Assert + that `::cast` returns an error. Also assert that `cast` followed + by `execute::` returns an error. +- `test_cast_kernel_without_nulls`. Case `without_nulls`. Assert that the kernel returns a DBP with + a non-nullable dtype. Assert with `assert_arrays_eq!` that `cast` gives the non-nullable + `DecimalArray` of the same values. +- `test_cast_rule_rejects_all_invalid`. MSP validity `AllInvalid`. Assert that `cast` to the + non-nullable dtype returns an error that contains `Cannot cast AllInvalid to NonNullable`. This + error comes from the rule, before execution. +- `test_cast_rule_reduces`, with named cases `nullable_all_valid_to_non_nullable` and + `non_nullable_to_nullable`. Assert that `cast` returns a DBP (`is::()`) with + the target dtype and the same values. This shows that the rule still avoids execution where it + can. +- `test_cast_kernel_keeps_lower_parts`. Build a DBP with + `DecimalByteParts::try_new_with_lower_parts` from an `i64` MSP `[1, 0, -1]` with an all-true + validity array and one `u64` lower part `[5, 6, 7]`, at `DecimalDType::new(38, 2)`. Assert with + `assert_arrays_eq!` that the cast to non-nullable gives the `i128` values + `[(1 << 64) + 5, 6, (-1 << 64) + 7]`. +- Keep `test_cast_decimal_byte_parts_nullability`, + `test_cast_decimal_byte_parts_nullable_to_non_nullable` and the conformance cases. + +Benchmarks: None. No benchmark covers DBP casts. A cast with a validity array now runs its null +check in the DBP kernel first, and the MSP kernel then finds the cached minimum. CodSpeed is +expected to show no change. + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-decimal-byte-parts` +- `cargo nextest run -p vortex-decimal-byte-parts` +- The rule returns `Ok(None)` whenever `trivially_cast_nullability` returns `Ok(None)`. +- A cast to non-nullable of a DBP with nulls fails, with or without a validity array. +- A cast to non-nullable without nulls keeps the DBP and its lower parts. + +Review focus: + +- The kernel is registered under `Cast.id()` for `DecimalByteParts`. +- The guard runs before any cast of the MSP. +- Both the rule and the kernel return `Ok(None)` for a cast that changes more than the + nullability. +- The comment beside the discarded `Validity` in the kernel is correct. +- The test session registers the crate, so the kernel tests do not pass through the decode + fallback. + +Size: small. One file of logic, one registration and focused tests. + +Open questions: None. + +### FX3: Dictionary `fill_null` no longer truncates the fill code + +- Goal: The dictionary `fill_null` kernel stops truncating the index of the fill value into the + codes type. When the index does not fit the codes dtype, the kernel declines, and the generic + path fills the decoded array. +- Depends on: None. Soft: None. Blocks: NR5. With NR5, `optimize()` unwraps a `Narrow` over + dictionary codes to a narrow child + (`af4b663059:vortex-array/src/arrays/narrow/tests/merge.rs:29-41`). This makes the bug easy to + reach. NR5 also carries the `Narrow` regression test of this fix. +- Decisions: None by ID. `decisions.md`, section "The `Narrow` pull requests", makes this fix its + own PR. +- Branch: `ct/dict-fill-code`, based on `develop`. +- Scope: The fix of local commit `7e2e5e4d7a` to + `vortex-array/src/arrays/dict/compute/fill_null.rs`, and its `Narrow`-free regression test, + adapted to the current test module. +- Out of scope: The `Narrow` test + `7e2e5e4d7a:vortex-array/src/arrays/narrow/tests/merge.rs:45-62`, which NR5 owns. Any change to + the "no fill value found" branch at `fill_null.rs:45-55`. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/dict/compute/fill_null.rs` | Checked fill code at `:57-76`, two unused imports removed (`:19`, `:21`), a regression test in `mod tests` (`:90-131`) | + +Design notes: + +How the truncation happens on `develop`, without `Narrow`: + +1. The kernel compares every dictionary value with the fill value (`fill_null.rs:33-40`). This + includes values that no code references. +2. It takes the first match as a `usize` index (`:43-44`). +3. It converts the index with `as P` into the integer type of the codes (`:61-69`). The + `#[expect(clippy::cast_possible_truncation)]` at `:63-66` assumes that the index fits. +4. A dictionary can hold more values than its codes type can index. `DictArray::try_new` and + `Dict::validate` check only the codes type, the lengths and the dtypes + (`dict/array.rs:303-315`, `dict/vtable/mod.rs:97-116`). +5. So `u8` codes `[0, null]` over the 301 values `0..=300` form a valid dictionary. With fill value + 300, `300 as u8` is 44, and the null row reads value 44. With fill value 256, it reads value 0. +6. With `i8` codes and index 128, the code becomes -128. Whether the read then returns an error or + panics is Not verified. + +Whether a producer on `develop` other than direct construction builds such a dictionary is Not +verified. + +The fix, as in `7e2e5e4d7a`: + +```rust +let codes = array.codes(); +// An unreferenced dictionary entry can exceed the codes' range. +let Ok(fill_code) = + Scalar::from(existing_fill_value_index).cast(&codes.dtype().as_nonnullable()) +else { + return Ok(None); +}; + +// Fill nulls in both the codes and the values. ... +let codes = codes.clone().fill_null(fill_code)?; +``` + +- `Scalar::from(usize)` builds a `u64` scalar + (`vortex-array/src/scalar/convert/primitive.rs:199-203`). The cast to an integer dtype is + checked. `PValue::cast` returns an error for a value that does not fit + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:287-303`). +- `Ok(None)` declines the kernel. `FillNull::execute` then decodes the dictionary and fills the + result (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:153-163`). +- The first match has the smallest index. If it does not fit, no later match fits. So declining on + the first match loses no fast path. +- Remove the imports `match_each_integer_ptype` (`:19`) and `ScalarValue` (`:21`). Only the + replaced code uses them. + +Steps: + +1. `fix(array): avoid truncating dictionary fill codes`. Apply the kernel change and the test in + one commit, as `7e2e5e4d7a` does. + +Tests: + +- Port `fill_unreferenced_value` from `7e2e5e4d7a:vortex-array/src/arrays/dict/compute/fill_null.rs` + into `mod tests` at `fill_null.rs:90-131`. PR #10681 rewrote this module (commit `d834a11faa`), + so adapt the test: + - Use the module `SESSION` (`:109`) with `SESSION.create_execution_ctx()`, not `array_session()`. + - Drop the explicit `.execute::` before `assert_arrays_eq!`. The macro executes + the lazy result (`AGENTS.md`, Tests), and PR #10458 removed the same pattern from this module. + - Add `use rstest::rstest;` and `use vortex_error::VortexResult;`. +- The test reaches the dictionary kernel. `VortexSession::empty()` installs the built-in kernels on + first use (`vortex-array/src/optimizer/kernels.rs:336-349`), including this one + (`dict/vtable/kernel.rs:25`). The codes are partially valid, so `short_circuit` does not answer + first (`vortex-array/src/scalar_fn/fns/fill_null/kernel.rs:67-93`). +- Cases. Each dictionary has the codes `[0, null]` and the values `0..=fill`. The results before the + fix come from reading the code, not from a run. + +| Case | Codes type | Fill | Before the fix | After the fix | +| --- | --- | --- | --- | --- | +| `u8_max` | `u8` | 255 | `[0, 255]`, passes | `[0, 255]` | +| `u8_overflow` | `u8` | 256 | `[0, 0]`, fails | `[0, 256]` | +| `u8_truncation` | `u8` | 300 | `[0, 44]`, fails | `[0, 300]` | +| `i8_max` | `i8` | 127 | `[0, 127]`, passes | `[0, 127]` | +| `i8_overflow` | `i8` | 128 | Code -128, error or panic Not verified | `[0, 128]` | + +- `nullable_codes_fill_in_values` stays as it is. + +Benchmarks: None. The change adds one checked scalar cast per kernel call. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array dict::compute::fill_null` +- With the kernel change reverted, `u8_overflow` and `u8_truncation` fail. + +Review focus: + +- No `as` conversion of the index remains, and the `#[expect(clippy::cast_possible_truncation)]` is + gone. +- The declined case gives the same result as the "no fill value found" branch. +- The test style matches the module after PR #10681. + +Size: small. One kernel branch and one parameterized test. + +Open questions: None. + +## Notes from planning + +These facts came up while the briefs were drafted. They do not change the scope of the PRs. + +1. FX1 decodes the protobuf field with `pb::PType::try_from` and the existing + `From for PType`, not with `PType::try_from(i32)`. The wire numbers then stay tied to + the protobuf enum. The numbers of the two enums match today. +2. The flatbuffer `PType` path is already strict + (`vortex-array/src/dtype/serde/flatbuffers.rs:559`). FX1 only adds the raw value to its message + and a test. +3. No Variant metadata field holds an enum. Variant and Parquet-variant metadata store protobuf + dtypes, so the protobuf fix covers them. +4. `vortex-layout/src/layouts/dict/writer.rs:537-550` defines a second `DictLayoutMetadata` that no + code uses. Its comment says to read the field with the generated getter, which FX1 makes wrong + advice. Its removal belongs in a separate change. +5. FX2 fixes a contract violation, not a wrong result that a caller can see. The rule does not + change the validity that a caller sees before execution, and the null check still runs at + execution. +6. The cast rules of RunEnd (`encodings/runend/src/compute/cast.rs:15-33`), ZigZag + (`encodings/zigzag/src/compute/cast.rs:14-25`), FoR + (`encodings/fastlanes/src/for_/compute/cast.rs`) and RLE + (`encodings/fastlanes/src/rle/compute/cast.rs`) push nullability casts down without the same + guard. A follow-up issue tracks them. FX2 does not change them. +7. The cast tests in `compute/cast.rs:60` use `VortexSession::empty()` without + `crate::initialize`. FX2 changes that session, or the tests never reach the new kernel. +8. Commit `7e2e5e4d7a` has two tests. `fill_unreferenced_value` already avoids `Narrow`, and FX3 + ports it. The `Narrow` test belongs to NR5, so NR5 lands after FX3. +9. On `develop`, the FX3 bug is shown only for a dictionary built directly. Whether the compressor + or a file read builds a dictionary with more values than its codes type can index is Not + verified. +10. The test modules in `vortex-array/src/dtype/serde/flatbuffers.rs:565` and + `encodings/alp/src/alp_rd/array.rs:563` are named `test`, which `AGENTS.md` discourages. FX1 + adds tests to them but does not rename them. diff --git a/research/decimal/agents/briefs/02-integer-narrow.md b/research/decimal/agents/briefs/02-integer-narrow.md new file mode 100644 index 00000000000..79dbc4b8465 --- /dev/null +++ b/research/decimal/agents/briefs/02-integer-narrow.md @@ -0,0 +1,931 @@ + + +# Integer `Narrow` PR briefs: NR1 to NR6 + +These briefs cover the integer `Narrow` track. Each PR is written from scratch from its brief +(D38). A local rewrite of the closed PRs #10223 and #10332 exists in +`/Users/connor/spiral/vortex-data/vortex3`. It is a reference, not a base: read its code and tests +for edge cases, reuse its test cases, and do not cherry-pick it. The rewrite predates D25, D26, D28 +and D36, so its structure differs from these briefs. The one exception is FX3, which ports the +dictionary fill fix of the rewrite as it is. + +No build, test or benchmark ran for these briefs. Every claim about code comes from reading it, +and it is marked "Not verified" where it matters. + +## Shared context for the track + +### The reference rewrite + +| Commit | Subject | Use it for | Do not copy | +| --- | --- | --- | --- | +| `f7408481ce` | feat(array): add in-memory Narrow integer encoding | NR1: the width selection of `encode`, `validate_dtypes`, the compare constant shortcut, the selection rules, and the cases of `narrow/tests/{encoding,compute}.rs` | The integer-only parts of the compare kernel (see below). The serialization stub | +| `8d548646ee` | feat(array): forward Narrow integer aggregates | NR3: the list of supported aggregates, and the cases of `narrow/tests/aggregates.rs` and `encodings/fastlanes/tests/narrow.rs` | The accumulator for the child dtype. NR3 uses the accumulator of the parent dtype (D36) | +| `74aa17644e` | feat(array): push arithmetic and scalar operations through Narrow | NR4: the interval arithmetic, the cast, fill-null, zip and between rules, and the cases of `narrow/tests/{numeric,pushdown}.rs` | The replacement of the NR1 cast and fill-null rules. NR1 writes those rules so that NR4 extends them | +| `af4b663059` | feat(array): retain Narrow storage through merges and indices | NR5: `TakeIndicesReduce` and `test_narrowed_take_indices` | The chunked and interleave merge rules in `narrow/merge.rs`, which put `Narrow` above `Chunked` and `Interleave` (D25) | +| `7e2e5e4d7a` | fix(array): avoid truncating dictionary fill codes | FX3 ports the fix as it is. NR5 reuses `test_fill_null_after_narrowing_take_indices` | Nothing | +| `adc1e83b9d` | feat(compress): retain narrow integer children through compression | NR1: `encode_values`. NR6: the list of call sites. WR1 to WR3: background | The `Narrow` wrappers for internal children, and the preview edition with only `vortex.narrow` | + +All these commits are reachable from every local worktree, because the worktrees share one bare +repository. The five rewrite commits are not on `origin`. Only the local branches +`ct/narrow-pushdown` (checked out in `vortex3`) and `ct/narrow-encoding` hold them, and both names +also exist on `origin` with the older commits of the closed PRs. Do not push, reset or delete +these two local branches. A remote agent cannot read the reference until it is pushed to a new +backup branch. `adc1e83b9d`, `0d94e44a91` and `07fafc2a5b` are on `origin/ct/narrow-compression`. + +### Changes on `develop` since the rewrite + +The rewrite is based on `61ff28d970`. These changes on `develop` matter when an agent reads it: + +| Change on `develop` | Commit | What new code does | +| --- | --- | --- | +| `vortex_array::array_session()` removed | `d834a11faa` (#10681) | Every test module uses a module `static SESSION`, with the pattern below. The tests of the rewrite call `array_session()`, so port their cases, not their setup | +| Interleave uses named slots | `96ac9f2cc5` (#10688) | `InterleaveArray::try_new(values, array_indices, row_indices)` keeps its signature. The slot order is the same: selectors at 0 and 1, values from 2. `InterleaveSlots` is `pub(super)` (`vortex-array/src/arrays/interleave/array.rs:24`). The NR5 pull-up rule builds a new array with `InterleaveArray::try_new` | +| Storage pushdown in the aggregate session | `d7562aa0e6` (#10566), `3bf14ba647` (#10616) | `AggregateFnSession::default` has `register_storage_ordered` calls (`vortex-array/src/aggregate_fn/session.rs:127-129`). Register the NR3 kernel after the `Dict` kernels. `register_aggregate_kernel` and the fallback lookup are unchanged (`:199-224`) | +| `fill_null` on an empty input | `633c52deab` (#10627) | `FillNull` returns an empty array for a non-constant fill on an empty input (`vortex-array/src/scalar_fn/fns/fill_null/mod.rs:145`). The `Narrow` rule receives a `Scalar`, so it is not affected | +| Kernel for constant dictionary codes | `a169338930` (#10643) | An execute-parent kernel handles `Constant` codes (`vortex-array/src/arrays/dict/take.rs:181-205`). The NR5 rule for `Narrow` codes is a reduce rule, so the two do not overlap | +| Deprecated APIs and `legacy_session` removed | `85ae34d5f0` (#10595) | None of the removed APIs is needed | +| Execution contexts in validation and patches | `c48457b572`, `7216cb5cb8`, `673fd2e064`, `b549eb5a97` | `ListViewArray::try_new`, `Patches::new`, `MaskedArray::try_new`, `RunEndArray::try_new` and `SparseArray::try_new` take a context. Reduce rules have none, so the NR5 brief names constructors without a context | + +`VTable`, `OperationsVTable`, `ValidityChild`, the kernel registry, `CompareKernel`, +`execute_compare`, `execute_numeric`, the reduce adaptors, `ChunkedSlots::CHUNKS_OFFSET`, +`DictSlots` and `BitBuffer::full_in` keep the signatures that the rewrite uses at `b70206818f`. +The execute-parent registry keeps a list for each key +(`vortex-array/src/optimizer/kernels.rs:263-276`). So the compare kernel of NR1 and the numeric +kernel of NR4 can both register for `Binary` on `Narrow`. + +#### Test session pattern + +PR #10681 gave each test module one session. `VortexSession::empty()` creates the vortex-array +session variables on first use, and the default `KernelSession` runs `arrays::initialize` +(`vortex-array/src/optimizer/kernels.rs:336-349`), so the `Narrow` kernels are present. + +```rust +// Module level, as in vortex-array/src/arrays/dict/compute/fill_null.rs:109 on develop. +use std::sync::LazyLock; + +use vortex_session::VortexSession; + +static SESSION: LazyLock = LazyLock::new(VortexSession::empty); +``` + +A test outside vortex-array that needs another crate registers it in the closure, as +`encodings/fastlanes/src/bitpacking/array/tests.rs:43-47` does: + +```rust +static SESSION: LazyLock = LazyLock::new(|| { + let session = VortexSession::empty(); + vortex_fastlanes::initialize(&session); + session +}); +``` + +### Stack and branches + +The closed PRs used `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and +`ct/narrow-compression` (#10333). All three still exist on `origin`. The new branches avoid them. + +| PR | Branch | Base | +| --- | --- | --- | +| NR1 | `ct/narrow-in-memory` | `develop` | +| NR2 | `ct/narrow-benchmarks` | NR1 | +| NR3 | `ct/narrow-aggregate-kernels` | NR2 | +| NR4 | `ct/narrow-arithmetic-kernels` | NR3 | +| NR5 | `ct/narrow-structural-rules` | NR4, after FX3 is on `develop` | +| NR6 | `ct/narrow-encode-values` | `develop` after NR1 merges. Merge it after NR5, so that the stack lands first (D35) | + +NR1 to NR5 form one linear stack (D35). NR3 needs no code from NR2, but it sits on NR2 so that the +benchmarks exist before the kernels change. NR6 needs only NR1. CodSpeed baselines come from pushes +to `develop` (`.github/workflows/codspeed.yml:13-17`). How CodSpeed picks a base for a PR whose base +branch is another PR branch is not verified. The safe order is to merge NR1 and NR2, then rebase NR3 +to NR5 on `develop` before their CodSpeed reports are read. + +`develop` squash-merges PRs. After a PR of the stack merges, a plain `git rebase develop` on the +next branch replays the commits of the merged PR again and conflicts. Rebase with +`git rebase --onto develop `, or with `gh stack`. + +### Write the kernels so that decimal support is additive + +CD4 opens `Narrow` for decimals. The rewrite has integer-only assumptions that break then (table +below). The new NR code avoids them from the start where that costs little: + +- No `as_ptype()` and no `DType::Primitive` on a path that a decimal child can reach once + `validate_dtypes` accepts decimals. For example, the compare of two `Narrow` operands picks the + wider child dtype without `as_ptype()`. +- The out-of-range compare decides the direction by comparing the constant with the bounds of the + child dtype, not through `i64::try_from` and `is_signed_int()`. +- Aggregates use the accumulator of the parent dtype (D36), so a decimal sum needs no special case. +- `execute` can still yield a cast for integers, so that the child fuses decoding and widening. + CD4 adds the decimal path, because the decimal cast of D15 returns a `Narrow`. +- Kernels that do not support decimals decline them with `Ok(None)`, so canonicalization handles + them. + +These sites of the rewrite assume integers. Most of them decline a decimal with `Ok(None)`, which +is safe. The others give a panic, a wrong result or a likely execution loop once `validate_dtypes` +accepts decimals: + +| Site | Assumption | Effect with a decimal `Narrow` | Safe for decimals | +| --- | --- | --- | --- | +| `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73-88` `validate_dtypes` | `PType::try_from`, integer, signedness, byte width | The gate. It rejects decimals | Gate | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:52-61` | `as_ptype()` and `DType::Primitive` for two `Narrow` operands | Panic: `DType::as_ptype` panics on a non-primitive dtype (`vortex-array/src/dtype/dtype_impl.rs:209-215`) | No | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:89` | `below = lhs.dtype().is_signed_int() && i64::try_from(&constant)? < 0` | Wrong result: `is_signed_int` is false for a decimal (`dtype_impl.rs:226-231`), so a negative constant below the child range answers `Lt`, `Lte`, `Gt` and `Gte` wrongly | No | +| `f7408481ce:vortex-array/src/arrays/narrow/compare.rs:76-86` | Constant cast to the child dtype | Correct when the child label is true (D10): the decimal scalar cast checks the precision (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:158-182`) | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:316-346` `BetweenReduce` | Constant cast to the child, else two compares | Inherits the wrong `below` result through `two_compares` | No | +| `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | Accumulate at the child dtype, then cast the partial | Wrong result for `Sum` and `SumV2`: the decimal sum state is `Decimal(p + 10, s)` (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`), so a child state overflows to null before the parent state does | No | +| `f7408481ce:vortex-array/src/arrays/narrow/vtable.rs:111-116` `execute` | Yields `values.cast(dtype)` | Today the decimal cast executes a copy (`vortex-array/src/arrays/decimal/compute/cast.rs`). D15 says a decimal cast to a wider canonical width returns a `Narrow` over its input. If CD4 implements that as a reduce rule, `execute` returns the same `Narrow` again and execution does not converge. Not verified | Depends on CD4 | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:111-143` `CastReduce` | `PType::try_from` for target and child | Declines decimals. CD4 adds the decimal cast rules of D15 | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:146-194` fill null, `integer_storage_type` | `PType::try_from`, `i64::try_from`, `min_*_ptype_for_value` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:240-278` `zip_values` | `PType::try_from`, `DType::Primitive` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | Not reachable for decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/numeric.rs:70` | `PType::try_from(array.dtype())` | Declines decimals | Yes | +| `af4b663059:vortex-array/src/arrays/narrow/encoding.rs:29` `encode` | Takes a `PrimitiveArray`, rejects non-integers | No decimal producer. CD4 adds one | Yes | + +## Briefs + +### NR1: In-memory integer `Narrow` encoding + +- Goal: Add `Narrow`, an in-memory encoding that stores an integer column as a strictly narrower + integer child with the same signedness and nullability. Selection, compare, mask, cast and fill + null pass to the child. `Narrow` has no wire form yet, so the serializer writes its canonical + array instead (D26). +- Depends on: None. Soft: None. Blocks: NR2, NR3, NR4, NR5, NR6, CD4, CD7. +- Decisions: D8 (one encoding with ID `vortex.narrow`), D15, D18, D22, D23, D26, D28, D38. +- Branch: `ct/narrow-in-memory`, based on `develop` (`b70206818f` or later). +- Scope: + - Write the encoding, `validate_dtypes`, the compare kernel, and the selection, cast and + fill-null rules. `f7408481ce` is the reference (D38). + - Add `NarrowArray::encode_values`, marked `#[deprecated]` (D28). `adc1e83b9d` has a reference + version. + - Add the serialization workaround of D26: a plugin hook that lowers an array before it is + serialized, which `Narrow` implements by executing itself to its canonical array. + - Add a reduce rule that flattens a `Narrow` child, so D15 holds after physical rewrites. +- Out of scope: Aggregate kernels (NR3). Arithmetic, between, zip, float and cross-signedness casts + (NR4). The rules that move structural encodings above `Narrow`, and the dictionary-codes rule + (NR5). The switch of the existing narrowing call sites to `encode_values` (NR6). The + `vortex.narrow` wire form and editions (WR1, WR2). Compressor changes (WR3). Decimals (CD4). The + dictionary fill-code fix (FX3). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/mod.rs` | New. Reference: `f7408481ce`. `Narrow`, `NarrowArray`, `NarrowSlots`, `try_new`, `validate_dtypes`, `initialize` | +| `vortex-array/src/arrays/narrow/vtable.rs` | New. Reference: `f7408481ce`. With `VTable::reduce` that flattens a `Narrow` child, and the lowering hook | +| `vortex-array/src/arrays/narrow/encoding.rs` | New. `NarrowArray::encode` and `encode_values`. References: `f7408481ce` and `adc1e83b9d:vortex-array/src/arrays/narrow/encoding.rs:45-90` | +| `vortex-array/src/arrays/narrow/compare.rs` | New. Reference: `f7408481ce`. Compare kernel | +| `vortex-array/src/arrays/narrow/rules.rs` | New. Reference: `f7408481ce`. Slice, filter, take, mask, cast and fill-null rules | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | New | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | New. Port the cases of the reference tests, and add the flatten, `encode_values` and serialization tests | +| `vortex-array/src/arrays/narrow/tests/compute.rs` | New. Port the cases of the reference tests | +| `vortex-array/src/array/plugin.rs:113-158` | Add the lowering hook to `ArrayPlugin`, with a default of `Ok(None)`. The blanket impl for `VTable` (`:165-205`) forwards it | +| `vortex-array/src/serde.rs:169-190` | `ArraySerializationTree::try_new` asks the hook when `array_serialize` returns `None` | +| `vortex-array/src/arrays/mod.rs:85-91` | Add `pub mod narrow;` with `pub use narrow::Narrow;` and `pub use narrow::NarrowArray;` between `masked` and `null` | +| `vortex-array/src/arrays/mod.rs:144-145` | Add `narrow::initialize(session);` between `map` and `null` | +| `vortex-array/src/session/mod.rs:31-32` | Import `crate::arrays::Narrow` | +| `vortex-array/src/session/mod.rs:107-108` | `this.register(Narrow);` after `Masked` | + +Design notes: + +```rust +pub struct Narrow; +pub type NarrowArray = Array; +#[array_slots(Narrow)] +pub struct NarrowSlots { pub values: ArrayRef } + +impl NarrowArray { + pub fn try_new(values: ArrayRef, dtype: DType) -> VortexResult; + + /// Stores `array` as a `Narrow` over a child of the smallest integer type with the same + /// signedness, or returns it unchanged when no narrower type holds its values. + pub fn encode(array: PrimitiveArray, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Returns `array` at the smallest integer type with the same signedness, without a `Narrow`. + /// Floats return unchanged. + #[deprecated(note = "changes the dtype; a later change decides whether each caller needs a Narrow")] + pub fn encode_values(array: PrimitiveArray, ctx: &mut ExecutionCtx) + -> VortexResult; +} +pub(super) fn validate_dtypes(storage: &DType, logical: &DType) -> VortexResult<()>; +``` + +- `validate_dtypes` implements D22: both dtypes are integer primitives, the signedness is equal, + the child byte width is strictly smaller, and the nullability is equal. `VTable::validate` and + `try_new` both call it. +- `encode` and `encode_values` share one private, non-deprecated function, so `encode` itself needs + no `#[expect(deprecated)]`. That function takes `min_max` of the valid values, picks the smallest + type of the same signedness for the min and for the max, and takes the wider of the two. An + empty or all-null input gets the 1-byte type of the same signedness. A 1-byte input returns + unchanged. `encode` fails on a float input, and `encode_values` returns a float unchanged. An + `I64` column with values 0 to 200 gets an `I16` child (D22). +- `encode_values` has no caller in this PR. CD7 (the DecimalByteParts MSP) and NR6 (the existing + call sites) call it, each with `#[expect(deprecated, reason = "...")]` (D28). +- `try_new` flattens a `Narrow` child. Physical rewrites keep only the slot dtype + (`vortex-array/src/array/erased.rs:595-620`), so a rewrite can replace a child with a `Narrow` of + the same dtype. The new `reduce` returns `NarrowArray::try_new(child.values().clone(), dtype)` + when the child is a `Narrow`, and `None` otherwise. +- `execute` yields `values.cast(dtype)`, so the child can fuse decoding and widening. `scalar_at` + casts the child scalar. Validity comes from the child. CD4 changes `execute` for decimals, + because the decimal cast of D15 would loop. +- The serialization workaround (D26): + - Today `ArraySerializationTree::try_new` fails with "Array … does not support serialization" + when `array_serialize` returns `None` (`vortex-array/src/serde.rs:171-175`). + - A plugin cannot return the wire form of another encoding. `array_serialize` checks that the + returned ID is one of the plugin's `serialized_ids` (`vortex-array/src/session/mod.rs:144-151`). + - So the fallback lives in the serializer. A new `ArrayPlugin` method, for example + `lower_for_serialization(&self, array, session) -> VortexResult>`, returns + `Ok(None)` by default. When `array_serialize` returns `None`, `try_new` calls it, and it builds + the tree from the returned array. Without a lowered array, it fails as today. + - `Narrow` lowers itself by executing to its canonical array with + `session.create_execution_ctx()`. No other encoding implements the hook. + - The doc of the hook and a comment in the `Narrow` impl say that this is a workaround. It stays + until `vortex.narrow` is in `core` (LT1). WR1 also uses it when the selected editions do not + permit `vortex.narrow`. +- `deserialize` fails with "Narrow deserialization is not supported", because no wire form exists + yet. `ArraySession::default` registers `Narrow` like `PiecewiseSequence` + (`vortex-array/src/session/mod.rs:108`). No compressor produces a `Narrow` in this PR. +- Compare kernel: two `Narrow` operands compare at the wider child width. A constant that casts to + the child dtype compares against the child. A constant outside the child range gives a constant + answer with the child validity. `Eq` is false and `NotEq` is true. `Lt`, `Lte`, `Gt` and `Gte` + follow from whether the constant is below the range. A null constant gives all nulls. Follow + "Write the kernels so that decimal support is additive" above. +- Cast rule in this PR: take an integer target of the same signedness. If it is no wider than the + child, the cast goes to the child with the same target. If it is wider, the rule returns a + `Narrow` over the child cast to the target nullability (D15 as amended). Other targets fall + back. Write the cast and fill-null rules so that NR4 can add target kinds without replacing them. +- Fill null declines when the fill value does not fit the child dtype, so canonical execution + handles it. + +Steps: + +1. Create `ct/narrow-in-memory` from `develop` in a new worktree. Do not reuse the `vortex3` + worktree or its branch. +2. Read `f7408481ce` and its tests with `git show f7408481ce`. +3. Write the module: the encoding, `validate_dtypes`, the vtable, the compare kernel and the rules. + Register it in `arrays/mod.rs` and `session/mod.rs`. +4. Add `encode_values` and the shared private function. +5. Add the lowering hook in `plugin.rs` and `serde.rs`, and the `Narrow` impl. +6. Add `VTable::reduce` and the flatten rule. +7. Add the tests, with the module `SESSION`. Port the cases of the reference tests. +8. Commit in reviewable steps. Each commit has the line + `Signed-off-by: Connor Tsui ` and no co-author line. + +Tests: + +- Port the cases of these tests in `f7408481ce:vortex-array/src/arrays/narrow/tests/encoding.rs`: + `test_signed_encoding` and + `test_unsigned_encoding` (`rstest` width boundaries), `test_null_payloads_and_empty`, + `test_reject_invalid_dtypes`, `test_validate_slots`, `test_flatten_and_scalar_dtype`, + `test_encoded_child_execution`, `test_smallest_width_unchanged`, `test_reject_float_encoding`, + `test_deserialization_unsupported`. +- `test_serialization_unsupported` becomes `test_serialization_writes_canonical`. Serialize a + nullable and a non-nullable `Narrow`, and decode the bytes. The result is a canonical primitive + array with the logical dtype and equal values, and its wire ID is the primitive ID. +- New, in `narrow/tests/encoding.rs`: + - `test_encode_values_keeps_signedness`, an `rstest`. `[0i64, 127]` gives `I8`, `[0i64, 200]` + gives `I16`, `[-1i32, 200]` gives `I16`, an empty `i32` gives `I8`, all-invalid `u32` gives + `U8`, and `[0u64, u64::MAX]` stays `U64`. Validity is kept in every case. + - `test_encode_values_preserves_floats`, from `adc1e83b9d`. + - `test_reduce_flattens_nested_narrow`. Build a `Narrow` with `try_from_parts` over a `Narrow` + child, and call `optimize()`. Check that the result is one `Narrow` over the inner child, with + the same values. +- In `vortex-array/src/serde.rs` tests: an encoding without a wire form and without the hook still + fails with "does not support serialization". +- Port the cases of these tests in `f7408481ce:vortex-array/src/arrays/narrow/tests/compute.rs`: + `test_selection_and_fill_null`, + `test_cast_preserves_checked_nullability`, `test_fill_null_preserves_result_dtype`, + `test_arithmetic_uses_logical_width`, and `test_comparison`. The last one is an `rstest` over six + operators. It covers constants on both sides of the child range, both operand orders, and a + second `Narrow`. + +Benchmarks: None in this PR. NR2 adds them. No change on existing benchmarks is expected. Record +the `take_patches` results for the NR6 investigation. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array narrow` and `cargo nextest run -p vortex-array serde` +- Serializing a `Narrow` writes its canonical array, and the read-back array has equal values. +- No file writer or compressor produces `vortex.narrow`. +- `encode_values` carries `#[deprecated]` and has no caller yet. + +Review focus: The constant shortcut of the compare kernel for both operand orders and null +constants. Checked nullability in the cast rule. Nullability of `take` and `mask` results, which +follow the child. The fill-null fallback. The new `reduce` rule. The lowering hook: it runs only +when an array has no wire form, and only `Narrow` implements it. Registration of the compare kernel +through `arrays::initialize`. + +Size: Large. The reference commit has about 550 lines of source and 435 lines of tests. +`encode_values`, the hook and their tests come on top. + +Open questions: None. + +### NR2: Benchmarks for compute on integer `Narrow` + +- Goal: Add one benchmark file that runs each `Narrow` operation on three inputs. They are the + `Narrow`, the equivalent wide canonical array, and the `Narrow` decoded inside the timed region. + CodSpeed on NR3 to NR5 then shows each kernel against the fallback. +- Depends on: NR1. Soft: None. Blocks: NR3, NR4 and NR5 (their CodSpeed evidence). +- Decisions: D19 (benchmarks land before the changes that they cover), D15. +- Branch: `ct/narrow-benchmarks`, based on NR1. +- Scope: A new `vortex-array/benches/narrow.rs` and its `[[bench]]` entry. +- Out of scope: Kernel changes (NR3 to NR5). Workflow changes. Decimal benchmarks (CD1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/benches/narrow.rs` | New | +| `vortex-array/Cargo.toml:335-336` | Add `[[bench]]` with `name = "narrow"` and `harness = false` after the `probe` entry | + +Design notes: + +- Follow the conventions of `vortex-array/benches/compare.rs:22-76` and + `vortex-array/benches/filter_fixed_width.rs:101-126`: + - An SPDX header and a module doc. + - The `MiMalloc` global allocator. + - A `main` that forces `SESSION` and then calls `divan::main()`. + - A `SESSION` with `ArraySession`, `KernelSession`, `DTypeSession`, `ScalarFnSession`, + `AggregateFnSession`, `MemorySession` and `StatsSession`. + - Each result wrapped in `divan::black_box`. +- Do not tag any case with `#[vortex_bench_support::cpu_features]`. These kernels are dispatch + work, so the cases stay in simulation. +- The name `narrow` matches no prefix in `.github/workflows/codspeed.yml:92-104`, so the cases run + in the "builders" shard. No workflow change is needed. +- Data: `LEN = 8_192` rows, so the wide input holds 64 KiB, as `filter_fixed_width.rs:57-64` + recommends. Values are `i16` from `StdRng::seed_from_u64`, uniform in `-10_000..=10_000`. The + `Narrow` input has logical dtype non-nullable `I64` over that `i16` child. The wide input holds + the same values as `i64`. +- Build every input inside `with_inputs`, so setup is not timed. Build fresh arrays for each + iteration. `sum` caches its result on the array + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:63-65`), and the NR4 arithmetic kernel calls + `min_max` on both operands, so reused inputs hide work. + +```rust +#[derive(Clone, Copy, Debug)] +enum Input { + /// The canonical `i64` array with the same values. + Wide, + /// A `Narrow` with logical dtype `i64` over an `i16` child. + Narrow, + /// The `Narrow`, executed to a `PrimitiveArray` inside the timed region, then the operation. + Decoded, +} + +const INPUTS: &[Input] = &[Input::Wide, Input::Narrow, Input::Decoded]; + +#[divan::bench(args = INPUTS)] +fn compare_in_range(bencher: Bencher, input: Input) { /* ... */ } +``` + +Workloads. Each one ends with `execute::`, or returns the aggregate scalar: + +| Bench | Operation | PR whose kernel changes the `Narrow` case | +| --- | --- | --- | +| `compare_in_range` | `binary(Constant(0i64), Lt)` | NR1 (child compare) | +| `compare_out_of_range` | `binary(Constant(100_000i64), Lt)`, above `i16::MAX` | NR1 (constant answer) | +| `between_in_range` | `between(Constant(-1_000i64), Constant(1_000i64))`, both bounds non-strict | NR4 | +| `filter` | `filter(mask)`, random mask with density 0.5 | NR1 | +| `take` | `take(indices)`, `LEN / 4` random `u32` indices | NR1 | +| `slice` | `slice(LEN / 4..3 * LEN / 4)` | NR1 | +| `sum` | `sum(&array, ctx)` | NR3 | +| `min_max` | `min_max(&array, ctx, NumericalAggregateOpts::default())` | NR3 | +| `add_constant` | `binary(Constant(1i64), Add)` | NR4 | +| `add_narrow` | `Narrow + Narrow`, second operand from another seed | NR4 | +| `add_wide` | `Narrow + wide i64 array`, only `Narrow` and `Decoded` inputs | NR4 | +| `chunked` | `ChunkedArray` of four inputs of `LEN / 4` rows | None. It guards the chunk kernels, which push work into each `Narrow` chunk | +| `interleave` | `InterleaveArray` of two inputs of `LEN / 2` rows, `array_indices = i % 2` as `u8`, `row_indices = i / 2` as `u32` | None. It guards the interleave kernels | + +- `Decoded` for `chunked` and `interleave` decodes each input, then builds the parent. +- `Wide` for `add_narrow` adds two `i64` arrays. `add_wide` omits `Wide`, because it is the same + case. + +Steps: + +1. Create `ct/narrow-benchmarks` on top of NR1. +2. Add `vortex-array/benches/narrow.rs` and the `[[bench]]` entry. +3. Commit as `bench(array): add Narrow compute benchmarks`, with the sign-off line. + +Tests: None. The benchmark compiles under `cargo clippy --all-targets`. + +Benchmarks: This PR only adds cases, so CodSpeed reports them as new. Qualitative expectations for +later PRs: + +- NR3: `sum[Narrow]` and `min_max[Narrow]` move from about the `Decoded` level toward or below + `Wide`. +- NR4: `add_constant`, `add_narrow`, `add_wide` and `between_in_range` change for `Narrow`. The + direction for `add_wide` is not known, because the NR4 kernel scans the wide operand for bounds. +- `chunked` and `interleave` stay flat. NR5 changes no tree that they build. +- `Wide` and `Decoded` cases must stay flat in NR3 to NR5. A move there points at noise or at a + side effect of session registration. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo bench -p vortex-array --bench narrow` runs every case. +- Each case stays under 1 ms in CodSpeed simulation. + +Review focus: The three inputs hold the same logical values. Inputs are built outside the timed +closure and fresh for each iteration. `Decoded` decodes inside the timed closure. + +Size: Medium. One file with about 13 workloads. + +Open questions: None. + +### NR3: Aggregates forward to the child + +- Goal: Register one fallback aggregate kernel for `Narrow`. It feeds the child to an accumulator + of the parent dtype (D36), so every result and every overflow is the same as on the parent. +- Depends on: NR1, NR2 (stack, D35: the benchmarks exist before the kernels change). Soft: None. + Blocks: NR4 (stack), CD4. +- Decisions: D15, D36, D38. +- Branch: `ct/narrow-aggregate-kernels`, based on NR2. +- Scope: Write the kernel. `8d548646ee` is the reference for the list of aggregates and for test + cases. +- Out of scope: Decimal aggregates (CD4). Pushdown of sum, min and max into the MSP (LT3). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/aggregates.rs` | New. Reference: `8d548646ee` | +| `vortex-array/src/arrays/narrow/mod.rs` | `mod aggregates;` and `pub(crate) use aggregates::register_aggregate_kernels;`, module doc names aggregates | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod aggregates;` | +| `vortex-array/src/arrays/narrow/tests/aggregates.rs` | New. Port the cases of the reference tests | +| `vortex-array/src/aggregate_fn/session.rs:51-52` | `use crate::arrays::narrow;` | +| `vortex-array/src/aggregate_fn/session.rs:135-136` | `narrow::register_aggregate_kernels(&this);` after the `Dict` kernels | +| `encodings/fastlanes/tests/narrow.rs` | New integration test with a `BitPacked` child, with the closure session pattern | + +Design notes: + +```rust +pub(crate) fn register_aggregate_kernels(session: &AggregateFnSession); +struct NarrowAggregateKernel; // impl DynAggregateKernel +``` + +- The kernel registers with aggregate id `None`, so it is the fallback for `Narrow`. A kernel for an + exact aggregate wins over it (`vortex-array/src/aggregate_fn/session.rs:199-224`). +- Supported: `MinMax`, `Min`, `Max`, `Sum`, `SumV2`, `IsConstant`, `IsSorted`, `First`, `Last`, + `Count`, `NullCount`, `AllNull`, `AllNonNull`. Every other aggregate returns `None` and uses the + default path. `UncompressedSizeInBytes` must stay on the default path, because it depends on the + logical width. +- Algorithm (D36): build the accumulator for the parent dtype, and feed it the child batch. First + confirm that each supported accumulator accepts a batch of the narrower child dtype. Not + verified. An aggregate whose accumulator rejects it leaves the supported list and uses the + default path. +- Integer correctness: exact widening keeps values and order. The integer `Sum` state is `I64` for + every signed input and `U64` for every unsigned input + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:114-122`). With the parent accumulator, overflow + matches the parent by construction. +- A decimal sum state depends on the input precision (`sum/mod.rs:81-86`). Because the + accumulator comes from the parent dtype, a decimal sum is correct once CD4 opens + `validate_dtypes`. CD4 only adds the decimal tests. + +Steps: + +1. Create `ct/narrow-aggregate-kernels` on the NR2 branch. +2. Write `narrow/aggregates.rs` and its registration in `aggregate_fn/session.rs`. +3. Port the test cases of `8d548646ee`, including the fastlanes integration test, with the module + sessions. That integration test needs no `#![cfg(test)]`. +4. Commit with the sign-off line. + +Tests: + +- `test_aggregate_partial_states` (`rstest`): six batch shapes (signed, unsigned, descending + boundary, equal boundary, empty, all null) times 14 bound aggregates. It checks the kernel + partial, the running partial and the final value against the canonical input. +- `test_unsupported_aggregate_keeps_logical_width`: `UncompressedSizeInBytes` returns `None` from + the kernel and reports the logical width. +- `encodings/fastlanes/tests/narrow.rs::test_narrow_bitpacked_child`: filter, take, compare, sum and + execution on a `BitPacked` child. + +Benchmarks: NR2 `sum` and `min_max` with the `Narrow` input are expected to improve. All other +cases stay flat. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-fastlanes --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-fastlanes` +- `cargo nextest run -p vortex-array narrow` +- `cargo nextest run -p vortex-fastlanes --test narrow` + +Review focus: The supported list. Each listed aggregate must give the same result on the child as +on the widened values. The cast of partial states that hold extrema or boundary values. The +fallback registration does not shadow kernels for exact aggregates. + +Size: Small. The reference has about 100 lines of source and 235 lines of tests. + +Open questions: None. + +### NR4: Arithmetic and scalar operations forward to the child + +- Goal: Execute checked add, subtract, multiply and divide at the smallest width that holds both + operands and the result interval. Push `between`, `zip`, `fill_null` and more casts through + `Narrow`. +- Depends on: NR3 (stack). Soft: NR2. Blocks: NR5. +- Decisions: D15, D22, D38. +- Branch: `ct/narrow-arithmetic-kernels`, based on NR3. +- Scope: Write the numeric kernel and the rules. `74aa17644e` is the reference for the interval + arithmetic and for test cases. Extend the cast and fill-null rules of NR1 instead of replacing + them. +- Out of scope: Structural rules and the dictionary-codes rule (NR5). Decimal arithmetic (D20). + Decimal casts (CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/numeric.rs` | New. Reference: `74aa17644e`. Execute-parent kernel for `Binary` | +| `vortex-array/src/arrays/narrow/rules.rs` | Extend the cast and fill-null rules. Add `ZipReduce`, `ZipFalseReduce` and `BetweenReduce` | +| `vortex-array/src/arrays/narrow/encoding.rs` | `integer_types` helper, used by `encode` and `numeric.rs` | +| `vortex-array/src/arrays/narrow/mod.rs` | `mod numeric;`, `initialize` registers the numeric kernel | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod numeric;` and `mod pushdown;` | +| `vortex-array/src/arrays/narrow/tests/numeric.rs` | New. Port the cases of the reference tests | +| `vortex-array/src/arrays/narrow/tests/pushdown.rs` | New. Port the cases of the reference tests | + +Design notes: + +- Numeric kernel: for `Add`, `Sub`, `Mul` and `Div`, unwrap `Narrow` operands to their children. + Take `min_max` bounds of both operands as `i128`. Compute the result interval. Pick the smallest + type of the same signedness and no wider than the logical type that holds both operands and the + interval. Cast both operands to it, run the checked `execute_numeric`, and wrap the result with + `wrap_values`. When no narrower type holds the interval, use the logical type, so the checked + kernel reports a real overflow exactly as canonical execution does. An empty parent gives an + empty canonical array. An all-null operand gives a null constant. +- Cast rule (D15): a float target goes to the child with the same target. An integer target + first picks a storage type. It is the child type with the signedness of the target. For an + unsigned child and a signed target, it is the next wider signed type. A target no wider than the + storage type goes to the child with the same target. A wider target returns a `Narrow` over the + child cast to the storage type. Non-primitive targets fall back. +- Fill null: `integer_storage_type` picks the child type, or the smallest type that holds the fill + value, without reading values. The child is cast to it before the fill. A fill value that needs + the logical width gives a plain array. +- Zip: both branches. The other branch is a `Narrow` or a constant. Otherwise fall back. +- Between: when both bounds are constants that fit the child, run `between` on the child. + Otherwise combine two compares with `And`, so each compare uses the NR1 kernel. + +Steps: + +1. Create `ct/narrow-arithmetic-kernels` on the NR3 branch. +2. Write the numeric kernel, extend the cast and fill-null rules, and add the zip and between + rules. +3. Port the test cases of `74aa17644e`, with the module `SESSION`. +4. Commit in reviewable steps, with the sign-off line. + +Tests: + +- `narrow/tests/numeric.rs`: + - `test_signed_arithmetic`: `rstest` over width growth, skipped widths, full width, division at + `i8::MIN` and across zero, and an overestimated interval. + - `test_signed_domain`: every `i8` value against small divisors, for all four operators. + - `test_logical_errors`: overflow and division by zero match the canonical errors. + - `test_unsigned_arithmetic`, `test_unsigned_underflow`, `test_nulls_and_empty`, + `test_encoded_child_and_operand_order`, `test_overflow_at_each_logical_width`, + `test_logical_overflow_behind_null`. +- `narrow/tests/pushdown.rs`: + - `test_between_bounds`: `rstest` over bounds inside and outside the child range, and both + strictness values. + - `test_cast_child`: unsigned to signed, signed to unsigned, and float. + - `test_fill_width`, `test_zip_width`, `test_between_nullable_bounds`, + `test_zip_false_branch_and_null_mask`, `test_signedness_cast_checks_values_and_nulls`. + +Benchmarks: NR2 `add_constant`, `add_narrow`, `add_wide` and `between_in_range` change for the +`Narrow` input. If `add_wide[Narrow]` is slower than `add_wide[Decoded]`, report it. A possible +fix is to decline operands that are neither `Narrow` nor constant. + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-array narrow` +- Every arithmetic result equals the canonical result, including errors and nulls. + +Review focus: Interval arithmetic for division across zero and for `i64::MIN / -1`. Overflow +checks stay with the checked kernel at the logical width. The cast rule for unsigned children and +signed targets. The `min_max` pass on a wide operand. Null payloads outside the chosen width. + +Size: Large. The reference has about 450 lines of source and 425 lines of tests. + +Open questions: None. + +### NR5: Structural encodings move above `Narrow`, and dictionary codes use the child + +- Goal: Keep generic structural encodings above `Narrow` (D25). Reduce rules turn a `Narrow` over + `Dict`, `RunEnd`, `Sparse`, `Constant`, `Masked`, `Chunked` or `Interleave` into that encoding + over `Narrow` children. Dictionary codes that are a `Narrow` use its child directly. +- Depends on: NR4 and FX3. Soft: NR2. Blocks: WR3. +- Decisions: D25, D15, D23, D38. +- Branch: `ct/narrow-structural-rules`, based on NR4, after FX3 is on `develop`. +- Scope: Seven pull-up rules, the dictionary-codes rule `TakeIndicesReduce` (reference: + `af4b663059`), and the case of the `Narrow` regression test from `7e2e5e4d7a`. +- Out of scope: The chunked and interleave merge rules of `af4b663059` (`narrow/merge.rs`). They + put `Narrow` above `Chunked` and `Interleave`, so they would loop against the new rules. The same + pull-up rule for every other encoding, which a follow-up issue tracks. Decimal tests (CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/rules.rs` | The pull-up rules for `Dict`, `Constant`, `Masked`, `Chunked` and `Interleave`. `TakeIndicesReduce` for `Dict` codes, from `af4b663059` | +| `vortex-array/src/arrays/narrow/tests/mod.rs` | `mod structural;` | +| `vortex-array/src/arrays/narrow/tests/structural.rs` | New. The pull-up tests, `test_narrowed_take_indices` from `af4b663059`, and `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`, with the module `SESSION` | +| `encodings/runend/src/` | A parent-reduce rule that moves `RunEnd` above a `Narrow` parent, with a test | +| `encodings/sparse/src/` | A parent-reduce rule that moves `Sparse` above a `Narrow` parent, with a test | + +Design notes: + +| Input | Output | +| --- | --- | +| `Narrow(Dict(codes, values))` | `Dict(codes, Narrow(values))` | +| `Narrow(RunEnd(ends, values))` | `RunEnd(ends, Narrow(values))` | +| `Narrow(Sparse(indices, values, fill))` | `Sparse(indices, Narrow(values), fill)`, with `fill` cast to the parent dtype | +| `Narrow(Constant(value))` | `Constant(value)`, with `value` cast to the parent dtype | +| `Narrow(Masked(child, validity))` | `Masked(Narrow(child), validity)` | +| `Narrow(Chunked(c1, ..., cn))` | `Chunked(Narrow(c1), ..., Narrow(cn))` | +| `Narrow(Interleave(selectors, v1, ..., vn))` | `Interleave(selectors, Narrow(v1), ..., Narrow(vn))` | + +- Every output is valid if each new `Narrow` gets the right dtype. The array that it wraps can have + another nullability than the structural array above it. `Masked` has the nullable dtype of its + child (`vortex-array/src/arrays/masked/array.rs:84`), so its child is often non-nullable. `Dict` + values can be non-nullable under nullable codes, and `Sparse` values can be non-nullable under a + null fill. So each new `Narrow` takes the parent dtype with the nullability of the array that it + wraps: `parent.with_nullability(inner.dtype().nullability())`. Then `validate_dtypes` holds. A + cast of a scalar from the child dtype to the parent dtype is exact. +- Reduce rules have no `ExecutionCtx`, but `MaskedArray::try_new`, `RunEndArray::try_new`, + `RunEndArray::new`, `SparseArray::try_new` and `Patches::new` need one. Use the constructors + without a context: + - `Masked`: `MaskedArray::new_unchecked`. Its safety needs a child without nulls. The new + `Narrow` has the validity of the old child, so it holds. + - `RunEnd`: `RunEnd::new_unchecked(ends, values, offset, length)`, with the ends, offset and + length of the input. + - `Sparse`: `patches.map_values(..)` (`vortex-array/src/patches.rs:972`), then + `SparseArray::try_new_from_patches`. + - `Dict`, `Chunked` and `Interleave`: their `try_new` takes no context. +- Where the rules live. `Dict`, `Constant`, `Masked`, `Chunked` and `Interleave` are in + `vortex-array`, so their rules are `Narrow` reduce rules that match the child encoding. `RunEnd` + and `Sparse` are in their own crates, which depend on `vortex-array`. So their rules are + parent-reduce rules on `RunEnd` and `Sparse` with `Narrow` as the parent. Whether the rule + registry supports both shapes as described is Not verified. Read how `vortex-array/src/arrays/` + registers parent rules before the first commit. +- No rule moves `Narrow` above a structural encoding. A rule in the other direction would loop. +- Dictionary codes: a `Narrow` codes child is replaced by its child, with the same `DictData`. + Narrowed codes can make an unreferenced fill index exceed the codes range. FX3 makes the + dictionary `fill_null` kernel decline in that case, so FX3 must be on `develop` first. +- The rules never read the child dtype, so they also hold for decimal `Narrow`. CD4 adds the + decimal tests. +- WR3 can wrap a column root in `Narrow`, because these rules then move a structural encoding back + to the top. The dict-layout probe and the Arrow and DuckDB exporters then see `Dict` and `RunEnd` + at the root, as today. + +Steps: + +1. Confirm that FX3 is on `develop`. Rebase NR1 to NR4 if needed. +2. Add the five rules in `vortex-array`, with their tests. +3. Add the `RunEnd` and `Sparse` rules in their crates, with one test each. +4. Write `TakeIndicesReduce`, with `af4b663059` as the reference, and port the cases of + `test_narrowed_take_indices`. +5. Port the case of `test_fill_null_after_narrowing_take_indices` from `7e2e5e4d7a`. + +Tests: + +- `test_pull_up_structural`, an `rstest` over the five `vortex-array` shapes. Each result has the + structural encoding at the root and a `Narrow` below it. Its values equal the input, checked with + `assert_arrays_eq!`. +- `test_pull_up_nullable`: nullable children, a null constant, `Masked` with nulls over a + non-nullable child, `Dict` with nullable codes over non-nullable values, and `Sparse` with a null + fill over non-nullable values. +- `test_pull_up_fixed_point`: `optimize()` on a `Chunked` of `Narrow` chunks returns the same tree. + So no rule pair loops. +- In `encodings/runend` and `encodings/sparse`: `Narrow` over the encoding optimizes to the encoding + over `Narrow` values, with equal values. +- `test_narrowed_take_indices`, from `af4b663059`. +- `test_fill_null_after_narrowing_take_indices`, from `7e2e5e4d7a`. It fails without FX3. + +Benchmarks: NR2 `chunked[Narrow]` and `interleave[Narrow]` keep their baseline. Their `Narrow` +input already is a `Chunked` or `Interleave` of `Narrow` chunks, so the new rules change nothing +for them. Kernels push work into each chunk. + +Acceptance checks: + +- `cargo clippy -p vortex-array -p vortex-runend -p vortex-sparse --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-runend -p vortex-sparse` +- `cargo nextest run -p vortex-array narrow`, `cargo nextest run -p vortex-runend` and + `cargo nextest run -p vortex-sparse` +- After `optimize()`, no `Narrow` sits directly above one of the seven encodings. + +Review focus: The fill and constant casts to the parent dtype. Nullability of each new `Narrow`. +No rule pair loops. The dictionary rule copies `DictData`. Each `unsafe` constructor has a SAFETY +comment that names why its precondition holds. + +Shares files with CD4: both edit `vortex-array/src/arrays/narrow/rules.rs`. Do not run NR5 and CD4 +at the same time. + +Size: Medium. Seven small rules in three crates, and their tests. + +Open questions: None. + +### NR6: The existing narrowing call sites move to `encode_values` + +- Goal: Replace `PrimitiveArrayExt::narrow` with the deprecated `NarrowArray::encode_values` of NR1 + at every call site. Each call site carries `#[expect(deprecated)]` with a reason, so every + narrowing that changes a dtype is marked for a later decision (D28). Internal children keep + their signedness. +- Depends on: NR1. Soft: NR5 (merge after the stack, D35). Blocks: None. +- Decisions: D28, D22, D23, D11. +- Branch: `ct/narrow-encode-values`, based on `develop` after NR1 merges. +- Scope: The call sites that part 1 of `adc1e83b9d` changed, with that commit and `0d94e44a91` as + references. The + removal of `PrimitiveArrayExt::narrow`, with its tests ported. Regenerated golden snapshots. A + check that the latest release reads files that this PR writes. +- Out of scope: `encode_values` itself (NR1). Part 2 of `adc1e83b9d` (WR1 to WR3). Which arrays a + compressor wraps in a `Narrow` (deferred, D27). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | Port the narrowing tests of `PrimitiveArrayExt::narrow` to `encode_values` | +| `vortex-array/src/arrays/primitive/array/mod.rs:157-241` | Remove `PrimitiveArrayExt::narrow` and imports that become unused | +| `vortex-array/src/arrays/primitive/array/cast.rs:38-213` | Remove the narrowing tests, which move to `narrow/tests/encoding.rs` | +| `vortex-array/src/builders/dict/mod.rs:19,87` | Dictionary codes | +| `encodings/runend/src/compress.rs:17,91` | Run ends | +| `encodings/fastlanes/src/rle/array/mod.rs:128,442` | Test call site | +| `vortex-btrblocks/src/schemes/patches.rs:9,21,42` | Patch indices and chunk offsets | +| `vortex-btrblocks/src/schemes/binary/varbin.rs:21,87` | VarBin offsets | +| `vortex-btrblocks/src/schemes/float/sparse.rs:13,101` | Sparse indices | +| `vortex-btrblocks/src/schemes/integer/rle.rs:13,65,78` | RLE indices and offsets | +| `vortex-btrblocks/src/schemes/integer/sparse.rs:15,192` | Sparse indices | +| `vortex-btrblocks/src/schemes/string/fsst.rs:17,86,100` | FSST lengths and code offsets | +| `vortex-btrblocks/src/schemes/string/onpair.rs:13,145` | OnPair children | +| `vortex-btrblocks/src/schemes/string/sparse.rs:13,109` | Sparse indices | +| `vortex-btrblocks/src/schemes/temporal.rs:16,95,105,115` | DateTimeParts days, seconds, subseconds | +| `vortex-compressor/src/builtins/dict/binary.rs:20,121` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/float.rs:22,139` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/integer.rs:22,131` | Dictionary codes | +| `vortex-compressor/src/builtins/dict/string.rs:20,121` | Dictionary codes | +| `vortex-compressor/src/compressor/structural.rs:19,54,87,99` | List offsets, ListView offsets and sizes | +| `vortex-btrblocks/tests/narrow.rs` | New. Only `test_internal_signed_buffers_roundtrip` from `0d94e44a91`, with the core edition and the module `SESSION` | +| `vortex-btrblocks/tests/snapshots/*.snap` | Regenerated | + +The first number in each call-site row is the `PrimitiveArrayExt` import. Remove an import only +when it becomes unused: `vortex-btrblocks/src/schemes/integer/bitpacking.rs:16`, +`float/alp.rs:18` and `float/alprd.rs:16` import the trait for other methods. + +`07fafc2a5b` pins dictionary compression in `test_array_stream_no_double_dict_encode`. `develop` +reworked that test with far-apart values (`vortex-file/src/tests.rs:1599-1623`). Leave +`07fafc2a5b` out unless that test fails with NR6. + +Design notes: + +- Each call site follows this shape. The reason names the open question, so a search finds every + site later: + + ```rust + #[expect( + deprecated, + reason = "TODO(connor): decide whether this child narrows its dtype or becomes a Narrow" + )] + let offsets = NarrowArray::encode_values(offsets, ctx)?; + ``` + +- What "internal buffers keep their signedness" changes in file contents. The old + `PrimitiveArrayExt::narrow` (`vortex-array/src/arrays/primitive/array/mod.rs:157-241`) picks + `U8`, `U16` or `U32` whenever no value is negative (`:189`), whatever the input signedness. It + also returns a zeroed `u8` buffer for an empty or all-null input (`:163-167`). `encode_values` + keeps the signedness: + - A signed child with only non-negative values gets a signed type. It is the same width when the + max fits the signed type. For example, `u16` becomes `i16` for list offsets below 32,768. It is + the next width when the max is between the signed and the unsigned limit. For example, values + 128 to 255 become `i16` instead of `u8`. + - A 1-byte signed child stays `i8` instead of becoming `u8`. + - An empty or all-null signed child becomes `i8` instead of `u8`. + - Unsigned children, children with negative values, and `u64` values above `i64::MAX` do not + change. + - Signed sources in practice: list offsets, ListView offsets and sizes, VarBin offsets, FSST + lengths and code offsets, OnPair children, and the DateTimeParts parts. Unsigned sources do not + change: patch and sparse indices, RLE indices and offsets, dictionary codes and run ends. + - The encoding IDs and metadata formats do not change. The dtype of these children changes in new + files, and an uncompressed child can grow by one width. +- Readers on `develop` accept signed children. List, ListView, VarBin, FSST, DateTimeParts and + OnPair require only `is_int()` for these children + (`vortex-array/src/arrays/list/array.rs:211-215`, `listview/array.rs:239-246`, + `varbin/array.rs:238-239`, `encodings/fsst/src/array.rs:842-848`, + `encodings/datetime-parts/src/array.rs:318-328`, `encodings/onpair/src/array.rs:419-426`). + ListView reinterprets signed offsets and sizes as unsigned before it uses them + (`listview/array.rs:270-280`). Whether earlier releases behave the same is Not verified, so an + acceptance check runs the latest release on the new files. +- The closed #10333 changed five snapshots this way. List offsets went from `u16` to `i16`. FSST + lengths and code offsets went from `u8` to `i8` and from `u32` to `i32`. Their byte counts did + not change. `develop` now writes `fastlanes.bitpacked.v2` and has more snapshot variants, so the + snapshots must be regenerated, not copied. + +Steps: + +1. Create `ct/narrow-encode-values` from `develop` after NR1 merges. +2. Replace every call site in the table, each with the `#[expect(deprecated)]` shape above. Remove + imports that become unused. +3. Remove `PrimitiveArrayExt::narrow` and port its tests. +4. Add `vortex-btrblocks/tests/narrow.rs` with the round-trip test only. +5. Regenerate the golden snapshots with + `INSTA_UPDATE=always cargo test -p vortex-btrblocks --all-features --test golden`. +6. Commit as `feat(compress): keep integer signedness when narrowing internal children`, with the + sign-off line. The PR body must say that `PrimitiveArrayExt::narrow` is removed from the public + API. + +Tests: + +- Port each case of `vortex-array/src/arrays/primitive/array/cast.rs:58-212` to an `rstest` on + `encode_values` in `narrow/tests/encoding.rs`. Expectations change where signedness changes: + - `[0i64, 127]` gives `I8` (was `U8`). + - `[Some(0i32), None, Some(127)]` gives nullable `I8` (was `U8`). + - An empty `i32` or `i64` gives `I8` (was `U8`). + - Unchanged: all-invalid `u32` gives `U8`, `[-1i32, 200]` gives `I16`, `[-100i16, 0, 100]` gives + `I8`, `[0u64, u64::MAX]` stays `U64`, and floats stay unchanged. + - Validity is kept in every case. +- `vortex-btrblocks/tests/narrow.rs::test_internal_signed_buffers_roundtrip`: lists and strings + compress with the compact preset, serialize, decode and match the input, with no `vortex.narrow` + in the array context. +- Regenerated `vortex-btrblocks/tests/snapshots/*.snap`. Only signed internal children change. + +Benchmarks: Expect changes in the btrblocks `compress` and `compress_listview` benchmarks, because +child dtypes change. The closed #10333 reported `take_map` regressions of 22% and 11% in Simulation +mode. The plan to investigate them: + +- `take_map` is in `vortex-array/benches/take_patches.rs:110-129`. It builds `Patches` with `u64` + indices and `u64` values directly (`:131-140`), draws random `u64` take indices, and times + `Patches::take_map` (`vortex-array/src/patches.rs:919-970`). No narrowing code runs. +- Step 1: compare the `take_map` cases on NR1, NR3 and NR4. These PRs add session registrations + and kernels but no narrowing code. A move there points at registry growth or code layout. +- Step 2: compare `take_map` on NR6 with its base. NR6 changes none of the code that `take_map` + runs, so a move there also points at code layout. +- Step 3: for a case that moves, open the CodSpeed flame graph diff. Find the symbols whose cost + changed, for example `hashbrown` inserts and lookups, or allocator calls. +- Step 4: check the `take_map` history on `develop` for variance. `HashMap` is `hashbrown` with + `DefaultHashBuilder` (`vortex-utils/src/aliases/hash_map.rs:5-9`). If that builder seeds per + process, the hash layout differs between runs. Not verified. +- Report the result in the NR6 PR. Do not change `take_map` in NR6. + +Acceptance checks: + +- The commands below pass: + + ```bash + cargo clippy -p vortex-array -p vortex-btrblocks -p vortex-compressor -p vortex-runend \ + -p vortex-fastlanes --all-targets --all-features -- -D warnings + cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-btrblocks -p vortex-compressor \ + -p vortex-runend -p vortex-fastlanes + cargo nextest run -p vortex-array narrow + cargo nextest run -p vortex-btrblocks --all-features + cargo nextest run -p vortex-compressor -p vortex-runend -p vortex-fastlanes + ``` + +- `git grep -n "\.narrow(" -- '*.rs'` finds no call of `PrimitiveArrayExt::narrow`. +- Every call of `encode_values` carries `#[expect(deprecated)]` with the `TODO(connor)` reason. +- Snapshot diffs show only dtype labels of signed internal children, plus byte counts where a + value range needs the next signed width. +- The latest release reads files that this PR writes: + 1. With this PR, run `cargo run -p vortex-compat --release -- generate --output `. + 2. In a separate worktree at the latest release tag, build `vortex-compat` and run its `check` + on ``. Read the modes in `vortex-test/compat-gen/src/check.rs:21-30` first, and use the + one that skips fixtures that only one side has. `compat.py --git-ref` does not build from the + ref (`vortex-test/compat-gen/scripts/compat.py:922-925`), so build the binary by hand. + +Review focus: Every byte-count change in the snapshot diff. Parents that receive a signed child +where they received an unsigned one before. The public API removal. Null payloads in the all-null +case, where the old code wrote zeros and the new code casts. + +Size: Medium. About 20 call sites, ported tests and snapshots. + +Open questions: None. + +## Notes from planning + +1. The claim that "the only integer-only check is the private `validate_dtypes`" holds only for + explicit dtype checks. Opening it for decimals without other changes gives a panic in the + compare kernel for two `Narrow` operands, wrong compare and between results for negative + constants below the child range, and early overflow in decimal `Sum` and `SumV2`. The table + "Integer-only sites" lists every site, and CD4 owns each fix. +2. `Narrow::execute` yields `values.cast(dtype)`. The decimal cast of D15 returns a `Narrow` for a + widening cast, so a decimal `Narrow` would execute to itself. CD4 widens decimals explicitly. + Any later integer cast rule that returns a `Narrow` has the same hazard. +3. The D15 text said that a cast of a `Narrow` passes to the child with the same target. For a + wider integer target, the rewrite returns a new `Narrow` over the child. The values are equal + and the data stays narrow. D15 now says this. +4. The rewrite flattens nested `Narrow` arrays only in `try_new`. Physical rewrites check only the + slot dtype (`vortex-array/src/array/erased.rs:595-620`), so NR1 adds a `VTable::reduce` flatten + rule. +5. The tests of the rewrite call `array_session()`, which `develop` removed. New tests use the + module session, so port their cases, not their setup. +6. The report cites `narrow/mod.rs:81-84` for the width rule. That matches the tip `af4b663059`. + In `f7408481ce` the same lines are `:76-79`. +7. `ct/narrow-encoding` (#10223), `ct/narrow-pushdown` (#10332) and `ct/narrow-compression` + (#10333) still exist on `origin`. The new branch names avoid all three. +8. `Narrow` is registered in `ArraySession::default`, while the in-memory `Filter`, `Slice` and + `Interleave` are not. `PiecewiseSequence` is registered and also has no wire form, so the + rewrite follows a precedent. +9. Part 2 of #10333 wraps every integer primitive that reaches `compress_canonical` in a `Narrow`, + including internal children. It also declares a preview edition with only `vortex.narrow`. WR2 + and WR3 must not reuse either as they are. diff --git a/research/decimal/agents/briefs/03-canonical-decimal.md b/research/decimal/agents/briefs/03-canonical-decimal.md new file mode 100644 index 00000000000..5bf4e09ac1b --- /dev/null +++ b/research/decimal/agents/briefs/03-canonical-decimal.md @@ -0,0 +1,1868 @@ + + +# Canonical decimal: CD1 to CD8c + +These briefs cover issue #10701: one canonical width for each decimal dtype, and decimal +`Narrow` in memory. The strict construction step is split into CD8a, CD8b and CD8c. + +- A `file:line` without a commit points at the `develop` checkout at `b70206818f`. +- A reference with a commit, for example `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73`, + points at the local integer `Narrow` rewrite in `/Users/connor/spiral/vortex-data/vortex3`. The + NR PRs are written from scratch with that rewrite as a reference (D38), so the landed code can + differ. Check each cited site against the NR code on `develop`. +- In the rewrite, commit `f7408481ce` holds the encoding, `validate_dtypes`, the compare kernel and + the cast, fill-null, filter, mask, slice and take rules (NR1). Commit `8d548646ee` holds the + aggregate kernel (NR3). Commit `74aa17644e` holds arithmetic, between, zip and the wider + fill-null and cast rules (NR4). Commit `af4b663059` holds merges and indices (NR5). +- Every commit carries `Signed-off-by: Connor Tsui ` and no AI co-author + line. +- Agents can run any command that the work needs, including the acceptance checks. Use the + narrowest command, as `AGENTS.md` describes. + +Two terms recur. The canonical width of a dtype is `DecimalDType::canonical_values_type()`, which +CD3 adds (D12). The name differs from the stored-width accessor on purpose, so that code cannot mix +the two up. The stored width of an array is `DecimalArray::values_type()`. The two are equal for +every array only after CD8a. + +## Shared helpers + +Each helper has one owner. Later PRs use it and do not add their own. + +| Helper | Owner | Users | Contract | +| --- | --- | --- | --- | +| `DecimalDType::canonical_values_type()` | CD3 | Every later PR | The canonical width of the precision (D12) | +| `copy_to_canonical_width` | CD3 | CD3 imports, CD4 execution, CD5 decoding, CD6 old nodes | Public in `vortex_array::arrays::decimal`. A narrowing copy writes zeros in null slots and fails on a valid value outside the target precision (D13, D21). A widening copy checks nothing | +| `check_precision` | CD3 | The zero-copy imports of CD3, `try_new` in CD8b | Public. Fails on a valid value outside the precision. It takes a `Mask`, so it needs no `ExecutionCtx` | +| The best child dtype for a range | CD4 | `NarrowArray::encode_decimal`, CD6, the `Narrow` scheme of WR3 | Public. From the min and max of the valid values, or no range, and a parent `DecimalDType`, it returns the dtype with the same scale and the smallest precision that holds the range and the scale (D10) | +| `NarrowArray::encode_decimal` | CD4 | WR3 | A `Narrow` over the child at the best precision, or the input when that precision has the canonical width of the input | +| `NarrowArray::encode_values` | NR1 | CD7 (the DecimalByteParts MSP), NR6 | Deprecated (D28). The narrowed child with the same signedness, without a `Narrow` | + +No decimal twin of `encode_values` exists. A caller that needs the child calls `encode_decimal` and +takes the values child of the `Narrow`. + +### CD1: Decimal benchmarks before the in-memory change + +- Goal: Add benchmarks for the decimal code that CD4, CD5 and CD7 change. CodSpeed then has a + base before those PRs land (D19). No library behavior changes. +- Depends on: None. Soft: None. Blocks: CD4, CD5, CD7. +- Decisions: D19 (benchmarks land first), D16 (DecimalByteParts decoding, changed by CD5), D17 + (the DecimalByteParts scheme, changed by CD7), D15 (the widening cast, changed by CD4). +- Branch: `ct/decimal-benchmarks`, based on `develop`. +- Scope: New benchmarks on DecimalByteParts (DBP) arrays with a narrow MSP and on a gap column. + Further benchmarks cover a compressed DBP tree, the cast that grows the canonical width, and the + Arrow and DuckDB exports. One `#[doc(hidden)]` re-export in `vortex-duckdb` lets a bench reach + the exporter. +- Out of scope: Benchmarks on a decimal `Narrow` (CD4, because `Narrow` does not exist on + `develop`). Changes to the CodSpeed shards in `.github/workflows/codspeed.yml` (decided: no CI + change). Changes to existing benchmark inputs, including the non-canonical input in `to_arrow.rs` + (CD8a). + +Existing decimal benchmarks at `b70206818f`: + +| Bench | Decimal cases | In CodSpeed | +| --- | --- | --- | +| `vortex-array/benches/compare.rs:166-171`, `:270-276` | `compare_decimal`: two `i128` arrays at `Decimal(38, 2)`, array against array | Yes, simulation, compute shard | +| `vortex-array/benches/binary_ops.rs:286-331`, `:394-403` | add, mul and div on `i64` at `Decimal(18, 2)` and nullable `i128` at `Decimal(38, 2)` | Yes, compute shard | +| `vortex-array/benches/cast_decimal.rs:96-133` | `in_place_*`: `Decimal(10, 2)` to `Decimal(18, 2)`, both `i64`. `copy_*`: `Decimal(18, 2)` to `Decimal(10, 2)` | Yes, compute shard | +| `vortex-array/benches/cast_primitive.rs:112-160` | integer to `Decimal(38, 2)`, `Decimal(9, 2)` and `Decimal(9, 0)` | Yes, compute shard | +| `vortex-array/benches/filter_fixed_width.rs:144-158` | filter on `i128` at `Decimal(19, 0)` and `i256` at `Decimal(39, 0)` | Yes, selection shard | +| `vortex-array/benches/expr/case_when_bench.rs:67-84` | `case_when_decimal_product` on `i64` at `Decimal(15, 2)` | Yes, compute shard (`expr_case_when`) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:35-101` | assembly kernels for `i128` and `i256`, including an `i8` MSP | Walltime CPU-feature legs only (`#[cpu_features]`) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:102-194` | `dbp_assemble`, `dbp_assemble_narrowed` on arrays | No, `#[cfg(not(codspeed))]` "due to noise" | +| `encodings/decimal-byte-parts/benches/dbp_split.rs` | split kernels, split on arrays | Kernels on CPU-feature legs, arrays not in CodSpeed | +| `vortex-arrow/benches/to_arrow.rs:100-122` | one `Decimal(19, 2)` column inside a struct export | No, `vortex-arrow` is in no shard (`codspeed.yml:52-61`) | + +No DuckDB decimal export bench exists. The only DuckDB bench is +`vortex-duckdb/benches/bool_export.rs`, a bit-unpack microbenchmark that needs no live DuckDB +vector. No benchmark covers sum, min or max on decimals, or decoding of a DBP array with a narrow +MSP in CodSpeed. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/benches/dbp_compute.rs` | New. Decode, compare, filter, sum and min-max on DBP arrays | +| `encodings/decimal-byte-parts/Cargo.toml` | Add `[[bench]] name = "dbp_compute"`, next to the entries at `:36-42` | +| `vortex/benches/common_encoding_tree_throughput.rs` | Add a DBP tree with a bit-packed `i32` MSP to the `decompress` arguments at `:454-466` | +| `vortex-array/benches/cast_decimal.rs` | Add casts that grow the canonical width, `i32` input | +| `vortex-arrow/benches/to_arrow.rs` | Add a `decimal_export` bench with its own cases. Leave `array()` at `:100-122` as it is | +| `vortex-arrow/Cargo.toml` | Add `vortex-decimal-byte-parts` to `[dev-dependencies]` at `:39-46` | +| `vortex-duckdb/src/lib.rs` | Add `#[doc(hidden)] pub mod _benchmarking` that re-exports `ArrayExporter` and `ConversionCache` from the private `exporter` module (`:29`) | +| `vortex-duckdb/benches/decimal_export.rs` | New. Exports decimal columns into a DuckDB `DataChunk` | +| `vortex-duckdb/Cargo.toml` | Add `[[bench]] name = "decimal_export"`, next to `:65-67` | +| `vortex-btrblocks/benches/compress.rs` | Optional. A decimal compress case inside the existing `#[cfg(not(codspeed))]` module | + +Design notes: + +- DBP inputs for `dbp_compute.rs`, built in memory with `DecimalByteParts::try_new` + (`encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:132-137`). One `Display` enum + names the cases for `divan` arguments: + + | Case | Dtype | MSP | Values | Why | + | --- | --- | --- | --- | --- | + | `msp_i32_p18` | `Decimal(18, 2)` | `i32` | seeded, within plus or minus 999,999,999 | The common old compressor shape. In the final state its child is `Decimal(9, 2)` | + | `msp_i32_p18_nullable` | `Decimal(18, 2)`, nullable | `i32` | as above, seeded nulls, zeros in null slots | Validity paths | + | `gap_msp_i16_p10` | `Decimal(10, 2)` | `i16` | 10,000 to 32,767 | The gap column of D19 | + | `msp_i64_p38` | `Decimal(38, 2)` | `i64` | within plus or minus (10^18 - 1) | No lower parts, canonical width `i128` | + | `msp_i64_p18_canonical` | `Decimal(18, 2)` | `i64` | within plus or minus (10^18 - 1) | Control: the MSP has the canonical width | + +- Operations in `dbp_compute.rs`, each a `#[divan::bench(args = ...)]` over the cases: + - `decode`: execute to `DecimalArray`. + - `compare_constant`: `Gte` against a constant that fits the MSP, executed to a `BoolArray`. + - `compare_constant_out_of_range`: `Gte` against a constant above the MSP range, on the + non-nullable cases only. The DBP kernel answers it without reading values only when all rows + are valid (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). + - `filter`: a seeded random mask, then execute to `Canonical`. + - `sum`: `vortex_array::aggregate_fn::fns::sum::sum` + (`vortex-array/src/aggregate_fn/fns/sum/mod.rs:50`). + - `min_max`: `vortex_array::aggregate_fn::fns::min_max::min_max` + (`vortex-array/src/aggregate_fn/fns/min_max/mod.rs:62`). +- These benches run in CodSpeed simulation (shard 7, "FastLanes & decimals"). Do not gate them with + `#[cfg(not(codspeed))]`. Size the arrays so that each case stays near the 1 ms simulation budget + that other array benches state (for example `vortex-array/benches/compare.rs:77-78`). +- The compressed tree in `common_encoding_tree_throughput.rs` mirrors the golden decimal tree + (`vortex-btrblocks/tests/snapshots/golden__regular__decimal_prices.snap`): a `Decimal(12, 2)` DBP + whose `i32` MSP is `fastlanes.bitpacked`. Build it by hand with `BitPacked::encode`, as + `setup::datetime_for_bp` (`:346`) does. A hand-built tree does not move when compressor + heuristics change. +- `cast_decimal.rs`: add `widen_non_nullable` and `widen_nullable`. Each casts an `i32`-backed + `Decimal(9, 2)` array to `Decimal(18, 2)` and executes to `Canonical`. The values stay below + 10^9, as the existing helper at `:66-83` already does. A second helper, or a generic one, builds + the `i32` buffer. +- `to_arrow.rs`: `decimal_export` exports one decimal array to a `Decimal128` field with + `SESSION.arrow().execute_arrow(array, Some(&field), ctx)`, as `string_export` does. Cases: + canonical `i64` at `Decimal(18, 2)`, canonical `i128` at `Decimal(38, 2)`, DBP `msp_i32_p18`, + DBP `gap_msp_i16_p10`, and a nullable variant. +- `decimal_export.rs` (DuckDB): wraps one decimal column in a `StructArray`, builds an + `ArrayExporter`, and exports into `DataChunk::new([LogicalType::decimal_type(p, s)?])` until the + exporter is exhausted. Two cases export without a copy: canonical `i64` at `Decimal(18, 2)` and + canonical `i16` at `Decimal(4, 2)`. Canonical `i8` at `Decimal(2, 1)` copies, because DuckDB + uses `i16`. The DBP cases are `msp_i32_p18` and `gap_msp_i16_p10`. The bench links DuckDB, so it + runs locally only. `ConversionCache` construction is Not verified. +- Optional `compress.rs` case: compress a canonical `Decimal(12, 2)` `i64` array with the default + `BtrBlocksCompressor`. It guards the write path that CD7 changes, outside CodSpeed. + +Steps: + +1. Add `dbp_compute.rs` and its `[[bench]]` entry. +2. Add the compressed DBP tree to `common_encoding_tree_throughput.rs`. +3. Add the widening cases to `cast_decimal.rs`. +4. Add `decimal_export` to `to_arrow.rs` and the dev-dependency. +5. Add the DuckDB `_benchmarking` module, `decimal_export.rs` and its `[[bench]]` entry. +6. Optional: add the decimal compress case. + +Tests: None. Benchmarks only. + +Benchmarks: + +- In this PR, every case is new, so CodSpeed shows additions only. +- After CD5, `decode`, `filter`, `sum` and `min_max` in `dbp_compute.rs` get slower for the cases + with a narrow MSP. Decoding then adds a widening copy, which D19 accepts. Compare cases stay the + same, because they run on the MSP. `msp_i64_p18_canonical` stays the same. The compressed tree + decompress gets slower by one widening copy. +- After CD7, these in-memory inputs do not change. The golden snapshot and the optional compress + case cover the scheme. +- After CD4, `widen_*` in `cast_decimal.rs` stays similar or gets faster, because the execution of + the returned `Narrow` widens without a per-value check. +- The Arrow and DuckDB export benches have no CodSpeed base. Run them locally with walltime before + and after CD5 and CD7. + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings`, and the + same for `vortex`, `vortex-array`, `vortex-arrow`, `vortex-duckdb` and `vortex-btrblocks`. +- `cargo +nightly-2026-09-10 fmt -p ` for each crate above. +- `cargo bench -p vortex-decimal-byte-parts --bench dbp_compute` runs every case. +- `cargo bench -p vortex --bench common_encoding_tree_throughput`, + `cargo bench -p vortex-array --bench cast_decimal`, `cargo bench -p vortex-arrow --bench + to_arrow` and `cargo bench -p vortex-duckdb --bench decimal_export` run. +- Every new bench input is a canonical decimal leaf, or a DBP array with an MSP that D16 allows. + No new input builds a `DecimalArray` at a non-canonical width, so CD8a does not break them. + +Review focus: The new inputs must stay legal through CD8a (canonical leaves and DBP MSPs only). The +CodSpeed benches must stay near the simulation budget. The DuckDB `_benchmarking` module must +expose only what the bench needs. + +Size: medium. Several bench files and one re-export, no library code. + +Open questions: None. The export benches run locally, because `vortex-arrow` and `vortex-duckdb` +have no CodSpeed shard (`codspeed.yml:52-61`), and the maintainer chose no CI change. If CodSpeed +shows noise on the new DBP benches, move them behind `#[cfg(not(codspeed))]`, as +`dbp_assemble.rs:102-104` does. + +### CD2: Compat fixtures for old decimal forms + +- Goal: Add compat fixtures for every old decimal form that the current writer can make. The + harness then compares old files against canonical expected arrays. A release before CD5 and CD6 + publishes them (D24, D31). +- Depends on: None. Soft: None. Blocks: CD5 and CD6 (the release gate of D31), CD8a (removes the + legacy overrides). +- Decisions: D24, D9 and D21 (the reads that the fixtures cover), D16 (DBP with a narrow MSP). +- Branch: `ct/decimal-compat-fixtures`, based on `develop`. +- Scope: A harness hook that separates the written form of a fixture from its expected array. Seven + new fixtures. Documentation of the hook in `DESIGN.md`. +- Out of scope: Error cases, such as a valid value outside the precision in a narrowing copy (unit + tests in CD3 and CD6, D24). Reader changes (CD5, CD6). The v2 DBP fixture, which stays disabled + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`). + +How the check works today: + +- `check` calls `fixture.write(tmp_dir, ctx)` for every fixture + (`vortex-test/compat-gen/src/check.rs:69-71`). For a flat fixture, `write` calls `build()` with + the current code and writes the result with the current flat writer + (`vortex-test/compat-gen/src/fixtures/mod.rs:90-100`). +- It then reads the published file and the fresh file with the current reader + (`check.rs:174-193`), and compares them with `assert_arrays_eq!` (`check.rs:195`). +- `assert_arrays_eq!` requires equal dtypes and lengths, and compares logical values of valid rows + only (`vortex-array/src/arrays/assertions.rs:84-113`, + `vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs:55-93`). Mixed stored widths compare + correctly (`all_non_distinct/decimal.rs:16-42`). So a canonical expected array matches an old + narrow file with the same values. +- So yes, the check rebuilds the expected array with the current code. A `build()` that makes a + narrow decimal leaf fails at runtime after CD8a. The `?` at `check.rs:70` then aborts the whole + check, not one fixture. The unit test `roundtrip_fixtures_to_bytes` also calls `build()` + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs:32-42`). +- `--git-ref` only sets the version. The code builds fixtures from the current tree + (`vortex-test/compat-gen/scripts/compat.py:922-925`, `:948-951`). The README says otherwise + (finding 1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-test/compat-gen/src/fixtures/mod.rs` | Add `FlatLayoutFixture::build_written` (trait at `:38-57`) and `Fixture::write_expected` (trait at `:26-35`). `FlatLayoutAdapter` (`:79-101`) writes `build_written` in `write` and `build` in `write_expected` | +| `vortex-test/compat-gen/src/check.rs` | `:70` calls `write_expected` instead of `write` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs` | The round-trip test (`:32-42`) writes `build_written` and compares the read-back array with `build`. Add the stored-width test | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal_widths.rs` | New. `decimal_canonical_widths.vortex` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal_legacy.rs` | New. A shared column spec and the five legacy-form fixtures | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/mod.rs` | Register the new fixtures in `fixtures()` (`:25-40`) | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_narrow_msp.rs` | New. `decimal_byte_parts_narrow_msp.vortex` | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs` | Register it in `fixtures()` (`:38-63`) | +| `vortex-test/compat-gen/DESIGN.md` | Document `build_written` in "Fixture Trait" and "The oracle question" | + +Design notes: + +- Harness hook, signatures only: + + ```rust + pub trait FlatLayoutFixture { + /// Build the arrays that `generate` writes. Defaults to [`Self::build`]. + /// + /// Override this only to write an old physical form that `build` cannot return. The + /// result has the dtype and the logical values of [`Self::build`]. + fn build_written(&self, ctx: &mut ExecutionCtx) -> VortexResult + } + + pub trait Fixture { + /// Write the file that `check` compares with a published file. Defaults to [`Self::write`]. + fn write_expected( + &self, + dir: &Path, + ctx: &mut ExecutionCtx, + ) -> VortexResult> + } + ``` + +- `generate` and `publish` write `build_written`. `check` writes `build`. So the published file + holds the old form, and the check compares it with a canonical array that never changes. This + keeps the contract of `DESIGN.md` that `build()` never changes its output. +- `check_expected_encodings` runs on the written form. `DatasetFixtureAdapter` keeps the default + `write_expected`. +- The legacy fixtures share one column spec: name, `DecimalDType`, stored `DecimalType`, the raw + value of every slot as `i128`, and an optional validity. `written()` builds + `DecimalArray::try_new::` at the stored width with the raw slot values. `expected()` builds + the canonical width with zeros in null slots. The canonical width comes from + `DecimalType::smallest_decimal_value_type` until CD3 migrates that call (CD3 hand-off). +- Survival through CD3 to CD8a: `build()` builds only canonical arrays, so `check` never builds an + illegal array. `build_written` uses `DecimalArray::try_new` and returns its error. After CD8a that + error stops `generate` and the round-trip test, which forces CD8a to delete the overrides. The DBP + fixture and the canonical-width fixture need no override, because D16 keeps a narrow MSP legal. +- After CD8a, later releases publish canonical files under the same names. The old forms stay in the + store under the versions published before CD8a. +- Fixtures. Each column holds the listed boundary values. Columns in one fixture have equal length. + The last column gives the result that CD5 and CD6 must produce, assuming that CD6 picks the best + precision of D9. + + `decimal_canonical_widths.vortex`, no override: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p2_s1` | `i8` | `Decimal(2, 1)` | plus or minus 99 | Zero-copy leaf | + | `i16_p4_s2` | `i16` | `Decimal(4, 2)` | plus or minus 9,999 | Zero-copy leaf | + | `i128_p19_s2` | `i128` | `Decimal(19, 2)` | plus or minus (10^19 - 1) | Zero-copy leaf | + | `i128_p38_s2` | `i128` | `Decimal(38, 2)` | plus or minus (10^38 - 1) | Zero-copy leaf | + | `i256_p39_s2` | `i256` | `Decimal(39, 2)` | plus or minus (10^39 - 1) | Zero-copy leaf | + | `i256_p76_s0` | `i256` | `Decimal(76, 0)` | plus or minus (10^76 - 1) | Zero-copy leaf | + | `i128_p38_s2_nullable` | `i128` | `Decimal(38, 2)`, nullable | with nulls | Zero-copy leaf | + + `decimal_narrow_storage.vortex`, written narrower than canonical: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p10_s2` | `i8` | `Decimal(10, 2)` | plus or minus 99 | `Narrow` over `Decimal(2, 2)`, buffer reused | + | `i16_p18_s2` | `i16` | `Decimal(18, 2)` | plus or minus 9,999 | `Narrow` over `Decimal(4, 2)`, reused | + | `i32_p18_s2` | `i32` | `Decimal(18, 2)` | plus or minus 999,999,999 | `Narrow` over `Decimal(9, 2)`, reused | + | `i64_p30_s2` | `i64` | `Decimal(30, 2)` | plus or minus (10^18 - 1) | `Narrow` over `Decimal(18, 2)`, reused | + | `i128_p40_s0` | `i128` | `Decimal(40, 0)` | plus or minus (10^38 - 1) | `Narrow` over `Decimal(38, 0)`, reused. This is the old v1 compressor form (`vortex-btrblocks/src/schemes/decimal.rs:133-137`) | + | `i8_p76_s0` | `i8` | `Decimal(76, 0)` | plus or minus 99 | `Narrow` over `Decimal(2, 0)`, reused | + + `decimal_gap_values.vortex`, values that fit the width but need the next precision: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p10_s2_gap` | `i8` | `Decimal(10, 2)` | 100 to 127, and -128 | `Narrow` over `Decimal(3, 2)` at `i16`, copy | + | `i8_p4_s2_gap` | `i8` | `Decimal(4, 2)` | up to 127 | Canonical leaf at `i16`, copy (the example in `02-design.md`) | + | `i16_p10_s2_gap` | `i16` | `Decimal(10, 2)` | 10,000 to 32,767 | `Narrow` over `Decimal(5, 2)` at `i32`, copy | + | `i32_p18_s2_gap` | `i32` | `Decimal(18, 2)` | 1,000,000,000 to `i32::MAX` | Canonical leaf at `i64`, copy | + | `i64_p38_s0_gap` | `i64` | `Decimal(38, 0)` | 10^18 to `i64::MAX` | Canonical leaf at `i128`, copy | + + `decimal_high_scale.vortex`, where the scale limits the child precision: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i8_p18_s10` | `i8` | `Decimal(18, 10)` | plus or minus 99 | Canonical leaf at `i64`, copy, because the child needs precision 10 | + | `i8_p18_s5` | `i8` | `Decimal(18, 5)` | plus or minus 99 | `Narrow` over `Decimal(5, 5)` at `i32`, copy | + | `i16_p38_s20` | `i16` | `Decimal(38, 20)` | plus or minus 9,999 | Canonical leaf at `i128`, copy | + | `i8_p4_s4` | `i8` | `Decimal(4, 4)` | plus or minus 99 | Canonical leaf at `i16`, copy | + + `decimal_wide_storage.vortex`, Arrow-style widths wider than canonical: + + | Column | Stored | Dtype | Values | Read result | + | --- | --- | --- | --- | --- | + | `i128_p5_s2` | `i128` | `Decimal(5, 2)` | plus or minus 99,999 | Canonical leaf at `i32`, narrowing copy | + | `i128_p18_s2` | `i128` | `Decimal(18, 2)` | plus or minus (10^18 - 1) | Canonical leaf at `i64` | + | `i64_p4_s2` | `i64` | `Decimal(4, 2)` | plus or minus 9,999 | Canonical leaf at `i16` | + | `i32_p2_s1` | `i32` | `Decimal(2, 1)` | plus or minus 99 | Canonical leaf at `i8` | + | `i256_p38_s2` | `i256` | `Decimal(38, 2)` | plus or minus (10^38 - 1) | Canonical leaf at `i128` | + + `decimal_null_slots.vortex`, nonzero values in null slots: + + | Column | Stored | Dtype | Null-slot values | Read result | + | --- | --- | --- | --- | --- | + | `canonical_i32_p9_s2` | `i32` | `Decimal(9, 2)` | `i32::MAX` | Zero-copy leaf. No override, legal under D14 | + | `narrow_i8_p10_s2` | `i8` | `Decimal(10, 2)` | 127 and -128, valid values within plus or minus 99 | `Narrow` over `Decimal(2, 2)`, reused, because the max ignores null slots | + | `narrow_i8_p4_s2` | `i8` | `Decimal(4, 2)` | 127, valid values within plus or minus 99 | `Narrow` over `Decimal(2, 2)`, reused. Without null slots ignored, this is a gap | + | `wide_i128_p5_s2` | `i128` | `Decimal(5, 2)` | 10^30 | Canonical leaf at `i32`, zeros in null slots, no error | + + `decimal_byte_parts_narrow_msp.vortex`, DBP v1, no override. The existing DBP fixture uses only + `i64` MSPs (`.../encodings/decimal_byte_parts.rs:35-108`): + + | Column | MSP | Dtype | Values | Decoded after CD5 | + | --- | --- | --- | --- | --- | + | `msp_i8_p10_s2` | `i8` | `Decimal(10, 2)` | full `i8` range | `i64` leaf | + | `msp_i16_p10_s2_gap` | `i16` | `Decimal(10, 2)` | 10,000 to 32,767 | `i64` leaf | + | `msp_i32_p18_s2` | `i32` | `Decimal(18, 2)` | plus or minus 999,999,999 | `i64` leaf | + | `msp_i32_p18_s2_gap` | `i32` | `Decimal(18, 2)` | 1,000,000,000 to `i32::MAX` | `i64` leaf | + | `msp_i64_p38_s2` | `i64` | `Decimal(38, 2)` | plus or minus (10^18 - 1) | `i128` leaf | + | `msp_i32_p18_s2_nullable` | `i32`, nullable | `Decimal(18, 2)` | nonzero null slots | `i64` leaf | + | `msp_i64_p4_s2_wide` | `i64` | `Decimal(4, 2)` | plus or minus 9,999 | `i16` leaf, narrowing copy (D16, D21) | + +- The last DBP column, an MSP wider than the canonical width, is beyond the list of D24. It is a + read that succeeds under D16, so the brief includes it. +- Publication. `publish.yml:119-127` calls `compat-gen-upload.yml` with the release tag on every + published release. That job checks out the tag and runs + `uv run vortex-test/compat-gen/scripts/compat.py publish --yes --git-ref `, which uploads + every fixture to `v/arrays/`. So: + 1. Merge CD2. + 2. Cut a release from a `develop` commit that has CD2. CD3 can be in it too: CD3 changes no form + that the CD2 fixtures write, because `build_written` builds its arrays directly and not + through an import. CD5 and CD6 must not be in it. + 3. Confirm that the upload job passed, and that + `uv run vortex-test/compat-gen/scripts/compat.py list --version ` lists the seven + new names. + 4. Merge CD5 and CD6 only after that. The library PRs CD3 and CD4 do not wait (D31). +- Fallback if CD5 or CD6 cannot wait for a release: check out the latest release tag, apply CD2, and + run `compat.py publish --update`. It uploads new fixtures only, and its hash check proves that the + other fixtures did not change (`compat.py:472-545`). Whether CD2 applies cleanly to that tag is + Not verified. +- Every PR runs `compat.py check --mode last` (`publish-dry-runs.yml:110-114`). In the PR of CD2, + the new names are missing from the store, so `superset` mode skips them (`check.rs:117-134`). + +Steps: + +1. Add `build_written` and `write_expected`, switch `check` to `write_expected`, and update the + round-trip test and `DESIGN.md`. +2. Add `decimal_canonical_widths.vortex`. +3. Add the column spec and the five legacy-form fixtures, with the stored-width test. +4. Add `decimal_byte_parts_narrow_msp.vortex`. +5. Generate and check locally. + +Tests: + +- `roundtrip_fixtures_to_bytes` (`synthetic/mod.rs:32-42`): for each fixture, check that + `build_written` and `build` have equal dtypes and logical values in memory. Then write + `build_written`, read it back, and compare it with `build`. +- New `legacy_decimal_fixtures_keep_stored_widths` in the same module: every decimal leaf of each + `build_written` has the stored width of its column spec. Without this test, a fixture can write + canonical data and test nothing. +- The read results in the tables are expectations for CD5 and CD6. Those PRs test them. + +Benchmarks: None. + +Acceptance checks: + +- `cargo nextest run -p vortex-compat` +- `cargo clippy -p vortex-compat --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-compat` +- `cargo run -p vortex-compat --release -- generate --output `, then + `cargo run -p vortex-compat --release -- check --dir --mode exact`. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` passes, with the new names + skipped. +- After the release, `compat.py list --version ` shows the seven new names. + +Review focus: `build()` builds only canonical arrays. Each written form holds the intended stored +width and values. Null-slot values in the written forms are nonzero where the table says so. +`check` uses `write_expected` and `generate` uses `write`. + +Size: medium. One small harness change and seven data fixtures. + +Open questions: None. The next regular release publishes the fixtures (D31). The DBP fixture +keeps `msp_i64_p4_s2_wide`, because it is a read that succeeds. + +### CD3: One width owner, and imports copy into the canonical width + +- Goal: `DecimalDType::canonical_values_type()` becomes the one owner of the width of each precision + (D12). The Arrow and DuckDB imports copy into that width, with zeros in null slots and an error on + a valid value outside the precision (D13). One shared helper performs every such copy (D21). +- Depends on: None. Soft: CD1, CD2. Blocks: CD4, CD5, CD6, CD8a. +- Decisions: D12, D13, D21 (the helper), D30 (imports check their own values), D3 (as far as + imports go). +- Branch: `ct/decimal-width-owner`, based on `develop`. +- Scope: Add `DecimalDType::canonical_values_type()`. Remove + `DecimalType::smallest_decimal_value_type` and migrate all its calls. Add the shared helpers + `copy_to_canonical_width` and `check_precision`. Change the Arrow and DuckDB imports, so that + every import checks its values (D30). +- Out of scope: Builders, including `DecimalBuilder::extend` and `append_decimal_array` + (`vortex-array/src/builders/decimal.rs:298-306`, `:160-178`), and a strict `try_new` (CD8a, CD8b). + DBP decoding (CD5). The read plugin (CD6). `narrowed_decimal` and the compressor (CD7). The + zero-copy widening cast (CD4). Exports. + +Count of `smallest_decimal_value_type` at `b70206818f`: 26 matches, one of them the definition +(`vortex-array/src/dtype/decimal/types.rs:46`), so 25 calls. This matches the notes at `5fc5cb8`. +No call exists outside Rust code. If CD2 or another PR adds calls before CD3, CD3 migrates them +too. Run `rg -n smallest_decimal_value_type` at the base of CD3. + +| # | Call | Use | New code | +| --- | --- | --- | --- | +| 1 | `vortex-array/src/dtype/decimal/types.rs:73` | `is_compatible_decimal_value_type` | `self >= dtype.canonical_values_type()` | +| 2 | `vortex-array/src/dtype/dtype_impl.rs:320` | element byte width | `decimal.canonical_values_type().byte_width()` | +| 3 | `vortex-array/src/array/haystack/mod.rs:119` | IN-list width | `decimal.canonical_values_type()` | +| 4 | `vortex-array/src/builders/mod.rs:416` | builder factory width | `decimal_type.canonical_values_type()` | +| 5 | `vortex-array/src/scalar/arbitrary.rs:164` | arbitrary scalar | `decimal_type.canonical_values_type()` | +| 6 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186` | `DecimalValue::zero` | `decimal_type.canonical_values_type()` | +| 7 | `vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209` | `DecimalValue::normalize` | `decimal_type.canonical_values_type()` | +| 8 | `vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112` | scalar work width | `work.canonical_values_type()` | +| 9 | `vortex-array/src/arrays/constant/vtable/canonical.rs:97` | constant canonicalization | `decimal_type.canonical_values_type()` | +| 10 | `vortex-array/src/arrays/constant/vtable/canonical.rs:356` | constant canonicalization | `decimal.canonical_values_type()` | +| 11 | `vortex-array/src/arrays/decimal/compute/cast.rs:155` | cast target width | `to_decimal_dtype.canonical_values_type()` | +| 12 | `vortex-array/src/arrays/arbitrary.rs:146` | arbitrary array | `decimal.canonical_values_type()` | +| 13 | `vortex-array/src/arrays/primitive/compute/cast.rs:144` | integer to decimal cast | `decimal_dtype.canonical_values_type()` | +| 14 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98` | arithmetic work width | `work_dtype.canonical_values_type()` | +| 15 | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416` | arithmetic result width | `decimal_dtype.canonical_values_type()` | +| 16 | `vortex-array/src/canonical.rs:188` | empty canonical array | `decimal_type.canonical_values_type()` | +| 17 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | size statistic | `array.decimal_dtype().canonical_values_type()` | +| 18 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365` | size statistic | `decimal_type.canonical_values_type()` | +| 19 | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103` | size statistic | `decimal.canonical_values_type()` | +| 20 | `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49` | sum accumulation width | `dtype.canonical_values_type()` | +| 21 | `vortex-cuda/src/kernel/arrays/constant.rs:73` | CUDA constant | `decimal_dtype.canonical_values_type()` | +| 22 | `fuzz/src/array/sum/mod.rs:54` | fuzz oracle | `output_dtype.canonical_values_type()` | +| 23 | `vortex-row/src/codec.rs:583` | row key width in `decimal_key_type` | `dt.canonical_values_type()`, keep the function and its doc | +| 24 | `encodings/sparse/src/canonical.rs:316` | sparse canonicalization | `decimal_dtype.canonical_values_type()` | +| 25 | `vortex-duckdb/src/convert/vector.rs:314` | DuckDB import narrowing | Removed. The import uses the helper | + +Producers and imports that create a width other than the canonical width today: + +| Producer | Location | Width today | Owner | +| --- | --- | --- | --- | +| Arrow import | `vortex-arrow/src/convert.rs:179-224` | The Arrow width, zero-copy. Wider than canonical for `Decimal32` at precision 1 to 4, `Decimal64` at 1 to 9, `Decimal128` at 1 to 18, `Decimal256` at 1 to 38 | CD3 | +| Python, C and C++ FFI, JNI, DataFusion, Parquet | `vortex-python/src/arrays/from_arrow.rs:40`, `:59`, `vortex-ffi/src/array.rs:510`, `vortex-jni/src/writer.rs:148`, `vortex-datafusion/src/lib.rs:209`, `vortex-datafusion/src/convert/scalars.rs:385` | All route through `ArrowSession::from_arrow_array`, which ends in `from_arrow_dyn` (`vortex-arrow/src/session.rs`, leaf fallback) | CD3, through the Arrow import. The Parquet route through arrow-rs is Not verified for every reader | +| DuckDB import | `vortex-duckdb/src/convert/vector.rs:303-338` | Canonical for every precision. Precision 1 to 2 narrows DuckDB `i16` to `i8` with a lossy `as_()` cast (`:312-322`, PR #10155) | CD3 | +| `DecimalBuilder::extend` and `append_decimal_array` | `vortex-array/src/builders/decimal.rs:298-306`, `:160-178` | Builder width from `T`. Accepts wider source buffers and panics on a null slot that does not fit | CD8a | +| User constructors | `vortex-array/src/arrays/decimal/array.rs:406-534` | Any `T` | CD8a | +| DBP decoding | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-93` | The MSP width | CD5 | +| Old `vortex.decimal` nodes | `vortex-array/src/arrays/decimal/vtable/mod.rs` | The stored width | CD6 | +| Compressor | `vortex-btrblocks/src/schemes/decimal.rs:133` (`narrowed_decimal`) | Narrower | CD7 | +| Zero-copy widening cast | `vortex-array/src/arrays/decimal/compute/cast.rs:122-143` | The source width | CD4 | +| Kernels that keep the input width | for example `vortex-array/src/arrays/decimal/compute/fill_null.rs:55-72` | The input width, or wider | CD8a | + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/dtype/decimal/mod.rs` | Add `DecimalDType::canonical_values_type()` in the impl block (`:43-110`). Tests in the module test block | +| `vortex-array/src/dtype/decimal/types.rs` | Remove `smallest_decimal_value_type` (`:45-57`). Update `is_compatible_decimal_value_type` (`:71-74`) | +| The 23 other files of the table above | Mechanical migration | +| `vortex-array/src/arrays/decimal/utils.rs` | Add `copy_to_canonical_width` and `check_precision`, with tests | +| `vortex-arrow/src/convert.rs` | The four `from_arrow_decimal*` functions (`:179-224`) copy when the Arrow width differs. Update the tests at `:1011-1110` | +| `vortex-duckdb/src/convert/vector.rs` | The decimal arm (`:303-338`) uses the helper. Add tests to the module at `:436` | + +Design notes: + +- The width owner, as `02-design.md:29-43` proposes: + + ```rust + impl DecimalDType { + /// The canonical width of the values buffer for this precision. + pub const fn canonical_values_type(&self) -> DecimalType { + match self.precision.get() { + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, + } + } + } + ``` + + Every `DecimalDType` built through `try_new` has a precision from 1 to 76 + (`vortex-array/src/dtype/decimal/mod.rs:48-77`). The serde derive at `:36` bypasses `try_new` + (finding 9). +- The shared helper, in `vortex-array/src/arrays/decimal/utils.rs` and exported as + `vortex_array::arrays::decimal::copy_to_canonical_width` through `pub use utils::*` + (`vortex-array/src/arrays/decimal/mod.rs:22-23`): + + ```rust + /// Copies unscaled decimal values into a new buffer at the canonical width of `target`. + pub fn copy_to_canonical_width( + values: &[F], + validity: &Mask, + target: DecimalDType, + ) -> VortexResult + ``` + + - The result holds `values.len()` values of `target.canonical_values_type()`. + - It returns an error when `validity.len()` differs from `values.len()`. + - Narrowing copy, when `target.canonical_values_type()` is narrower than `F`: every valid value + must be within `F::MIN_BY_PRECISION[p]` and `F::MAX_BY_PRECISION[p]` for the precision `p` of + `target` (`vortex-array/src/dtype/decimal/types.rs:102-108`). Otherwise the copy fails, and the + error names the row, the value and `target`. Null slots of the result hold zeros. + - Widening copy, when `target.canonical_values_type()` is wider than `F`: each value is widened, + and no value is checked. The caller must know that every valid value fits `target`. This holds + when `target` has at least the precision of the source dtype, or when the caller chose `target` + from the values (CD6). + - Same width: a plain copy without a check. A caller that owns a `Buffer` reuses it instead. + - The helper never rescales. The caller keeps the scale. + - Use the lane kernels for the checked loop (`vortex_compute::lane_kernels::IndexedSourceExt`, as + `vortex-array/src/arrays/decimal/compute/cast.rs:217-241` does), then write zeros in the null + rows. Follow the hidden-cost rules in `STYLE.md`: no per-row `Mask::value(i)`. + - `converted_buffer` (`utils.rs:40-89`) stays. It checks the width, not the precision, and keeps + null-slot values that fit. Its callers are `vortex-array/src/array/haystack/mod.rs:123`, `:216` + and `vortex-row/src/codec.rs:958`. +- Users of the helper: CD3 (Arrow and DuckDB imports), CD4 (`Narrow` execution and the decimal + encoder), CD5 (DBP decoding in both directions), CD6 (old nodes). CD8c can use it in + `append_decimal_array`. +- Arrow import. One private function in `vortex-arrow/src/convert.rs` serves all four + `from_arrow_decimal*` functions: + - It builds the dtype with `DecimalDType::try_new`, not `new` (`:184`, `:195`, `:206`, `:217`). + An invalid precision or scale from FFI then returns an error, not a panic. + - It builds a `Mask` from the Arrow `NullBuffer`, which is all valid when absent. + - When the Arrow native width equals `decimal_dtype.canonical_values_type()`, it keeps the + zero-copy buffer, calls `check_precision`, and builds the array with the unchecked constructor. + This covers `Decimal32` at precision 5 to 9, `Decimal64` at 10 to 18, `Decimal128` at 19 to 38, + and `Decimal256` at 39 to 76. + - Otherwise it calls `copy_to_canonical_width` and builds the array with the unchecked + constructor. `Decimal256` keeps the transmute of `:219-222` before the copy. + - The error adds context with `.with_context()`: the import rejects values outside the precision, + and arrow-rs `PrimitiveArray::null_if_overflow_precision` turns them into nulls before the + import (`arrow-array-59.3.1/src/array/primitive_array.rs:1672`). + - Doc comments change from "Zero-copy conversion" to "zero-copy when the Arrow width is the + canonical width, otherwise a checked copy". +- Options hook (D13). No import options exist. `ArrowExportOptions` + (`vortex-arrow/src/options/mod.rs:22-51`) covers exports only, and the import API takes no options + (`ArrowSession::from_arrow_array`, `from_arrow_dyn`). The smallest scope is the default of D13: + an error, with the arrow-rs pre-processing named in the error. An import options type is future + work. +- DuckDB import. In the decimal arm, each DuckDB width calls `copy_to_canonical_width` on the vector + slice with the mask of `:307`. Precision 1 to 2 then becomes a checked narrowing copy from `i16` + into `i8`, with zeros in null slots. The other widths are equal widths, so the call is a plain + copy, as `Buffer::copy_from` is today, and the import then calls `check_precision`. The `i128` + path keeps `Buffer::from_iter(vector_i128_values(vector, len))` (`:331-334`), which already builds + a new buffer at the canonical width. +- Every import checks its values in CD3, so D13 holds for every import from CD3 on. A zero-copy + import now scans every valid value once. The imports use the unchecked constructors after their + own check, so the `ExecutionCtx` that CD8b adds to `try_new` never reaches the import API (D30). +- `check_precision(values: &[T], validity: &Mask, decimal_dtype: DecimalDType) -> VortexResult<()>` + runs a branchless fold first, and it finds the failing row only on failure, as + `vortex-array/src/arrays/decimal/utils.rs:50-80` does. The error names the row, the value and the + dtype. + +Steps: + +1. Add `DecimalDType::canonical_values_type()` with tests. +2. Migrate the 25 calls, remove `smallest_decimal_value_type`, and update + `is_compatible_decimal_value_type`. +3. Add `copy_to_canonical_width` with tests. +4. Change the Arrow import and its tests. +5. Change the DuckDB import and add its tests. + +Tests: + +- `vortex-array/src/dtype/decimal/mod.rs` tests: `test_values_type_boundaries`, an `rstest` over + the precisions 1, 2, 3, 4, 5, 9, 10, 18, 19, 38, 39 and 76. Each case expects the width of D12. +- `vortex-array/src/arrays/decimal/utils.rs`, a new `tests` module: + - `test_copy_to_canonical_width_widens`: `rstest` over the widening pairs, values unchanged. + - `test_copy_to_canonical_width_narrowing_boundaries`: `rstest` over the narrowing pairs. + Plus or minus (10^p - 1) passes, and plus or minus 10^p fails. + - `test_copy_to_canonical_width_zeros_null_slots`: null slots hold values outside the target + width and precision, the copy succeeds, and the null slots are zero. + - `test_copy_to_canonical_width_error_names_row`: the error names the first failing valid row. + - `test_copy_to_canonical_width_all_null`, and `..._length_mismatch` (an error). + - `test_check_precision_boundaries`: plus or minus (10^p - 1) passes, and plus or minus 10^p + fails, for each width. + - `test_check_precision_ignores_null_slots`: a value outside the precision in a null slot + passes. +- `vortex-arrow/src/convert.rs` tests: + - `test_decimal_import_canonical_width`: `rstest` over `Decimal32(4, 2)`, `Decimal64(9, 2)`, + `Decimal128(5, 2)`, `Decimal128(18, 2)`, `Decimal128(38, 2)`, `Decimal256(38, 2)` and + `Decimal256(40, 2)`. The result has `values_type() == decimal_dtype.canonical_values_type()` and + equal values. The equal-width cases keep the Arrow buffer pointer. + - `test_decimal_import_zeros_null_slots`: an Arrow `Decimal128(5, 2)` array with 10^30 in a null + slot imports, and the null slot holds zero. + - `test_decimal_import_rejects_value_outside_precision`: `Decimal128(5, 2)` with a valid + 1,000,000 fails. + - `test_decimal_import_zero_copy_rejects_value_outside_precision`: `Decimal128(20, 2)` with a + valid 10^20 fails. This is the zero-copy path. + - `test_decimal_import_rejects_invalid_precision`: an invalid precision or scale returns an error. + - The existing `test_decimal128_array_conversion` and `test_decimal256_array_conversion` + (`:1013`, `:1063`) build their expected arrays at `i128` for precision 10 and `i256` for + precision 38. Change the expected arrays to the canonical widths (`i64` and `i128`), and assert + the width. D12 fixes those widths, so this change does not follow the code under test. +- `vortex-duckdb/src/convert/vector.rs` tests, in the style of `test_vector_with_nulls` (`:676`): + - `test_decimal_vector_conversion`: `rstest` over the precisions 1, 2, 3, 4, 9, 18 and 38. The + width is canonical and the values are equal. + - `test_decimal_vector_null_slot_garbage`: precision 2 with 300 in a null slot imports, and the + slot holds zero. + - `test_decimal_vector_value_outside_precision`: precision 2 with a valid 150 fails. + - `test_decimal_vector_equal_width_outside_precision`: precision 4 with a valid 10,000 fails. + +Benchmarks: None. The Arrow import of `Decimal128` at precision 18 or below now copies, which D13 +accepts. `vortex-arrow` is in no CodSpeed shard. + +Acceptance checks: + +- `cargo nextest run -p vortex-array`, `-p vortex-arrow`, `-p vortex-duckdb`, `-p vortex-row` and + `-p vortex-sparse`. +- `cargo test --doc -p vortex-array` +- `cargo clippy -p --all-targets --all-features -- -D warnings` for `vortex-array`, + `vortex-arrow`, `vortex-duckdb`, `vortex-row`, `vortex-sparse` and `vortex-fuzz`. For + `vortex-cuda`, run it where the CUDA toolchain exists. Whether `vortex-cuda` builds without it is + Not verified. +- `cargo +nightly-2026-09-10 fmt -p ` for each crate above. +- `rg -n smallest_decimal_value_type` returns nothing. +- Every Arrow and DuckDB decimal import returns an array at the canonical width of its dtype. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` still passes, including the + CD2 fixtures and the TPC-H and ClickBench fixtures, whose decimals come through the Arrow import. + +Review focus: The helper checks precision, not width, on a narrowing copy. Null slots never cause an +error. The equal-width Arrow import stays zero-copy. Every import checks its values, then uses the +unchecked constructors. The migration is +mechanical, and every call names the right dtype (the work dtype in arithmetic, the target dtype in +casts). The changed Arrow test expectations follow D12. + +Size: medium. Twenty-five mechanical sites, one helper, and two imports with tests. + +Open questions: None. No import options exist, and the maintainer chose none for now. The import +fails on an invalid value, and the error names `null_if_overflow_precision`. Zero-copy imports check +their values in CD3 (D30). + +### CD4: Decimal `Narrow` in memory + +- Goal: The `Narrow` encoding accepts decimal children under D10, and its kernels and the decimal + cast follow D15. Compare, sum, min and max run on the narrower child, with the result of the + parent. +- Depends on: NR1, NR3, NR4, CD3, CD1. Soft: NR2 (bench file layout), NR5 (its structural rules + also hold for decimals). Blocks: CD6, CD8a, WR1, WR3. +- Decisions: D10, D15, D4, D8 (one encoding for integers and decimals, in memory only), D19, D25, + D26, D36. +- Branch: `ct/decimal-narrow`, based on `develop` after NR1, NR3, NR4, CD1 and CD3 merge. +- Scope: Open `validate_dtypes` for decimals, and add decimal execution and a decimal encoder. Add + decimal branches to the compare, aggregate, cast and fill-null kernels. Change the decimal cast + kernel for D15. Add tests and benchmarks. +- Out of scope: The `vortex.narrow` wire form (WR1). The compressor scheme (WR3). The read plugin + (CD6). A strict `try_new` (CD8a). Arithmetic, zip and merges on decimal `Narrow`, which decline + and fall back. A cast rule that wraps a DBP or other encoded decimal in a `Narrow` without + decoding. Pushdown into the MSP (LT3). Exports keep their own widening code. Replacing it with a + `Narrow` execution (D13) is not part of CD4. + +Places in the rewrite that assume an integer child. CD4 handles each one: + +| # | Place | Assumption | CD4 action | +| --- | --- | --- | --- | +| 1 | `af4b663059:vortex-array/src/arrays/narrow/mod.rs:73-88` `validate_dtypes` | `PType::try_from`, `is_int`, signedness and byte width | Add the decimal rule of D10 | +| 2 | `af4b663059:.../narrow/mod.rs:4-10`, `:35-54` | Docs say integer only | Update the docs | +| 3 | `af4b663059:.../narrow/vtable.rs:111-116` `execute` | Yields `values.cast(dtype)` | Widen explicitly for decimals, or the D15 cast loops (design notes) | +| 4 | `af4b663059:.../narrow/vtable.rs:130-139` `scalar_at` | None, a scalar cast | Test only | +| 5 | `af4b663059:.../narrow/encoding.rs:23-63` `NarrowArray::encode` | Takes a `PrimitiveArray` | Add `NarrowArray::encode_decimal` | +| 6 | `af4b663059:.../narrow/compare.rs:52-70` | Narrow against Narrow uses `as_ptype()`, which panics for decimals | Decimal branch at the wider child precision | +| 7 | `af4b663059:.../narrow/compare.rs:88-95` | Out-of-range sign uses `is_signed_int()` and `i64::try_from` | Decimal sign from the `DecimalValue` | +| 8 | `af4b663059:.../narrow/compare.rs:111-122` `compare_values` | Fast path only for `Primitive` | Also take it for `Decimal` leaves, if `execute_compare` accepts them (Not verified) | +| 9 | `af4b663059:.../narrow/rules.rs:111-144` `CastReduce` | `PType::try_from`, declines decimals | D15: pass the cast to the child | +| 10 | `af4b663059:.../narrow/rules.rs:146-194` `FillNullReduce`, `integer_storage_type` | `PType::try_from`, declines decimals | Decimal branch: fill the child when the fill value fits its precision, otherwise decline | +| 11 | `af4b663059:.../narrow/rules.rs:240-279` `zip_values` | `PType::try_from`, declines | Keep the decline. Test the fallback | +| 12 | `af4b663059:.../narrow/rules.rs:316-346` `BetweenReduce` | None, a scalar cast and `two_compares` | Test only. Relies on 6 and 7 | +| 13 | `af4b663059:.../narrow/rules.rs:281-314` `TakeIndicesReduce` | Dictionary codes are integers | None. A decimal cannot be a code | +| 14 | `8d548646ee:vortex-array/src/arrays/narrow/aggregates.rs:83-89` | The child partial, cast to the parent partial dtype, assumes the same result dtype | NR3 builds every accumulator from the parent dtype (D36). CD4 adds the decimal tests | +| 15 | `af4b663059:.../narrow/numeric.rs:69-72` | `PType::try_from`, declines | Keep the decline. Test the fallback | + +The rows cite the tip `af4b663059`, which also holds NR5 code. CD4 does not need NR5, so on its base +some of these lines are elsewhere, and `TakeIndicesReduce` (row 13) exists only after NR5. The +rewrite file `narrow/merge.rs` never lands: NR5 drops its chunked and interleave merge rules (D25). +So CD4 has no merge code to change. + +Files. The `narrow` module comes from NR1 to NR4, so its lines are not on `b70206818f`: + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/mod.rs` | `validate_dtypes` for decimals, docs | +| `vortex-array/src/arrays/narrow/vtable.rs` | Decimal `execute` | +| `vortex-array/src/arrays/narrow/encoding.rs` | `NarrowArray::encode_decimal` and the precision helper | +| `vortex-array/src/arrays/narrow/compare.rs` | Decimal branches | +| `vortex-array/src/arrays/narrow/rules.rs` | Decimal `CastReduce` and `FillNullReduce` | +| `vortex-array/src/arrays/narrow/aggregates.rs` | No change expected: NR3 builds every accumulator from the parent dtype (D36). Change it only if a decimal accumulator rejects the child batch | +| `vortex-array/src/arrays/decimal/compute/cast.rs` | The D15 rule in `CastReduce` (`:45-82`) and `CastKernel` (`:84-170`) | +| `vortex-array/src/arrays/narrow/tests/decimal.rs` | New tests, registered in `narrow/tests/mod.rs` | +| The `Narrow` bench file of NR2, or `vortex-array/benches/` | Decimal `Narrow` benches | + +Design notes: + +- `validate_dtypes(child, parent)` under D10: + - Two primitive dtypes follow the integer rule of D22, unchanged. + - Two decimal dtypes need the same scale, the same nullability, and + `child.canonical_values_type() < parent.canonical_values_type()`. `DecimalType` orders by width + (`vortex-array/src/dtype/decimal/types.rs:25-42`). + - Any other pair fails, with an error that names both dtypes. + - A strictly narrower canonical width implies a lower precision. `DecimalDType::try_new` already + rejects a positive scale above the precision (`vortex-array/src/dtype/decimal/mod.rs:68-74`), so + a child precision is always at least the scale. No extra check is needed. + - `NarrowArray::try_new` flattens a `Narrow` over a `Narrow` + (`af4b663059:.../narrow/mod.rs:55-61`), which gives one `Narrow` (D15). The check of the inner + child holds by transitivity. + - `try_new` trusts the child dtype. It reads no values. +- Execution. The integer path yields a cast to the parent dtype. For decimals that cast returns a + `Narrow` again under the new rule below, so execution never ends. The decimal path therefore + executes the child to a `DecimalArray`, computes its validity mask, and calls + `copy_to_canonical_width` (CD3) with the parent `DecimalDType`. The result keeps the validity of + the child. The call dispatches on the stored width of the child (`DecimalArray::values_type()`), + so a child at a non-canonical width before CD8a is still correct. A child stored at the parent + width reuses its buffer. A child stored wider than the parent width gets a checked narrowing + copy. +- Encoder, the decimal twin of `NarrowArray::encode`: + + ```rust + impl NarrowArray { + /// Stores a decimal array at the best precision that its valid values and its scale allow. + /// + /// Returns the input unchanged when that precision has the canonical width of the input + /// dtype. + pub fn encode_decimal(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult + } + ``` + + - It takes the min and max of the valid values with `min_max`. + - A public function returns the best child dtype (see Shared helpers): the smallest precision + `p`, at least 1 and at least a positive scale, with `MIN_BY_PRECISION[p] <= min` and + `max <= MAX_BY_PRECISION[p]`. An empty or all-null array gets that lower bound. + - When the child width is narrower than the parent width, it copies with + `copy_to_canonical_width` and wraps the child in a `Narrow`. + - This is the producer rule of D10. CD6 and WR3 reuse the function, so that one function owns + the child label. A false label gives wrong answers in the compare shortcut and in casts. +- Compare with a constant (the constant has the parent dtype, `af4b663059:.../compare.rs:48-50`): + - If the constant casts to the child dtype, compare the child with it. The decimal scalar cast + checks the precision (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:158-182`, + `:208-217`). + - Otherwise the constant is outside the range of the child precision. The answer needs no values: + `Eq` is false, `NotEq` is true. For `Lt` and `Lte` the answer is true when the constant is + positive. For `Gt` and `Gte` it is true when the constant is negative. + - The sign comes from the decimal value of the constant, for example `as_i256() < i256::ZERO`. + - The result keeps the validity of the child, cast to the result nullability, as the integer path + does (`af4b663059:.../compare.rs:96-107`). +- Compare two decimal `Narrow` arrays: cast both children to the higher child precision at the + common scale, then compare. The narrower child becomes a `Narrow` or a relabeled leaf under the + cast rule. The comparison then falls back to its canonical form at the child width. +- Sum (decided, D36). The result dtype of a decimal sum depends on the input precision + (`sum_decimal_dtype`, `vortex-array/src/aggregate_fn/fns/sum/mod.rs:81-86`). An accumulator for + the child dtype uses `sum_decimal_dtype(child)`, which overflows where the parent result does not. + A `Narrow` must never fail where its parent would not (D36). So: + - NR3 already builds every accumulator from the dtype of the parent, and it feeds that + accumulator the child batch. `Sum` and `SumV2` need no `Narrow` code, and CD4 adds only tests. + - `accumulate_decimal` reads any input width into the width of the result dtype + (`vortex-array/src/aggregate_fn/fns/sum/decimal.rs:27-61`). `multiply_constant` handles a + constant child of any precision (`vortex-array/src/aggregate_fn/fns/sum/constant.rs:69-89`). + So the sum never widens the child buffer. + - First, confirm that the accumulator accepts a batch whose decimal dtype differs from its input + dtype. Not verified. If it does not, stop and report, because the fix then touches the shared + accumulator code. +- Min, max, first and last: the NR3 path stands. The child partial casts to the parent partial + dtype, so each value becomes a value of the parent dtype. `IsSorted`, `IsConstant` and the counts + need no change. Whether the struct partial of `MinMax` casts field by field is Not verified, so + CD4 tests it. +- Casts (D15): + - A cast of a decimal `Narrow` passes to the child with the same target dtype: decimal, integer + or float. When the target equals the dtype of the array, the rule returns the array. + - Decimal leaf, same scale, a higher precision, and a canonical width that grows: return + `NarrowArray::try_new(input, target)`. The input first takes the target nullability. In + `CastReduce` this uses `trivially_cast_nullability`, and the rule declines when that returns + `None`. In `CastKernel` it uses `cast_nullability` with the context. + - Same scale, a higher precision, and the same canonical width: change only the dtype, as the + zero-copy path at `cast.rs:122-143` does. + - A lower precision checks every value, as today. A scale change rescales, as today. + - Before CD8a, an input can have a stored width other than its canonical width. The rule compares + the canonical widths of the two dtypes. For such an input with an equal canonical width, the + kernel keeps its current behavior. So only the case where the canonical width grows changes. + - The existing tests `cast_upcast_precision_succeeds` (`cast.rs:969-996`) and + `cast_widening_same_physical_type_is_zero_copy` (`:999-1029`) still pass. The first now goes + through a `Narrow` and its execution. +- Fill null: when the fill value casts to the child dtype, fill the child and wrap the result. + Otherwise decline. Canonical execution then handles a fill value that needs a wider precision. +- Filter, take, slice and mask need no decimal code (D15). Their rules wrap the child result with + `wrap_values` (`af4b663059:.../narrow/rules.rs:73-79`), which now validates decimal pairs. + +Steps: + +1. Open `validate_dtypes` for decimals and update the docs. +2. Add decimal execution and `encode_decimal`. +3. Add the decimal compare branches. +4. Confirm with tests that decimal sums through the NR3 kernel use the parent result dtype (D36). +5. Add the D15 cast rules on `Narrow` and on the decimal leaf, and the fill-null branch. +6. Add the conformance and fallback tests. +7. Add the benchmarks. + +Tests, in `vortex-array/src/arrays/narrow/tests/decimal.rs`, as functions that return +`VortexResult<()>`: + +- `test_try_new_decimal_dtypes`: an `rstest` over pairs of child and parent dtypes. + - Accepted: `Decimal(9, 2)` under `Decimal(18, 2)`, and `Decimal(38, 0)` under `Decimal(76, 0)`. + - Rejected for the same width: `Decimal(10, 2)` under `Decimal(18, 2)`, and `Decimal(75, 0)` + under `Decimal(76, 0)`. + - Rejected for other reasons: another scale, another nullability, or a higher child precision. + - Rejected for mixed types: an integer child under a decimal parent, and the reverse. +- `test_narrow_over_narrow_flattens`: one `Narrow` remains, with the inner child. +- `test_decimal_execute_widens`: nullable and non-nullable, values and validity equal the canonical + array, and the width is the canonical width of the parent. Include a child at a non-canonical + stored width. +- `test_decimal_execute_terminates`: a decimal `Narrow` executes to `Canonical`. This is the + regression test for the cast loop. +- `test_encode_decimal_best_precision`: an `rstest` over values and scales. Cases include a + positive scale that raises the precision, negative values, all nulls and an empty array. One case + needs the canonical width and returns no `Narrow`. +- `test_decimal_compare_constant`: an `rstest` over all six operators. The constant is inside the + child range, above it, below it, or null. One case uses a nullable child. Each result equals the + comparison on the canonical array with `assert_arrays_eq!`. +- `test_decimal_compare_two_narrow`: two children with different precisions. +- `test_decimal_between`: bounds inside and outside the child range. +- `test_decimal_sum`: `Sum` and `SumV2` equal the canonical results, with the dtype + `sum_decimal_dtype(parent)`. Include nulls and all nulls. +- `test_decimal_sum_uses_parent_precision`: a `Narrow` of `Decimal(18, 0)` over a `ConstantArray` + of `Decimal(2, 0)` with the value 99. Its length pushes the child sum past precision 12, the + precision of `sum_decimal_dtype(Decimal(2, 0))`. The sum is the exact product, not null. The + constant child is never materialized. +- `test_decimal_min_max_first_last`: results are values of the parent dtype and equal the canonical + results. +- `test_decimal_cast_rules`: an `rstest` over the leaf rules. + - The canonical width grows: the result is a `Narrow` before execution. + - The width stays: the result reuses the buffer pointer. + - The precision drops: values are checked, and a value that does not fit fails. + - The nullability changes, with and without nulls. +- `test_decimal_narrow_cast_passes_to_child`: decimal, integer and `f64` targets equal the casts of + the canonical array. +- `test_decimal_fill_null`: a fill value that fits the child, and one that needs a wider precision. +- `test_decimal_fallbacks`: arithmetic, zip, chunked and interleave on decimal `Narrow` arrays + equal the canonical results. +- `test_decimal_structural_pull_up`: the NR5 rules move `Dict`, `Constant` and `Chunked` above a + decimal `Narrow`, with equal values. +- `test_decimal_narrow_serializes_as_canonical`: a decimal `Narrow` serializes as a canonical + `vortex.decimal` node at the parent width (D26). +- Conformance: run `test_array_consistency`, `test_cast_conformance`, `test_filter_conformance`, + `test_take_conformance` and `test_mask_conformance` (`vortex-array/src/compute/conformance/`) on + nullable and non-nullable decimal `Narrow` arrays. + +Benchmarks (CodSpeed compute shard). Follow the file layout of NR2 (hand-off). Each case has a +canonical baseline in the same file: + +| Case | `Narrow` input | Baseline | +| --- | --- | --- | +| `p18_child_p9` | `encode_decimal` of `Decimal(18, 2)` values within plus or minus 999,999,999, child `Decimal(9, 2)` at `i32` | The canonical `i64` array | +| `gap_p10_child_p5` | `Decimal(10, 2)` values from 10,000 to 32,767, child `Decimal(5, 2)` at `i32` | The canonical `i64` array | +| `p38_child_p18` | `Decimal(38, 2)` values within plus or minus (10^18 - 1), child `Decimal(18, 2)` at `i64` | The canonical `i128` array | + +Operations: + +- `compare_constant`, with a constant inside the child range. +- `compare_constant_out_of_range`. +- `sum` and `min_max`. +- `filter`, with a seeded mask, then execute. +- `execute`, which widens to canonical. +- `cast_widen`, to `Decimal(38, 2)` or to `Decimal(76, 2)` for the `p38` case, then execute. + +Expected in CodSpeed: `compare_constant`, `sum`, `min_max` and `filter` are equal to or faster than +the baseline, because the child is narrower. `compare_constant_out_of_range` is much faster, +because it reads no values. `execute` costs one widening copy. Compare it with the DBP `decode` of +CD1 when the D19 cost comes up. The CD1 case `widen_*` in `cast_decimal.rs` stays similar or gets +faster. + +Acceptance checks: + +- `cargo nextest run -p vortex-array`, `-p vortex-decimal-byte-parts` and `-p vortex-arrow`. +- `cargo test --doc -p vortex-array` +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- Every decimal `Narrow` kernel result equals the result on the canonical array. +- `NarrowArray::try_new` rejects every pair that D10 forbids. +- A cast of a decimal leaf to a precision with the same scale and a wider canonical width returns a + `Narrow` and reads no values. +- `uv run vortex-test/compat-gen/scripts/compat.py check --mode last` passes. + +Review focus: The child label (D10) comes only from a dtype or from `encode_decimal`. Execution +never goes through the cast rule. The sum overflows only against the precision of the parent. The +compare shortcut gets the sign right for negative constants and keeps nulls. Casts that wrap set the +nullability first. Inputs at a non-canonical stored width before CD8a still give correct results. +`DecimalArray::values_type()` (stored) and `DecimalDType::canonical_values_type()` (canonical) have +the same name. + +Size: large. One encoding opens for a second type, with four kernels, the cast kernel and broad +tests. + +Open questions: None. Whether a cast of a DBP array, or of another encoded decimal, to a wider +canonical width wraps it in a `Narrow` without decoding is left to the design pass of WR3. + +### CD5: DecimalByteParts decodes to the canonical width + +- Goal: Decoding DecimalByteParts produces the canonical decimal array of its dtype, and every lower + part has the dtype `u64`. Today decoding reuses the MSP buffer at the MSP width, a form that D3 + removes. +- Depends on: CD1, CD3, and the published CD2 fixtures (D31). Soft: FX2. Blocks: CD8a. +- Shares files with CD7 (see the CD7 brief). Do not run CD5 and CD7 at the same time. +- Decisions: D16, D21, D29, D3, D12, D14. +- Branch: `ct/dbp-canonical-decode`, based on `develop` after CD3 merges, or stacked on CD3. +- Scope: + - Decoding to the canonical width in `assemble_decimal` and in the CUDA decoder. + - The `u64` rule for lower parts, and the bound on the lower-part count (D29), in the + constructor, the v2 reader and the assembler. + - The module and type docs, and the tests and the bench that build narrow lower parts. +- Out of scope: + - The max pass that returns a `Narrow` (LT2). This PR adds only its `TODO(connor)`. + - Sum, min and max pushdown into the MSP (LT3). + - The compressor scheme (CD7). + - The nullability cast fix in `compute/cast.rs` (FX2). + - Strict `DecimalArray` construction (CD8a). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:16` | The doc line that allows narrower parts changes. The MSP is a signed integer of at most 64 bits, and every lower part is `u64` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:62-63` | `LOWER_PART_DTYPE` doc: the dtype of every lower part, not "before any narrowing" | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:57-58` | Type doc: only the MSP can use a narrower dtype | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:105-118` | `DecimalBytePartsData::validate` requires `part.dtype() == &LOWER_PART_DTYPE`, and a lower-part count whose assembled width is at most the canonical width (D29) | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:139-149`, `:167-173` | Constructor docs and the `new_unchecked` safety text name `u64` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:372` | Comment on "narrowed lower value" goes | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:32-160` | Canonical-width assembly (design notes), the same `u64` check as the constructor, the `TODO(connor)`, and removal of the lower-part cast and its `TODO(mk)` at `:107-116` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs:92-104` | The v2 reader requires `PType::U64` for every lower part | +| `encodings/decimal-byte-parts/src/lib.rs:7-14` | Crate doc: narrowing applies to the MSP only | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs`, `assemble.rs`, `plugin/tests.rs` | Tests (see Tests) | +| `encodings/decimal-byte-parts/benches/dbp_assemble.rs:158-179` | `dbp_assemble_narrowed` uses `u64` lower parts | +| `vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:43-118` | Canonical width on the GPU, and tests | +| `vortex-cuda/src/arrow/canonical.rs:507` | Move `decimal_cast` to a crate-private module that the export and the DecimalByteParts decoder share | +| `vortex-cuda/kernels/src/decimal_cast.cu:46-102` | Add an `int16_t` output and the `decimal_cast__i16` kernels | + +Design notes: + +`assemble_decimal(msp, lower_parts, decimal_dtype, ctx)` keeps its signature. It returns a decimal +array whose width is `T = decimal_dtype.canonical_values_type()`. The assembled width `A` is the MSP +width with no lower parts, `i128` with one lower part, and `i256` with two or three. + +| Lower parts | Case | Result | +| --- | --- | --- | +| 0 | `A == T` | Reuse the MSP buffer. No copy and no scan | +| 0 | `A < T` | Widening copy into `T`. This branch holds the `TODO(connor)` | +| 0 | `A > T` | `copy_to_canonical_width` into `T` (only for precision 1 to 9, because the MSP is at most `i64`) | +| 1 | `T == i128` | Assemble into `i128`, as today | +| 1 | `T == i256` | Assemble straight into `i256` with `assemble_wide_decimal::`, which sign-extends the MSP | +| 2 or 3 | `T == i256` | Assemble into `i256`, as today | + +- The constructor rejects a lower-part count whose assembled width is wider than the canonical + width (D29). So precision up to 18 allows no lower parts, precision 19 to 38 allows at most one, + and precision 39 to 76 allows any count. Assembly with lower parts then never needs a narrowing + copy. No file breaks, because no edition permits v2. +- A path without a copy trusts the precision (D14). A narrowing copy checks it (D21). A widening + copy from a narrower width cannot leave the precision. +- Build every result with `DecimalArray::new_unchecked`. The SAFETY comment says that the width is + the canonical width and that reads trust the precision. So CD8b adds no scan to decoding. +- The `TODO(connor)` at the widening branch reads: + + ```rust + // TODO(connor): Find the max of a narrow MSP and return a `Narrow` over a child that reuses + // its buffer, once benchmarks show that this widening copy costs too much. + ``` + +- The `u64` rule: the constructor, `assemble_decimal` and the v2 reader reject `u8`, `u16` and `u32` + lower parts. The error text names `u64`. No file breaks: no edition permits v2, and the v2 compat + fixture is disabled + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`). +- `scalar_at` (`array.rs:360-401`) keeps returning `DecimalValue::I64` with no lower parts. A scalar + accepts any width that fits its precision (`vortex-array/src/scalar/validate.rs:58-60`). +- CUDA: `decimal_byte_parts.rs:49-65` reuses the MSP buffer at the MSP width today. Keep the error + for lower parts at `:45-47`. With `A == T`, reuse the buffer. With `A < T`, widen on the device + with `decimal_cast::`. Its kernel has outputs `i32`, `i64`, `i128` and `i256` + (`decimal_cast.cu:80-102`), so precision 3 to 4 with an `i8` MSP needs the new `i16` output. With + `A > T`, return an error, as for lower parts. A device narrowing copy needs an overflow flag that + synchronizes with the host (`vortex-cuda/src/arrow/canonical.rs:433-437`). +- Keep the CUDA change minimal and in its own commit, and ask for a review from someone who + knows the CUDA code. If it grows beyond the device widening and the new `i16` output, move it + to its own PR. + +Steps: + +1. Require `u64` lower parts in the constructor, the assembler and the v2 reader. Fix the docs. + Remove the lower-part cast. Update the tests and the bench that build narrow lower parts. +2. Make `assemble_decimal` return the canonical width, as the table shows, with the `TODO(connor)`. + Add the CPU tests. +3. In a separate commit, make the CUDA decoder return the canonical width. Add the `i16` kernel + output and the GPU tests. + +Tests: + +- New in `assemble.rs`: + - `test_assemble_returns_canonical_width` (`rstest`): MSP ptype and precision `(I8, 2)` reuse, + `(I8, 4)`, `(I32, 18)`, `(I64, 38)`, `(I16, 76)` widen, and `(I64, 9)`, `(I32, 2)` narrow, each + with and without nulls. It asserts `values_type()` against + `decimal_dtype.canonical_values_type()` and the values. + - `test_assemble_reuses_msp_buffer`: pointer equality when the MSP width is the canonical width. + - `test_assemble_narrowing_copy_zeroes_null_slots`: an `i64` MSP at `Decimal(2, 0)` with 1000 in a + null slot gives 0 in that slot. + - `test_assemble_narrowing_copy_rejects_value_outside_precision`: an `i64` MSP at `Decimal(2, 0)` + with the valid value 100 fails. 100 fits `i8` but not precision 2. + - `test_assemble_widening_sign_extends`: an `i8` MSP with -1 at precision 40 gives `i256` -1. + - `test_assemble_lower_parts_canonical_width` (`rstest`) with these cases: + - One lower part at precision 40 gives `i256`. + - One lower part at precision 38 gives `i128`. + - Three lower parts at precision 76 give `i256`. +- New in `array.rs`: cases `u8_lower_part`, `u16_lower_part` and `u32_lower_part` in + `test_rejects_invalid_parts` (`:661-679`). Also `one_lower_part_at_p18` and + `three_lower_parts_at_p38`, which break the bound of D29. +- New in `plugin/tests.rs`: case `v2_narrow_lower_part` with `PType::U32` in + `decoder_rejects_malformed_payloads` (`:151-207`). +- Changed: + - `array.rs:579-651` `test_independently_narrowed_parts` becomes + `test_narrow_msp_with_u64_lower_parts`. It keeps every MSP ptype and lower-part count, with the + high bit set in each word, and drops the narrow lower ptypes. + - `plugin/tests.rs:66-85` cases `narrowed_lower_parts` and `nullable_mixed_lower_parts` keep their + narrow MSPs and use `u64` lower parts. + - `plugin/tests.rs:46-60`: case `wider_i64_storage` builds an `i64` MSP with no lower parts at + `Decimal(2, 0)`, so it covers the narrowing decode without a non-canonical `DecimalArray`. Cases + `wider_i128_storage` and `wider_i256_storage` become rejection cases, because one or three + lower parts at precision 2 break the bound of D29. + - `plugin/tests.rs:61-65` builds three lower parts at precision 38. It moves to precision 76. + - `plugin/tests.rs:182-187` case `v2_signed_lower_part` expects the new message. + - `assemble.rs:369-386` `test_split_narrow_decimal_reuses_values` uses `i32` at `Decimal(9, 0)` + and asserts reuse on split and on decode. + - `assemble.rs:388-409` `test_assemble_rejects_invalid_lower_dtype` gets a `u32` case. + - CUDA `decimal_byte_parts.rs:90-118` asserts the canonical width. New cases: `i8` at precision 3 + (the new `i16` kernel), `i64` at precision 40, `i32` at precision 9 (reuse), and `i64` at + precision 4 (error). +- No existing test asserts a narrow decoded width. The `values_type()` asserts at `array.rs:703`, + `:707`, `:716`, `:729`, `assemble.rs:266` and `prop_tests.rs:144` expect canonical widths. Outside + the crate, a search for `values_type()` asserts found no decoded DecimalByteParts width. Not + verified for tests that read buffers with `buffer::()` after a decode. +- `prop_tests.rs` needs no change, because `draw_encoded` draws `u64` lower parts (`:113-119`). + +Benchmarks: CD1's DecimalByteParts decoding with a narrow MSP gets slower, because a widening copy +replaces the reuse. Compute on decoded DecimalByteParts runs at the canonical width, so it gets +slower for columns with a narrow MSP. D19 accepts both until the compressor narrows the decimal type +first (WR3). An Arrow export of such a column copies twice: once to decode and once to widen for +Arrow. An MSP at the canonical width and arrays with lower parts show no change. The +`dbp_assemble_kernel*` benches call slice kernels and do not change. `dbp_assemble_narrowed` is not +in CodSpeed (`dbp_assemble.rs:105`). + +Acceptance checks: + +- `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings` +- `cargo nextest run -p vortex-decimal-byte-parts` and + `cargo test --doc -p vortex-decimal-byte-parts` +- `cargo +nightly-2026-09-10 fmt -p vortex-decimal-byte-parts -p vortex-cuda` +- `cargo nextest run -p vortex-btrblocks --all-features` and `cargo nextest run -p vortex-file`, + whose round trips decode DecimalByteParts. +- On a CUDA host: `cargo clippy -p vortex-cuda --all-targets --all-features -- -D warnings`, + `cargo nextest run -p vortex-cuda`, and + `clang-format --style=file -i vortex-cuda/kernels/src/decimal_cast.cu`. +- `cargo run -p vortex-compat --release -- check --dir --mode exact`, where + `` holds the files that the release with CD2 published, or files generated at a + commit before CD5. Files generated with the CD5 code only test the new code against itself. +- Decoding gives `values_type() == decimal_dtype.canonical_values_type()` for every MSP width and + part count. +- The reuse path copies nothing. A narrowing copy writes zeros in null slots and fails on a valid + value outside the precision. The constructor and the v2 reader reject a lower part that is not + `u64`. + +Review focus: + +- Sign extension from an `i8` MSP into `i256`, and one lower part at precision 39 or more. +- Null slots in the narrowing copy, and no scan on the reuse path. +- The place of the `TODO(connor)`, and the CUDA kernel names (`decimal_cast_i8_i16`). + +Size: medium. One crate changes, plus a small CUDA change, with many test updates. + +Open questions: None. The constructor bounds the lower-part count (D29). On the GPU, an MSP wider +than the canonical width returns an error, because the btrblocks scheme never writes one. + +### CD6: The `vortex.decimal` read plugin converts old nodes + +- Goal: An old `vortex.decimal` node has a stored width that differs from the canonical width of its + dtype. The reader converts it into the canonical array or a decimal `Narrow`. After this PR no + read produces a decimal leaf with a non-canonical width. +- Depends on: FX1, CD3, CD4, and the published CD2 fixtures (D31). Soft: CD5. Blocks: CD8a. +- Decisions: D7, D9, D21, D14, D10, D24, D31, D37. +- Branch: `ct/decimal-read-plugin`, based on `develop` after FX1, CD3 and CD4 merge, or stacked on + CD4. +- Scope: + - A new `DecimalPlugin` for `vortex.decimal` with the canonical, wider-node and narrower-node + paths. + - Its `is_supported_encoding` override and its registration. + - A `Decimal` VTable `deserialize` that rejects a non-canonical width. + - Unit tests with hand-built old nodes. +- Out of scope: + - Stored statistics in plugins (rejected, D9). + - DecimalByteParts (CD5) and strict construction (CD8a). + - The `vortex.narrow` wire format and a serializer for `Narrow` (WR1). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:151-186` | Move the body of `deserialize` into a module-private function that returns the stored width, the buffer and the validity. `Decimal::deserialize` accepts only the canonical width, and otherwise fails with an error that names `DecimalPlugin` | +| `vortex-array/src/arrays/decimal/vtable/plugin/mod.rs` (new) | `DecimalPlugin` | +| `vortex-array/src/arrays/decimal/vtable/plugin/tests.rs` (new) | Tests | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:31-33` | `mod plugin` and `pub use plugin::DecimalPlugin` | +| `vortex-array/src/arrays/decimal/mod.rs:13-16` | `pub use vtable::DecimalPlugin` | +| `vortex-array/src/session/mod.rs:24`, `:92` | Register `DecimalPlugin` in place of `Decimal` | + +Design notes: + +```rust +/// Reads every `vortex.decimal` wire form into a canonical decimal array or a decimal `Narrow`. +pub struct DecimalPlugin + +impl ArrayPlugin for DecimalPlugin { + fn id(&self) -> ArrayId // `vortex.decimal` + fn serialize(&self, array: &ArrayRef, session: &VortexSession) + -> VortexResult> // delegates to `Decimal` + fn deserialize(&self, parts: ArrayDeserialization<'_>, session: &VortexSession) + -> VortexResult + fn is_supported_encoding(&self, id: &ArrayId) -> bool // `Decimal` or `Narrow` +} +``` + +The ALP plugin is the model (`encodings/alp/src/alp/plugin.rs:33-94`). The deserializer asserts +`is_supported_encoding` (`vortex-array/src/serde.rs:412-417`), and so does the Python CUDA path +(`vortex-python-cuda/src/lib.rs:328-333`). Statistics attach after the plugin returns +(`serde.rs:419-424`), so the plugin scans the buffer (D9). + +`deserialize` with the stored width `W_s` and the canonical width +`W_c = decimal_dtype.canonical_values_type()`: + +1. Decode the metadata with FX1. Check that the buffer has `len * W_s.byte_width()` bytes and the + alignment of `W_s`. A malformed node returns an error and never panics (D14). +2. If `W_s == W_c`, return today's leaf. No copy and no scan. +3. Otherwise, copy a device buffer to the host with `BufferHandle::try_to_host_sync` + (`vortex-array/src/buffer.rs:295-312`). CUDA reads decode device segments through plugins + (`vortex-cuda/src/layout.rs:241-247`). Then execute the validity mask once with + `session.create_execution_ctx()`, as the ALP plugin does (`plugin.rs:84`). +4. If `W_s > W_c` (a wider node), `copy_to_canonical_width` into `W_c` and return the canonical + array. This path needs no max scan. +5. If `W_s < W_c` (a narrower node): + 1. Make one pass over the valid values for their min and max. Use a fold for an all-true mask and + `for_each_set_index` otherwise (`STYLE.md:108-133`). Do not use the `min_max` aggregate: it + builds decimal scalars and panics on a value outside the precision (see CD7). + 2. Pass the range, or no range when no value is valid, to the best-child-dtype function of CD4 + (see Shared helpers). It returns `Decimal(p*, s)`, with the smallest precision that holds the + range and the scale. + 3. `p*` never exceeds `p`, because the largest magnitude of a width below `W_c` is below + `10^p - 1`. + 4. Let `W* = canonical_values_type(p*)`. If `W* == W_c`, return the canonical array, a widening + copy from `W_s`. + 5. Otherwise build the child `Decimal(p*, s)` at `W*`. Reuse the buffer if `W* == W_s`, make a + widening copy if `W* > W_s`, and `copy_to_canonical_width` if `W* < W_s`. Return + `NarrowArray::try_new(child, parent_dtype)`. +6. Build every decimal array with the unchecked constructors. Reuse trusts the precision (D14). A + narrowing copy checks it, and a widening copy from a narrower width cannot leave it. + +The brief picks the best precision exactly, not a larger one. D9 allows a larger one. But D21 makes +a narrowing copy when the best precision has a smaller width than the stored width. A tight child +label also serves the compare shortcut of CD4. + +| Case | Result | +| --- | --- | +| All values null | `p*` comes from the scale alone. A child at `canonical_values_type(p*)`, a copy when that width differs from `W_s` | +| Empty array | The same rule, with an empty child | +| Negative values | The min counts: `i8` -128 needs precision 3, so the child is `i16` | +| Scale 0 or negative | No lower bound from the scale | +| `Decimal(18, 10)` at `i8` with values up to 99 | `p* = 10`, `W* = i64 = W_c`, so the canonical array | +| `Decimal(4, 2)` at `i8` with values up to 127 | `p* = 3`, `W* = i16 = W_c`, so the canonical array at `i16` | +| `Decimal(18, 2)` at `i16` with values up to 9,999 | `p* = 4`, child `Decimal(4, 2)` at `i16`, buffer reused | +| `Decimal(18, 2)` at `i32` with values up to 99 | `p* = 2`, child `Decimal(2, 2)` at `i8`, narrowing copy | +| `Decimal(18, 2)` at `i16` with 32,767 | `p* = 5`, child `Decimal(5, 2)` at `i32`, widening copy | +| `Decimal(40, 2)` at `i128` (old v1 compressor output) | A child at `i128` with reuse, or the canonical `i256` array when the values need 39 digits | +| `Decimal(4, 2)` at `i32` with the valid value 20,000 | A wider node. The narrowing copy fails | +| Nonzero values in null slots | The scan ignores them. Reuse and widening keep them, and a narrowing copy writes zeros | +| Unknown stored width | An error (FX1) | + +Steps: + +1. Add `DecimalPlugin` with the canonical path, the `is_supported_encoding` override and the + registration. Make `Decimal::deserialize` reject a non-canonical width. +2. Add the wider-node path. +3. Add the narrower-node path. +4. Add the tests. + +Tests: + +- Change `legacy_decimal_fixtures_keep_stored_widths` in + `vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs`, which CD2 (#10753) adds. Its + file round-trip check asserts that a read keeps the stored width of each old column. After CD6 + the read converts old nodes, so the check must assert the read results of the CD2 tables + instead. Keep its in-memory width check. +- New in `vtable/plugin/tests.rs`. They build nodes with `ArrayDeserialization::new` and + hand-written `DecimalMetadata`, as + `encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs:317-328` does. They build no + non-canonical `DecimalArray`, so they survive CD8a. + - `reads_old_node` (`rstest`) with every row of the table above. It asserts the encoding + (`Decimal` or `Narrow`), the child dtype and width, buffer reuse by pointer, and the values. + - `wider_node_rejects_value_outside_precision`, `narrower_node_ignores_null_slots`, + `unknown_values_type_fails` and `short_buffer_fails`. + - `decode_through_serialized_array`: serialize a canonical `Decimal(2, 0)` array and decode it as + `Decimal(10, 0)`, which is an old narrow node. Serialize a canonical `Decimal(38, 0)` array and + decode it as `Decimal(5, 0)`, which is a wider node. This covers the assert at + `serde.rs:412-417` without a non-canonical array. +- Unchanged: `vtable/mod.rs:241-273` `test_array_serde` builds `i128` at `Decimal(10, 2)`. It now + reads through the narrowing copy and passes. CD8a rewrites it. +- The compat check reads CD2's fixtures through this plugin. + +Benchmarks: None. Old narrow nodes are rare, and the speed of this path matters little (D9). + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo nextest run -p vortex-array` and `cargo test --doc -p vortex-array` +- `cargo +nightly-2026-09-10 fmt -p vortex-array` +- `cargo nextest run -p vortex-file` +- `cargo run -p vortex-compat --release -- check --dir --mode exact` +- A node at the canonical width reads with no copy and no scan. Every old node reads into the + canonical array or a `Narrow` with a true child label. A malformed node returns an error. + +Review focus: + +- The child label, where both the min and the max count, and the scale bound. +- The choice between reuse and copy. +- The `is_supported_encoding` override, no panic on malformed input, and the host copy of device + buffers. +- The CUDA read of an old narrow node (decided, D37). The host converts old narrow nodes, also on + CUDA read paths. A GPU scan of such a node copies the buffer to the host and gets a host-side + `Narrow`. State this in the PR, and ask a CUDA reviewer to confirm that such a scan still works. + +Size: medium. One new module and one registration change. + +Open questions: None. A read can now return an in-memory `Narrow`. Until WR1, the serializer writes +its canonical array (D26). + +### CD7: One DecimalByteParts scheme on the canonical array + +- Goal: The btrblocks decimal scheme takes the canonical array and chooses the number of lower parts + from the values. It narrows the MSP as the compressor narrows other internal children. + `narrowed_decimal` goes. File contents stay the same, except for precision 39 to 76 with values + that need more than 64 bits in v1 mode. +- Depends on: NR1 (`encode_values`), CD1 (benchmarks first, D19). Soft: CD3, CD5. Blocks: CD8a, + WR3. +- Shares files with CD5: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in + `assemble.rs` and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`, + which CD5's tests call. Do not run CD5 and CD7 at the same time. +- Decisions: D17, D23, D22, D19, D11, D28. +- Branch: `ct/dbp-scheme-canonical`, based on `develop` after NR1 and CD1 merge. +- Scope: `lower_part_count`, `split_decimal` with an explicit count, `DecimalByteParts::encode`, + `DecimalScheme::compress`, the removal of `narrowed_decimal`, docs, golden corpus entries and + tests. +- Out of scope: + - The decimal `Narrow` scheme (WR3) and DecimalByteParts v2 in an edition (WR2). + - Narrowing the MSP when lower parts exist. It changes the v2 output, so WR2 or WR3 decides. + - A shape with two lower parts, and changes to the scheme estimate. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-btrblocks/src/schemes/decimal.rs:43-66`, `:121`, `:125-165` | Docs, the estimate comment, and the new `compress` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs:30-142` | `lower_part_count`, `split_decimal` with a count, a split of `i256` into one lower part, and `DecimalByteParts::encode` | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:38-47` | Export `lower_part_count` | +| `encodings/decimal-byte-parts/src/lib.rs:7-14` | Crate doc: the MSP narrows, the decimal does not | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/testing.rs:23-38` | `i128_parts` and `i256_parts` pass the counts 1 and 3 | +| `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:257`, `:303`, `:318`, `:376`, `split.rs:271`, `:295` | Callers of `split_decimal` pass a count | +| `encodings/decimal-byte-parts/benches/dbp_split.rs:130-138`, `dbp_assemble.rs:145-156` | Callers pass the natural count of the width | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_v2.rs:26-37` | `encode_byte_parts` calls `DecimalByteParts::encode` | +| `vortex-array/src/arrays/decimal/utils.rs:4-5`, `:91-144` | Remove `narrowed_decimal`, `try_downcast!` and their imports | +| `vortex-btrblocks/tests/golden.rs:142-165`, `:369-374` | New decimal corpus entries | +| `vortex-btrblocks/tests/snapshots/` | New snapshots | +| `vortex-btrblocks/tests/decimal_config.rs` | New scheme tests | + +Design notes: + +```rust +/// The number of lower parts that the valid values need: 0 when every valid value fits `i64`, 1 +/// when every valid value fits `i128`, and 3 otherwise. +pub fn lower_part_count(decimal: &DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult + +/// Split a decimal array into an MSP and `lower_part_count` lower parts. +pub fn split_decimal( + decimal: &DecimalArray, + lower_part_count: usize, + ctx: &mut ExecutionCtx, +) -> VortexResult +``` + +- `lower_part_count` returns 0 without a scan for a width up to `i64`. For `i128` and `i256` it + executes the mask once and scans the valid values. It must not use the `min_max` aggregate: that + aggregate builds decimal scalars with `Scalar::decimal` + (`vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:61-80`), which panics on a value outside + the precision (`vortex-array/src/scalar/constructor.rs:68-78`). Decoded data trusts the precision, + and `prop_tests.rs:104-132` draws such values. +- `split_decimal` accepts these shapes: + - A width up to `i64` with count 0. The MSP reuses the buffer, as today (`split.rs:108-111`). + - `i128` with count 0, a copy into `i64` with zeros in null slots, or count 1 with `split_wide`. + - `i256` with count 0, count 1 with a new `i256_to_i128_parts`, or count 3, as today. + - A count below the need of a valid value returns an error, because the copy reads every value. + Other counts return an error. + - No valid row gives constant parts with the requested count, as today (`split.rs:123-142`). +- `DecimalByteParts::encode` is `split_decimal(decimal, lower_part_count(decimal, ctx)?, ctx)`. +- `DecimalScheme::compress`: + 1. Execute the input to the canonical `DecimalArray`. + 2. Compute `lower_part_count`. + 3. In v1 mode with a count above 0, return the input array. The cascade keeps its input when the + output is not smaller (`vortex-compressor/src/compressor/cascade.rs:338-351`), so the file + holds the canonical leaf. + 4. Call `split_decimal` with the count. + 5. With count 0, narrow the MSP with `NarrowArray::encode_values`. The call carries + `#[expect(deprecated)]` with the `TODO(connor)` reason of NR6 (D28). + 6. Compress the children and build the array, as today (`decimal.rs:140-164`). +- The MSP narrowing uses `NarrowArray::encode_values` of NR1, not `PrimitiveArrayExt::narrow`. + `narrow` picks an unsigned type for non-negative values + (`vortex-array/src/arrays/primitive/array/mod.rs:209-230`). An unsigned MSP fails + `DecimalBytePartsData::validate` (`array.rs:97-99`). The widths also change: values from 0 to 200 + get `U8`, where `narrowed_decimal` gives `I16`. `encode_values` keeps the signedness, so it gives + the width of `narrowed_decimal`: the smallest signed type that holds the min and the max + (`utils.rs:91-121`). So CD7 lands after NR1. CD7 changes no golden snapshot of other internal + children. + +File contents, compared with today: + +| Input | Today | After CD7 | +| --- | --- | --- | +| Width up to `i64` | MSP at the width of `narrowed_decimal` | The same width from `encode_values` | +| `i128` or `i256` with values that fit `i64` | Narrowed decimal, then the MSP | A copy into `i64`, then `encode_values`: the same width | +| Precision 19 to 38, values above 64 bits | v1: canonical `i128` leaf. v2: `i64` MSP and one lower part | The same | +| Precision 39 to 76, values from 65 to 128 bits | v1: `vortex.decimal` at `i128`. v2: `i64` MSP and one lower part | v1: canonical `i256` leaf (the accepted change, D17). v2: the same | +| Precision 39 to 76, values above 128 bits | v1: canonical `i256`. v2: `i64` MSP and three lower parts | The same | +| Null slots that hold values outside the range of the valid values | `narrowed_decimal` counts them (`utils.rs:96`) | `encode_values` ignores them, so the MSP can be narrower. The maintainer accepts this | +| One-row arrays | `narrowed_decimal` does not narrow them (`utils.rs:98-99`) | Unreachable: the cascade handles empty, all-null and constant arrays first (`cascade.rs:281-298`) | + +- Compact mode uses the same scheme. Only the excluded schemes differ + (`vortex-btrblocks/src/builder.rs:46-63`), so the MSP can use Pco. The MSP is the same array, so + the output is the same. +- The CUDA mode excludes the v2 ID (`builder.rs:94-101`), so it takes the v1 path. + +Steps: + +1. Add the decimal golden corpus entries on the old code, in their own commit. +2. Add `lower_part_count` and the explicit count of `split_decimal`. Update `encode`, the test + helpers, the benches and the compat fixture. +3. Rewrite `DecimalScheme::compress`. Delete `narrowed_decimal`. Fix the docs. +4. Accept the snapshot change of the precision-40 entry only. + +Tests: + +- New golden entries in `vortex-btrblocks/tests/golden.rs`, added in step 1 on the old code: + - `decimal_wide_dtype_small_values`: `Decimal(38, 2)` with `i128` values below 10,000,000. + - `decimal_nullable`: `Decimal(12, 2)` with a tenth of the rows null and zeros in null slots. + - `decimal_negative`: `Decimal(10, 2)` with values from -100,000 to 100,000. + - `decimal_wide_p40`: `Decimal(40, 2)` with values from 2^64 to 2^100. +- After step 3, only `regular__decimal_wide_p40` and `compact__decimal_wide_p40` change, from an + `i128` leaf to an `i256` leaf. `regular__decimal_prices` and `compact__decimal_prices` do not + change. +- New in `vortex-btrblocks/tests/decimal_config.rs`: + - `decimal_scheme_part_shapes` (`rstest`) for v1 and v2. It asserts the encoding, the lower-part + count and the MSP ptype, and it round-trips through serialization. Its cases: + - `Decimal(12, 2)` with small values. + - `Decimal(38, 2)` with values that fit `i64`, and with values above 64 bits. + - `Decimal(40, 2)` with values that fit `i64`, and with values from 65 to 128 bits. + - `Decimal(76, 2)` with values above 128 bits. + - A nullable case. + - `msp_keeps_signedness`: values from 0 to 200 at `Decimal(10, 0)` give an `I16` MSP. +- New in `split.rs`: + - `lower_part_count` cases for each width, a null slot that holds `1 << 100`, all nulls, and an + empty array. + - `lower_part_count` on values outside the precision does not panic. + - `split_decimal` with a count that is too small fails. + - `i256` values that fit `i128` split into one lower part and round-trip. +- Unchanged: `decimal_scheme_follows_allowed_ids` and `wide_decimal_parts_roundtrip` in + `decimal_config.rs`, `refine_picks_v2_when_v2_id_is_allowed` in `decimal.rs`, + `decimal_writer_uses_default_or_session_restrictions` in `vortex-file/src/tests.rs`, and both + property tests in `prop_tests.rs`. +- The golden tests are `golden_regular`, `golden_onpair` and `golden_compact` + (`vortex-btrblocks/tests/golden.rs:469-499`). `golden_onpair` holds no decimal entry. + `golden_compact` needs the `zstd` and `pco` features. + +Benchmarks: No CodSpeed bench covers decimal compression. A search of `vortex-btrblocks/benches` and +`vortex-compressor/benches` for "decimal" found none. File contents and decoding cost do not change. +Compression of a wide canonical array with small values makes more passes than today. It scans, +copies into `i64`, computes a min and max, and casts. + +Acceptance checks: + +- `cargo clippy -p vortex-btrblocks --all-targets --all-features -- -D warnings`, and the same for + `-p vortex-decimal-byte-parts` and `-p vortex-array` +- `cargo nextest run -p vortex-btrblocks --all-features` +- `cargo nextest run -p vortex-decimal-byte-parts`, `cargo nextest run -p vortex-file` +- `cargo +nightly-2026-09-10 fmt -p vortex-btrblocks -p vortex-decimal-byte-parts`, and the same for + `-p vortex-array` and `-p vortex-compat` +- `cargo clippy -p vortex-compat --all-targets -- -D warnings` (the v2 fixture compiles) +- The snapshot diff of step 3 touches only the two `decimal_wide_p40` snapshots. + +Review focus: + +- Equal file contents, and the null-slot difference. +- The signedness of the MSP, and the v1 early return. +- No `min_max` aggregate on decimal arrays. + +Size: medium. Two crates change, with new tests and snapshots. + +Open questions: None. The maintainer accepts the null-slot difference in the file contents. + +### CD8a: Every `DecimalArray` has the canonical width + +- Goal: Every decimal array has the canonical width of its dtype. Typed constructors reject another + width, the iterator constructors and the builder convert, and the last non-canonical producers go. +- Depends on: CD3, CD5, CD6, CD7. Soft: CD4. Blocks: CD8b, CD8c. +- Decisions: D3, D12, D14 (the width half), X8. +- Branch: `ct/decimal-canonical-width`, based on `develop` after CD3, CD5, CD6 and CD7 merge. +- Scope: + - The width check in `VTable::validate` and in every constructor. + - `from_iter` and `from_option_iter` that convert, and `DecimalBuilder` without a width parameter. + - Constant canonicalization, and the removal of `upcast_decimal_values` and the fill-null upcast + branch. + - Docs, tests, and the compat fixture builders of CD2. +- Out of scope: + - The value check (CD8b), and branches that this PR makes unreachable (CD8c). + - Scalar normalization at construction (future work), and export rewrites (CD3 or CD4). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/vtable/mod.rs:123-149` | `validate` checks `values_type == decimal_dtype.canonical_values_type()` | +| `vortex-array/src/arrays/decimal/array.rs:188-343` | `DecimalData` constructors check the width and become crate-private. No caller outside the decimal module uses them | +| `vortex-array/src/arrays/decimal/array.rs:404-531` | `Array` constructors: the width rule, and conversion in `from_iter` and `from_option_iter` | +| `vortex-array/src/arrays/decimal/array.rs:53-109` | Type doc: one width for each precision | +| `vortex-array/src/builders/decimal.rs:33-145` | The builder takes its width from the dtype | +| `vortex-array/src/builders/mod.rs:412-418`, `vortex-array/src/arrays/arbitrary.rs:146-147` | Builder calls without the type parameter | +| `vortex-array/src/arrays/constant/vtable/canonical.rs:113-121` | The buffer uses the canonical type, not the type of the scalar value | +| `vortex-array/src/arrays/decimal/compute/cast.rs:439-489` | Remove `upcast_decimal_values` and its tests at `:1065-1170` | +| `vortex-array/src/arrays/decimal/compute/fill_null.rs:54-72` | `cast::().vortex_expect(...)` in place of the upcast branch | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/` | Delete the `build_written` overrides of CD2, because they can no longer build narrow leaves. The published files keep the old forms | +| Tests across the workspace | See Tests | + +Design notes: + +- The width rule is `values_type == decimal_dtype.canonical_values_type()`: + - `VTable::validate` checks it. This covers `Array::try_from_parts`, deserialization and + `with_buffers`. + - `try_new::` and `try_new_handle` return an error that names the dtype, the canonical width + and the given width. + - `new::` and `new_handle` panic through `vortex_expect`, as today. + - `new_unchecked::`, `new_unchecked_handle` and `new_unchecked_from_byte_buffer` keep the check + with `vortex_panic!`. The check costs nothing for each row, and a mismatch is a kernel bug + (`STYLE.md:54`). +- `DecimalArray::new::` with a `T` other than the canonical type panics, and `try_new::` + fails. The constructors do not convert silently, because a silent copy hides a wrong `T`. +- `from_iter` and `from_option_iter` keep their generic `T` and collect into the canonical type with + `BigCast`. They panic when a value does not fit the canonical width. CD8b adds the precision + check. +- `DecimalBuilder::new`, `new_in`, `with_capacity` and `with_capacity_in` lose their type parameter. + The buffer variant comes from `decimal.canonical_values_type()`. `append_value::` keeps + converting. +- Constant canonicalization builds `Buffer::full(value.cast::(), len)` with `T` the canonical + type. A scalar keeps any width that fits its precision (`scalar/validate.rs:58-60`), so a + `DecimalValue::I128` at precision 5 gives an `i128` buffer today. Scalars from DataFusion + (`vortex-datafusion/src/convert/scalars.rs`), DuckDB, Python, FFI and JNI reach arrays through + this path. If CD3 already changed it, skip this item. +- `upcast_decimal_values` builds a decimal array wider than the canonical width by design. Its only + non-test caller is the fill-null upcast branch, which is unreachable: a fill value fits its + precision, so it fits the canonical type. If CD4 started to use the function, replace that use + with the decimal cast. + +Producers of a non-canonical width on `develop`: + +| Producer | Location | Owner | +| --- | --- | --- | +| Arrow import | `vortex-arrow/src/convert.rs:179-226` | CD3 | +| DuckDB import, including the `i16` to `i8` copy of PR #10155 | `vortex-duckdb/src/convert/vector.rs:303-338` | CD3 | +| DecimalByteParts decoding on the CPU | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:82-160` | CD5 | +| DecimalByteParts decoding on the GPU | `vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:49-65` | CD5 | +| `vortex.decimal` reads | `vortex-array/src/arrays/decimal/vtable/mod.rs:151-186` | CD6 | +| `narrowed_decimal` in the btrblocks scheme | `vortex-array/src/arrays/decimal/utils.rs:91-144`, `vortex-btrblocks/src/schemes/decimal.rs:133-137` | CD7 | +| Typed and iterator constructors | `vortex-array/src/arrays/decimal/array.rs:188-531` | CD8a | +| The builder width parameter | `vortex-array/src/builders/decimal.rs:97-145` | CD8a | +| Constant canonicalization | `vortex-array/src/arrays/constant/vtable/canonical.rs:113-121` | CD8a, unless CD3 | +| `upcast_decimal_values` | `vortex-array/src/arrays/decimal/compute/cast.rs:439-489` | CD8a | + +Checked, and already canonical when the input is canonical: + +- Arithmetic writes the canonical width of the result + (`scalar_fn/fns/binary/numeric/decimal.rs:410-429`). +- The integer to decimal cast targets the canonical width (`arrays/primitive/compute/cast.rs:144`). + Its buffer reuse at `:147-158` applies only when the source width equals it. +- The decimal to decimal cast copies into the canonical target width + (`arrays/decimal/compute/cast.rs:148-169`). The reuse branch at `:128-142` keeps the width only + when it is already the target width. +- Take, filter, mask, slice and masked validity keep the input width + (`arrays/decimal/compute/fixed_width.rs:14-35`, `arrays/filter/execute/take/fixed_width.rs:78`, + `arrays/decimal/compute/mask.rs:20`, `arrays/decimal/compute/rules.rs:47-82`, + `arrays/masked/execute.rs:89-99`). CUDA filter and dictionary kernels keep the input width + (`vortex-cuda/src/kernel/filter/decimal.rs:21-36`, + `vortex-cuda/src/kernel/arrays/dict.rs:220-305`). +- Fill null and patch keep the input width (`arrays/decimal/compute/fill_null.rs:76-85`, + `arrays/decimal/array.rs:537-579`). +- Sparse, run-end and constant-list canonicalization use the canonical type + (`encodings/sparse/src/canonical.rs:314-327`, `encodings/runend/src/compress.rs:386`, + `arrays/constant/vtable/canonical.rs:354-372`). So do the CUDA constant kernel + (`vortex-cuda/src/kernel/arrays/constant.rs:70-75`), the builder factory + (`builders/mod.rs:412-418`) and `arbitrary` (`arrays/arbitrary.rs:146`). +- Python, FFI, JNI and DataFusion build no decimal arrays directly. A search for `DecimalArray::` + and `DecimalBuilder` in `vortex-python/src`, `vortex-ffi/src`, `vortex-jni/src` and + `vortex-datafusion/src` finds only scalar code and the Python subclass. + +Steps: + +1. Add the width rule to `validate` and to every constructor. +2. Convert in `from_iter` and `from_option_iter`. Remove the width parameter of `DecimalBuilder`. +3. Fix constant canonicalization. Remove `upcast_decimal_values` and the fill-null upcast branch. +4. Update the docs, the tests and the compat fixture builders. + +Tests: + +- New: + - `try_new` rejects each non-canonical `T`, with the canonical width in the error. + - `new_unchecked` panics on a wrong width. + - `from_iter` converts a narrower and a wider `T`. + - The builder writes the canonical width. + - Constant canonicalization of a `DecimalValue::I128` at precision 5 gives `i32`. + - `VTable::validate` rejects a wrong width through `Array::try_from_parts`. +- Changed, because they build a non-canonical array with a typed constructor or the builder. This + list comes from a search. The test run finds the rest: + - `vortex-array/src/arrays/decimal/compute/mod.rs:30-118`, including `decimal_narrow_storage` at + `:83-87` (the report cites `:82-86` at `5fc5cb8`). + - `vortex-array/src/arrays/decimal/compute/cast.rs` tests (`:742-1000`), `fill_null.rs:171-195` + (`fill_null_overflow_upcasts` goes), `vtable/operations.rs:32-90`, `vtable/mod.rs:241-273`. + - `vortex-array/src/aggregate_fn/fns/sum/decimal.rs:113-290`, `mean/mod.rs:340-400`, + `all_non_distinct/tests.rs:283-292`, `uncompressed_size_in_bytes/mod.rs:672`, `:782`, + `min_max/mod.rs:945`. + - `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs:421`, `:542`, `:667`, `:818`, + `vortex-array/src/scalar_fn/fns/between/mod.rs:690-740`, + `vortex-array/src/arrays/fixed_width/take/tests.rs:178`. + - `vortex-array/src/builders/decimal.rs:340-440`, `vortex-array/src/builders/tests.rs:96-98`. + - `vortex-arrow/src/executor/decimal.rs:276-400`: the cases over `T` at precision 2 collapse, so + they iterate over precisions instead. + - `vortex-row/src/tests.rs:834`, `encodings/runend/src/array.rs:910-918`, + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:81-89`, + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs:174-179`. + - `vortex-cuda/src/arrow/canonical.rs:2190-2330`. `test_export_decimal_narrowing_errors` and + `test_export_decimal_narrowing_from_arrow_import` go, because `from_iter` now converts their + inputs and the export no longer fails. + - `vortex-cuda/src/kernel/filter/decimal.rs:62-90`: the cases use one precision for every `T`, so + after the conversion they lose their width coverage. Use the largest precision of each width. + - `fuzz/src/array/fill_null.rs:415-508`. +- Unchanged but weaker: tests that use `from_iter` with a non-canonical `T` keep passing on values. + `vortex-array/src/scalar_fn/fns/binary/compare/tests.rs:577-616` no longer mixes widths. CD8c + deletes those tests with their code. + +Benchmarks: `vortex-arrow/benches/to_arrow.rs:117` builds `i64` at `Decimal(19, 2)` with +`from_iter`. The input becomes canonical `i128`, so the export skips its widening copy. CodSpeed +then shows a change that comes from the input, not from the export code. The other decimal benches +already build canonical inputs (`vortex-array/benches/cast_decimal.rs:66-82`, `compare.rs:167-170`, +`binary_ops.rs:395-403`, `filter_fixed_width.rs:145-157`, `expr/case_when_bench.rs:70-80`). + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings` for each changed crate: `vortex-array`, + `vortex-arrow`, `vortex-btrblocks`, `vortex-decimal-byte-parts`, `vortex-runend`, `vortex-row`, + `vortex-duckdb`, `vortex-fuzz`, `vortex-compat`, and `vortex-cuda` on a CUDA host +- `cargo nextest run` for the same crates, and `cargo test --doc -p vortex-array` +- `cargo +nightly-2026-09-10 fmt` for the same crates +- `cargo run -p vortex-compat --release -- check --dir --mode exact` +- No code path builds a decimal array with a width other than the canonical width. + +Review focus: completeness of the producer audit, the panic in the unchecked constructors, the test +rewrites that keep their intent, and the compat fixture builders. + +Size: large. The API change touches many tests in many crates. + +Open questions: None. + +### CD8b: `DecimalArray::try_new` checks every valid value + +- Goal: `try_new` checks that every valid value fits the precision (D14). Kernels that keep values + legal use `new_unchecked`. The builder and the iterator constructors check values. +- Depends on: CD8a. Soft: CD1. Blocks: None. +- Shares files with CD8c and WR1. This PR adds `ctx` to `DecimalArray::new` and `try_new`, which + CD8c and WR1 call, and it edits the same builder code and tests as CD8c. Land this PR first, or + run the three one after the other. +- Decisions: D14, D13, D30, XQ3, XQ4. +- Branch: `ct/decimal-value-check`, based on `develop` after CD8a merges, or stacked on CD8a. +- Scope: + - A `ctx` parameter on the checked constructors, and the value check. + - Kernel call sites that move to the unchecked constructors. + - The precision check in the builder and the iterator constructors. + - Docs and tests. +- Out of scope: the `ExtensionArray` check (X7, another track), file reads (they keep trusting the + precision), unreachable code (CD8c). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/array.rs:404-531` | `ctx` on `new`, `try_new`, `new_handle`, `try_new_handle`, the value check, and the `new_unchecked*` docs | +| `vortex-array/src/arrays/decimal/utils.rs` | Reuse `check_precision` from CD3 | +| `vortex-array/src/builders/decimal.rs:147-190` | Precision check in `append_value` and `append_n_values`, `new_unchecked` in `finish_into_decimal` | +| Kernel call sites (table below) | Unchecked constructors | +| Callers of the checked constructors | `ctx`, or a move to `from_iter` and `from_option_iter` | + +Design notes: + +```rust +pub fn try_new( + buffer: Buffer, + decimal_dtype: DecimalDType, + validity: Validity, + ctx: &mut ExecutionCtx, +) -> VortexResult + +pub fn try_new_handle( + values: BufferHandle, + values_type: DecimalType, + decimal_dtype: DecimalDType, + validity: Validity, + ctx: &mut ExecutionCtx, +) -> VortexResult +``` + +- `new` and `new_handle` take the same `ctx` and panic on the same errors. +- The check executes the validity mask once (`vortex-array/src/validity.rs:256`). It tests + `MIN_BY_PRECISION[p] <= v <= MAX_BY_PRECISION[p]` for every valid value of the canonical type. A + branchless fold runs first, and a second pass finds the index only on failure, as + `vortex-array/src/arrays/decimal/utils.rs:50-80` does. The error names the index, the value and + the dtype. +- The `ctx` follows `VarBinViewArray::try_new` + (`vortex-array/src/arrays/varbinview/array.rs:230-237`), which executes its validity for its view + check. +- A device buffer skips the value check, as VarBin and ListView skip their host-only checks + (`vortex-array/src/arrays/varbin/array.rs:271-280`, + `vortex-array/src/arrays/listview/array.rs:265-266`). +- `check_precision` serves `try_new`, the imports of CD3 and the integer cast + (`vortex-array/src/arrays/primitive/compute/cast.rs:176-226` has its own check today). +- `from_iter` and `from_option_iter` check each valid value while they convert, and panic on + failure. +- `append_value` and `append_n_values` check the precision and panic, as they already panic on a + value that does not fit the width (`builders/decimal.rs:264-288`). The builder computes the bounds + once. `append_scalar`, `append_decimal_array`, zeros and nulls need no check, because their values + are legal. +- The `new_unchecked*` docs say: every valid value must fit the precision. A violation gives wrong + results, not memory unsafety. File reads use these constructors and trust the precision (D14). + +Kernel call sites that move to the unchecked constructors: + +| Location | Why the values are legal | +| --- | --- | +| `vortex-array/src/arrays/decimal/compute/fixed_width.rs:29` | Take and filter copy legal values | +| `vortex-array/src/arrays/decimal/compute/fill_null.rs:84` | Legal values and a fill value that fits the precision | +| `vortex-array/src/arrays/decimal/compute/cast.rs:274` | `cast_decimal_values` checks the target bounds | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:418`, `:427` | The kernel checks the precision bound | +| `vortex-array/src/arrays/primitive/compute/cast.rs:252` | The cast checks the precision | +| `vortex-array/src/builders/decimal.rs:187` | `append_value` checks | +| `vortex-array/src/arrays/constant/vtable/canonical.rs:369` | Scalars fit their precision | +| `encodings/runend/src/compress.rs:386` | Run-end decoding copies legal values | +| `vortex-cuda/src/kernel/filter/decimal.rs:31`, `vortex-cuda/src/kernel/arrays/dict.rs:294`, `vortex-cuda/src/kernel/arrays/constant.rs:146`, `:182` | Device kernels copy legal values | +| The imports of CD3 | They check their values themselves (D13) | + +Steps: + +1. Reuse `check_precision` from CD3. +2. Add `ctx` and the value check to the checked constructors. +3. Move the kernels in the table to the unchecked constructors. +4. Add the check to the builder and the iterator constructors. +5. Update the callers and the tests. + +Tests: + +- New: + - `try_new` rejects a valid value outside the precision with `NonNullable`, `AllValid` and + `Validity::Array`. + - `try_new` accepts such a value in a null slot, and anything with `AllInvalid`. + - `try_new` accepts `MIN_BY_PRECISION[p]` and `MAX_BY_PRECISION[p]`. + - `from_iter` and `append_value` panic on a value outside the precision. +- Changed: every caller of `new` and `try_new` passes `ctx` or moves to `from_iter` and + `from_option_iter`. A search finds 219 lines that call `DecimalArray::new(` or + `DecimalArray::try_new(`, most of them in tests. +- Changed: tests that build values outside the precision on purpose. + `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:328-367` uses `i128::MAX` at + precision 38 and `i256::MAX` at precision 76. `split.rs:263-278` uses `i256::MIN` at precision + 76. Test these extremes on the slice kernels `split_wide` and `assemble_wide_decimal`. Keep the + round trips through `DecimalArray` inside the precision. Other candidate files mix decimals with + integer extremes. Not verified line by line: + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs`, `plugin/tests.rs`, + `encodings/runend/src/compute/sum/tests.rs`, `fuzz/src/array/sum/tests.rs`, + `vortex-array/src/aggregate_fn/fns/all_non_distinct/tests.rs`, `sum_v2/tests.rs`, + `sum/decimal.rs`, `vortex-array/src/arrays/decimal/compute/cast.rs`, + `vortex-array/src/arrays/primitive/compute/cast.rs`, + `vortex-array/src/scalar_fn/fns/between/mod.rs`, + `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs`, + `vortex-array/src/scalar_fn/fns/list_contains/prepared/tests.rs`, + `vortex-cuda/src/arrow/canonical.rs`, `vortex-duckdb/src/convert/vector.rs`, + `vortex-file/src/tests.rs`, `vortex-row/src/tests.rs`. +- The property tests in `prop_tests.rs` need only the `ctx` argument. + `decoded_survives_encode_then_decode` draws values inside the precision (`:25-30`). + `encoded_survives_decode_then_encode` draws arbitrary parts (`:104-132`), and decoding uses the + unchecked constructors. + +Benchmarks: The expected CodSpeed result is no change in +`vortex-array/benches/filter_fixed_width.rs`, `cast_decimal.rs`, `binary_ops.rs`, `compare.rs`, +`expr/case_when_bench.rs`, `vortex-arrow/benches/to_arrow.rs` and the CD1 benches. A regression +points at a kernel that still calls a checked constructor. + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings`, `cargo nextest run` and + `cargo +nightly-2026-09-10 fmt` for each changed crate, as in CD8a +- `cargo test --doc -p vortex-array` +- Kernels run no value scan. A checked constructor fails on a valid value outside the precision and + accepts any value in a null slot. + +Review focus: + +- Every kernel in the table uses an unchecked constructor. +- The host-only rule and the null-slot rule. +- No per-element hidden cost in the check (`STYLE.md:108-133`). + +Size: large. The signature change touches many call sites. + +Open questions: None. `try_new` takes an `ExecutionCtx`, as `VarBinViewArray::try_new` does (D30). + +### CD8c: Remove the code for mixed decimal widths + +- Goal: Delete the code that handles a decimal array whose width differs from its dtype. CD8a makes + this code unreachable. +- Depends on: CD8a. Soft: CD8b. Blocks: None. +- Shares files with CD8b: `vortex-array/src/builders/decimal.rs`, + `vortex-array/src/arrays/decimal/utils.rs` and several tests. Start after CD8b merges, or rebase + on it. +- Decisions: D3, D14. +- Branch: `ct/decimal-remove-width-workarounds`, based on `develop` after CD8a merges. +- Scope: the items in the table, and their tests. +- Out of scope: the narrowing arms of the Arrow export + (`vortex-arrow/src/executor/decimal.rs:52-221`), which a caller still reaches with an Arrow type + narrower than the precision. The MSP checks of DecimalByteParts compare and between + (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs:111-150`, + `between.rs:59-83`), because the MSP is a narrow integer by design (D16). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/decimal/array.rs:537-609` | `patch`: one dispatch over the canonical type in place of two, and no panic at `:594-600` | +| `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:6-8`, `:114-153` | No widening of mismatched operands, and no "constant does not fit" branch at `:137-150` | +| `vortex-array/src/arrays/decimal/compute/between.rs:63-105` | The bounds cast to the canonical type always succeed, so `Option` becomes `T` | +| `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:167-176` | No common width | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:431-462` | No widening of a mismatched pair at `:445-449` | +| `vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:12-45` | Only the same-width path | +| `vortex-array/src/arrays/decimal/utils.rs:19-89` | Remove `widened_buffer` and `converted_buffer` | +| `vortex-array/src/array/haystack/mod.rs:118-126`, `:216`, `vortex-row/src/codec.rs:958` | `buffer::()` in place of `converted_buffer` | +| `vortex-array/src/builders/decimal.rs:160-179`, `:298-307` | `append_decimal_array` copies the same width. `DecimalBuffer::extend` stops converting | +| `vortex-cuda/src/arrow/canonical.rs:461-473` | The export target comes from the precision (`vortex-cuda/src/arrow/mod.rs:841-849`) and is never narrower than the canonical width, so the rejection goes | +| `vortex-array/src/dtype/decimal/types.rs:71-74` | Remove `is_compatible_decimal_value_type` if no caller remains | +| Tests | `vortex-array/src/scalar_fn/fns/binary/compare/tests.rs:575-616`, `vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs:421`, `:818`, `vortex-array/src/builders/decimal.rs:347` (`test_mixed_extend`), `vortex-array/src/aggregate_fn/fns/all_non_distinct/tests.rs:283-292` | + +Design notes: Each change replaces a width dispatch with one dispatch over +`decimal_dtype.canonical_values_type()`. Where a `None` becomes impossible, the code uses +`vortex_expect` with a message that names the invariant. Two operands of one decimal dtype share one +width after CD8a. Scalars of a dtype fit its precision (`scalar/validate.rs:58-60`), so they fit the +canonical type. + +Steps: + +1. Remove the branches in the kernels, one commit for each file group. +2. Remove `widened_buffer`, `converted_buffer` and `is_compatible_decimal_value_type`, with their + callers updated. +3. Delete the tests of the removed behavior. + +Tests: The deleted tests are in the table. The conformance suites (`test_array_consistency`, +`test_binary_numeric_array`, `test_cast_conformance`, `test_filter_conformance`) cover the kept +paths. + +Benchmarks: `vortex-array/benches/compare.rs`, `binary_ops.rs` and `filter_fixed_width.rs` stay the +same or improve a little, because a dispatch level goes. Not verified. + +Acceptance checks: + +- `cargo clippy --all-targets --all-features -- -D warnings`, `cargo nextest run` and + `cargo +nightly-2026-09-10 fmt` for `vortex-array`, `vortex-row` and `vortex-cuda` (on a CUDA + host) +- No function in `vortex-array` dispatches on two decimal widths of one dtype. + +Review focus: no removed branch is reachable with a canonical array, and the kept paths keep their +coverage. + +Size: medium. Many files, each with a small deletion. + +Open questions: None. + +## Notes from planning + +These facts came up while the briefs were drafted. Each one is either handled by a brief or +tracked elsewhere. + +1. The compat harness rebuilds the expected arrays with the current code (`check.rs:69-71`). One + broken fixture aborts the whole check (`check.rs:70`, `:195`), and no workflow runs + `compat.py check --mode all`. The README also says that `--git-ref` builds from that ref, but the + code does not (`scripts/compat.py:922-925`). A follow-up issue tracks these gaps. +2. `vortex-arrow/benches/to_arrow.rs:117` builds an `i64` buffer at `Decimal(19, 2)`, whose + canonical width is `i128`, so CD8a changes that input. Its `schema()` says `Decimal(19, 10)` + (`:94`), which does not match. +3. `dbp_assemble_narrowed` (`encodings/decimal-byte-parts/benches/dbp_assemble.rs:158-184`) uses + `u8`, `u16` and `u32` lower parts, which D16 forbids. CD5 changes it. +4. PR #10155 (`659c7f76e6`) changed only `vortex-duckdb/src/convert/vector.rs`, where it added the + lossy `i16` to `i8` narrowing. The builder accepted wider buffers before it. +5. `DecimalDType` derives `serde::Deserialize` (`vortex-array/src/dtype/decimal/mod.rs:36`), which + bypasses `try_new`. The wire readers use `try_new`. +6. `vortex-arrow/src/convert.rs:184`, `:195`, `:206` and `:217` call `DecimalDType::new`, which + panics on an invalid precision or scale from FFI. CD3 switches them to `try_new`. +7. The decimal `min_max` aggregate panics on a valid value outside the precision + (`vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:61-80`, through `Scalar::decimal`). File + reads trust the precision, so a file can trigger it. CD6 and CD7 avoid the aggregate. A + follow-up issue tracks the general case. +8. The CUDA decoder of DecimalByteParts also produces narrow leaves + (`vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:63-65`). CD5 changes it. +9. Constant canonicalization uses the width of the scalar value + (`vortex-array/src/arrays/constant/vtable/canonical.rs:113-121`), and `upcast_decimal_values` + (`vortex-array/src/arrays/decimal/compute/cast.rs:439-489`) builds arrays wider than canonical. + CD8a fixes both. Scalars still keep any width that fits their precision. +10. After CD6, Python sees `Narrow` arrays for old narrow nodes. The binding keys its + `DecimalArray` subclass on the encoding ID (`vortex-python/src/arrays/native.rs:174-175`), so + such arrays get no decimal subclass until the binding maps `Narrow`. +11. `docs/specs/versioning/design.md:264-267` says that the decimal scheme writes only single-child + arrays, but the scheme has a v2 mode. A follow-up issue tracks the stale versioning docs. +12. CD6 copies device buffers to the host with `BufferHandle::try_to_host_sync`, because CUDA reads + hand device segments to plugins (`vortex-cuda/src/layout.rs:241-247`). Whether this works for + every device buffer that reaches the plugin is Not verified. +13. Research line references that moved on `develop`: `decimal_narrow_storage` is at + `vortex-array/src/arrays/decimal/compute/mod.rs:83-87`, and the `DecimalBuffer` enum is at + `vortex-array/src/builders/decimal.rs:48-55`. diff --git a/research/decimal/agents/briefs/04-wire.md b/research/decimal/agents/briefs/04-wire.md new file mode 100644 index 00000000000..5a31cb7e42d --- /dev/null +++ b/research/decimal/agents/briefs/04-wire.md @@ -0,0 +1,491 @@ + + +# Wire change: briefs WR1 to WR3 + +WR1 needs NR1, CD3, CD4 and CD8a on `develop`. WR3 also needs NR5 and CD7. In practice NR1 to NR5 +and CD3 to CD8a are all on `develop` by then. Integer and decimal `Narrow` then exist in memory with +no wire form, and one DecimalByteParts (DBP) scheme takes the canonical decimal array. + +Paths under `vortex-array/src/arrays/narrow/` do not exist at `b70206818f`. NR1 creates them. The +briefs cite the local rewrite (`f7408481ce`) or the closed #10333 (`adc1e83b9d`) for them. Every +other `file:line` is checked against the `develop` checkout at `b70206818f`. + +The wire change lands as a stack of three PRs that merge in order, in one release, right after +CD8a. +The section [Lockstep](#lockstep) gives the reasons. + +### WR1: `vortex.narrow` wire format for integers and decimals + +- Goal: Give `Narrow` the wire form of D8. The metadata is a prost message with a typed `oneof` + (D32): the child `PType` for an integer parent, or the child precision for a decimal parent. The + reader builds and checks the child dtype before it decodes the child, so invalid metadata is an + error and never a panic. When the selected editions do not permit `vortex.narrow`, the writer + still writes the canonical array (D26). +- Depends on: NR1 (encoding, `validate_dtypes`, the lowering hook, registration), CD3 + (`DecimalDType::canonical_values_type()`), CD4 (decimal rules in `validate_dtypes`, decimal + `Narrow`). Soft: FX1, NR3, NR4, NR5, CD6. Blocks: WR2, WR3. +- Decisions: D8, D10, D14, D15 (one `Narrow` over a `Narrow`), D22, D26, D32. Invariants 1, 2 and 6 + of `docs/specs/versioning/design.md:219-235`. +- Branch: `ct/narrow-wire-format`, based on `develop` after CD8a. +- Shares an API with CD8b: CD8b adds `ctx` to `DecimalArray::new` and `try_new`, which the decimal + tests of this PR call. Start after CD8b merges, or rebase on it. +- Scope: The serializer and deserializer of `vortex.narrow`, the documented wire contract, the + lowering when the context does not permit the ID, and tests for round trips and for every + rejected input. +- Out of scope: Edition membership (WR2). Compressor output (WR3). Compat fixtures (LT1). Any + change to the child rules of `validate_dtypes` (NR1 and CD4 own them). A second wire ID: + `vortex.narrow` has one wire form, so the blanket `ArrayPlugin` impl for `VTable` + (`vortex-array/src/array/plugin.rs:165-205`) is enough. + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-array/src/arrays/narrow/wire.rs` | New. `NarrowMetadata`, `metadata` and `child_dtype`, with the contract in the module doc | +| `vortex-array/src/arrays/narrow/vtable.rs` | Created by NR1. Replace the `deserialize` stub (`vortex_bail!`), and make `serialize` return the `vortex.narrow` form | +| `vortex-array/src/arrays/narrow/mod.rs` | Created by NR1. Declare `mod wire;`. Remove "cannot be serialized" from the module doc | +| `vortex-array/src/serde.rs:161-256` | The tree builder also lowers an array when the context does not permit its serialized ID | +| `vortex-array/src/arrays/narrow/tests/encoding.rs` | Created by NR1. Remove `test_deserialization_unsupported`. Change `test_serialization_writes_canonical` to use a context that leaves out `vortex.narrow` | +| `vortex-array/src/arrays/narrow/tests/wire.rs` | New. Round-trip and rejection tests. Add `mod wire;` to `tests/mod.rs` | +| `vortex-file/src/tests.rs` | One file round-trip test with integer and decimal `Narrow` columns | + +Design notes: + +The wire contract of `vortex.narrow`: + +| Part | Rule | +| --- | --- | +| Dtype | An integer `DType::Primitive` or a `DType::Decimal`. Any other dtype is an error | +| Buffers | None | +| Children | Exactly one, the values child | +| Metadata | `NarrowMetadata`, a prost message with one `oneof`: the child `PType` for an integer parent, or the child precision for a decimal parent | +| Child dtype | The reader builds it from the metadata. It takes the nullability of the parent, and the scale of the parent for a decimal. It must pass `validate_dtypes`: D22 for integers, D10 for decimals | +| Child encoding | Any encoding whose dtype is the child dtype, including a `Narrow` produced by another plugin | +| Values | No scan. A read trusts the child dtype (D14). A legal child value is a legal parent value | + +```rust +// vortex-array/src/arrays/narrow/wire.rs + +#[derive(Clone, prost::Message)] +pub(super) struct NarrowMetadata { + #[prost(oneof = "NarrowChild", tags = "1, 2")] + pub child: Option, +} + +#[derive(Clone, Copy, prost::Oneof)] +pub(super) enum NarrowChild { + /// The child `PType` of an integer parent. + #[prost(enumeration = "PType", tag = "1")] + Integer(i32), + /// The child precision of a decimal parent. + #[prost(uint32, tag = "2")] + Decimal(u32), +} + +/// Returns the metadata of a `vortex.narrow` node whose values child has `child` dtype. +pub(super) fn metadata(child: &DType) -> VortexResult + +/// Returns the child dtype that `metadata` names for a parent of dtype `parent`. +/// +/// Fails on any metadata that does not name a valid child. Never panics. +pub(super) fn child_dtype(parent: &DType, metadata: &[u8]) -> VortexResult +``` + +- A missing `oneof` is an error, so an empty metadata buffer can never decode silently as `U8`. + The integer `PType` decodes with `PType::try_from`, not the prost getter, as FX1 does for other + metadata. An unknown value is an error. +- `serialize` returns the encoded `NarrowMetadata` with the `vortex.narrow` ID. A valid in-memory + `Narrow` always serializes. +- The lowering of D26 also covers a context that does not permit `vortex.narrow`. Today the tree + builder has no context, and `try_write_node` rejects a non-permitted ID with "not permitted by + ctx" (`vortex-array/src/serde.rs:251-256`). So the tree builder gets the permitted IDs, and it + asks the NR1 hook when an array's serialized ID is not permitted. The doc keeps the workaround + note and its end at LT1. + +`deserialize` runs these checks in this order. Every check returns an error: + +1. `buffers` is empty, and `children.len() == 1`. `SerializedArray::child` panics on a missing + child index (`vortex-array/src/serde.rs:480-493`), so this check comes before any `get`. +2. `child_dtype(dtype, metadata)`: + - The metadata decodes, and it holds a `oneof` variant. + - The variant matches the parent: `Integer` for an integer parent, `Decimal` for a decimal + parent. + - Integer parent: `PType::try_from` of the value. An unknown value is an error. + - Decimal parent: convert the stored `u32` with `u8::try_from`, and fail on a value above 255. + Never use `as u8`: 258 would wrap to 2 and decode as a valid-looking child. Then call + `DecimalDType::try_new(precision, parent.scale())`. This rejects 0, a value above 76, and a + precision below a positive scale (`vortex-array/src/dtype/decimal/mod.rs:48-78`). + - `validate_dtypes(&child, parent)`. This rejects a float child, other signedness, a width that + is not smaller, and a decimal child whose canonical width is not narrower (D10, D22). +3. `children.get(0, &child, len)`. The child dtype is valid at this point. The slice impl of + `ArrayChildren` asserts on a dtype mismatch (`vortex-array/src/serde.rs:314-318`), so the tests + catch any check that runs too late. +4. Flatten a `Narrow` child into its values, as `NarrowArray::try_new` does. A child can be a + `Narrow` without a nested `vortex.narrow` node: the CD6 read plugin returns a `Narrow` for an old + narrow `vortex.decimal` node. +5. Return `ArrayParts` with the parent dtype. The generic adapter runs `validate` again. + +Edge cases: `I8`, `U8` and `Decimal(1..=2, s)` parents have no narrower child, so every metadata +fails. A negative scale puts no lower bound on the child precision. Nullability is never read from +the file. + +Steps: + +1. Add `wire.rs` with `NarrowMetadata`, `metadata`, `child_dtype` and the module doc of the + contract. Add unit tests for both functions. +2. Replace the `deserialize` stub and change `serialize` in `vtable.rs`. Add `tests/wire.rs`. +3. Pass the permitted IDs to the tree builder, and lower a non-permitted `Narrow` with the NR1 + hook. +4. Add the `vortex-file` round-trip test. + +Tests: + +- `tests/wire.rs`, `round_trip_integer`, `rstest` over parent and child pairs: `I64` to `I8`, + `I16` and `I32`, `I32` to `I8`, `I16` to `I8`, `U64` to `U8`, `U32` to `U16`, each nullable and + non-nullable. Serialize with a context that permits `vortex.narrow`, decode, and check that the + result is a `Narrow` with the same child dtype. Compare values with `assert_arrays_eq!`. +- `round_trip_decimal`, `rstest`: `Decimal(12, 2)` to `Decimal(7, 2)`, `Decimal(38, 2)` to + `Decimal(3, 2)`, `Decimal(76, 0)` to `Decimal(31, 0)`, `Decimal(10, -2)` to `Decimal(4, -2)`, and + a nullable case. +- `metadata_variant`: an `I16` child gives `Integer` with the `I16` value, and a `Decimal(7, 2)` + child gives `Decimal(7)`. The serialized ID is `vortex.narrow`. +- `reject_metadata`, `rstest`: an empty buffer, bytes that do not decode, an unknown `PType`, a + `Decimal` variant under an integer parent, an `Integer` variant under a decimal parent, precision + 0, precision 77, precision 258 (which `as u8` would wrap to 2) under `Decimal(12, 2)`, a precision + below the scale, a precision whose canonical width is not narrower, + `F32`, `U8` under `I64` (signedness), and `I64` under `I64` (same width). +- `reject_parent_dtype`, `rstest`: `F64`, `Utf8`, `Bool`, `I8`, `Decimal(2, 1)`. +- `reject_child_and_buffer_counts`: zero children, two children, one buffer. +- `flattens_narrow_child`: a child that is itself a `Narrow` decodes to one `Narrow`. +- `non_permitted_context_writes_canonical`: serialization with a context whose allowed IDs leave + out `vortex.narrow` writes the canonical array, and it reads back with equal values. Build the + context with `ArrayContext::new(..).with_allowed_ids(..)`, as the private `restricted_context` + helper does (`vortex-array/src/serde.rs:907-909`). +- All rejection tests assert `is_err()`. A panic fails them, which pins the check order. +- `vortex-file/src/tests.rs`, `narrow_round_trips_through_a_file`: write a struct with an `I64` + column and a `Decimal(12, 2)` column of `Narrow` arrays. The test session permits every + registered array (`vortex-file/src/tests.rs:120-129`). Read back, find `Narrow` nodes with + `depth_first_traversal`, and compare values. + +Benchmarks: None for the wire format. WR3 needs a decode case for `Narrow` over FoR and BitPacked +and for `Narrow` over DBP. If NR2 and CD1 do not have them, add them to +`vortex/benches/common_encoding_tree_throughput.rs` as the first commit of this PR (D19). + +Acceptance checks: + +- `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` +- `cargo clippy -p vortex-file --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-file` +- `cargo nextest run -p vortex-array`, `cargo nextest run -p vortex-file` +- `cargo test --doc -p vortex-array` +- An integer or decimal `Narrow` round-trips through serialization with its child dtype. +- Every invalid metadata, parent dtype, child count or buffer count returns an error. +- A write with the default `core` edition writes the canonical array instead of `vortex.narrow`. + +Review focus: The check order before `children.get`. No `unwrap` or `vortex_expect` on metadata. A +missing `oneof` and a variant that does not match the parent both fail. The decimal child takes the +parent scale. The module doc states the full contract, because it freezes when an edition first +permits the ID. After that, relaxing D10 or D22 is a reader-visible change and needs a new wire ID +(invariant 1). + +Size: Medium. One small module, two vtable methods, one serializer change, and many small test +cases. + +Open questions: None. + +### WR2: One preview edition with `vortex.narrow` and DecimalByteParts v2 + +- Goal: Declare one new draft `preview` edition that permits `vortex.narrow` and + `vortex.decimal_byte_parts.v2` (D18). Writers that select it get DBP with lower parts at once, + and `Narrow` after WR3. +- Depends on: WR1 (D8 and invariant 6: the release that first permits the ID reads both forms), + CD5 (the v2 contract with `u64` lower parts, D16), CD7 (the one DBP scheme on the canonical + array, whose v2 mode this edition turns on). Soft: None. Blocks: WR3. +- Decisions: D8, D16, D17, D18. +- Branch: `ct/narrow-preview-edition`, based on WR1. +- Scope: The edition declaration, its export, its generated record and the default preview + constant. Edition tests. The versioning docs that the new edition makes false. +- Out of scope: Compressor changes (WR3). Any `core` edition (LT1). Compat fixtures, which can hold + only `core` formats (LT1). Rewriting the "Planned scheme configuration" text (a follow-up issue). + +Files (checked at `b70206818f`): + +| Path | Change | +| --- | --- | +| `vortex-edition/src/declarations/preview/v2026_10.rs` | New. `PREVIEW_2026_10_0` and its `DECLARATION` | +| `vortex-edition/src/declarations/preview/mod.rs:17-19` | Add `pub mod v2026_10;` and `pub use v2026_10::PREVIEW_2026_10_0;` | +| `vortex-edition/src/declarations/mod.rs:24-33` | Add `&preview::v2026_10::DECLARATION` after line 32 | +| `vortex/src/editions/mod.rs:44-54` | Re-export `PREVIEW_2026_10_0`. Set `DEFAULT_PREVIEW_EDITION` to it | +| `vortex/editions/preview/preview2026.10.0.toml` | New. Generated by `cargo run -p xtask -- generate-editions` | +| `vortex/src/editions/tests.rs` | New tests (see below). `preview_starts_empty` (`:148-159`) stays: it pins `preview2026.08.0` | +| `docs/specs/versioning/editions.md:63-72` | Add a `preview2026.10.0` entry under "Draft editions" | +| `docs/specs/versioning/editions.md:125-126` | Remove "The current `preview` edition is empty, so its first component must go into a new edition." | +| `docs/specs/versioning/design.md:259-267` | Say that the decimal scheme writes v2 when an enabled edition permits it, as `DecimalScheme::refine` does (`vortex-btrblocks/src/schemes/decimal.rs:99-105`) | +| `docs/specs/versioning/design.md:276-278` | The footnote says no edition permits v2. Name the draft `preview2026.10.0` instead | +| `vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:15-18`, `:46-47` | Change "joins an edition" to "joins a `core` edition". Fixtures are written with the default session (`vortex-test/compat-gen/src/adapter.rs:66-67`) | + +Design notes: + +```rust +/// The October 2026 draft edition of the `preview` family. +pub const PREVIEW_2026_10_0: EditionId = EditionId::new("preview", 2026, 10, 0); + +/// The declaration of [`PREVIEW_2026_10_0`] and the components that join the family at it. +pub static DECLARATION: EditionDeclaration = EditionDeclaration { + edition: Edition { id: PREVIEW_2026_10_0, min_library_version: None }, + added: &[ + EditionMember::array(&"vortex.decimal_byte_parts.v2"), + EditionMember::array(&"vortex.narrow"), + ], +}; +``` + +- The name uses the month in which the PR merges. `preview2026.10.0` is the name of #10333 + (`adc1e83b9d:vortex-edition/src/declarations/preview/v2026_10.rs`). Rename all files and + constants if the PR merges in a later month. `check-editions` requires a new record to follow + its family's chronology (`xtask/src/check_editions.rs:160-180`). +- The edition is a draft: `min_library_version` is `None`. Members are sorted by ID, as in the + `core` declarations. +- The default session registers the edition but does not enable it + (`vortex/src/editions/mod.rs:53-83`). Nothing changes for default writers. +- A writer that enables it gets DBP v2 for wide decimal values, because + `BtrBlocksCompressorBuilder::allowed_schemes` calls `refine` + (`vortex-btrblocks/src/builder.rs:234-245`). No compressor produces `Narrow` until WR3. +- The process in `docs/specs/versioning/editions.md:116-126` and `:147-150` starts a new component + in its own family. D18 puts both IDs straight into `preview`, and this brief follows D18. + +The edition adds only `vortex.narrow` and `vortex.decimal_byte_parts.v2`. Whether other IDs +join it is a decision for the owners of those encodings, not part of this plan. + +Steps: + +1. Add the declaration, the module, the re-exports and `DEFAULT_PREVIEW_EDITION`. Regenerate the + record. +2. Add the edition tests. +3. Update the docs and the compat-gen comments. + +Tests (all in `vortex/src/editions/tests.rs`): + +- `preview_2026_10_0_adds_narrow_and_decimal_byte_parts_v2`: the array set of the new edition is + exactly the two IDs. Its other kinds are empty. +- `preview_edition_ids_are_registered_array_encodings`: the same check as + `core_edition_ids_are_registered_array_encodings` (`:232-250`), for `DEFAULT_PREVIEW_EDITION`. +- `preview_edition_writes_decimal_byte_parts_v2`: a default session with the preview edition + enabled writes `Decimal(38, 2)` values at or above 2^70 as DBP with lower parts. With `core` + only, the read-back tree has no lower parts. These values need precision 22, whose canonical + width is the column width, so WR3 adds no `Narrow` and the test stays valid. +- `every_declared_edition_validates` (`:70-77`) covers the new declaration with no change. + +Benchmarks: None. Default writers do not change. + +Acceptance checks: + +- `cargo run -p xtask -- generate-editions`, then `git status --porcelain` is empty. +- `cargo run -p xtask -- check-editions --base origin/develop` +- `cargo clippy -p vortex-edition -p vortex --all-targets --all-features -- -D warnings` +- `cargo +nightly-2026-09-10 fmt -p vortex-edition -p vortex` +- `cargo nextest run -p vortex-edition`, `cargo nextest run -p vortex` +- The default session enables `core2026.08.3` only (`vortex/src/editions/tests.rs:216-230`). +- Docs build as `CONTRIBUTING.md#documentation` describes, if the user wants a docs check. + +Review focus: No frozen record changes. The edition is a draft. The docs no longer say that no +edition permits v2. WR1 is in the same release or an earlier one. + +Size: Small. One declaration, generated records, three tests and doc edits. + +Open questions: The edition name follows the merge month. Rename all files and constants if the +PR merges after October 2026. + +### WR3: A `Narrow` scheme in the compressor + +This brief needs a design pass before a coding agent starts. Two questions are still open: which +arrays the scheme may narrow (deferred, D27), and how the scheme fits the `AlwaysUse` contract and +the cascade budget (D33). The facts below are the input for that pass. + +- Goal: When the selected editions permit `vortex.narrow`, the compressor can store an integer or + decimal array at a narrower width under a `Narrow`. For decimals, the scheme narrows the decimal + type to the best precision first, and the DecimalByteParts scheme then compresses the child + (D17). Integers narrow by width with the same signedness (D22). This replaces part 2 of #10333. +- Depends on: WR1 (serializer), WR2 (the edition that tests and snapshots select), NR1 + (`NarrowArray::encode`), NR5 (the rules that move structural encodings above `Narrow`), CD3 + (`canonical_values_type()` and the narrowing copy), CD4 (`NarrowArray::encode_decimal` and the + best child dtype), CD7 (the DBP scheme on the canonical array). Soft: NR2, CD1, CD8a. Blocks: LT1. +- Decisions: D4, D10, D11, D17, D18, D19, D22, D23, D25, D27, D33. +- Branch: `ct/narrow-compressor`, based on WR2. +- Scope: A `Narrow` scheme for integers and decimals, its gate in the BtrBlocks builder, the CUDA + exclusion, tests and a `preview` golden variant. Test expectations that change because + permissive sessions now permit `vortex.narrow`. +- Out of scope: `Narrow` on internal children: offsets, codes, run ends, indices, the DBP MSP + (D23). The `encode_values` switch of the existing call sites (NR6). Any `core` edition (LT1). + Fuzzing under the preview edition. + +Decided: + +- It is a real scheme (D33). If it later turns out that the compressor always wants it, it can + become a hard-coded step. +- The scheme runs only when `vortex.narrow` passes the same edition test as the IDs that a scheme + produces. `BtrBlocksCompressorBuilder::build` already applies that test + (`vortex-btrblocks/src/builder.rs:228-245`). `empty()` and the CUDA preset leave it out, because + CUDA has no `Narrow` decoder (`builder.rs:93-102`). +- Placement follows D25. If the child compresses to `Dict`, `RunEnd` or `Sparse`, the NR5 rules + move that encoding above the `Narrow`. The compressor output must pass through those rules before + it is written or probed. Then the dict-layout probe, which checks `is::()` on its output + (`vortex-layout/src/layouts/dict/writer.rs:157-166`), still sees `Dict` at the root. The Arrow and + DuckDB exporters keep their fast paths (`vortex-duckdb/src/exporter/mod.rs:250`, + `vortex-arrow/src/executor/dictionary.rs:36`, `run_end.rs:39`). +- For decimals, the scheme writes no `Narrow` when the best precision has the canonical width of + the array. The array then goes to the DBP scheme unchanged (D17). +- The decimal child dtype comes from `NarrowArray::encode_decimal` of CD4, so one function owns the + child label. A false label breaks the compare shortcut. + +To design: + +1. Which arrays the scheme may narrow (D27). Every array whose dtype comes from outside enters + `CascadingCompressor::compress` (`vortex-compressor/src/compressor/cascade.rs:72-88`): columns, + struct fields, list and fixed-size-list elements, extension storage, union and variant children + (`cascade.rs:143-245`, `vortex-compressor/src/compressor/structural.rs:45`, `:78`, `:144`). + Arrays that layouts create also enter there: dict-layout codes, which are `U16` by default and + fixed by layout metadata (`vortex-layout/src/layouts/dict/writer.rs:69-73`), and zone-map + statistics tables (`vortex-file/src/strategy.rs:202-206`). Scheme children enter through + `compress_child` (`cascade.rs:99-119`). +2. Precedence with the DBP scheme. The DBP scheme returns `AlwaysUse` + (`vortex-btrblocks/src/schemes/decimal.rs:115-123`). The contract says that `AlwaysUse` schemes + must be mutually exclusive for each canonical type + (`vortex-compressor/src/scheme/estimate.rs:63-72`), and pass 1 returns the first match in + registration order (`vortex-compressor/src/compressor/select.rs:76-96`). A decimal `Narrow` + scheme must run before the DBP scheme when the array narrows, and not otherwise. The design + chooses how the contract changes, for example an explicit order between `AlwaysUse` schemes. +3. The cascade level. A scheme uses one level of `MAX_CASCADE = 3` + (`vortex-compressor/src/scheme/ctx.rs:14-16`). So the MSP under a `Narrow` gets one level less + than in a file without `Narrow`. The design chooses whether that is acceptable, or whether the + `Narrow` scheme does not count against the budget. +4. Integer estimates. An integer `Narrow` scheme competes with BitPacking and FoR, which already + store the narrow bits. The design decides when `Narrow` wins: for example only when it improves + the estimate, or always for in-memory compute. + +Facts and tests to keep from the draft: + +- Expected shapes under the preview edition, from the value ranges in + `vortex-btrblocks/tests/golden.rs` (Not verified by a run): + + | Column | Values | Result | + | --- | --- | --- | + | `I64` | 0 to 200 | `Narrow(I64)` over a compressed `I16` child | + | `Decimal(12, 2)` | 0 to 9,999,999 | `Narrow(Decimal(12, 2))` over DBP of `Decimal(7, 2)` | + | `Decimal(10, 2)` (gap) | 10,000 to 32,767 | `Narrow` over DBP of `Decimal(5, 2)`, whose canonical width is `i32` (D19) | + | `Decimal(18, 2)` | up to 2,000,000,000 | No `Narrow`: precision 10 has the column width `i64`. DBP with a narrowed MSP | + | `Decimal(76, 2)` | about 2^100 | `Narrow` over DBP v2 of `Decimal(31, 2)` with one lower part | + +- Golden snapshots. `regular`, `onpair` and `compact` select `core2026.08.3`, plus `zstd2026.02.0` + for `compact` (`vortex-btrblocks/tests/golden.rs:469-497`). `vortex.narrow` is not permitted + there, so no existing snapshot changes. A change there means that the gate leaks. A new + `golden_preview` variant selects `core` and the preview edition, and writes one snapshot for each + corpus entry. +- Permissive sessions. `disable_editions()` and `unrestricted()` allow every registered ID + (`vortex-btrblocks/src/builder.rs:212-226`, `vortex-file/src/writer.rs:252-256`). NR1 registers + `Narrow` in the default `ArraySession`, so from WR3 on these settings can write `Narrow`. Many + `vortex-file` tests use a test edition with every registered array (`vortex-file/src/lib.rs:200`), + and two compression benchmarks use `unrestricted()`. `test_stock_ticker_file_size` + (`vortex-file/src/tests.rs:150-191`) is one known change. Review every changed expectation + against D23. Do not accept a change only because it is new output. +- `test_array_stream_no_double_dict_encode` (`vortex-file/src/tests.rs:1598-1622`) stays unpinned. + The closed #10333 pinned it to an explicit compressor (`07fafc2a5b`), which hid the probe + problem. With the NR5 rules, the test is the regression test for placement. +- Tests to port from `adc1e83b9d:vortex-btrblocks/tests/narrow.rs`, with the module `SESSION`: + round trips under `core` and preview selections, an edition without `vortex.narrow`, the CUDA + preset, and internal children that never narrow. +- The closed #10333 showed `take_map` regressions with no known cause. Check `take_map` in this + CodSpeed run, and compare it with the NR6 run. + +Acceptance checks, once the design is settled: + +- For each `` in `vortex-compressor`, `vortex-btrblocks`, `vortex-layout`, `vortex-file` + and `vortex`: `cargo clippy -p --all-targets --all-features -- -D warnings`, + `cargo +nightly-2026-09-10 fmt -p `, and `cargo nextest run -p --all-features`. +- With `core` only, compressor output is the same as before WR3. +- With the preview edition, no `Narrow` sits directly above a structural encoding, no internal + child is a `Narrow`, and decoding gives the input. + +Size: Large. + +Open questions: The four items under "To design". + +## Lockstep + +Recommendation: a stack of three PRs, WR1 then WR2 then WR3, each based on the previous one. The +stack merges in order in one release, right after CD8a, so the wire change ships with the in-memory +change (D11). + +Reasons: + +- No PR needs temporary code (D11). After WR1, readers read a format that no edition permits. + After WR2, preview writers write DBP v2 and no `Narrow`. After WR3, preview writers write + `Narrow`. Each state is consistent, so a release cut between two PRs does no harm. It only + delays the D19 benefit for preview writers. +- The review topics differ: a frozen-to-be wire contract, edition records and docs, and + compressor behavior with many expectation updates. One PR would bury the wire review under + snapshot churn. +- The one hard order is WR1 before WR2. D8 and invariant 6 need the reader for both forms in + the release that first permits `vortex.narrow`. +- WR2 adds no golden variant. WR3 adds it, so preview snapshots change once. + +One PR is acceptable if the maintainer wants one revert unit. If the release cut cannot wait, only +WR3 can slip. WR3 also waits for its design pass. + +Hard dependencies on the NR and CD PRs: + +| PR | Needs | For | +| --- | --- | --- | +| WR1 | NR1 | `Narrow`, `validate_dtypes`, the lowering hook, registration in the default `ArraySession` (`f7408481ce:vortex-array/src/session/mod.rs`) | +| WR1 | CD3 | `DecimalDType::canonical_values_type()` for the width check | +| WR1 | CD4 | Decimal rules in `validate_dtypes` and decimal `Narrow` arrays | +| WR2 | CD5 | The v2 contract with `u64` lower parts and canonical decoding (D16) | +| WR2 | CD7 | The DBP scheme on the canonical array, which v2 mode runs once the edition permits v2 | +| WR3 | NR1 | `NarrowArray::encode` | +| WR3 | NR5 | The rules that move structural encodings above `Narrow` (D25) | +| WR3 | CD3 | The narrowing copy behind the precision-drop cast | +| WR3 | CD4 | `NarrowArray::encode_decimal` and the best child dtype | +| WR3 | CD7 | The DBP scheme that compresses the narrowed child | + +## Later work (pointers only) + +- LT1, both formats enter `core`: a new `core` edition with `vortex.decimal_byte_parts.v2` and + `vortex.narrow` (D18). Other wire improvements from WR2 can join with them. Bump + `DEFAULT_CORE_EDITION` (`vortex/src/editions/mod.rs:51`). Re-enable the DBP v2 compat fixture + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:46-47`) and add integer + and decimal `Narrow` fixtures. The `regular` and `compact` golden snapshots change then. + `vortex-duckdb/src/lib.rs:52` and `vortex-jni/src/session.rs:25` pin `CORE_2026_08_3` and need a + decision. Default writers stop writing plain `i256` for wide values (D17). Decide then whether + the serialization workaround of D26 stays for writers that select an older `core` edition. +- LT2, the max pass in DBP decoding (D16): resolve the `TODO(connor)` that CD5 adds in + `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs`. Decoding returns a `Narrow` + that reuses a narrow MSP buffer. No wire change. It waits for the CD1 benchmarks to show a large + regression. +- LT3, sum, min and max pushdown into the MSP (D19): kernels under + `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/` for gap columns. No wire change. + It waits for benchmarks. + +## Notes from planning + +1. The documented process starts a new component in its own family draft before `preview` + (`docs/specs/versioning/editions.md:116-126`, `:147-150`). D18 puts both IDs straight into + `preview`, and WR2 follows D18. +2. Some versioning docs are stale on `develop`. `docs/specs/versioning/design.md:264-267` says that + the decimal scheme writes only single-child arrays, but `DecimalScheme::refine` selects v2 when + it is allowed (`vortex-btrblocks/src/schemes/decimal.rs:99-105`). `design.md:269-274` and + `compatibility.md:78-83` say that per-edition scheme configuration does not exist, but + `Scheme::refine` implements it (`vortex-compressor/src/scheme/mod.rs:138-145`). WR2 fixes only + the lines that its edition makes false. A follow-up issue tracks the rest. +3. Compat fixtures are written with the default session, which enables only `core` + (`vortex-test/compat-gen/src/adapter.rs:66-67`). So no fixture can pin `vortex.narrow` or DBP v2 + while they are preview-only. The compat harness covers them from LT1 on. +4. D10 says that the child rule can be relaxed later. Once an edition permits `vortex.narrow`, older + readers reject a relaxed child. So a relaxation must happen before WR2, or it needs a new wire + ID (invariant 1 in `docs/specs/versioning/design.md:221-222`). +5. Before WR1, a `vortex.narrow` node fails with "Narrow deserialization is not supported", not + "Unknown encoding", because NR1 registers `Narrow`. This is harmless. +6. CUDA has no `Narrow` decoder, so WR3 leaves `vortex.narrow` out of the CUDA preset. What a CUDA + reader does with a preview file that holds `Narrow` is Not verified. diff --git a/research/decimal/agents/decimal-call-sites.md b/research/decimal/agents/decimal-call-sites.md new file mode 100644 index 00000000000..89cda07df3e --- /dev/null +++ b/research/decimal/agents/decimal-call-sites.md @@ -0,0 +1,350 @@ + + +# Decimal call-site inventory + +Generated from `develop` at `5fc5cb8`. It supports `report.md` in the same directory. Line +counts split code from tests. The lines of a file after its first `#[cfg(test)]` count as tests, +and files named `tests.rs` count as tests. + +## 1. Decimal-specific files + +The first column is non-test lines, the second is test lines. "Fate under the leaf design" uses +these labels. The leaf design is decision D2 and D3 in `../../decisions.md`. + +- **delete**: the logic becomes a call to an integer kernel through the integer view. +- **shrink**: a smaller decimal-specific piece remains (dtype rules, rescaling, legacy wire). +- **keep**: the logic stays, possibly moved. +- **generalize**: the logic becomes a generic wide-integer component. + +```text + 410 353 encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs + 199 262 encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs + 106 149 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/between.rs + 36 98 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs + 151 173 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs + 21 67 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/filter.rs + 63 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs + 26 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/kernel.rs + 22 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/mask.rs + 13 121 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/mod.rs + 21 0 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/slice.rs + 30 73 encodings/decimal-byte-parts/src/decimal_byte_parts/compute/take.rs + 25 38 encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs + 21 77 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/mod.rs + 0 324 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/tests.rs + 85 0 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v1.rs + 112 0 encodings/decimal-byte-parts/src/decimal_byte_parts/plugin/v2.rs + 0 177 encodings/decimal-byte-parts/src/decimal_byte_parts/prop_tests.rs + 19 0 encodings/decimal-byte-parts/src/decimal_byte_parts/rules.rs + 238 73 encodings/decimal-byte-parts/src/decimal_byte_parts/split.rs + 34 0 encodings/decimal-byte-parts/src/decimal_byte_parts/testing.rs + 35 0 encodings/decimal-byte-parts/src/lib.rs + 42 0 vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs + 13 0 vortex-array/src/aggregate_fn/fns/is_constant/decimal.rs + 63 0 vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs + 82 0 vortex-array/src/aggregate_fn/fns/min_max/decimal.rs + 112 353 vortex-array/src/aggregate_fn/fns/sum/decimal.rs + 36 0 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs + 609 0 vortex-array/src/arrays/decimal/array.rs + 154 0 vortex-array/src/arrays/decimal/compute/between.rs + 490 751 vortex-array/src/arrays/decimal/compute/cast.rs + 86 135 vortex-array/src/arrays/decimal/compute/fill_null.rs + 36 0 vortex-array/src/arrays/decimal/compute/fixed_width.rs + 29 0 vortex-array/src/arrays/decimal/compute/mask.rs + 10 111 vortex-array/src/arrays/decimal/compute/mod.rs + 85 129 vortex-array/src/arrays/decimal/compute/rules.rs + 23 0 vortex-array/src/arrays/decimal/mod.rs + 144 0 vortex-array/src/arrays/decimal/utils.rs + 25 0 vortex-array/src/arrays/decimal/vtable/kernel.rs + 218 84 vortex-array/src/arrays/decimal/vtable/mod.rs + 31 70 vortex-array/src/arrays/decimal/vtable/operations.rs + 16 0 vortex-array/src/arrays/decimal/vtable/validity.rs + 326 114 vortex-array/src/builders/decimal.rs + 60 0 vortex-array/src/dtype/decimal/macros.rs + 623 0 vortex-array/src/dtype/decimal/max_precision.rs + 137 178 vortex-array/src/dtype/decimal/mod.rs + 135 16 vortex-array/src/dtype/decimal/precision.rs + 262 0 vortex-array/src/dtype/decimal/types.rs + 171 0 vortex-array/src/scalar/typed_view/decimal/arithmetic.rs + 294 0 vortex-array/src/scalar/typed_view/decimal/dvalue.rs + 40 0 vortex-array/src/scalar/typed_view/decimal/float_cast.rs + 63 0 vortex-array/src/scalar/typed_view/decimal/integer_cast.rs + 18 2 vortex-array/src/scalar/typed_view/decimal/mod.rs + 258 0 vortex-array/src/scalar/typed_view/decimal/scalar.rs + 0 1486 vortex-array/src/scalar/typed_view/decimal/tests.rs + 185 0 vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs + 468 0 vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs + 222 196 vortex-arrow/src/executor/decimal.rs + 167 38 vortex-btrblocks/src/schemes/decimal.rs + 68 46 vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs + 38 101 vortex-cuda/src/kernel/filter/decimal.rs + 131 120 vortex-duckdb/src/exporter/decimal.rs + 7637 5915 TOTAL (code, tests) +``` + +| File or group | Role today | Fate under the leaf design | +| --- | --- | --- | +| `vortex-array/src/arrays/decimal/array.rs` | `DecimalData { decimal_dtype, values, values_type }`, constructors, `patch` | shrink: one buffer at the width that the precision fixes | +| `vortex-array/src/arrays/decimal/vtable/*` | `vortex.decimal` vtable, `DecimalMetadata { values_type }` serde | shrink: a width check, and readers that convert old widths | +| `vortex-array/src/arrays/decimal/compute/cast.rs` | decimal to decimal, integer, and f64 casts, plus `upcast_decimal_values` | shrink: rescale rules over integer casts | +| `vortex-array/src/arrays/decimal/compute/{between,fill_null,mask,rules,fixed_width}.rs` | per-width kernels and storage-fit special cases | delete: integer kernels through the integer view | +| `vortex-array/src/arrays/decimal/utils.rs` | `widened_buffer`, `converted_buffer`, `narrowed_decimal` | delete: no mixed widths in canonical form | +| `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs` | same-dtype compare with width widening | delete: storage compare | +| `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs` | checked arithmetic at a working width | shrink: rescale and precision check over integer ops | +| `vortex-array/src/aggregate_fn/fns/*/decimal.rs` | sum, min/max, is_sorted, is_constant, all_non_distinct, size | delete: storage aggregates plus result wrapping | +| `vortex-array/src/builders/decimal.rs` | `DecimalBuilder` with six buffer variants | shrink: one width for each dtype, over `PrimitiveBuilder` | +| `vortex-array/src/dtype/decimal/*` | `DecimalDType`, `DecimalType`, `NativeDecimalType`, i256 bound tables | keep `DecimalDType`, and keep `DecimalType` only for legacy metadata | +| `vortex-array/src/scalar/typed_view/decimal/*` | `DecimalValue`, `DecimalScalar`, Arrow arithmetic rules | shrink: scalar view over a storage integer | +| `encodings/decimal-byte-parts/src/*` | MSP plus 64-bit lower parts, v1 and v2 wire | keep as the compressed form. Its MSP can stay narrow, and its lower parts are `u64` (D16). Decoding widens a narrower MSP to the canonical width of the dtype, and it narrows a wider one (D21). One compressor scheme builds it from the canonical array (D17) | +| `vortex-arrow/src/executor/decimal.rs` | width adaptation to Decimal32/64/128/256 | keep, reads the one canonical width | +| `vortex-duckdb/src/exporter/decimal.rs` | width adaptation to DuckDB widths | keep, reads the one canonical width | +| `vortex-btrblocks/src/schemes/decimal.rs` | narrow then split | shrink: no narrowing, which moves to `Narrow` and integer encodings | +| `vortex-cuda/src/kernel/*decimal*` | device filter and DBP | keep or generalize to wide-integer lanes | + +## 2. Uses of `DecimalType::smallest_decimal_value_type` + +This function is the de facto canonical width. Every site below already treats the width as a +function of precision. The leaf design makes that rule the contract instead of a convention. +The first `types.rs` entry is the definition, so the list holds 25 call sites. + +```text +vortex-duckdb/src/convert/vector.rs:314 +vortex-array/src/dtype/dtype_impl.rs:320 +vortex-array/src/dtype/decimal/types.rs:46 +vortex-array/src/dtype/decimal/types.rs:73 +vortex-array/src/array/haystack/mod.rs:119 +vortex-array/src/builders/mod.rs:414 +vortex-array/src/scalar/arbitrary.rs:164 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:186 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:209 +vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112 +vortex-array/src/arrays/constant/vtable/canonical.rs:97 +vortex-array/src/arrays/constant/vtable/canonical.rs:356 +vortex-array/src/arrays/decimal/compute/cast.rs:155 +vortex-array/src/arrays/arbitrary.rs:146 +vortex-array/src/arrays/primitive/compute/cast.rs:144 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:416 +vortex-array/src/canonical.rs:188 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:365 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:103 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:49 +vortex-cuda/src/kernel/arrays/constant.rs:73 +fuzz/src/array/sum/mod.rs:54 +vortex-row/src/codec.rs:583 +encodings/sparse/src/canonical.rs:316 +``` + +## 3. Sources that produce a `values_type` other than the canonical width + +| Source | Location | Width produced | +| --- | --- | --- | +| Compressor narrowing | `vortex-btrblocks/src/schemes/decimal.rs:133` calls `narrowed_decimal` | narrower than canonical. V1 mode returns the narrowed `vortex.decimal` array for I128 and I256 values (`:134-137`) | +| DBP assembly | `encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:83-93` | the MSP ptype (I8 to I64) for any precision | +| Arrow import | `vortex-arrow/src/convert.rs:204-277` | the Arrow width (32, 64, 128, 256 bits) for any precision | +| DuckDB import | `vortex-duckdb/src/convert/vector.rs:303-330` | DuckDB width, with I16 narrowed to I8 for p <= 2 | +| Zero-copy precision-widening cast | `vortex-array/src/arrays/decimal/compute/cast.rs:128-141` | the source width, which can exceed the target canonical width | +| User constructors | `DecimalArray::new::` (`vortex-array/src/arrays/decimal/array.rs:406`) | any `T` | + +## 4. Consumers that must handle mixed widths today + +| Consumer | Location | Workaround | +| --- | --- | --- | +| Compare | `vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:114-126` | widen both sides to the wider width | +| Compare with constant | `.../compare/decimal.rs:128-153` | sign test when the constant does not fit the storage | +| Between | `vortex-array/src/arrays/decimal/compute/between.rs:48-126` | sign test when a bound does not fit the storage | +| Arithmetic | `vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:446-450` | widen mismatched pairs | +| Fill null | `vortex-array/src/arrays/decimal/compute/fill_null.rs:55-72` | upcast the whole array when the fill value does not fit | +| Patch | `vortex-array/src/arrays/decimal/array.rs:594-600` | panics when the storage cannot hold the dtype | +| all_non_distinct | `vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs` | triple-nested width dispatch | +| Row codec | `vortex-row/src/codec.rs:579-584` | keys use the canonical width so chunks stay comparable | +| Haystack (IN list) | `vortex-array/src/array/haystack/mod.rs:118-125` | converts to the canonical width | +| Size statistic | `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | reports canonical width, not the stored width | +| Bloom filter | `vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:34-38` | unsupported, citing issue 5820 | + +## 5. Canonical dispatch sites that name `Canonical::Decimal` (non-test) + +Each site needs a decision when the decimal canonical form changes. With one width for each +dtype, most of them reduce to a call on the integer view. + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:38 +vortex-duckdb/src/exporter/canonical.rs:30 +vortex-array/src/builders/decimal.rs:255 +vortex-array/src/arrays/constant/vtable/canonical.rs:110 +vortex-array/src/arrays/constant/vtable/canonical.rs:123 +vortex-array/src/arrays/masked/execute.rs:54 +vortex-array/src/arrays/dict/execute.rs:53 +vortex-array/src/arrays/filter/execute/mod.rs:141 +vortex-array/src/arrays/filter/execute/mod.rs:142 +vortex-array/src/scalar_fn/fns/cast/mod.rs:218 +vortex-array/src/scalar_fn/fns/fill_null/mod.rs:197 +vortex-array/src/canonical.rs:158 +vortex-array/src/canonical.rs:190 +vortex-array/src/canonical.rs:360 +vortex-array/src/canonical.rs:368 +vortex-array/src/canonical.rs:667 +vortex-array/src/canonical.rs:674 +vortex-array/src/canonical.rs:844 +vortex-array/src/canonical.rs:851 +vortex-array/src/canonical.rs:1202 +vortex-array/src/canonical.rs:1222 +vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs:516 +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:291 +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:387 +vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs:284 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs:332 +vortex-array/src/aggregate_fn/fns/is_constant/mod.rs:421 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:457 +vortex-array/src/aggregate_fn/fns/sum/mod.rs:285 +vortex-array/src/aggregate_fn/fns/min_max/mod.rs:474 +vortex-cuda/src/kernel/arrays/dict.rs:294 +vortex-cuda/src/kernel/arrays/constant.rs:146 +vortex-cuda/src/kernel/arrays/constant.rs:182 +vortex-cuda/src/kernel/slice/mod.rs:53 +vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs:63 +vortex-cuda/src/kernel/filter/decimal.rs:31 +vortex-cuda/src/kernel/filter/mod.rs:78 +vortex-cuda/src/arrow/canonical.rs:306 +vortex-cuda/src/canonical.rs:145 +vortex-cuda/src/canonical.rs:154 +vortex-btrblocks/src/schemes/decimal.rs:87 +fuzz/src/array/scalar_at.rs:49 +fuzz/src/array/sum/mod.rs:135 +fuzz/src/array/mask.rs:90 +fuzz/src/array/fill_null.rs:45 +fuzz/src/row.rs:370 +vortex-row/src/codec.rs:299 +vortex-row/src/codec.rs:410 +vortex-compressor/src/compressor/cascade.rs:123 +vortex-compressor/src/compressor/cascade.rs:124 +vortex-compressor/src/compressor/constant/mod.rs:80 +``` + +## 6. Shared files with decimal arms (non-test), with reference counts + +The count is the number of lines that mention a decimal type, array, value, or macro. + +```text +74 vortex-array/src/arrays/primitive/compute/cast.rs +31 vortex-array/src/canonical.rs +29 vortex-datafusion/src/convert/scalars.rs +28 vortex-cuda/src/arrow/canonical.rs +26 vortex-array/src/aggregate_fn/fns/sum/mod.rs +26 vortex-array/src/aggregate_fn/fns/mean/mod.rs +26 fuzz/src/array/fill_null.rs +19 vortex-array/src/scalar/proto.rs +19 vortex-array/src/arrays/constant/vtable/canonical.rs +19 vortex-array/src/aggregate_fn/fns/sum/constant.rs +18 vortex-arrow/src/scalar.rs +18 encodings/runend/src/array.rs +16 vortex-python/src/scalar/factory.rs +15 vortex-row/src/codec.rs +15 vortex-cuda/src/kernel/arrays/dict.rs +13 vortex-ffi/src/scalar.rs +13 encodings/sparse/src/canonical.rs +12 vortex-duckdb/src/convert/vector.rs +12 vortex-arrow/src/convert.rs +12 vortex-array/src/scalar_fn/fns/between/mod.rs +12 vortex-array/src/scalar/convert/into_scalar.rs +11 vortex-jni/src/expression.rs +10 vortex-duckdb/src/convert/scalar.rs +9 vortex-cuda/src/arrow/mod.rs +9 vortex-array/src/aggregate_fn/fns/min_max/mod.rs +9 vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs +8 vortex-cuda/src/kernel/arrays/constant.rs +8 vortex-array/src/compute/conformance/binary_numeric.rs +8 fuzz/src/array/sum/mod.rs +8 fuzz/src/array/mask.rs +7 vortex-array/src/stats/stats_set.rs +7 vortex-array/src/scalar/downcast.rs +7 vortex-array/src/dtype/dtype_impl.rs +7 vortex-array/src/array/haystack/mod.rs +7 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs +7 fuzz/src/array/take.rs +6 vortex-ffi/src/dtype.rs +6 vortex-array/src/scalar/validate.rs +6 vortex-array/src/scalar/arbitrary.rs +6 vortex-array/src/arrays/filter/execute/take/fixed_width.rs +6 vortex-array/benches/binary_ops.rs +6 fuzz/src/array/sort.rs +6 fuzz/src/array/slice.rs +6 fuzz/src/array/search_sorted.rs +6 fuzz/src/array/filter.rs +6 fuzz/src/array/compare.rs +6 encodings/runend/src/compress.rs +5 vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts_v2.rs +5 vortex-duckdb/src/convert/dtype.rs +5 vortex-array/src/scalar_fn/fns/binary/compare/nested.rs +5 vortex-array/src/scalar/typed_view/primitive/scalar.rs +5 fuzz/src/array/scalar_at.rs +4 vortex-test/e2e-cuda/src/lib.rs +4 vortex-python/src/scalar/into_py.rs +4 vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs +4 vortex-cuda/src/canonical.rs +4 vortex-array/src/scalar_fn/fns/fill_null/mod.rs +4 vortex-array/src/scalar/scalar_value.rs +4 vortex-array/src/dtype/serde/proto.rs +4 vortex-array/src/builders/mod.rs +4 vortex-array/src/arrays/masked/execute.rs +4 vortex-array/src/arrays/dict/execute.rs +4 vortex-array/src/arrays/constant/compute/cast.rs +4 vortex-array/src/arrays/chunked/compute/mod.rs +4 vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs +4 vortex-array/src/aggregate_fn/fns/is_constant/mod.rs +4 fuzz/src/row.rs +3 vortex-duckdb/src/duckdb/logical_type.rs +3 vortex-cuda/src/kernel/filter/mod.rs +3 vortex-compressor/src/compressor/constant/mod.rs +3 vortex-arrow/src/dtype.rs +3 vortex-arrow/benches/to_arrow.rs +3 vortex-array/src/scalar_fn/fns/cast/mod.rs +3 vortex-array/src/scalar_fn/fns/case_when.rs +3 vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs +3 vortex-array/src/scalar/constructor.rs +3 vortex-array/src/arrays/constant/vtable/mod.rs +3 vortex-array/src/arrays/arbitrary.rs +3 vortex-array/benches/filter_fixed_width.rs +3 vortex-array/benches/expr/case_when_bench.rs +3 vortex-array/benches/cast_primitive.rs +2 vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs +2 vortex-python/src/arrays/native.rs +2 vortex-layout/src/layouts/zoned/aggregates/bloom_filter/scalar_fn.rs +2 vortex-compressor/src/compressor/cascade.rs +2 vortex-array/src/scalar_fn/fns/binary/mod.rs +2 vortex-array/src/scalar/scalar_impl.rs +2 vortex-array/src/flatbuffers/generated/dtype.rs +2 vortex-array/src/dtype/serde/serde.rs +2 vortex-array/src/compute/conformance/consistency.rs +2 vortex-array/src/arrays/validation_tests.rs +2 vortex-array/src/arrays/struct_/compute/cast.rs +2 vortex-array/src/arrays/mod.rs +2 vortex-array/src/arrays/filter/execute/mod.rs +2 vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs +2 vortex-array/benches/compare.rs +1 vortex-python/src/scalar/mod.rs +1 vortex-python/src/python_repr.rs +1 vortex-python/src/dtype/mod.rs +1 vortex-python/src/dtype/factory.rs +1 vortex-layout/src/layouts/zoned/zone_map.rs +1 vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs +1 vortex-jni/src/writer.rs +1 vortex-file/src/writer.rs +1 vortex-duckdb/src/table_function.rs +1 vortex-duckdb/src/exporter/canonical.rs +1 vortex-datafusion/src/convert/stats/mod.rs +1 vortex-cuda/src/kernel/slice/mod.rs +1 vortex-array/src/scalar_fn/fns/binary/compare/mod.rs +1 vortex-array/src/scalar/typed_view/mod.rs +1 vortex-array/src/scalar/display.rs +1 vortex-array/src/scalar/convert/from_scalar.rs +1 vortex-array/src/scalar/cast.rs +1 vortex-array/src/dtype/serde/flatbuffers.rs +1 vortex-array/src/dtype/arbitrary/mod.rs +1 vortex-array/src/aggregate_fn/fns/all_non_distinct/mod.rs +1 fuzz/src/array/mod.rs +``` diff --git a/research/decimal/agents/ptype-match-inventory.md b/research/decimal/agents/ptype-match-inventory.md new file mode 100644 index 00000000000..9bef4ef1fbd --- /dev/null +++ b/research/decimal/agents/ptype-match-inventory.md @@ -0,0 +1,618 @@ + + +# PType blast-radius inventory + +Generated from `develop` at `5fc5cb8` with `grep`. Paths are relative to the repository root. A +site marked `[test]` is in a test file or a `tests` module file. This file supports `report.md` +in the same directory. + +## Summary of dispatch macros + +| Macro | Uses | Files | Crates | +| --- | ---: | ---: | --- | +| `match_each_native_ptype!` | 70 | 54 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb vortex-row | +| `match_each_integer_ptype!` | 169 | 94 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb vortex-layout vortex-python vortex-row | +| `match_each_unsigned_integer_ptype!` | 118 | 53 | encodings vortex-array vortex-cuda vortex-duckdb | +| `match_each_signed_integer_ptype!` | 3 | 2 | encodings vortex-array | +| `match_each_float_ptype!` | 9 | 9 | vortex-array vortex-duckdb vortex-layout vortex-tensor | +| `match_each_native_simd_ptype!` | 5 | 5 | vortex-cuda vortex-duckdb | +| `match_each_alp_float_ptype!` | 7 | 6 | encodings vortex-cuda | +| `match_each_pvalue!` | 5 | 3 | encodings | +| `match_each_decimal_value_type!` | 77 | 49 | encodings fuzz vortex-array vortex-compressor vortex-cuda vortex-duckdb | +| `match_each_decimal_value!` | 9 | 8 | encodings vortex-array vortex-python | + +## Trait bound mentions + +| Trait | Mentions | Files | +| --- | ---: | ---: | +| `NativePType` | 598 | 169 | +| `IntegerPType` | 169 | 49 | +| `UnsignedPType` | 124 | 23 | +| `PhysicalPType` | 29 | 6 | +| `OffsetBuilderPType` | 68 | 18 | +| `NativeDecimalType` | 155 | 40 | + +## PType helper call sites + +| Method | Call sites | +| --- | ---: | +| `is_int()` | 53 | +| `is_signed_int()` | 25 | +| `is_unsigned_int()` | 31 | +| `is_float()` | 44 | +| `to_signed()` | 14 | +| `to_unsigned()` | 63 | +| `max_value_as_u64()` | 14 | + +## Wire, FFI, and language mirrors of the PType enum + +Each of these must change, or explicitly reject the new variants, when PType gains variants. + +```text +vortex-array/flatbuffers/vortex-dtype/dtype.fbs:15: F64, +vortex-array/proto/dtype.proto:22: F64 = 10; +vortex-ffi/cinclude/vortex.h:168: PTYPE_F64 = 10, +vortex-cuda/kernels/src/dynamic_dispatch.h:55: PTYPE_F64 = 9, +vortex-cuda/kernels/src/dynamic_dispatch.h:75: case PTYPE_F64: +vortex-cuda/kernels/src/dynamic_dispatch.h:96: case PTYPE_F64: +lang/cpp/include/vortex/dtype.hpp:55: F64 = PTYPE_F64, +lang/cpp/include/vortex/common.hpp:102: return PTYPE_F64; +vortex-python/python/vortex/_lib/dtype.pyi:65: F64: PType +vortex-cuda/src/dynamic_dispatch/mod.rs:57:pub fn ptype_to_tag(ptype: PType) -> PTypeTag { +vortex-cuda/src/dynamic_dispatch/mod.rs:74:pub fn tag_to_ptype(tag: PTypeTag) -> PType { +vortex-ffi/src/ptype.rs:35:impl From for PType { +vortex-ffi/src/ptype.rs:53:impl From for vx_ptype { +vortex-python/src/dtype/ptype.rs:64:impl From for PyPType { +vortex-python/src/dtype/ptype.rs:82:impl From for PType { +vortex-array/src/dtype/serde/flatbuffers.rs:525:impl From for fb::PType { +vortex-array/src/dtype/serde/flatbuffers.rs:543:impl TryFrom for PType { +vortex-array/src/dtype/serde/proto.rs:229:impl From for PType { +vortex-array/src/dtype/serde/proto.rs:248:impl From for pb::PType { +vortex-duckdb/src/convert/dtype.rs:100: DUCKDB_TYPE::DUCKDB_TYPE_HUGEINT => vortex_bail!("I128 is not in Vortex type system"), +vortex-duckdb/src/convert/dtype.rs:101: DUCKDB_TYPE::DUCKDB_TYPE_UHUGEINT => vortex_bail!("U128 is not in Vortex type system"), +``` + +## Exhaustive PType matches outside ptype.rs (proxy: an arm naming PType::F64) + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:39: PType::F16 | PType::F32 | PType::F64 => { +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:27: PType::F16 | PType::F32 | PType::F64 => accumulate_primitive_float(array, partial, ctx)?, +vortex-array/src/dtype/serde/proto.rs:262: PType::F64 => F64, +vortex-array/src/dtype/serde/flatbuffers.rs:538: PType::F64 => Self::F64, +vortex-array/src/dtype/serde/flatbuffers.rs:558: fb::PType::F64 => Self::F64, +vortex-array/src/dtype/decimal/types.rs:259: | PType::F64) => vortex_bail!("cannot convert ptype {p} to DecimalType"), +vortex-array/src/compute/conformance/consistency.rs:894: PType::F64 => return None, [test] +vortex-array/src/compute/conformance/consistency.rs:955: PType::F64 => { [test] +vortex-array/src/compute/conformance/cast.rs:64: PType::F16 | PType::F32 | PType::F64 => test_cast_from_floating_point_types(array, ctx), [test] +vortex-array/src/compute/conformance/binary_numeric.rs:137: PType::F64 => test_binary_numeric_conformance::(array, ctx), [test] +vortex-array/src/compute/conformance/binary_numeric.rs:376: PType::F64 => test_binary_numeric_edge_cases_float::(array, ctx), [test] +vortex-array/src/array/haystack/mod.rs:412: PType::F64 => PType::U64, +vortex-array/src/builders/tests.rs:804: PType::F64 => Scalar::primitive(i as f64 * 1.5, *n), [test] +vortex-array/src/scalar/proto.rs:312: ftype @ (PType::F16 | PType::F32 | PType::F64) => vortex_bail!( +vortex-array/src/scalar/proto.rs:345: ftype @ (PType::F32 | PType::F64) => vortex_bail!( +vortex-array/src/scalar/arbitrary.rs:152: PType::F64 => PValue::F64(u.arbitrary()?), +vortex-array/src/scalar/typed_view/decimal/tests.rs:542: PType::F16 | PType::F32 | PType::F64 => panic!("Unexpected type {ptype}"), [test] +vortex-array/src/scalar/typed_view/decimal/scalar.rs:116: PType::F64 => Scalar::primitive(actual_value, *nullability), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:256: PType::F64 => PValue::F64(0.0), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:347: PType::F64 => f64::from_bits(*v).into(), +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:363: PType::F64 => f64::from_bits(i64::cast_unsigned(*v)).into(), +vortex-array/src/arrays/listview/tests/operations.rs:281: PType::F64 => buffer![1f64, 2., 3., 4.].into_array(), [test] +vortex-array/src/arrays/listview/tests/operations.rs:290: PType::F64 => (buffer![0u32, 2].into_array(), buffer![2u32, 2].into_array()), [test] +vortex-array/src/arrays/bool/compute/cast.rs:155: PType::F64 => buffer![1f64, 0., 1.].into_array(), +vortex-array/src/arrays/arbitrary.rs:142: PType::F64 => random_primitive::(u, *n, chunk_len), +vortex-array/src/arrays/primitive/compute/cast.rs:346: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/arrays/primitive/compute/cast.rs:360: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/arrays/primitive/compute/cast.rs:374: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/aggregate_fn/fns/sum_v2/mod.rs:96: DType::Primitive(PType::U64 | PType::I64 | PType::F64, _) +vortex-array/src/aggregate_fn/fns/sum/mod.rs:123: PType::F16 | PType::F32 | PType::F64 => { +vortex-array/src/aggregate_fn/fns/sum/mod.rs:343: PType::F16 | PType::F32 | PType::F64 => SumState::Float(0.0), +vortex-array/src/aggregate_fn/fns/min_max/mod.rs:169: PType::F64 => Scalar::primitive(f64::NAN, Nullability::NonNullable), +vortex-cuda/src/dynamic_dispatch/mod.rs:69: PType::F64 => PTypeTag_PTYPE_F64, +vortex-cuda/src/dynamic_dispatch/mod.rs:393: PType::U64 | PType::I64 | PType::F64 => PType::U64, +vortex-cuda/src/dynamic_dispatch/plan_builder.rs:91: return matches!(arr.dtype().as_ptype(), PType::F32 | PType::F64); +vortex-btrblocks/src/schemes/float/alprd.rs:72: PType::F64 => RDEncoder::new(primitive_array.as_slice::()), +vortex-ffi/src/ptype.rs:66: PType::F64 => vx_ptype::PTYPE_F64, +fuzz/src/row.rs:351: PType::F64 => a +vortex-python/src/dtype/ptype.rs:77: PType::F64 => Self::F64, +vortex-python/src/dtype/ptype.rs:95: PyPType::F64 => PType::F64, +vortex-python/src/scalar/into_py.rs:54: PType::F64 => p.typed_value::().into_pyobject(py), +vortex-python/src/python_repr.rs:51: PType::F16 | PType::F32 | PType::F64 => { +vortex-arrow/src/scalar.rs:109: PType::F64 => value_to_arrow_scalar!(scalar.typed_value(), Float64Array), +vortex-arrow/src/dtype.rs:331: PType::F64 => DataType::Float64, +vortex-arrow/src/convert.rs:1111: PType::F64 => ( +vortex-datafusion/src/convert/scalars.rs:54: PType::F64 => ScalarValue::Float64(pscalar.typed_value::()), +vortex-compressor/src/stats/float.rs:116: PType::F64 => typed_float_stats::(input, opts.count_distinct_values, ctx), +encodings/pco/src/array.rs:288: PType::F64 => NumberType::F64, +encodings/alp/src/alp/compress.rs:38: PType::F64 => { +encodings/alp/src/alp/compress.rs:54: PType::F64 => alp_encode_components_typed::(parray, exponents, ctx)?, +``` + +## Sites of `match_each_native_ptype!` + +```text +vortex-duckdb/src/exporter/primitive.rs:34 +vortex-array/src/dtype/ptype.rs:768 +vortex-array/src/dtype/ptype.rs:781 +vortex-array/src/builders/mod.rs:406 +vortex-array/src/builders/dict/mod.rs:65 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:96 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:175 +vortex-array/src/scalar/typed_view/primitive/scalar.rs:357 +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:212 +vortex-array/src/scalar/typed_view/primitive/pvalue.rs:220 +vortex-array/src/arrays/patched/compute/compare.rs:68 +vortex-array/src/arrays/patched/compute/take.rs:49 +vortex-array/src/arrays/patched/array.rs:284 +vortex-array/src/arrays/patched/vtable/mod.rs:232 +vortex-array/src/arrays/patched/vtable/mod.rs:290 +vortex-array/src/arrays/constant/vtable/mod.rs:209 +vortex-array/src/arrays/constant/vtable/canonical.rs:81 +vortex-array/src/arrays/constant/vtable/canonical.rs:341 +vortex-array/src/arrays/list/array.rs:304 +vortex-array/src/arrays/bool/compute/cast.rs:68 +vortex-array/src/arrays/interleave/execute/primitive.rs:42 +vortex-array/src/arrays/primitive/compute/cast.rs:119 +vortex-array/src/arrays/primitive/compute/cast.rs:120 +vortex-array/src/arrays/primitive/compute/between.rs:39 +vortex-array/src/arrays/primitive/compute/zip.rs:64 +vortex-array/src/arrays/primitive/compute/fill_null.rs:32 +vortex-array/src/arrays/primitive/array/patch.rs:34 +vortex-array/src/arrays/primitive/array/mod.rs:491 +vortex-array/src/arrays/primitive/array/top_value.rs:31 +vortex-array/src/arrays/primitive/vtable/operations.rs:26 +vortex-array/src/arrays/primitive/vtable/mod.rs:197 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:40 +vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs:43 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:165 +vortex-array/src/scalar_fn/fns/binary/numeric/row.rs:77 +vortex-array/src/canonical.rs:180 +vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs:20 +vortex-array/src/aggregate_fn/fns/sum_v2/grouped.rs:89 +vortex-array/src/aggregate_fn/fns/all_non_distinct/primitive.rs:15 +vortex-array/src/aggregate_fn/fns/is_constant/primitive.rs:54 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:44 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:49 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:54 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:140 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:153 +vortex-array/src/aggregate_fn/fns/sum/primitive.rs:166 +vortex-array/src/aggregate_fn/fns/sum/grouped.rs:83 +vortex-array/src/aggregate_fn/fns/min_max/primitive.rs:27 +vortex-cuda/src/kernel/encodings/runend.rs:75 +vortex-cuda/src/kernel/encodings/runend.rs:90 +fuzz/src/array/scalar_at.rs:45 +fuzz/src/array/filter.rs:61 +fuzz/src/array/sort.rs:45 +fuzz/src/array/slice.rs:50 +fuzz/src/array/search_sorted.rs:86 +fuzz/src/array/compare.rs:74 +fuzz/src/array/fill_null.rs:115 +fuzz/src/array/take.rs:83 +vortex-row/src/codec.rs:774 +vortex-row/src/codec.rs:1297 +vortex-compressor/src/compressor/constant/mod.rs:62 +encodings/sequence/src/compute/is_sorted.rs:39 +encodings/sparse/src/canonical.rs:310 +encodings/runend/src/compress.rs:70 +encodings/runend/src/compress.rs:79 +encodings/runend/src/compress.rs:203 +encodings/runend/src/compute/sum/kernel.rs:100 +encodings/fastlanes/src/rle/array/rle_decompress.rs:40 +encodings/fastlanes/src/rle/array/rle_compress.rs:37 +encodings/pco/src/probe.rs:135 +``` + +## Sites of `match_each_integer_ptype!` + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:48 +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:67 +vortex-duckdb/src/exporter/dict.rs:99 +vortex-duckdb/src/exporter/run_end.rs:57 +vortex-duckdb/src/exporter/list.rs:91 +vortex-duckdb/src/exporter/list_view.rs:129 +vortex-duckdb/src/exporter/list_view.rs:130 +vortex-array/src/array/haystack/mod.rs:113 +vortex-array/src/builders/listview.rs:342 +vortex-array/src/builders/listview.rs:428 +vortex-array/src/builders/listview.rs:446 +vortex-array/src/builders/listview.rs:598 +vortex-array/src/builders/list.rs:224 +vortex-array/src/builders/list.rs:285 +vortex-array/src/builders/list.rs:286 +vortex-array/src/builders/dict/bytes.rs:262 +vortex-array/src/scalar/typed_view/decimal/scalar.rs:85 +vortex-array/src/arrays/listview/rebuild.rs:349 +vortex-array/src/arrays/listview/compute/take.rs:56 +vortex-array/src/arrays/listview/compute/take.rs:59 +vortex-array/src/arrays/listview/array.rs:380 +vortex-array/src/arrays/listview/array.rs:401 +vortex-array/src/arrays/listview/conversion.rs:54 +vortex-array/src/arrays/listview/conversion.rs:123 +vortex-array/src/arrays/list/compute/filter.rs:165 +vortex-array/src/arrays/list/array.rs:225 +vortex-array/src/arrays/bool/compute/take.rs:70 +vortex-array/src/arrays/decimal/compute/cast.rs:100 +vortex-array/src/arrays/decimal/compute/cast.rs:633 +vortex-array/src/arrays/decimal/compute/cast.rs:661 +vortex-array/src/arrays/varbin/compute/filter.rs:67 +vortex-array/src/arrays/varbin/compute/filter.rs:174 +vortex-array/src/arrays/varbin/compute/compare.rs:64 +vortex-array/src/arrays/varbin/compute/compare.rs:70 +vortex-array/src/arrays/varbin/compute/compare.rs:77 +vortex-array/src/arrays/varbin/compute/take.rs:176 +vortex-array/src/arrays/varbin/array.rs:300 +vortex-array/src/arrays/varbin/array.rs:364 +vortex-array/src/arrays/varbin/vtable/mod.rs:255 +vortex-array/src/arrays/varbin/vtable/canonical.rs:29 +vortex-array/src/arrays/varbin/builder.rs:258 +vortex-array/src/arrays/varbin/builder.rs:359 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:122 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:240 +vortex-array/src/arrays/primitive/compute/cast.rs:168 +vortex-array/src/arrays/primitive/array/patch.rs:33 +vortex-array/src/arrays/dict/compute/fill_null.rs:67 +vortex-array/src/arrays/dict/array.rs:159 +vortex-array/src/arrays/dict/array.rs:163 +vortex-array/src/arrays/dict/vtable/mod.rs:296 +vortex-array/src/arrays/dict/vtable/mod.rs:330 +vortex-array/src/arrays/varbinview/compute/take.rs:55 +vortex-array/src/arrays/null/compute/take.rs:27 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:42 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:67 +vortex-array/src/arrays/filter/execute/take/rank.rs:26 +vortex-array/src/arrays/filter/execute/take/rank.rs:36 +vortex-array/src/arrays/filter/execute/take/rank.rs:46 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:264 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:779 +vortex-array/src/scalar_fn/unstable/row/types/element/utf8.rs:202 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:81 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:82 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:117 +vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/list_view.rs:118 +vortex-array/src/aggregate_fn/accumulator_grouped.rs:386 +vortex-cuda/benches/fsst_cuda.rs:76 [test] +vortex-cuda/src/kernel/arrays/dict.rs:88 +vortex-cuda/src/kernel/arrays/dict.rs:110 +vortex-cuda/src/kernel/arrays/dict.rs:237 +vortex-cuda/src/kernel/arrays/dict.rs:342 +vortex-cuda/src/kernel/arrays/list.rs:87 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:113 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:114 +vortex-cuda/src/kernel/encodings/date_time_parts.rs:115 +vortex-cuda/src/kernel/encodings/onpair.rs:192 +vortex-cuda/src/kernel/encodings/onpair.rs:421 +vortex-cuda/src/kernel/encodings/onpair.rs:649 +vortex-cuda/src/kernel/encodings/bitpacked.rs:129 +vortex-cuda/src/kernel/encodings/fsst.rs:144 +vortex-cuda/src/kernel/encodings/fsst.rs:390 +vortex-cuda/src/kernel/encodings/for_.rs:74 +vortex-cuda/src/kernel/encodings/for_.rs:86 +vortex-cuda/src/arrow/offsets.rs:44 +vortex-cuda/src/arrow/list_view.rs:143 +vortex-cuda/src/arrow/list_view.rs:144 +vortex-cuda/src/arrow/list_view.rs:470 +vortex-cuda/src/arrow/list_view.rs:471 +fuzz/src/array/cast.rs:34 +fuzz/src/array/cast.rs:37 +fuzz/src/array/sum/mod.rs:126 +fuzz/src/row.rs:356 +vortex-python/src/arrays/mod.rs:516 +vortex-python/src/arrays/range_to_sequence.rs:149 +vortex-row/src/codec.rs:475 +vortex-row/src/codec.rs:486 +vortex-compressor/src/stats/integer.rs:260 +encodings/sequence/src/compress.rs:72 +encodings/sequence/src/compress.rs:103 +encodings/sequence/src/compute/filter.rs:28 +encodings/sequence/src/compute/take.rs:75 +encodings/sequence/src/compute/take.rs:94 +encodings/sequence/src/array.rs:195 +encodings/sequence/src/array.rs:269 +encodings/sequence/src/model_tests.rs:45 +encodings/sequence/src/model_tests.rs:91 +encodings/sequence/src/model_tests.rs:105 +encodings/sparse/src/canonical.rs:161 +encodings/sparse/src/canonical.rs:204 +encodings/sparse/src/canonical.rs:532 +encodings/sparse/src/canonical.rs:822 +encodings/runend/src/compute/take.rs:50 +encodings/bytebool/src/compute.rs:101 +encodings/fastlanes/src/bitpacking/compute/is_constant.rs:50 +encodings/fastlanes/src/bitpacking/compute/compare.rs:61 +encodings/fastlanes/src/bitpacking/compute/cast.rs:110 +encodings/fastlanes/src/bitpacking/compute/cast.rs:116 +encodings/fastlanes/src/bitpacking/compute/cast.rs:239 +encodings/fastlanes/src/bitpacking/compute/between.rs:59 +encodings/fastlanes/src/bitpacking/compute/take.rs:90 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:41 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:255 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:313 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:586 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:616 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:656 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:35 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:72 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:211 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/mod.rs:40 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:206 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:216 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:226 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:236 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:296 +encodings/fastlanes/src/bitpacking/vtable/mod.rs:194 +encodings/fastlanes/src/for_/compute/compare.rs:40 +encodings/fastlanes/src/for_/array/for_compress.rs:44 +encodings/fastlanes/src/for_/array/for_compress.rs:55 +encodings/fastlanes/src/for_/array/for_decompress.rs:76 +encodings/fastlanes/src/for_/array/for_decompress.rs:105 +encodings/fastlanes/src/for_/array/for_decompress.rs:181 +encodings/fastlanes/src/for_/array/for_decompress.rs:264 +encodings/fastlanes/src/for_/vtable/operations.rs:45 +encodings/fastlanes/src/delta/tests.rs:62 [test] +encodings/fastlanes/src/delta/tests.rs:163 [test] +encodings/datetime-parts/src/canonical.rs:48 +encodings/datetime-parts/src/canonical.rs:67 +encodings/datetime-parts/src/canonical.rs:85 +encodings/alp/src/alp_rd/mod.rs:160 +encodings/fsst/src/compress.rs:118 +encodings/fsst/src/compress.rs:169 +encodings/fsst/src/compress.rs:251 +encodings/fsst/src/compute/mod.rs:143 +encodings/fsst/src/compute/like.rs:63 +encodings/fsst/src/array.rs:367 +encodings/fsst/src/array.rs:407 +encodings/fsst/src/canonical.rs:34 +encodings/fsst/src/canonical.rs:84 +encodings/fsst/src/canonical.rs:110 +encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs:117 +encodings/onpair/src/tests.rs:417 [test] +encodings/onpair/src/tests.rs:419 [test] +encodings/onpair/src/tests.rs:449 [test] +encodings/onpair/src/array.rs:666 +encodings/onpair/src/array.rs:705 +encodings/onpair/src/decode.rs:51 +encodings/onpair/src/canonical.rs:42 +encodings/onpair/src/canonical.rs:73 +``` + +## Sites of `match_each_unsigned_integer_ptype!` + +```text +vortex-duckdb/src/exporter/rle.rs:64 +vortex-duckdb/src/exporter/rle.rs:65 +vortex-array/src/patches.rs:659 +vortex-array/src/patches.rs:694 +vortex-array/src/patches.rs:862 +vortex-array/src/patches.rs:864 +vortex-array/src/patches.rs:868 +vortex-array/src/patches.rs:948 +vortex-array/src/patches.rs:949 +vortex-array/src/patches.rs:1020 +vortex-array/src/patches.rs:1042 +vortex-array/src/arrays/patched/compute/take.rs:48 +vortex-array/src/arrays/patched/array.rs:283 +vortex-array/src/arrays/listview/rebuild.rs:146 +vortex-array/src/arrays/listview/rebuild.rs:277 +vortex-array/src/arrays/listview/array.rs:278 +vortex-array/src/arrays/listview/array.rs:279 +vortex-array/src/arrays/listview/array.rs:442 +vortex-array/src/arrays/listview/array.rs:443 +vortex-array/src/arrays/listview/array.rs:773 +vortex-array/src/arrays/listview/array.rs:774 +vortex-array/src/arrays/listview/array.rs:807 +vortex-array/src/arrays/listview/array.rs:808 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:130 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:131 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:132 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:153 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:154 +vortex-array/src/arrays/piecewise_sequence/vtable.rs:155 +vortex-array/src/arrays/list/compute/take.rs:66 +vortex-array/src/arrays/list/compute/take.rs:67 +vortex-array/src/arrays/list/compute/take.rs:203 +vortex-array/src/arrays/list/compute/take.rs:228 +vortex-array/src/arrays/list/compute/take.rs:250 +vortex-array/src/arrays/list/compute/take.rs:275 +vortex-array/src/arrays/list/compute/take.rs:301 +vortex-array/src/arrays/bool/compute/take.rs:97 +vortex-array/src/arrays/bool/compute/take.rs:108 +vortex-array/src/arrays/bool/compute/take.rs:109 +vortex-array/src/arrays/bool/patch.rs:30 +vortex-array/src/arrays/interleave/execute/bool.rs:53 +vortex-array/src/arrays/interleave/execute/bool.rs:54 +vortex-array/src/arrays/interleave/execute/primitive.rs:90 +vortex-array/src/arrays/interleave/execute/primitive.rs:91 +vortex-array/src/arrays/decimal/array.rs:559 +vortex-array/src/arrays/fixed_width/take/mod.rs:156 +vortex-array/src/arrays/fixed_width/take/mod.rs:186 +vortex-array/src/arrays/fixed_width/take/mod.rs:200 +vortex-array/src/arrays/fixed_width/take/mod.rs:201 +vortex-array/src/arrays/fixed_width/take/avx2/mod.rs:61 +vortex-array/src/arrays/fixed_width/take/avx2/mod.rs:64 +vortex-array/src/arrays/varbin/compute/take.rs:152 +vortex-array/src/arrays/varbin/compute/take.rs:175 +vortex-array/src/arrays/varbin/compute/take.rs:272 +vortex-array/src/arrays/varbin/compute/take.rs:273 +vortex-array/src/arrays/varbin/compute/take.rs:424 +vortex-array/src/arrays/varbin/compute/take.rs:492 +vortex-array/src/arrays/varbin/compute/take.rs:515 +vortex-array/src/arrays/chunked/compute/take.rs:395 +vortex-array/src/arrays/chunked/compute/take.rs:406 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:148 +vortex-array/src/arrays/fixed_size_list/compute/take.rs:159 +vortex-array/src/arrays/varbinview/compute/take.rs:90 +vortex-array/src/arrays/varbinview/compute/take.rs:101 +vortex-array/src/arrays/varbinview/compute/take.rs:102 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:531 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:532 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:681 +vortex-array/src/scalar_fn/fns/list_contains/mod.rs:682 +vortex-cuda/src/kernel/encodings/runend.rs:91 +vortex-cuda/src/kernel/encodings/fsst.rs:158 +vortex-cuda/src/kernel/encodings/fsst.rs:202 +vortex-cuda/src/kernel/encodings/zigzag.rs:64 +vortex-cuda/src/kernel/encodings/sequence.rs:51 +vortex-cuda/src/kernel/encodings/delta.rs:135 +vortex-cuda/src/kernel/patches/types.rs:85 +vortex-cuda/src/dynamic_dispatch/mod.rs:396 +encodings/sparse/src/canonical.rs:397 +encodings/runend/src/compress.rs:204 +encodings/runend/src/compress.rs:377 +encodings/runend/src/compress.rs:409 +encodings/runend/src/compute/filter.rs:69 +encodings/runend/src/compute/sum/kernel.rs:97 +encodings/runend/src/compute/take.rs:96 +encodings/runend/src/ops.rs:48 +encodings/runend/src/ops.rs:70 +encodings/runend/src/decompress_bool.rs:51 +encodings/runend/src/decompress_bool.rs:62 +encodings/zigzag/src/array.rs:254 +encodings/fastlanes/src/bitpacking/compute/filter.rs:55 +encodings/fastlanes/src/bitpacking/compute/stream_predicate.rs:64 +encodings/fastlanes/src/bitpacking/compute/compare_fused.rs:133 +encodings/fastlanes/src/bitpacking/compute/is_constant.rs:163 +encodings/fastlanes/src/bitpacking/compute/take.rs:68 +encodings/fastlanes/src/bitpacking/compute/take.rs:89 +encodings/fastlanes/src/bitpacking/compute/take.rs:124 +encodings/fastlanes/src/bitpacking/array/tests.rs:107 [test] +encodings/fastlanes/src/bitpacking/array/tests.rs:142 [test] +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:128 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:229 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/blocked.rs:400 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:158 +encodings/fastlanes/src/bitpacking/array/bitpack_decompress/global.rs:177 +encodings/fastlanes/src/bitpacking/array/mod.rs:101 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:157 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:212 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:270 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/blocked.rs:372 +encodings/fastlanes/src/bitpacking/array/bitpack_compress/global.rs:129 +encodings/fastlanes/src/rle/array/rle_decompress.rs:32 +encodings/fastlanes/src/for_/array/for_decompress.rs:411 +encodings/fastlanes/src/delta/array/delta_decompress.rs:42 +encodings/fastlanes/src/delta/array/delta_compress.rs:33 +encodings/fastlanes/src/delta/array/mod.rs:118 +encodings/alp/src/alp/decompress.rs:117 +encodings/alp/src/alp/decompress.rs:120 +encodings/fsst/src/compute/mod.rs:99 +encodings/fsst/src/compute/mod.rs:101 +``` + +## Sites of `match_each_signed_integer_ptype!` + +```text +vortex-array/src/arrays/primitive/compute/cast.rs:149 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:90 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:118 +``` + +## Sites of `match_each_float_ptype!` + +```text +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/partial/scalar.rs:40 +vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/primitive.rs:39 +vortex-duckdb/src/column_statistics.rs:40 +vortex-array/src/scalar_fn/fns/is_nan.rs:43 +vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs:17 +vortex-tensor/src/scalar_fns/l2_norm.rs:93 +vortex-tensor/src/scalar_fns/inner_product.rs:89 +vortex-tensor/src/scalar_fns/cosine_similarity.rs:89 +vortex-tensor/src/scalar_fns/l2_normalize.rs:128 +``` + +## Sites of `match_each_native_simd_ptype!` + +```text +vortex-duckdb/src/convert/scalar.rs:108 +vortex-cuda/src/kernel/arrays/dict.rs:87 +vortex-cuda/src/kernel/arrays/constant.rs:66 +vortex-cuda/src/kernel/encodings/for_.rs:98 +vortex-cuda/src/kernel/filter/mod.rs:74 +``` + +## Sites of `match_each_decimal_value_type!` + +```text +vortex-duckdb/src/exporter/decimal.rs:55 +vortex-duckdb/src/exporter/decimal.rs:63 +vortex-duckdb/src/exporter/decimal.rs:64 +vortex-array/src/array/haystack/mod.rs:122 +vortex-array/src/builders/decimal.rs:136 +vortex-array/src/builders/decimal.rs:165 +vortex-array/src/builders/mod.rs:413 +vortex-array/src/scalar/arbitrary.rs:163 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:31 +vortex-array/src/scalar/typed_view/decimal/dvalue.rs:210 +vortex-array/src/scalar/typed_view/decimal/arithmetic.rs:112 +vortex-array/src/arrays/constant/vtable/canonical.rs:100 +vortex-array/src/arrays/constant/vtable/canonical.rs:357 +vortex-array/src/arrays/decimal/compute/cast.rs:157 +vortex-array/src/arrays/decimal/compute/cast.rs:158 +vortex-array/src/arrays/decimal/compute/cast.rs:188 +vortex-array/src/arrays/decimal/compute/cast.rs:219 +vortex-array/src/arrays/decimal/compute/cast.rs:228 +vortex-array/src/arrays/decimal/compute/cast.rs:474 +vortex-array/src/arrays/decimal/compute/cast.rs:476 +vortex-array/src/arrays/decimal/compute/cast.rs:625 +vortex-array/src/arrays/decimal/compute/rules.rs:47 +vortex-array/src/arrays/decimal/compute/rules.rs:167 +vortex-array/src/arrays/decimal/compute/between.rs:42 +vortex-array/src/arrays/decimal/compute/fill_null.rs:41 +vortex-array/src/arrays/decimal/compute/fill_null.rs:66 +vortex-array/src/arrays/decimal/array.rs:309 +vortex-array/src/arrays/decimal/array.rs:561 +vortex-array/src/arrays/decimal/array.rs:563 +vortex-array/src/arrays/decimal/utils.rs:25 +vortex-array/src/arrays/decimal/utils.rs:48 +vortex-array/src/arrays/decimal/vtable/operations.rs:22 +vortex-array/src/arrays/decimal/vtable/mod.rs:170 +vortex-array/src/arrays/arbitrary.rs:146 +vortex-array/src/arrays/primitive/compute/cast.rs:169 +vortex-array/src/arrays/primitive/compute/cast.rs:312 +vortex-array/src/arrays/filter/execute/take/fixed_width.rs:65 +vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:121 +vortex-array/src/scalar_fn/fns/binary/compare/decimal.rs:134 +vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:171 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:98 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:357 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:369 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:421 +vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:446 +vortex-array/src/scalar_fn/fns/list_contains/prepared/tests.rs:195 [test] +vortex-array/src/canonical.rs:187 +vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs:19 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:21 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:27 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:28 +vortex-array/src/aggregate_fn/fns/all_non_distinct/decimal.rs:29 +vortex-array/src/aggregate_fn/fns/is_constant/decimal.rs:10 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:50 +vortex-array/src/aggregate_fn/fns/sum/decimal.rs:51 +vortex-array/src/aggregate_fn/fns/min_max/decimal.rs:27 +vortex-cuda/src/kernel/arrays/dict.rs:236 +vortex-cuda/src/kernel/arrays/constant.rs:74 +vortex-cuda/src/kernel/filter/mod.rs:79 +vortex-cuda/src/arrow/canonical.rs:478 +fuzz/src/array/scalar_at.rs:50 +fuzz/src/array/filter.rs:76 +fuzz/src/array/sort.rs:66 +fuzz/src/array/slice.rs:61 +fuzz/src/array/search_sorted.rs:111 +fuzz/src/array/compare.rs:105 +fuzz/src/array/sum/tests.rs:144 [test] +fuzz/src/array/sum/mod.rs:55 +fuzz/src/array/sum/mod.rs:135 +fuzz/src/array/mask.rs:92 +fuzz/src/array/fill_null.rs:161 +fuzz/src/array/take.rs:94 +fuzz/src/row.rs:372 +vortex-compressor/src/compressor/constant/mod.rs:81 +encodings/sparse/src/canonical.rs:318 +encodings/runend/src/compress.rs:376 +encodings/decimal-byte-parts/src/decimal_byte_parts/assemble.rs:246 +``` + diff --git a/research/decimal/agents/report.md b/research/decimal/agents/report.md new file mode 100644 index 00000000000..10f59fa7378 --- /dev/null +++ b/research/decimal/agents/report.md @@ -0,0 +1,2051 @@ + + +# Decimal and the Vortex type roadmap (agent reference) + +This file is the detailed report behind the summary in `../README.md`. It is research, not an +accepted design, and it changes no code. Paths and line numbers are for `develop` at `5fc5cb8` +(2026-10-17), the same commit as the extension-type notes in `../../extension-types/`. + +The maintainer reviewed these notes on 2026-10-19, 2026-10-22 and 2026-10-23. +[`decisions.md`](../../decisions.md) records the decisions (D1 to D20, X1 to X9, C1) and the open +questions (DQ5 to DQ7, XQ1 to XQ6, CQ1, CQ2). This report follows that file. If the two disagree, +the decisions file wins. Issue #10701 tracks the fix of the canonical decimal encoding. + +D20 makes the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions +and scales, and casts that lose information future work. Sections 3.2, 3.4, 4, 5 and 6 describe +that work. Their recommendations are not reviewed. + +Audience: Vortex maintainers and coding agents who implement the type-system work. + +Companion files in this directory: + +- `ptype-match-inventory.md` lists every PType dispatch macro use, every exhaustive PType match, + and every place where another language or wire format mirrors the PType enum. +- `decimal-call-sites.md` lists every decimal-specific file with code and test line counts, every + use of the canonical-width function, and every `Canonical::Decimal` dispatch site. + +Conventions: + +- Paths are relative to the repository root. A `file:line` reference points at the commit above. +- "Settled" marks a decision of the maintainer in the decisions file. "Recommendation" marks a + proposal of these notes that the maintainer has not confirmed. +- "Canonical width" means the buffer width that the precision fixes (D3). D12 settles the existing + function `DecimalType::smallest_decimal_value_type` for it + (`vortex-array/src/dtype/decimal/types.rs:46-57`). It maps precision 1-2 to I8, 3-4 to I16, + 5-9 to I32, 10-18 to I64, 19-38 to I128, and 39-76 to I256. +- "Gap value" means a value that fits a width but needs a precision whose canonical width is + wider, for example 2,000,000,000 in an `i32` (section 3.3). A gap column holds such values. +- "Integer view" means a `PrimitiveArray` over the buffer and the validity of a decimal array, + without a copy (section 3.2). +- "Storage" means the storage child of an extension array. +- "Lane integer" means U8 to U64 and I8 to I64. "Wide integer" means I128, I256, U128 and U256. +- Issue and pull request numbers refer to `vortex-data/vortex` on GitHub. Section 10 links + each one. +- Two claims come from running Rust code, with probes R1 and R6 in + `../../extension-types/agents/probe_followup.rs`. They are the cast in section 1.1 and the + pushdown rules in section 3.3. The other claims come from reading code, RFC drafts, issues, and + pull requests. Appendix A holds the one Python script that ran. Section 10 lists what is not + verified. + +## 1. Summary + +### 1.1 What is wrong today + +`DecimalArray` stores unscaled integers in one typed buffer. The buffer width (`values_type`) is +independent of the precision (`vortex-array/src/arrays/decimal/array.rs:70-75`). One logical +dtype therefore has up to six physical forms. That single fact explains most of the strangeness: + +- Kernels must accept mixed widths, widen on the fly, or test whether a constant fits the + storage. Some paths still panic, for example `DecimalArray::patch` + (`vortex-array/src/arrays/decimal/array.rs:594-600`). Issue #5820 collects the fuzzer crashes. +- The width that every consumer actually wants is a function of precision. The code computes it + at 25 call sites, so it is a convention, not a contract (`decimal-call-sites.md`, section 2). +- Decimal duplicates much of Primitive. It has its own canonical variant (51 dispatch sites), + builder, compare, arithmetic, and aggregates. DecimalByteParts is a wide-integer limb split that + only decimals can use. Decimal-specific files hold about 7,600 non-test lines. +- Arrays never validate precision (`array.rs:301-319`), while scalars do + (`vortex-array/src/scalar/validate.rs:159-167`). +- The scalar path and the array path disagree. A decimal scalar casts to `f32` and `f16` + (`vortex-array/src/scalar/typed_view/decimal/scalar.rs:111-115`). The array kernel only handles + `f64` (`vortex-array/src/arrays/decimal/compute/cast.rs:104`), so the array cast fails with + "No CastKernel" (`vortex-array/src/scalar_fn/fns/cast/mod.rs:143`). Probe R1 confirms it: the + array path fails, and the row path returns `[1f32, 2.5f32]` for `Decimal(10, 2) [1.00, 2.50]`. + +### 1.2 Recommendation in short + +Each point names the settled decision or the open question behind it. + +1. Keep `DecimalArray` a leaf: one values buffer and validity, and no data child (D2). Give each + decimal dtype exactly one canonical encoding, so the precision fixes the buffer width (D3). + Use the existing smallest-width mapping (D12). `try_new` checks the width and every valid + value, and imports copy into the canonical width (D13, D14). +2. Store narrow columns with a `Narrow` encoding (D4). A `Decimal(p, s)` column holds one child of + dtype `Decimal(p', s)` whose canonical width is narrower (D10). One `Narrow` encoding serves + integers and decimals (D8). An integer child keeps the signedness of its parent (D22). Its + kernels pass work to the child (D15). The wire form is only for columns with a fixed logical + dtype (D23). +3. Keep DecimalByteParts (DBP) as the compressed decimal form. Its MSP is any signed integer up to + 64 bits, and its lower parts are `u64` (D16). Operations run on the parts where they can. DBP + decodes to the canonical array of its own dtype. It widens a narrower MSP, and it narrows a + wider one (D21). +4. Convert old narrow `vortex.decimal` nodes by their max, found with a scan, into a `Narrow` or a + copy (D9). Every narrowing copy writes zeros in null slots and fails on a value outside the + precision (D21). +5. Build DBP with one scheme from the canonical array, with parts as narrow as the values allow, + so that file contents stay the same (D17). Once an edition permits `vortex.narrow`, a decimal + `Narrow` scheme narrows the decimal type first. +6. Keep every decimal wire format. Generic structural encodings sit above the decimal storage + encodings (X5, recommendation XQ1). The wire forms of today already have this shape. + `vortex.decimal` keeps its wire ID (D7). The only new wire ID is `vortex.narrow`, which serves + integers and decimals. It enters one preview edition together with DBP v2 (D18). +7. Keep `Decimal` a built-in type that behaves like a hardcoded extension type (D1). A move to an + extension type is an optional later step. + +Future work (D20), with recommendations that are not reviewed: + +8. Give decimal kernels a zero-copy integer view: a `PrimitiveArray` over the same buffer and + validity. Decimal kernels delegate to integer kernels through the view. Rewrite rules that the + type owns add the bookkeeping for the scale (X4). +9. Add `I128`, `I256`, `U128` and `U256` as primitive integer types (D5). Build them as a separate + wide-integer tier with typed dispatch. Recommendation (DQ5): user-visible behind an edition, + with a small kernel set first. The integer view needs `I128` and `I256` for precision 19 to 76. + The leaf does not need them, because `DecimalType` already has both widths. +10. Add FixedSizeBinary as a core type for opaque bytes (D6). Decimal and wide integers do not + depend on it. Recommendation (DQ7): a new core dtype tag behind an edition. + +Section 7 gives the dependency graph. Section 8 gives the recommended order of work. + +## 2. Current state of Decimal + +### 2.1 Logical dtype + +- `DType::Decimal(DecimalDType, Nullability)` (`vortex-array/src/dtype/mod.rs:85`). +- `DecimalDType { precision: NonZero, scale: i8 }` + (`vortex-array/src/dtype/decimal/mod.rs:35-40`). +- `MAX_PRECISION` and `MAX_SCALE` both come from the i256 implementation and equal 76 + (`vortex-array/src/dtype/decimal/mod.rs:26-29`, `types.rs:215-218`). +- `DecimalDType::try_new` checks precision in 1..=76, scale at most 76, and a positive scale at + most the precision (`mod.rs:48-77`). Nothing bounds a negative scale except `i8::MIN`. The doc + comment names a `MIN_SCALE` that does not exist (`mod.rs:47`). Tests use scale -128 + (`vortex-array/src/arrays/decimal/compute/cast.rs`, case `zero_extreme_scale`). +- Wire: flatbuffer `table Decimal { precision: uint8, scale: int8, nullable: bool }` with union + tag 4 (`vortex-array/flatbuffers/vortex-dtype/dtype.fbs:29,88`). Protobuf `message Decimal` + with `uint32 precision`, `int32 scale`, and oneof field 4 + (`vortex-array/proto/dtype.proto:36,96`). + +### 2.2 Physical width types + +- `DecimalType { I8 = 0, I16, I32, I64, I128, I256 = 5 }` derives `prost::Enumeration` + (`vortex-array/src/dtype/decimal/types.rs:25-42`). +- `DecimalType::smallest_decimal_value_type` is the canonical width (`types.rs:46-57`). + `is_compatible_decimal_value_type` accepts any width at least that wide (`types.rs:72-74`). +- `NativeDecimalType` is implemented for `i8` through `i128` and for `i256`. It carries the + per-precision bound tables `MIN_BY_PRECISION` and `MAX_BY_PRECISION` and visitor hooks + (`types.rs:80-229`). The i256 tables are a 623-line literal table because the bounds cannot be + computed in a `const` context (`vortex-array/src/dtype/decimal/max_precision.rs:4-5`). +- `TryFrom for DecimalType` only covers I8 to I64 (`types.rs:244-262`). There is no + PType for 128 or 256 bits, so decimals cannot share the Primitive machinery at those widths. +- Dispatch macros: `match_each_decimal_value_type!` (77 uses in 49 files) and + `match_each_decimal_value!` (9 uses) (`vortex-array/src/dtype/decimal/macros.rs:13-60`). + +### 2.3 The i256 type + +- `#[repr(transparent)] pub struct i256(arrow_buffer::i256)` + (`vortex-array/src/dtype/bigint/mod.rs:34-40`). The Arrow type is + `#[repr(C)] struct i256 { low: u128, high: i128 }`, and its `Ord` compares `high` and then + `low` (arrow-buffer 59.3.0, `src/bigint/mod.rs:59-66` and `:171-183`). +- Size 32 bytes, alignment equal to `u128` alignment, which is 16 bytes on x86_64 and aarch64 + with the pinned toolchain (`rust-toolchain.toml`: 1.98.0, workspace `rust-version` 1.95). +- It implements the arithmetic operators, `CheckedAdd`, `CheckedSub`, `CheckedMul`, and + `CheckedDiv`. It also implements wrapping add and sub, `Zero`, `One`, `ToPrimitive` with a + full-range `to_f64`, and `AsPrimitive` to and from `i8` through `i128` + (`bigint/mod.rs:153-350`). `BigCast` and `ToI256` give checked conversions up to 256 bits + (`bigint/bigcast.rs`). It is a CUDA `DeviceRepr` (`bigint/mod.rs:42-43`). +- It does not implement `Num`, `NumCast`, `FromPrimitive`, `Bounded`, or `PrimInt`. Those are + the traits that `NativePType` and `IntegerPType` require (`vortex-array/src/dtype/ptype.rs:68` + and `:124-140`). + +### 2.4 DecimalArray layout and vtable + +- `DecimalData { decimal_dtype, values: BufferHandle, values_type: DecimalType }` + (`vortex-array/src/arrays/decimal/array.rs:110-115`). One slot holds validity (`:46-51`). +- The docs allow any width for any precision. Precision 39 can be stored in `i8`, and + precision 2 can be stored in `i256` (`array.rs:70-75`). +- `DecimalData::validate` checks only that the buffer length is a multiple of the width and + that the buffer is aligned (`array.rs:301-319`). The safety docs of the unchecked + constructors say values must fit the precision (`array.rs:256-262`), but nothing checks it. + The arithmetic kernel documents the consequence: out-of-precision values can reach kernels + (`vortex-array/src/scalar_fn/fns/binary/numeric/decimal.rs:13-16`). +- `DecimalData::buffer::()` panics when `T` is not the stored width (`array.rs:360-369`). +- Encoding id `vortex.decimal` (`vortex-array/src/arrays/decimal/vtable/mod.rs:78-81`). The + metadata is `DecimalMetadata { values_type }` with the comment "The type of the values can be + determined by looking at the type info...right?" (`vtable/mod.rs:52-57`). +- `ArrayEq` and `ArrayHash` include `values_type` (`vtable/mod.rs:59-70`). Two arrays with the + same logical values at different widths are not `ArrayEq`. +- Deserialization reads the width with the prost getter `metadata.values_type()` + (`vtable/mod.rs:170`). prost-derive 0.14.4 generates that getter as + `TryFrom::try_from(raw).unwrap_or(default)`, and the default of a prost enumeration is its + first variant (prost-derive `src/field/scalar.rs:286-300`, `src/lib.rs:308`). An unknown + width therefore decodes silently as `I8`. No such value exists today, but the pattern blocks + safe extension of this metadata. +- Take and filter are shared with Primitive through the `FixedWidthArray` adapter + (`vortex-array/src/arrays/fixed_width/array.rs:20-41`, + `vortex-array/src/arrays/decimal/compute/fixed_width.rs:14-36`). PR #8864 introduced it. The + record-width dispatch already covers 1, 2, 4, 8, 16, and 32 bytes + (`vortex-array/src/arrays/fixed_width/mod.rs:16-47`). + +### 2.5 Scalar values + +- `DecimalValue { I8(i8), I16, I32, I64, I128, I256(i256) }` + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:48-61`). Its `PartialEq`, + `PartialOrd`, and `Hash` upcast to i256 (`dvalue.rs:255-281`) because one dtype allows several + widths. `normalize` and `zero` produce the canonical width (`dvalue.rs:185-216`). +- `ScalarValue::Decimal(DecimalValue)` (`vortex-array/src/scalar/scalar_value.rs:31-32`). +- `DecimalScalar` equality and ordering require equal dtypes apart from nullability + (`vortex-array/src/scalar/typed_view/decimal/scalar.rs:188-204`). +- Decimal scalars serialize as protobuf `bytes_value` with the byte length of the variant + (`vortex-array/src/scalar/proto.rs:87-98`). Readers infer the width from the byte length and + do not check precision, which a TODO notes (`scalar/proto.rs:402-445`, TODO at `:407`). +- Scalar construction validates precision (`vortex-array/src/scalar/validate.rs:58-60`, + `:159-167`). Array construction does not. + +### 2.6 Builders + +`DecimalBuilder` holds an enum of six `BufferMut` variants and coerces appended values into the +builder width (`vortex-array/src/builders/decimal.rs:38-56`, `:160-180`). The generic builder +factory picks the canonical width (`vortex-array/src/builders/mod.rs:412-416`). + +### 2.7 Compute kernels and aggregates + +| Operation | Location | Behavior | +| --- | --- | --- | +| Cast, nullability only | `arrays/decimal/compute/cast.rs:45-80` | Reduce rule, same decimal dtype | +| Cast to integer | `cast.rs:97-103`, `:172-196` | Truncate toward zero. A value outside the target range fails before truncation | +| Cast to float | `cast.rs:104-107`, `:198-245` | `f64` only, one rounding step | +| Cast decimal to decimal | `cast.rs:108-170`, plan at `:348` | Zero-copy when the scale is equal, the precision widens, and the width already fits. Otherwise rescale, exact scale-down only | +| Integer to decimal | `arrays/primitive/compute/cast.rs:87-89`, `:127-173` | Floats rejected. Zero-copy when the width equals the canonical width and the scale is 0 | +| Compare | `scalar_fn/fns/binary/compare/decimal.rs` | Equal dtypes only. Widens mismatched widths (`:114-126`). Sign test for constants that do not fit (`:128-153`) | +| Arithmetic | `scalar_fn/fns/binary/numeric/decimal.rs` | Equal dtypes only (`scalar_fn/fns/binary/mod.rs:146-153`). Arrow result rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`). Checked at a working width, then narrowed (`numeric/decimal.rs:410-430`) | +| Between | `arrays/decimal/compute/between.rs` | Constant bounds only, storage-fit logic | +| Fill null | `arrays/decimal/compute/fill_null.rs:55-72` | Upcasts the whole array when the fill value does not fit | +| Mask | `arrays/decimal/compute/mask.rs` | Metadata-only | +| Patch | `arrays/decimal/array.rs:537-609` | Panics when the width cannot hold the dtype | +| Sum | `aggregate_fn/fns/sum/mod.rs:78-86`, `sum/decimal.rs:49` | Result precision p+10, capped at 76. Overflow gives null | +| Min, max | `aggregate_fn/fns/min_max/decimal.rs` | Per-width | +| is_sorted, is_constant | `aggregate_fn/fns/is_sorted/decimal.rs`, `is_constant/decimal.rs` | Per-width | +| all_non_distinct | `aggregate_fn/fns/all_non_distinct/decimal.rs` | Triple-nested width dispatch | +| Uncompressed size | `aggregate_fn/fns/uncompressed_size_in_bytes/decimal.rs:21` | Reports the canonical width, not the stored width | +| IN list | `array/haystack/mod.rs:118-125` | Converts to the canonical width | + +Paths in the table are relative to `vortex-array/src/`. Shared code also has decimal arms: + +- The row codec (`vortex-row/src/codec.rs:579-584`, `:822-860`). +- Sparse canonicalization (`encodings/sparse/src/canonical.rs:314-327`). +- Run-end decoding (`encodings/runend/src/compress.rs:358-388`). +- Constant canonicalization (`vortex-array/src/arrays/constant/vtable/canonical.rs:96-125`). +- Dictionary execution (`vortex-array/src/arrays/dict/execute.rs:53`, `:102-111`). +- Masked execution (`vortex-array/src/arrays/masked/execute.rs:54`, `:89-95`). + +### 2.8 Encodings and compression + +- DecimalByteParts (DBP) stores a signed most significant part (MSP) and up to three unsigned + 64-bit lower parts (`encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:6-15`). The + MSP holds validity (`decimal_byte_parts/array.rs:404-409`). Each part is a normal integer array + that any integer scheme can compress. +- DBP carries `DType::Decimal` itself (`decimal_byte_parts/array.rs:102-104`). So it is a decimal + storage encoding, not an integer encoding. +- Two wire ids exist. `vortex.decimal_byte_parts` (v1, MSP only) is in the first core edition. + `vortex.decimal_byte_parts.v2` (with lower parts) is in no edition yet + (`decimal_byte_parts/plugin/mod.rs:30-40`, + `vortex-edition/src/declarations/core/v2025_05.rs:30-31`). The plugin writes v1 whenever there + are no lower parts (`decimal_byte_parts/plugin/mod.rs:62-79`). +- DBP assembly with no lower parts reuses the MSP buffer, so the result width is the MSP width + for any precision (`decimal_byte_parts/assemble.rs:82-93`). +- The DBP compare kernel only handles the MSP-only case. A TODO describes the missing limb-wise + comparison (`decimal_byte_parts/compute/compare.rs`, top of `compare`). +- The btrblocks `DecimalScheme` narrows first (`vortex-btrblocks/src/schemes/decimal.rs:133`). + In v1 mode it returns the narrowed `vortex.decimal` array unchanged when values still need + 128 or 256 bits (`:134-137`). Files written today can therefore contain `vortex.decimal` + nodes whose width is smaller than the canonical width. + +### 2.9 Statistics and zone maps + +- Array statistics store `min`, `max`, and `sum` as serialized `ScalarValue` bytes + (`vortex-array/flatbuffers/vortex-array/array.fbs`, `table ArrayStats`). +- Sum statistics merge with a precision check (`vortex-array/src/stats/stats_set.rs:480-492`). +- Zone maps use the column dtype, falling back to the extension storage dtype + (`vortex-layout/src/layouts/zoned/schema.rs:112-121`, `:177-188`). +- The bloom filter aggregate rejects decimals and cites issue #5820 + (`vortex-layout/src/layouts/zoned/aggregates/bloom_filter/canonical/mod.rs:34-49`). + +### 2.10 Serialization and editions + +- Editions gate four component kinds: array ids, layout ids, extension dtype ids, and + aggregates (`vortex-edition/src/lib.rs:161-178`). +- Core dtype tags and PType values are not gated. The writer only checks extension ids inside + the schema (`vortex-file/src/writer.rs:429-475`). Union, Variant, and Map dtype tags reach the + wire without an edition check, although their array ids are gated. +- An unknown flatbuffer dtype tag or PType is a loud error for old readers + (`vortex-array/src/dtype/serde/flatbuffers.rs:326`, `:559`). An unknown protobuf oneof is + also loud ("Unrecognized DType", `vortex-array/src/dtype/serde/proto.rs:32`). An unknown + protobuf PType value is silent: `p.r#type()` returns `U8` (`proto.rs:36`). Protobuf dtypes are + persisted on disk in Variant shredding metadata + (`vortex-array/src/arrays/variant/vtable/mod.rs:54`) and Parquet-variant metadata + (`encodings/parquet-variant/src/vtable.rs:73`). +- Unknown extension dtype ids read as `ForeignExtDType`, which keeps the storage dtype and the + raw metadata (`vortex-array/src/dtype/extension/foreign.rs:26-75`). Old readers can read new + extension types as their storage. +- `ArrayPlugin` separates the in-memory id from the serialized ids. A plugin can read several + historical ids into one in-memory array. It can also write an older form as a "lossless + structural downgrade" (`vortex-array/src/array/plugin.rs:26-41`, `:101-157`). DBP uses this + today. +- Compatibility fixtures exist for decimals, DBP v1, and DBP v2 + (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/`). The decimal fixture only covers + canonical widths (i32 and i64). + +### 2.11 Engine mappings + +| Engine | Decimal mapping | Wide integers | +| --- | --- | --- | +| Arrow | Export Decimal128 for p <= 38 and Decimal256 above. Decimal32 and Decimal64 are commented out for DataFusion (`vortex-arrow/src/dtype.rs:333-347`). Import keeps the Arrow width zero-copy (`vortex-arrow/src/convert.rs:204-277`) | Arrow has no Int128 or Int256 | +| DuckDB | DECIMAL both ways (`vortex-duckdb/src/convert/dtype.rs:106-110`, `:237-241`). Widths 16, 32, 64, and 128 bits, zero-copy when equal (`vortex-duckdb/src/exporter/decimal.rs:122-130`). Import narrows i16 to i8 for p <= 2 (`convert/vector.rs:303-330`) | HUGEINT and UHUGEINT are rejected (`convert/dtype.rs:100-101`) | +| DataFusion | Scalars to Decimal32, 64, 128, or 256 by precision (`vortex-datafusion/src/convert/scalars.rs:57-110`) | No Int128 scalar | +| Python | `DecimalDType` class, Python `Decimal` scalars built from the unscaled value and exponent (`vortex-python/src/scalar/into_py.rs:153-171`). `DecimalArray` subclass is keyed by encoding id (`vortex-python/src/arrays/native.rs:174-175`) | NumPy has no 128-bit integer dtype | +| FFI | `DTYPE_DECIMAL = 8`. Scalar constructors and getters for i8 to i64 and 16-byte little-endian i128 (`vortex-ffi/src/scalar.rs:265-330`) | `vx_ptype` is a `#[non_exhaustive]` C enum (`vortex-ffi/src/ptype.rs:6-31`) | +| JNI | Statistics as `java.math.BigDecimal` (`vortex-jni/src/writer.rs:254-270`). Literals from big-endian bytes (`vortex-jni/src/expression.rs:511-545`) | Not applicable | +| CUDA | Arrow device widths 32, 64, 128, 256 (`vortex-cuda/src/arrow/mod.rs:840-849`). Decimal filter and DBP kernels | Separate C `PTypeTag` enum (`vortex-cuda/kernels/src/dynamic_dispatch.h:45-56`) | + +Four different width functions exist: + +- The Vortex canonical width, with an I8 minimum. +- DuckDB, with an I16 minimum. +- Arrow device export, with an I32 minimum. +- Arrow dtype export, with an I128 minimum below precision 39. + +### 2.12 The PType enum and its blast radius + +- `PType { U8 = 0, U16, U32, U64, I8, I16, I32, I64, F16, F32, F64 = 10 }`, `#[repr(u8)]`, + `prost::Enumeration` (`vortex-array/src/dtype/ptype.rs:33-60`). +- `NativePType` requires `Num`, `NumCast`, `FromPrimitiveOrF16`, byte conversions, and a seal + (`ptype.rs:124-140`). `IntegerPType` adds `PrimInt` and `AsPrimitive` (`ptype.rs:68`). +- `match_each_native_ptype!` is exhaustive over 11 arms. `match_each_integer_ptype!` and its + unsigned, signed, and float siblings end with `other => panic!` (`ptype.rs:355-737`). +- Counts at this commit (details in `ptype-match-inventory.md`): + +| Item | Count | +| --- | ---: | +| `match_each_native_ptype!` uses | 70 in 54 files | +| `match_each_integer_ptype!` uses | 169 in 94 files | +| `match_each_unsigned_integer_ptype!` uses | 118 in 53 files | +| Other PType macros (signed, float, SIMD) | 17 | +| Exhaustive PType matches outside `ptype.rs` (proxy) | about 45, 10 of them in tests | +| `is_int()` call sites | 53, about 40 of them guard offsets, codes, or indices | +| `NativePType` mentions | 598 in 169 files | +| Mirrors in other languages and formats | flatbuffer, protobuf, C header, C++, Python stub, Python enum, CUDA C enum and mapping, FFI enum, JNI codes, DuckDB, Arrow, DataFusion | + +- `is_int()` routes a type into 64-bit-only code in many places. Examples are list, list-view, + and varbin offsets, dictionary codes, take indices, and every btrblocks integer scheme + (`vortex-btrblocks/src/schemes/integer/*.rs`, where `matches` uses `is_int()`). If `is_int()` + returns true for I128, those guards admit wide integers, which then reach a + `match_each_integer_ptype!` panic. +- Primitive assumes alignment equals byte width in deserialization + (`vortex-array/src/arrays/primitive/vtable/mod.rs:158`) and in `from_values_byte_buffer` + (`vortex-array/src/arrays/primitive/array/mod.rs:505`). That is wrong for I256, whose Rust + alignment is 16, not 32. + +### 2.13 What is strange, in one list + +1. The canonical form has up to six physical shapes per dtype. The narrowing is compression that + leaked into the canonical form (issue #5820, Epic #7706). +2. The width that consumers need is a function of precision, but it is a convention repeated at + 25 call sites, not a contract. The row codec says so directly: keys use the declared dtype "so + keys from differently compressed chunks stay memcmp-comparable" + (`vortex-row/src/codec.rs:579-584`). +3. Kernels carry storage-fit special cases: constant sign tests in compare and between, the + fill-null upcast, the patch panic, widening in compare and arithmetic. +4. Arrays skip precision validation, scalars enforce it. Kernels must defend. +5. The scalar and array cast paths disagree for `f32` and `f16` targets. +6. Decimal duplicates Primitive in builders, kernels, aggregates, canonical dispatch, and + compression. Epic #7706 lists this as an open problem. +7. DBP is a wide-integer limb split that only decimals can use, and it carries a decimal dtype. +8. Decimal metadata decodes with a prost getter that turns unknown values into `I8`. +9. Bloom filters, IN lists, size statistics, and the row codec each have decimal special cases. +10. DuckDB HUGEINT cannot be imported although Vortex already stores i128 decimal lanes. +11. Decimal scalars on the wire carry the width of whatever variant the writer used, and + readers do not check precision. +12. Sparse canonicalization supports decimals with a dedicated path but rejects all extension + arrays (`encodings/sparse/src/canonical.rs:361`). No shared rule decides where structural + encodings sit relative to a type. X5 and XQ1 address this (section 3.3). + +## 3. Target: one canonical decimal leaf + +### 3.1 Definition + +The logical dtype does not change. `DType::Decimal(DecimalDType, Nullability)` keeps its Rust +variant and its wire tag (D1, section 3.8). + +`DecimalArray` stays a leaf (D2). It holds one values buffer and validity, and it has no data +child. This is also the shape of the `vortex.decimal` wire node: one buffer, an optional validity +child, and no data child (`vortex-array/src/arrays/decimal/vtable/mod.rs:155-170`). + +The precision fixes the width of the buffer (D3). The width function gets one owner: + +```rust +impl DecimalDType { + /// The width of the values buffer for this precision. + /// + /// This is the existing `DecimalType::smallest_decimal_value_type` (D12). + pub const fn canonical_values_type(&self) -> DecimalType { + match self.precision.get() { + 1..=2 => DecimalType::I8, + 3..=4 => DecimalType::I16, + 5..=9 => DecimalType::I32, + 10..=18 => DecimalType::I64, + 19..=38 => DecimalType::I128, + _ => DecimalType::I256, + } + } +} + +impl Array { + /// Fails unless `T::DECIMAL_TYPE` equals `decimal_dtype.canonical_values_type()`. + pub fn try_new( + buffer: Buffer, + decimal_dtype: DecimalDType, + validity: Validity, + ) -> VortexResult; + + /// The unscaled integers over the same buffer and validity, without a copy (section 3.2). + pub fn unscaled(&self) -> PrimitiveArray; +} +``` + +`try_new` exists today with this signature, but it accepts any width +(`vortex-array/src/arrays/decimal/array.rs:433-443`). `DecimalType` already has `I128` and `I256` +(`vortex-array/src/dtype/decimal/types.rs:25-42`). So the leaf needs no new `PType`, and the +one-canonical-form fix (node N11) does not wait for the wide integers (node N9). + +Invariants: + +- **I1, width.** The buffer width equals `decimal_dtype.canonical_values_type()`. + `DecimalData::validate` checks it next to the length and alignment checks of today + (`vortex-array/src/arrays/decimal/array.rs:301-319`). The check costs nothing for each row. +- **I2, value.** Every valid unscaled value `v` satisfies `|v| <= 10^p - 1`. `try_new` checks + every valid value, and `new_unchecked` skips that check for kernels that keep values legal + (D14, the rule of X7 for decimals). The Arrow, DuckDB, FFI, and Python imports use `try_new` + (D13). File reads trust the precision and never panic. Kernels that produce new values + (arithmetic, rescaling casts, sums) check their own outputs, as `numeric/decimal.rs` does today + with `DecimalValueBounds`. +- **I3, leaf.** The array has one buffer and one validity slot + (`vortex-array/src/arrays/decimal/array.rs:46-51`). It has no data child. + +The value of a row is `unscaled * 10^(-scale)`. Nothing else is stored. + +D12 keeps the I8 minimum. Section 2.11 lists three other width functions in the engines. A +function with an I128 minimum below precision 39 matches the Arrow export. It keeps Arrow +`Decimal128` imports zero-copy, but it costs up to 16 times the memory at small precisions. With an +engine width, no precision maps to `i8` or `i16`, so decimal `Narrow` children lose those widths. +From precision 5, the D12 width equals the width of the smallest Arrow decimal type +that holds the precision (section 3.7). + +### 3.2 Canonical form and the integer view + +`Canonical::Decimal(DecimalArray)` stays a variant. With I1, one decimal dtype has one canonical +form. So a kernel never receives a width that it does not expect. The `patch` panic and the "does +the constant fit" special cases go away (section 2.13, items 1 and 3). + +Issue #5820 also proposed a decimal array with an integer child "that is separately decompressed, +like list offsets". D2 rejects that shape for the canonical form. The narrow representation lives +in a separate `Narrow` encoding instead (section 3.3). + +The rest of this section is future work (D20), and its recommendations are not reviewed. The +`Narrow` work does not need the view, because `Narrow` kernels pass work to the decimal kernels of +the child (D15). + +The integer view is a `PrimitiveArray` over the same buffer and validity. It costs no copy. Its +dtype is `Primitive(I_w, nullability)`, where `I_w` is the signed integer type of the canonical +width: + +```rust +// Zero-copy. For `Decimal(9, 2)`, the dtype of the view is `Primitive(I32, nullability)`. +let unscaled: PrimitiveArray = decimal.unscaled(); +``` + +Decimal kernels delegate to integer kernels through the view. A result of an integer kernel +becomes a decimal again without a copy, when its buffer has the canonical width of the result +dtype. + +The view for precision 1 to 18 uses I8 to I64, which exist today. The view for precision 19 to 76 +needs I128 and I256 (D5, node N9). Until N9 lands, the kernels for those precisions stay +decimal-specific. Kernels can move onto the view one at a time, at any point in the order of work. + +For extension types, `ext_storage` is the explicit unwrap to the storage, and it stays. For +`Decimal`, the integer view plays the same role. + +### 3.3 Storage encodings and the placement rule + +#### `Narrow` + +A `Narrow` encoding stores a `Decimal(p, s)` column as one child of dtype `Decimal(p', s)` (D4). +The child follows three rules (D10): + +- It has the same scale and the same nullability as the parent. +- Its canonical width is strictly narrower than the canonical width of `p`. A `Narrow` from + `Decimal(76, s)` to `Decimal(75, s)` is not allowed, because both use `i256`. The integer + `Narrow` has the same rule: its `validate_dtypes` requires a strictly smaller byte width + (`vortex-array/src/arrays/narrow/mod.rs:81-84` in the local rewrite of #10223). The rule can be + relaxed later. +- The producer chooses `p'`: the best precision that the values and the scale allow, or a larger + one. + +`DecimalDType::try_new` rejects a positive scale larger than the precision +(`vortex-array/src/dtype/decimal/mod.rs:68`), so `p'` is at least `s`. The narrowest child is `i8` +for a scale up to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, and `i128` for 19 to +38. This limit matters only for a column whose values all have fewer unscaled digits than the +scale. arrow-rs has the same scale rule (`arrow-array-59.3.0/src/types.rs:1436`). + +`Narrow` has three properties: + +- It has no buffers. Validity comes from the child. +- It decodes by widening the child buffer to the canonical width of `p`. +- It needs no value check, because a legal `Decimal(p', s)` value is a legal `Decimal(p, s)` value. + +One `Narrow` encoding, with one wire ID `vortex.narrow`, serves integers and decimals (D8). +Integers narrow by width (issue #10020), and decimals narrow by precision. An integer child has +the same signedness and nullability as its parent, and a strictly smaller width (D22). This is +the rule of `validate_dtypes` in the local rewrite. `vortex.narrow` stays in memory only until the +maintainer decides to write it to files (D18). + +`vortex.narrow` is for arrays with a fixed logical dtype (D23). Which arrays the compressor wraps is +deferred (D27). Internal integer children keep changing their own dtype. Today they use +`PrimitiveArrayExt::narrow`, and NR6 moves them to the deprecated `encode_values` (D28). The +btrblocks schemes call it for internal children of the patch, sparse, RLE, temporal, VarBin, FSST +and OnPair schemes (`vortex-btrblocks/src/schemes/{patches,temporal}.rs`, `integer/{sparse,rle}.rs`, +`float/sparse.rs`, `binary/varbin.rs`, `string/{fsst,onpair,sparse}.rs`). The structural compressor +calls it for list offsets (`vortex-compressor/src/compressor/structural.rs:54`). No call narrows a +DBP part. Issue #10733 keeps open whether `List` and `ListView` offsets get a fixed `I64` dtype with +`Narrow`. + +`Narrow` kernels pass work to the child, and the fallback is to canonicalize (D15): + +| Operation | Kernel on a decimal `Narrow` | +| --- | --- | +| Compare with a constant | A constant outside the range of the child dtype gives the answer without the values. Otherwise, the child is compared with the constant at the child precision | +| Min and max | The result of the child, as a value of the parent dtype | +| Sum | The sum of the child, with the result dtype of the parent (`sum_decimal_dtype`, `aggregate_fn/fns/sum/mod.rs:81-86`) | +| Filter, take and slice | Run on the child | +| Cast | Passes to the child with the same target dtype. A `Narrow` holds exactly the values of its child, and a cast depends only on the values and the target dtype | + +A `Narrow` over a `Narrow` becomes one `Narrow`. Another decimal cast to a precision with the same +scale returns a `Narrow` over its input when the canonical width grows. It changes only the dtype +when the width stays the same, and it checks every value when the precision drops. To widen an +array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it. + +Precision and integer width do not line up, because precision limits are powers of ten and widths +are powers of two. The table gives the largest precision for each width: + +| Width | Largest precision for the width | Largest unscaled value at that precision | Largest value of the width | +| --- | --- | --- | --- | +| `i8` | 2 | 99 | 127 | +| `i16` | 4 | 9,999 | 32,767 | +| `i32` | 9 | 999,999,999 | 2,147,483,647 | +| `i64` | 18 | 10^18 - 1 | about 9.2 * 10^18 | +| `i128` | 38 | 10^38 - 1 | about 1.7 * 10^38 | + +A gap value fits the width but needs a precision of the next width. For example, 2,000,000,000 +fits `i32`, but it needs precision 10, whose canonical width is `i64`. The largest value of each +width is always a gap value. So a narrow buffer alone does not give `p'`, and each producer finds +`p'` from the values. + +A false label gives wrong results, because the `Narrow` rules trust the child dtype. In the +`Narrow` stack, the compare kernel answers a constant that does not cast to the child dtype +without reading the values (`vortex-array/src/arrays/narrow/compare.rs`). For integers this is +correct, because an `i32` child never holds a value above `i32::MAX`. For a decimal child with a +false label, `x < 15,000,000.00` on a `Decimal(9, 2)` child returns true for the row +20,000,000.00. The cast rule passes a cast to the child when the child fits the target +(`narrow/rules.rs`). So a cast to `Decimal(9, 2)` returns that child unchecked. + +In the final state, a gap column uses the next width in memory (D19). For example, a +`Decimal(10, 2)` column with values from 100.00 to 327.67 has unscaled values from 10,000 to +32,767. Today DBP decodes it to an `i16` buffer. In the final state, its child has precision 5 and +an `i32` buffer. Disk size does not change, and exports already widen. Pushdown of sum, min and max +into the MSP can remove the cost for those operations later. + +#### Reading old narrow data + +Old writers chose the width from the values. `narrowed_decimal` +(`vortex-array/src/arrays/decimal/utils.rs:124-143`) picks the smallest integer type that holds +the minimum and the maximum, not the width of a lower precision. + +- **Detection (D7).** A reader compares the stored `values_type` with the canonical width of the + dtype. This costs nothing for each row, and it does not depend on which writer wrote the node. + An unknown `values_type` must fail first (section 8.3, P1), or the reader treats it as an old + `i8` node. +- **Narrower node (D9).** The reader scans the non-null values for their max. It chooses the best + precision that the values and the scale allow, or a larger one. If the canonical width of that + precision is narrower than the canonical width of the column, the result is a `Narrow` over a + child at that width. Otherwise, the result is the canonical array. The result reuses the buffer + when its width is the stored width, and it copies otherwise. Null slots can hold any value + (D14), so they do not count. +- **Wider node (D9).** The reader makes a narrowing copy (D21). +- **Narrowing copies (D21).** Every narrowing copy writes zeros in null slots, and it fails on a + valid value outside the precision of the target dtype. Three paths make one: a wider node, a + narrower node whose best precision has a smaller width than the stored width, and DBP decoding + with an MSP wider than the canonical width. The copy reads every value anyway, so the precision + check costs nothing extra. A read without a copy trusts the precision. A buffer narrower than + the canonical width cannot hold a value above the precision, because the largest value of each + narrower width is below `10^p - 1`. + +In a gap, the result is a copy. An old `Decimal(4, 2)` node with an `i8` buffer and values up to +1.27 holds unscaled values up to 127, which need precision 3. Precision 3 and precision 4 both use +`i16`, so no legal array keeps the `i8` buffer. The result is a plain `Decimal(4, 2)` array at +`i16`. + +The `vortex.decimal` read plugin returns a `Narrow` array. So it overrides `is_supported_encoding` +(`vortex-array/src/array/plugin.rs:154`), as the ALP plugin does for `Patched` +(`encodings/alp/src/alp/plugin.rs:90`). The deserializer asserts this +(`vortex-array/src/serde.rs:413`). `Decimal` and `Narrow` both live in `vortex-array`, so no other +crate needs to replace the plugin. + +The plugin does not see stored statistics. The deserializer attaches them after the plugin returns +(`vortex-array/src/serde.rs:419-424`). So the plugin scans the buffer, which is already in memory +(D9). A change that passes the statistics to plugins would change the `ArrayPlugin` trait for +every plugin, and the maintainer rejected it for this rare path. + +Such nodes are rare, and the speed of the conversion matters little (D9). The btrblocks +`DecimalScheme` always applies (`vortex-btrblocks/src/schemes/decimal.rs:122`). It narrows first +(`:133`), returns the narrowed leaf only for `I128` and `I256` in v1 mode (`:134-137`), and puts +every other width into DBP. This routing exists since commit `1b89b2f950` (2025-05-28). The compact +compressor did the same from commit `0036b1566d` (2025-09-08). On `develop` it is no longer +separate code. `with_compact` keeps the same `DecimalScheme` and changes only the excluded +schemes: the default mode excludes Pco, and the compact mode allows it +(`vortex-btrblocks/src/builder.rs:46-63`, `:190-193`). So a narrow `vortex.decimal` node is `I128` +at precision 39 to 76, or it comes from a write that skipped the decimal scheme. + +#### The integer `Narrow` PRs + +The integer `Narrow` was a draft stack: PRs #10223, #10332 and #10333. The maintainer closed all +three on 2026-10-23, and new PRs replace them. Section 8.3 and the decisions file give the state on +2026-10-23. A local rewrite of #10223 and #10332 makes `Narrow` in-memory only. The new PRs are +written from scratch, with that rewrite as a reference for tests and edge cases (D38). The rewrite +assumes integers in more places than `validate_dtypes`, so the new code avoids those assumptions +where that costs little, and CD4 handles the rest. + +The closed PR #10333 adds `vortex.narrow` to a new draft edition `preview2026.10.0`, with one +metadata byte that holds the child `PType`. Under D8 and D32, the metadata is a prost message with +a typed `oneof`: the child `PType` for an integer parent, or the child precision for a decimal +parent. A missing variant is an error. Every release that permits an ID must read every valid +instance of it (invariant 6 in `docs/specs/versioning/design.md`). So the decimal reader ships in +the same release as the first edition that permits `vortex.narrow`. The second part of #10333 +comes back as a new PR in the wire change, which puts `vortex.narrow` and DBP v2 into one preview +edition (D11, D18). + +#### DecimalByteParts + +DBP stays the compressed decimal form. Its semantics: + +- **Value.** Each unscaled value is its two's complement bits, cut into 64-bit words. The value + is `msp * 2^(64k) + lower[0] * 2^(64(k-1)) + ... + lower[k-1]`, with `k` from 0 to 3 + (`encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:6-16`). +- **Parts.** The most significant part (MSP) is any signed integer up to 64 bits, with the + nullability of the decimal, and it holds the validity (`array.rs:97-99`). Each lower part is a + non-nullable `u64` (D16), and `split_decimal` writes zeros in its null rows. Today the + constructor accepts a lower part of any unsigned width (`array.rs:111-118`), and the module doc + says that each part can use a narrower dtype (`mod.rs:16`). But `split_decimal` makes only `u64` + parts (`split.rs:68-71`), and the compressor never narrows the dtype of a part. D16 tightens the + constructor and the doc. No file has lower parts: no edition permits v2, and the v2 compat + fixture is disabled (`vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs:47`). + So the change breaks no file. Each part can use any integer encoding. The parts are integers, so + they promise only a width, never a number of digits. +- **Split.** `split_decimal` (`split.rs:98`) keeps a buffer of `i8` to `i64` as the MSP, with no + lower parts (`split.rs:60`). It splits `i128` into an `i64` MSP and one `u64` part, and `i256` + into an `i64` MSP and three `u64` parts. +- **Wire.** One in-memory array, whose ID is `vortex.decimal_byte_parts.v2` + (`plugin/mod.rs:38`), has two wire forms. v1 (`vortex.decimal_byte_parts`, `plugin/mod.rs:32`) + has exactly one child, and its metadata holds the MSP `PType` (`plugin/v1.rs:28`, `:70`). It is + in `core2025.05.0`. v2 adds the lower parts and their `PType`s, and no edition permits it yet. + The serializer picks v1 when there are no lower parts (`plugin/mod.rs:73`). +- **Decoding.** DBP is not canonical. `execute` decodes each part to a primitive array + (`array.rs:289-301`) and assembles a decimal leaf. With no lower parts, the leaf reuses the MSP + buffer at the MSP width (`assemble.rs:82-93`). One lower part gives `i128`, and two or three give + `i256`. +- **Kernels.** Cast (nullability only), filter, mask, slice and take reduce into the parts + (`rules.rs:14-18`). Compare and between run on the parts (`compute/kernel.rs:16-25`). Every + other operation, such as sum, min, max and arithmetic, decodes DBP first. + +The compressor narrows the values before it splits them. So DBP for `Decimal(18, 2)` often has an +`i32` MSP, and its decoded result is a leaf with an `i32` buffer. D3 removes that form. Under D3: + +- **Reading.** No change. A narrow MSP is already correct, because the MSP is an integer (D16). +- **Decoding (D16).** Decoding produces the canonical array of the DBP dtype. It widens an MSP + that is narrower than that width. For precision up to 18, the MSP rule also allows an MSP wider + than the canonical width, and decoding narrows it with the copy of D21. The btrblocks scheme + never writes such an MSP, because it splits the narrowed buffer. A `TODO(connor)` in + `assemble.rs` marks a later optimization: find the max of the MSP, and return a `Narrow` that + reuses its buffer. It waits until benchmarks show large regressions. +- **Operations (D16).** They run on the parts wherever they can. Decoding is the fallback. +- **Writing (D17).** The compressor has one DBP scheme. It takes the canonical array, uses the + value range to choose the number of lower parts (v1 has none), and narrows the MSP dtype as the + compressor narrows every internal child. The parts get the same widths as today, so file + contents do not change. The scheme stays after the wire change, for writers whose edition does + not permit `vortex.narrow`. When an edition permits it, a decimal `Narrow` scheme runs first, and + the DBP scheme compresses the child. The result is a `Narrow` of dtype `Decimal(p, s)` over a + DBP child of dtype `Decimal(p', s)`. The MSP of that child usually has the canonical width of + `p'`, so decoding reuses its buffer. In the gaps, decoding widens it. + +The scheme cannot keep `narrowed_decimal`. Today the scheme narrows the canonical array +(`vortex-btrblocks/src/schemes/decimal.rs:133`) and returns the narrowed leaf for `I128` and `I256` +in v1 mode (`:134-137`). Under D3 and D14, that leaf is illegal in memory. + +So DBP decoding needs no check of the values, and DBP needs no conversion at read time. + +Until the compressor narrows the decimal type first, decoding a compressed decimal runs at the full +width in memory. This applies to existing files, and to files from writers that do not select an +edition with `vortex.narrow`. The maintainer accepts this cost, because the bar of no compute +regression applies to the final state (D19). + +The stored statistics of the MSP do not help here. A review on 2026-10-23 traced them, and the +2026-10-22 recommendation DQ8, to always store min and max on the MSP, is dropped (D20): + +- The compressor computes min and max on the canonical MSP to estimate the integer schemes + (`vortex-btrblocks/src/schemes/integer/bitpacking.rs:129`, `for_.rs:155`). +- A BitPacked MSP keeps them (`bitpacking.rs:155`, `:201-206`). +- A FoR MSP has none, because the FoR node copies statistics from a newly built array + (`for_.rs:233-234`). Its inner BitPacked child holds the min and max of the shifted values only. +- A Pco MSP has none (`vortex-btrblocks/src/schemes/integer/pco.rs:55-69`). +- The layout writer computes statistics on the uncompressed chunk, but it does not pass them to + the DBP node or its MSP (`vortex-layout/src/layouts/compressed.rs:73`, `:114-127`). +- A read of part of a chunk slices it (`vortex-layout/src/layouts/flat/reader.rs:150-152`), and a + slice keeps only IsConstant, IsSorted and IsStrictSorted (`vortex-array/src/array/erased.rs:238-249`). +- FastLanes registers no min or max kernel, so `min_max` on a FoR or BitPacked array without + statistics decompresses it (`encodings/fastlanes/src/lib.rs:77-91`). + +The DBP cast rule has a bug that is independent of D3. For a cast that changes only the +nullability, it casts the MSP for every target nullability (`compute/cast.rs:16-35`). The contract +of cast rules in the optimizer asks them to call `Validity::trivially_cast_nullability`, to return +`None` when that call cannot prove the cast safe, and to pair the rule with a cast kernel for +execution (`vortex-array/src/validity.rs:447-500`). DBP has no cast kernel (`rules.rs:14`). PR +#10230 added the guard in 8 lines. The fix lands as its own PR, with a cast kernel and tests, and +it needs no issue (section 8.3, P2). Whether the current rule gives a wrong validity before +execution is not verified. + +#### Placement of structural encodings + +Rule X5 pulls generic structural arrays out from under the extension wrapper. Recommendation XQ1 +names them: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. So +generic structural encodings sit above the type layer. The decimal storage encodings, DBP and +`Narrow`, sit below them. For `Narrow`, reduce rules move these seven encodings above it (D25). +The rewrite's chunked and interleave merge rules, which put `Narrow` on top, are dropped. + +For `Decimal`, the wire forms of today already follow this rule: + +- The compressor writes a constant decimal column as `vortex.constant` with a decimal scalar + (`vortex-compressor/src/compressor/cascade.rs:272-273`). When some rows are null, it writes + `vortex.masked` over that node (`vortex-compressor/src/compressor/constant/mod.rs:175-193`). +- Run-end arrays already decode decimal values (`encodings/runend/src/array.rs:488-494`). +- A dictionary over decimal values works (`vortex-array/src/arrays/dict/execute.rs:102-111`). +- Sparse and masked arrays have decimal paths (`encodings/sparse/src/canonical.rs:314-327`, + `vortex-array/src/arrays/masked/execute.rs:89-95`). + +So `Decimal` needs no write plugin that translates an in-memory form back to the legacy forms. An +earlier version of these notes needed one for a wrapper design. + +`DateTimeParts` is the temporal twin of DBP. It is a storage encoding below the structural +encodings, and it carries the extension dtype today +(`encodings/datetime-parts/src/array.rs:294-305`). + +Probe R6 in the extension notes checks the related `ext_storage` pushdown rules. Existing rules +push `ext_storage` into dictionary values and chunks. `ext_storage` decodes a `Masked` input and +fails on a `Sparse` input with extension values. + +### 3.4 Decimal semantics as rewrite rules + +This section is future work (D20), and its recommendations are not reviewed. The casts between +precisions with the same scale are settled in D15 (section 3.3). + +`Decimal` implements its semantics as rewrite rules on arrays that the type owns (X4). Each rule +runs integer kernels on the integer view and builds a decimal result. There is no `ext_wrap` +expression, because a rule builds its result array directly. + +Three settled rules frame the table. Vortex runs an operation only when the operand dtypes already +match, and engines convert operands to a common dtype (X3). A cast exists only where a rule for the +pair of dtypes exists (C1). `Decimal` ships built-in rules for the common pairs below. + +| Operation on `Decimal(p, s)` | Rule over the integer view | Notes | +| --- | --- | --- | +| Compare, same `(p, s)` | `compare(view(a), view(b))` | A derive for one dtype (X2). Valid because the unscaled value is unique within one `(p, s)` after D3 | +| Compare across `(p1, s1)` and `(p2, s2)` | None in Vortex | The engine casts both operands to one dtype first (X3). Rejected today (`scalar_fn/fns/binary/mod.rs:162-168`) | +| Compare a column cast to a larger scale with a constant | Fold the cast into the constant and adjust the operator. `cast(x, s = 3) < 1.005` at `s = 2` becomes `view(x) <= 100` | An optimization for engine casts. Needs floor and ceiling rounding | +| Add, subtract, same dtype | Widen to the result width, checked operation, precision check | Arrow rules (`scalar/typed_view/decimal/arithmetic.rs:49-101`). Equal dtypes only today (`scalar_fn/fns/binary/mod.rs:146-153`) | +| Arithmetic across `(p, s)` | Open (DQ6). Recommendation: accept it, rescale to the larger scale, and use the Arrow rules for the result type | Comparisons keep the rule of equal dtypes | +| Multiply | Widen, checked multiply, precision check | Result scale `s1 + s2` | +| Divide | Multiply the dividend by `10^k`, checked divide toward zero, precision check | As today | +| Cast to `Decimal(p2, s2)` | Rescale with exactness, integer width cast, precision check | Replaces `DecimalCastPlan`. A wider precision with the same scale needs no copy. It relabels the buffer, or returns a `Narrow` over the source when the width grows | +| Cast to an integer | For a positive scale, `check_range` on the unscaled values with the inclusive bounds `MIN · 10^s` and `MAX · 10^s` of the target, then divide by `10^s` toward zero. A bound that does not fit the storage width saturates. For a negative scale, multiply by `10^-s` with a checked multiply at a width that holds the result. Then do a checked narrowing to the target. A zero value needs no factor, so it succeeds even when `10^-s` fits no integer width | Replaces `DecimalToIntegerCast` and keeps its rule: a value outside the range fails before truncation, so `127.1` fails for `i8` (`scalar/typed_view/decimal/integer_cast.rs:14-53`, test `cast_decimal_to_integer_bounds` in `arrays/decimal/compute/cast.rs:651`). The current code computes in `i256` and rejects a factor `10^s` that does not fit (`integer_cast.rs:37-39`). Under recommendation CQ1, truncation becomes a separate `trunc` step | +| Cast to a float | Integer to `f64`, divide by `10^s`, then narrow to `f32` or `f16` | One path for scalars and arrays. CQ1 also decides whether this rounding cast stays a default cast | +| Cast from an integer | Widen, multiply by `10^s`, precision check | Replaces `cast_to_decimal` | +| Sum | Integer sum with accumulator `canonical_values_type(min(p + 10, 76))`, precision check | `sum_decimal_dtype` (`aggregate_fn/fns/sum/mod.rs:81-86`) | +| Min, max, min_max, bounded_min, bounded_max | Integer aggregate on the view, then a decimal of the input dtype for each result value | Matches `min_max/extension.rs:17-35`. `min_max` returns a struct of two values. Truncation applies only to `Utf8` and `Binary`, so the bounded variants return exact values for decimals | +| is_sorted, is_constant | Integer aggregate on the view. The `Bool` result stays as it is | Valid within one `(p, s)` | +| Hash, group by, distinct, join keys | Integer hash and equality on the view | Valid within one `(p, s)` | +| Zone-map pruning | Rewrite the predicate onto the integer statistics | Decimal statistics hold the same integers. Recommendation XQ2 writes rules over `ReduceNode`, so they also rewrite bound expressions for pruning | +| IN list | Integer set over the needles, which have the column dtype (X3) | Replaces `haystack/mod.rs:118-125` | +| Row encoding | The canonical width, big-endian with the sign bit flipped | Already done (`vortex-row/src/codec.rs:19`, `:1466-1480`) | + +These rules need a small set of integer kernels that do not exist as reusable pieces today. +They are the scaled-integer family (node N8): + +```rust +/// Rounding when a rescale divides by a power of ten. +pub enum RescaleRounding { + /// Fail when a nonzero remainder would be dropped. + Exact, + /// Round toward zero. Used by decimal to integer casts and by division. + TowardZero, + /// Round toward negative infinity. Used for upper bounds of folded predicates. + Floor, + /// Round toward positive infinity. Used for lower bounds of folded predicates. + Ceil, +} + +/// Options of the `vortex.int.rescale` scalar function. +pub struct RescaleOptions { + /// Positive to multiply by 10^delta, negative to divide by 10^-delta. + pub delta: i16, + pub rounding: RescaleRounding, + /// The integer type of the result. The kernel fails on overflow. + pub target: PType, +} + +/// Options of the `vortex.int.check_range` scalar function, which fails when a valid value +/// falls outside `[min, max]` and otherwise returns its input unchanged. +pub struct CheckRangeOptions { + pub min: Scalar, + pub max: Scalar, +} +``` + +Two points matter for the design: + +- **Temporal types need the same kernels.** Timestamp, Time, and Duration units are powers of + ten apart, except Days. A Timestamp cast from milliseconds to nanoseconds (issue #8570) is + `rescale(delta = 6)`. A Duration cast across units (PR #10467) needs the same rescale as a + decimal cast across scales. One kernel family serves both. +- **Fusion keeps performance.** The current decimal arithmetic kernel checks the operation and + the precision bound in one pass (`numeric/decimal.rs`, `DecimalOpConstants`). A chain of + separate array operations makes several passes. So generalize the fused kernel from + `DecimalValue` widths to integer PTypes, with optional pre-scale factors and bounds. Plain + integer arithmetic is then the special case with type bounds and unit factors. + +### 3.5 Wire compatibility + +The leaf is the `vortex.decimal` wire node, so it needs no new wire ID and no new dtype tag +(D7). The versioning spec asks for a new ID only for "a change to what a reader must understand" +(`docs/specs/versioning/design.md`). D3 removes forms and adds none: + +- Every canonical leaf is a valid instance of the frozen `vortex.decimal` contract, so old readers + read new files. +- The format selection rule makes a serializer choose "the oldest supported writable format that + preserves the array's representation". For a canonical leaf that is always `vortex.decimal`, so + a writer never selects a v2 ID. +- Invariant 3 keeps readers of frozen formats forever, so a v2 ID removes no reader code. +- The existing v2 IDs each add a shape that old readers cannot parse. `fastlanes.for.v2` stores a + reference for each chunk as a child, and `fastlanes.bitpacked.v2` gives each block its own bit + width (`encodings/fastlanes/src/{for_,bitpacking}/plugin/mod.rs`). + `vortex.decimal_byte_parts.v2` adds lower parts. D3 is the opposite case. + +The only new ID is `vortex.narrow` (D8). The `ArrayPlugin` mechanism already lets one plugin read +several historical wire forms, and return an in-memory array of another encoding +(`vortex-array/src/array/plugin.rs:102-157`). + +Reading: + +| Wire node | In-memory result | +| --- | --- | +| `vortex.decimal`, `values_type` equals the canonical width | The leaf, zero-copy | +| `vortex.decimal`, `values_type` narrower (compressor narrowing) | The conversion of D9: a `Narrow` over a child with the best precision, or the canonical array in a gap (section 3.3) | +| `vortex.decimal`, `values_type` wider (Arrow Decimal128 data at p 5, written uncompressed) | The leaf after the narrowing copy of D21, with zeros in null slots. A value outside the precision is an error, not a panic | +| `vortex.decimal`, unknown `values_type` | Error. Decode with `DecimalType::try_from(raw)`, not the prost getter | +| `vortex.decimal_byte_parts` (v1) and `vortex.decimal_byte_parts.v2` | DBP, unchanged. Decoding widens a narrower MSP (D16) and narrows a wider one (D21) | +| `vortex.narrow`, after the wire change | `Narrow` over its child | + +Writing: + +| In-memory shape | Wire node | Readable by old readers | +| --- | --- | --- | +| The leaf | `vortex.decimal`, `values_type` equals the canonical width | Yes | +| DBP with an MSP only | `vortex.decimal_byte_parts` v1 | Yes, core 2025.05 | +| DBP with lower parts | `vortex.decimal_byte_parts.v2` | Only when an edition allows v2. It enters one preview edition together with `vortex.narrow` (D18) | +| `Narrow` | `vortex.narrow` | Only when an edition allows it. Otherwise the compressor does not produce it (D17) | +| `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked` or `Masked` over decimal values | The same nodes as today | Yes | + +Consequences: + +- Old readers read every file that a new writer produces, with the same edition rules as today. +- Wide PTypes never appear on the wire for decimal columns. The leaf carries `DecimalType`, and + DBP carries lane-integer PTypes. Only top-level wide-integer columns (DuckDB HUGEINT and + UHUGEINT, node N27) need new wire values and edition gating. +- Decimal statistics and zone-map entries keep the `Decimal` dtype tag. They stay readable. +- A writer without an edition for `Narrow` or DBP v2 loses one compact form. Today the v1 + compressor writes `vortex.decimal` at I128 for precision 39 to 76 when the values need more than + 64 bits (`vortex-btrblocks/src/schemes/decimal.rs:133-137`). Under D3, that writer stores I256 + values (D17). The `core` family holds the formats of the default writer + (`docs/specs/versioning/using-editions.md:47`), so default writers pay this cost until + `vortex.narrow` and DBP v2 enter `core`. A writer that selects the preview edition of D18 uses + DBP v2 instead. +- Extend the compat fixtures before the reader changes (D24). The harness publishes fixtures from + a released version, and the current reader must read all of them + (`vortex-test/compat-gen/README.md`). Only the old writer can make narrow leaves, so the new + fixtures are published from a release before the in-memory change. Add narrowed widths (I8 at + p 10) and wide widths (I128 at p 20, I256 at p 40). Also add Arrow-style wider widths (I128 at + p 5) and the narrow 128-bit form (I128 at p 40). Add gap values, such as 100 in an I8 buffer at + p 10, so that the copy path of D9 has a fixture. Add a high scale at a narrow width, such as I8 + at `Decimal(18, 10)`. Add nonzero values in null slots, and DBP with a narrow MSP. +- Test the error cases with unit tests, not compat fixtures, because the harness expects every + read to succeed. Examples are a value outside the precision in a narrowing copy, and an unknown + `values_type`. + +### 3.6 What happens to `values_type` narrowing + +Narrowing leaves the canonical form. Three mechanisms replace it: + +- **Parts.** One DBP scheme builds the parts from the canonical array, as narrow as the values + allow (D17). The parts get the same widths as today, so file contents do not change. +- **Precision.** Once an edition permits `vortex.narrow`, the compressor narrows the decimal type + first. A `Narrow` stores the column at a lower precision with the same scale (section 3.3). The + compressor finds `p'` from the values. +- **Bits.** The integer encodings compress the MSP of DBP. FoR with BitPacking, ZigZag, Delta, + RLE, and Dict already narrow integers better than a fixed smaller width does. + +`narrowed_decimal` and the first step of the btrblocks `DecimalScheme` go away (section 8.3). +Legacy reads keep working. A reader converts old narrow leaves into a `Narrow` or a copy (D9). Old +DBP arrays read unchanged, and their decoding widens a narrow MSP (D16). + +`Narrow` replaces the lazy `widen` and `checked_narrow` nodes of an earlier version of these notes. +It carries the generic comparison rule that DBP already has +(`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs`). A comparison of a +narrow column with a constant compares the child when the constant fits the child. Otherwise the +result is a constant. PR #10223 adds this rule for integers. The rule helps every narrow integer +column, not only decimals. For a decimal child, the rule is correct only when the child dtype is +true. Every producer of a decimal `Narrow` finds the child precision from the values (D9, D10). + +### 3.7 Builders, scalars, engines, and the public API + +- **Builder.** `DecimalBuilder` takes its width from the dtype and never changes it. The + six-variant buffer enum (`vortex-array/src/builders/decimal.rs:38-56`) can stay, because the + width depends on the dtype. `append_value::` converts with a checked cast and fails when the + value is outside the precision. The generic builder factory already picks the canonical width + (`vortex-array/src/builders/mod.rs:412-416`). +- **Scalars.** `DecimalScalar` keeps its API. `DecimalValue` is normalized to the canonical width + at construction with the existing `normalize` + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:208-216`). `scalar_at` reads the buffer + at the canonical width. +- **Accessors.** `DecimalArray::buffer::()` keeps its signature, and `T` is always the type of + the canonical width. `values_type()` always returns the canonical width. `unscaled()` is new and + returns the integer view (future work, D20). `DecimalArray::new::` fails for any other `T` + (step 6 of the in-memory change in section 8.3). +- **Engines.** The Arrow, DuckDB, and CUDA exporters read the leaf as today. Arrow export already + widens buffers below 128 bits for `Decimal128` (`vortex-arrow/src/executor/decimal.rs:152-165`). + It uses `Decimal128` for every precision up to 38, because DataFusion does not fully support + `Decimal32` and `Decimal64` (`vortex-arrow/src/dtype.rs:337-346`). Arrow import copies when the + Arrow width differs from the canonical width, for example for `Decimal128` data at precision 18 + or below. Today that import is zero-copy (`vortex-arrow/src/convert.rs:204-277`). D13 accepts + this cost. The DuckDB export uses `i16` for precision 1 to 4 + (`vortex-duckdb/src/exporter/decimal.rs:124`). +- **Arrow model.** Arrow has the same model as Vortex: a precision, a scale, and unscaled two's + complement integers. In Arrow, the width is part of the type. `Decimal32`, `Decimal64`, + `Decimal128` and `Decimal256` hold precision up to 9, 18, 38 and 76 + (`arrow-data-59.3.1/src/decimal.rs:894`, `:820`, `:706`, `:56`). From precision 5, the D12 width + equals the width of the smallest Arrow type for the precision. Arrow has no 8-bit or 16-bit + decimal. The format comment that arrow-rs ships still says that only 128-bit and 256-bit values + are used (`arrow-ipc-59.3.1/src/gen/Schema.rs:3100-3102`). +- **Arrow gap.** Arrow has the same gap at the top of each width. arrow-rs does not check decimal + values when it builds an array: `with_precision_and_scale` checks only the precision and scale + (`arrow-array-59.3.0/src/array/primitive_array.rs:1642-1648`). `validate_decimal_precision` + checks the values (`:1652`), and `null_if_overflow_precision` turns invalid values into nulls + (`:1671`). So the Vortex import fails on such values by default (D13), and the Arrow import + logic can be configured to handle them in another way. +- **Python.** The binding keys its `DecimalArray` subclass on the in-memory encoding id + (`vortex-python/src/arrays/native.rs:174-175`). The id stays `vortex.decimal`. + +### 3.8 Decimal stays a built-in type + +D1 settles this question. `Decimal` stays a built-in type, and it behaves like a hardcoded +extension type: + +- It is opaque by default. No code applies integer semantics to a decimal value unless a rule of + the type says so, as X1 requires of extension types. +- It derives `eq` and `ord` for one `(p, s)`, as X2 describes. After D3, the unscaled value is + unique within one dtype, so integer equality and order on the view are value equality and order. +- Its other operations are rewrite rules that the type owns (X4, section 3.4). + +A move to an extension type adds no capability. If the built-in type and an extension type become +identical, a later change can remove the built-in type. That step is optional (node N15). + +RFC 0062 (branch `ngates/types`, `rfcs/0062-value-types-and-representations.md`) proposes +`vortex.decimal` as a well-known extension with signed-integer storage. It states the precondition +directly: "Moving Decimal requires integer storage capable of its currently supported precisions, +including wide lanes." D1 does not adopt that move. + +If the optional move happens, it waits for the extension fixes. Today an extension array cast to +any other dtype casts its storage (`vortex-array/src/arrays/extension/compute/cast.rs:17-22`). A +scalar cast into an extension casts to the storage and rewraps +(`vortex-array/src/scalar/cast.rs:56-61`). A comparison with a lone extension side compares raw +storage (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-194`). If Decimal moves now, a +cast from `decimal(10, 2)` to `i64` returns the unscaled integers, and `price < 5` compares `500` +with `5`. So the move waits for nodes N1 to N4. The API cost is large: 269 lines in 97 files +mention `DType::Decimal`. Built-in registration is not an obstacle, because Date, Time, and +Timestamp are already default extensions (`vortex-array/src/dtype/session.rs:36-39`). + +## 4. The buffer type of the leaf + +D2 and D3 fix the shape of the leaf: one buffer, with one width for each precision. This section +keeps the earlier comparison of integer and FixedSizeBinary representations as background. The +verdict now concerns the type of the values in that buffer, and the type of the view over it. + +### 4.1 The options + +- **(a) Integer lanes with an integer view.** The buffer holds little-endian signed integers, as + today. The view is `Primitive(I8 .. I256)`, which needs `I128` and `I256` in `PType`. +- **(b) FixedSizeBinary values.** The buffer holds `FixedSizeBinary(n)` values with `n` in 1, 2, + 4, 8, 16, 32, or only 16 and 32 for the wide widths. +- **(c) Hybrid.** Integer lanes up to 64 bits, `FixedSizeBinary(16)` or `(32)` above. +- **(d) Separate wide-integer family.** A new canonical family for 128 and 256 bits, outside + `Primitive`. An example is a new dtype variant with a new canonical array. +- **(e) Integer lanes without a view.** The buffer of option (a), with the width fixed to the + canonical width, but with decimal-specific kernels instead of an integer view. + +### 4.2 Byte order is not integer order + +A FixedSizeBinary value orders by byte-wise comparison (memcmp). Arrow, DuckDB, and Vortex store +decimal lanes as little-endian two's complement. Memcmp of those bytes is not integer order: + +- `1` and `256` as 16-byte little-endian values: memcmp says `256 < 1`. +- `-1` and `0`: memcmp says `-1 > 0`. +- On 19,999 random pairs of 128-bit values, little-endian memcmp agreed with integer order + 10,060 times, which is chance. Big-endian bytes with the sign bit flipped agreed 19,999 times. + Appendix A has the script. + +Vortex already uses the sign-flip transform where it needs byte order. The row codec writes +"fixed-width integers big-endian, with the sign bit flipped for signed types" +(`vortex-row/src/codec.rs:19`). The DuckDB UUID exporter flips the top bit of a big-endian `u128` +to get the ordered HUGEINT form of DuckDB (`vortex-duckdb/src/exporter/uuid.rs`, export loop). + +So FixedSizeBinary values can reuse memcmp order only if they hold big-endian sign-flipped bytes. +That layout breaks zero-copy exchange with Arrow `Decimal128` and `Decimal256` and with DuckDB +`DECIMAL`. It also forces a conversion before every arithmetic operation. With the native +little-endian layout, every ordered operation on the values needs a decimal-specific +reinterpretation. Either way, the decimal logic stays duplicated. + +### 4.3 Comparison + +Compute: + +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | +| --- | --- | --- | --- | --- | --- | +| Compare | Generic primitive kernel at 128 and 256 bits | Decimal reinterpretation | Integer to 64 bits, reinterpretation above | New kernels, or shared generic code as in (a) | Existing decimal kernel | +| Checked arithmetic | `i128` native. `i256` through num-traits `Checked*` | Reinterpretation | Mixed | As (a) | Existing | +| Sum, min, max, sorted | Integer aggregates with a wider accumulator | Decimal-specific | Mixed | As (a) | Existing | +| Casts | Integer width casts plus rescale | Decimal-specific | Mixed | As (a) | Existing | +| Hash | Native | Byte hash, correct at a fixed width | Both | Native | Decimal-specific | +| Lane layout | Array of little-endian lanes. A limb split gives a SIMD-friendly layout | Same bytes, no typed lanes | Mixed | As (a) | As today | + +Data: + +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | +| --- | --- | --- | --- | --- | --- | +| Compression | `Narrow`, DBP, and all lane schemes on the MSP. Dict, RLE, run-end, constant, sparse, zstd | Dict, zstd, byte split. No FoR or BitPacking without reinterpretation | Mixed | As (a) | As (a) | +| Statistics | Integer min, max, and sum are meaningful | Memcmp min and max are wrong for numbers | Mixed | As (a) | Existing | +| Ordering | Native | Wrong for little-endian bytes | Wrong above 64 bits | Native | Native | + +Interop: + +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | +| --- | --- | --- | --- | --- | --- | +| Arrow | Decimal export as today. A top-level `Primitive(I128)` needs a policy, because Arrow has no Int128 | Decimal export as today | As today | As (a) | As today | +| DuckDB | DECIMAL as today. HUGEINT becomes possible | DECIMAL needs reinterpretation. No HUGEINT | No HUGEINT | HUGEINT possible | No HUGEINT | +| DataFusion | Decimals as today. No Int128 scalar exists | As today | As today | As today | As today | +| Python, NumPy | Python `int` scalars. NumPy has no 128-bit integer | As today | As today | As (a) | As today | +| Scalar value | `PValue::I128`, `PValue::I256` | Binary payload | Two shapes | New scalar variant | `DecimalValue` | + +Cost: + +| Dimension | (a) Integer lanes | (b) FSB values | (c) Hybrid | (d) Wide family | (e) No view | +| --- | --- | --- | --- | --- | --- | +| Decimal wire | No change | Needs FSB first | Needs FSB first | No change | No change | +| New wire values | Flatbuffer enum values, a new protobuf oneof, edition gating. Only for top-level wide columns | FSB tag or extension descriptor | As (b) | New dtype tag if user-visible | None | +| Blast radius | Largest: about 380 macro uses, about 45 exhaustive matches, 53 `is_int()` sites, 12 mirrors. The compiler finds most of them | An FSB type: the closed prototype PR #8865 changed 118 files | As (b), plus two decimal paths | A dtype variant and a canonical variant: 51 canonical sites plus dtype matches | Smallest | +| Alignment | 16 bytes for all wide widths. Fix two `byte_width == alignment` assumptions | 1 byte by default. Typed access needs 16 | Mixed | As (a) | Unchanged | +| Duplicated logic | Lowest. Decimal kernels become rules over the view | High | Medium | Medium | High, about 7,600 lines stay | + +### 4.4 Verdict + +Choose (a). The leaf keeps little-endian integer lanes, and the integer view gives decimal kernels +the integer kernels. D5 adds the wide `PType` values that the view needs. The wide-integer tier +(section 5) keeps the useful part of (d). Wide integers opt into a small, explicit kernel set +instead of pretending to be lane integers. + +Option (e) is the state between nodes N11 and N9. It fixes the crash class of issue #5820. But the +kernels for precision 19 to 76 stay decimal-specific until the wide `PType` values land. + +Wide PTypes do not need FixedSizeBinary: + +- Rust stores `i128` natively, and Vortex already has `i256` with checked arithmetic. +- The structural layer that both need already exists. `FixedWidthArray` moves records of 1, 2, + 4, 8, 16, and 32 bytes for take and filter (`vortex-array/src/arrays/fixed_width/mod.rs:16-47`). +- Arrow export of decimals uses the Arrow decimal types. Only a top-level wide-integer column + needs an export policy, and `FixedSizeBinary(16)` is one possible policy, not a prerequisite. +- Compression of wide integers is a limb split into lane integers, which DBP already implements. + +The two share the fixed-width structural layer and, later, the "execution layouts" idea of +RFC 0062. A UUID can use 128-bit lanes for equality and hashing, and a wide integer can use a +limb split for comparisons. Neither type carries the semantics of the other. + +Evidence that changes the decision: + +- Benchmarks can show that the generic integer kernels at 128 and 256 bits are clearly slower + than the current decimal kernels. Then keep the decimal kernels as specializations behind the + rules. This result is unlikely, because the current decimal kernels already run on `i128` + and `i256` lanes. +- If a requirement appears for precisions above 76 or arbitrary precision (for example PostgreSQL + NUMERIC), no fixed-width buffer works, and a variable-width representation is needed. + +## 5. Design of the wide-integer tier + +D5 settles that Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types. This +section describes how. + +### 5.1 Types + +```rust +#[repr(u8)] +pub enum PType { + U8 = 0, U16 = 1, U32 = 2, U64 = 3, + I8 = 4, I16 = 5, I32 = 6, I64 = 7, + F16 = 8, F32 = 9, F64 = 10, + I128 = 11, I256 = 12, + U128 = 13, U256 = 14, +} + +/// Integers that fit a 64-bit lane. Only these can be offsets, codes, sizes, or take indices, +/// and only these reach FastLanes and other lane kernels. +pub enum LaneIntPType { U8, U16, U32, U64, I8, I16, I32, I64 } + +/// Integers wider than 64 bits. +pub enum WideIntPType { I128, I256, U128, U256 } + +impl PType { + pub const fn as_lane_int(self) -> Option; + pub const fn as_wide_int(self) -> Option; + /// True for every integer, lane or wide. + pub const fn is_integer(self) -> bool; + /// The required buffer alignment. Equal to the byte width up to 16 bytes, then 16. + pub const fn alignment(self) -> Alignment; +} +``` + +Trait plan: + +- Implement `NativePType` for `i128`, `u128`, `i256` and a new `u256`. `i128` and `u128` already + have every supertrait. `i256` needs `Num`, `NumCast`, `FromPrimitive`, `Bounded`, and the byte + conversion traits, about two hundred lines over the Arrow type. Most + `match_each_native_ptype!` bodies are generic over `NativePType`, so they then compile and run + correctly at the new widths. +- Vortex has no native `u256` type. `primitive_types::U256` appears only in the fuzzing support for + `i256` (`vortex-array/src/dtype/arbitrary/decimal.rs:14-20`). So `U256` needs a new type, built + like `i256`. +- Keep `IntegerPType` (which requires `PrimInt`) for lane integers only. +- Add one trait for any-width integer kernels, for example `NativeIntPType: NativePType + Ord + + Hash + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv + BigCast`, implemented for the eight + lane integers and the four wide ones. Compare, arithmetic, sum, min and max, casts, and the + scaled-integer family bound on it. +- `DecimalType` stays the width type of the leaf metadata. After N9, each of its values maps onto + one signed `PType`. `NativeDecimalType` can later fold into the new trait, with the precision + bound tables as a `Pow10` trait on the signed integers. + +### 5.2 Dispatch macros + +| Macro | Arms | Use | +| --- | --- | --- | +| `match_each_native_ptype!` | All 15 | Generic code over `NativePType` | +| `match_each_lane_int_ptype!` | 8, takes `LaneIntPType` | Offsets, codes, indices, lane encodings. Exhaustive, no panic arm | +| `match_each_int_ptype!` | 12 | Any-width integer kernels | +| `match_each_wide_int_ptype!` | 4 | Wide-only code, such as the limb split | + +Retire `match_each_integer_ptype!` and its siblings after the migration, because their +`other => panic!` arm turns a type mistake into a run-time panic +(`vortex-array/src/dtype/ptype.rs:562`, `:589`, `:616`, `:640`). A caller that holds a `PType` +converts with `as_lane_int()` and returns a typed error. + +### 5.3 Migration technique + +Let the compiler enumerate the sites: + +1. Add `LaneIntPType`, `WideIntPType`, and the new macros while `PType` still has 11 variants. +2. Remove `PType::is_int` and add `is_lane_int` and `is_integer`. Every one of the 53 call sites + stops compiling and must choose. About 40 of them guard offsets, codes, indices, or lane + encodings and choose `is_lane_int`. +3. Migrate `match_each_integer_ptype!` uses (169) and `match_each_unsigned_integer_ptype!` uses + (118) to the typed macros, crate by crate. +4. Add the four variants last. Every exhaustive match (about 45 outside `ptype.rs`) and every + mirror then fails to compile until it handles them. +5. Teach the fuzzer to generate decimal views and wide-integer columns at every width before + exposing wide columns. + +### 5.4 Wire encoding and gating + +- **Flatbuffers.** Append `I128`, `I256`, `U128` and `U256` to `enum PType: uint8` + (`vortex-array/flatbuffers/vortex-dtype/dtype.fbs:4-16`). Old readers fail loudly with + "Unknown PType variant" (`vortex-array/src/dtype/serde/flatbuffers.rs:559`). +- **Protobuf.** Do not append to `enum PType` in `dtype.proto`. Old readers decode an unknown + value as `U8` without an error (`vortex-array/src/dtype/serde/proto.rs:36`, prost-derive + getter semantics in section 2.4). Variant shredding and Parquet-variant metadata persist + protobuf dtypes on disk, so a silent misread corrupts data. Add a new oneof variant + instead, for example `WideInteger wide_integer = 14`, which old readers reject with + "Unrecognized DType" (`proto.rs:32`). Also change new readers to decode `PType` with + `TryFrom` and fail on unknown values. +- **Editions.** Add a component kind for core dtype features, or treat `vortex.primitive.i128` + and the other wide types as dtype members, and check them in `validate_dtype_editions` + (`vortex-file/src/writer.rs:429-475`). Use the same mechanism for any later core dtype tag, + including the FixedSizeBinary tag that DQ7 recommends. Decimal columns do not need it, because + the leaf carries `DecimalType` (section 3.5). + +### 5.5 Scalars + +- Add `PValue::I128`, `PValue::I256`, `PValue::U128` and `PValue::U256`. `PValue` equality and + hashing upcast within each signedness family today + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-178`). Extend the upcast targets to + `i256` for signed values and to `u256` for unsigned values. +- Serialize them as protobuf `bytes_value` holding 16 or 32 little-endian bytes, decoded by the + dtype, the same way decimals work (`vortex-array/src/scalar/proto.rs:402-445`). Old readers + reject `bytes_value` for a primitive dtype (`proto.rs:442-444`), which is a loud failure. +- Decide later whether `DecimalValue` becomes a storage `PValue` plus the dtype. Until then it + stays, normalized to the canonical width at construction. + +### 5.6 Engine policy for top-level wide columns + +| Engine | `Primitive(I128)` | `Primitive(U128)` | `Primitive(I256)`, `Primitive(U256)` | +| --- | --- | --- | --- | +| Arrow | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `i128` needs 39 digits) or `FixedSizeBinary(16)` through an export plugin | Error by default. Opt-in `Decimal256(39, 0)` (lossless, `u128` needs 39 digits) or `FixedSizeBinary(16)` | Error by default. Opt-in `FixedSizeBinary(32)`. `Decimal256` holds at most 76 digits, `i256` needs 77, and `u256` needs 78 | +| DuckDB | HUGEINT, zero-copy little-endian | UHUGEINT | No type. Error | +| DataFusion | Error | Error | Error | +| Python | Python `int` scalars. Arrays follow the Arrow policy | Same | Same | +| FFI, C++, Java | New enum values in `vx_ptype` and mirrors | Same | Same | +| CUDA | Not supported in dynamic dispatch at first | Same | Same | + +### 5.7 U128 and U256 + +D5 adds them, which overrides the earlier recommendation of these notes to defer them. Decimal +needs only the signed widths. DuckDB UHUGEINT is the first consumer of `U128` +(`vortex-duckdb/src/convert/dtype.rs:101`). `U256` needs a new native type (section 5.1). + +With all four types, `to_unsigned()` and `to_signed()` have a target for each wide type. Audit +their 77 call sites during step 3 of section 5.3 anyway, because many of them reinterpret lane +integers for indices and codes. + +Recommendation (DQ5): wide integers are user-visible columns behind an edition. They get a small +kernel set first: compare, arithmetic, cast, sum, min, max, hash, and structural operations. + +### 5.8 What wide integers deliberately do not get + +- No FastLanes BitPacking, FoR, or Delta on 128-bit lanes. The `fastlanes` lanes stop at 64 + bits. Compression goes through `Narrow` or the limb split. +- Never offsets, sizes, codes, or take indices. +- No CUDA dynamic dispatch at first. The decimal device kernels keep their own width handling. + +## 6. FixedSizeBinary design + +### 6.1 Where the discussion stands + +- RFC 0055 (branch `ct/types`) and its earlier draft RFC 0029 (branch `ngates/rfc-0029-types`) + apply a two-step test. The first step asks if the type is semantically distinct, or if a + refinement predicate gates operations. The second step asks if the type is core or an + extension. Both drafts leave FixedSizeBinary unresolved. +- RFC 0062 makes `FixedSizeBinary(n)` a core variant with contiguous fixed-width canonical + storage and whole-value validity, and makes UUID an extension over `FixedSizeBinary(16)`. It + states that "Any UUID ordering must be specified separately before a byte or integer kernel can + implement it." +- Issue #9902 asks for first-class support. One user stores trigrams as `FixedSizeBinary(3)`. + The discussion there raises power-of-two over-alignment and the cost of filtering a contiguous + layout. +- PR #8865 (closed, not merged) prototyped it in 118 files (+4,021, -2,004), stacked on the + merged `FixedWidthArray` refactor (PR #8864). It used `DType::FixedSizeBinary(u32, Nullability)`, + a contiguous buffer with an explicit length for zero-width values, a binary scalar payload, + byte-lexicographic ordering, and a new flatbuffer tag. +- PR #9886 (closed) shows the current gap: a DataFusion `FixedSizeBinary` literal had to become + a `FixedSizeList` scalar to compare against a UUID column. +- UUID today is `vortex.uuid` over `FixedSizeList(Primitive(U8, non-null), 16)` + (`encodings/uuid/src/vtable.rs:53-78`), frozen in edition core 2026.08.3 + (`vortex-edition/src/declarations/core/v2026_08_3.rs:23`). Its scalar unpacking walks sixteen + `PValue::U8` list elements (`vtable.rs:78-115`). + +### 6.2 Is it justified, and is it core? + +Apply the RFC 0055 test: + +- **Refinement predicate.** `FixedSizeBinary(n) = { b : Binary | len(b) = n }`. The predicate + gates a fixed-width canonical layout, constant-time row addressing, and scalar-compatible + values, exactly as `FixedSizeList` refines `List`, and `FixedSizeList` is already core. +- **Semantic distinctness from `FixedSizeList`.** Opaque bytes have byte-string + operations (memcmp order, prefix, conversion to `Binary`). A list of bytes has element + operations. The distinction is real but weak. +- **Core or extension.** Core Vortex owns the operations that matter: the canonical layout, + equality, hashing, ordering, Arrow and Parquet mapping, and scalar representation. RFC 0056 + (branch `ngates/scalar-values`) keeps `Scalar` only for scalar-compatible values and moves + nested values, including extensions over nested storage, to arrays. Under that rule a UUID + over `FixedSizeList` has no cheap scalar, while a UUID over `FixedSizeBinary(16)` does. + +D6 settles the question: FixedSizeBinary is a core type, as RFC 0062 proposes. The test above is +the background for that decision. + +### 6.3 Canonical layout + +- One buffer of `len * n` bytes, whole-value validity, and an explicit length so that `n = 0` + works. This is the PR #8865 layout. +- Implement `FixedWidthArray` so take and filter come from the shared kernels. Those kernels + already have typed paths for 1, 2, 4, 8, 16, and 32 bytes and a fallback for other widths. +- Alignment: 1 byte by default. Allow a writer to request power-of-two alignment for + power-of-two widths. The file format records alignment per buffer + (`vortex-array/flatbuffers/vortex-array/array.fbs`, `struct Buffer`), so this is free on the wire. + +### 6.4 Operations + +| Operation | Definition | +| --- | --- | +| Equality, hashing | Bytes | +| Ordering | Byte-lexicographic (memcmp), the same as `Binary` and Arrow | +| Cast to and from `Binary` | Explicit. From `Binary` validates every length | +| Cast to and from `FixedSizeList` | Zero-copy when the elements are a flat non-nullable `u8` array | +| Arithmetic, numeric casts | None. Never implicit | + +FixedSizeBinary must never carry integer semantics. A type that wants integer behavior over +fixed bytes uses an integer type, or it asks for an explicit reinterpretation with a declared +byte order. A 128-bit lane can implement equality and hashing for UUID, because those do not +depend on byte order. It must not implement ordering without the big-endian sign-flip transform. + +### 6.5 Scalars and compression + +- Scalars: `ScalarValue::Binary` with a validated length. The value is scalar-compatible under + RFC 0056. +- Compression: Dict, constant, run-end, sparse, zstd, and a byte-lane split. The split + transposes the bytes into `n` lane columns, or into 64-bit words, and integer schemes compress + the lanes. The limb split for wide integers (node N14) and the lane split for FixedSizeBinary + can share code at the word level. They keep different dtypes. + +### 6.6 Wire representation + +| Option | Old readers | Gating | +| --- | --- | --- | +| New core tag (the next free flatbuffer union value and protobuf oneof field) and a new `vortex.fixed_size_binary` array id | Fail loudly on the schema | Needs the core dtype feature gate of section 5.4 | +| Extension descriptor `vortex.fixed_size_binary` over `FixedSizeList` storage, and the array written in that storage form | Read it as a foreign extension over `FixedSizeList`, so the data stays readable (`vortex-array/src/dtype/extension/foreign.rs:26-75`) | The existing dtype edition kind | + +Recommendation (DQ7): a new core dtype tag behind an edition. Then the wire form matches the Rust +form, and no translation is needed. The extension descriptor keeps old readers working. But then +the wire form differs from the Rust form, and node N16 must translate at the serialization +boundary. The question stays open (section 9). + +### 6.7 UUID on FixedSizeBinary + +- In memory, `vortex.uuid` storage becomes `FixedSizeBinary(16)`. +- `Uuid::validate_dtype` accepts both `FixedSizeBinary(16)` and the frozen + `FixedSizeList` storage, and the reader normalizes the second form zero-copy. +- The writer keeps writing the frozen `FixedSizeList` storage until a new edition declares the + new storage. The conversion is zero-copy for flat data. The existing Arrow plugin already does + the same conversion (`encodings/uuid/src/arrow.rs`, `try_fsl_to_fsb`). +- The DuckDB exporter keeps its big-endian sign-flip transform. +- XQ6 asks whether `Uuid` orders by its bytes. It is open, and there is no recommendation yet. + +### 6.8 Relation to decimals and wide integers + +None is required. FixedSizeBinary, Primitive (with wide integers), and Decimal storage share only +the fixed-width structural layer. That layer is the right abstraction to grow. PR #8864 started +it, and Epic #7706 suggests it ("a `Flat` encoding that allows for any aligned fixed-size binary +values"). + +## 7. The type-system dependency graph + +### 7.1 Nodes + +Kind: **impl** means implementation work with a known shape. A **research** node needs a design +decision first. An **urgent** node fixes a correctness bug that exists today. No settled decision +needs an **optional** node. + +| Id | Node | Kind | Notes | +| --- | --- | --- | --- | +| N1 | Extension opacity fixes | impl, urgent | The urgent changes U1 to U8 of the extension notes. They cover storage casts, lone-side storage compares, errors from reads, nullability, temporal validation, storage checks, zero values and debug output | +| N2 | Rewrite rules that the type owns | research, then impl | X4: a type implements its behavior as rewrite rules on arrays, registered with the type. There is no `ext_wrap`. X5 and X6 add the placement rule and the removal of nested extension layers. Recommendation (XQ2): the rules live on `ExtVTable`, written over `ReduceNode` | +| N3 | Derives for one dtype | impl | X2: `eq` and `ord` are properties of one dtype and never apply across dtypes. Operations across dtypes need a rule of the type (N2) or a cast by the engine (X3) | +| N4 | Cast rules (#8570) | impl | C1: a cast exists only if a rule for the pair exists. Extension types get `cast_to` and `cast_from` hooks. CQ1, the default for inexact values, is open | +| N5 | Value checks in `try_new` | impl | X7: a checked `try_new` and a `new_unchecked` for kernels. `ExtensionArray::try_new` checks every non-null value. Checks: UTF-8 and FixedSizeBinary length. The decimal precision check moves into N11 (D14) | +| N6 | Nullability restructure | research, optional | Remove nullability duplication between wrapper and storage dtypes | +| N7 | Fixed-width structural layer | impl | Extend `FixedWidthArray` (PR #8864) to run-end, sparse, dict, constant, and builders, keyed by record width | +| N8 | Scaled-integer kernels | impl | `rescale`, `check_range`, fused scaled arithmetic and compare | +| N9 | Wide-integer tier (I128, I256, U128, U256) | impl | Section 5 and D5. Recommendation (DQ5): user-visible behind an edition, with a small kernel set first | +| N10 | Wire safety and core dtype gating | impl | Strict enum decoding, protobuf oneof for wide PTypes, edition kind for core dtype features | +| N11 | One canonical width for each decimal dtype | impl, urgent | D3, tracked by issue #10701. An in-memory change and then a wire change (D11). Producers take the width from the precision, readers convert old buffers, DBP decodes to its own canonical width, and `DecimalArray::try_new` accepts only the canonical width and checks every value. Removes the crash class of issue #5820 | +| N12 | Zero-copy integer view | impl, future work (D20) | Section 3.2. Precision 1 to 18 needs no new `PType`, so that part can land at any point. Precision 19 to 76 needs N9 | +| N13 | Decimal semantics as rewrite rules | impl, future work (D20) | Section 3.4. Decimal kernels delegate to integer kernels through the view. Most decimal-specific kernels go away | +| N14 | Limb split for wide-integer columns | impl, optional, future work (D20) | The split of DBP, offered as an encoding of wide-integer columns. DBP itself keeps its decimal dtype (D16) | +| N15 | Decimal as an extension type | impl, optional, later | D1 keeps the built-in type. A later change can remove it if the two become identical. Wire tag unchanged | +| N16 | Wire translation layer for well-known types | research, then impl | Rust representation and wire representation chosen separately | +| N17 | FixedSizeBinary core type | impl | D6 and section 6. Recommendation (DQ7): a new core dtype tag behind an edition | +| N18 | UUID on FixedSizeBinary | impl | Accept frozen storage, write frozen storage. XQ6, the byte order of `Uuid`, is open | +| N19 | Utf8 as a well-known extension | research, low priority | 776 lines in 212 files mention `DType::Utf8` | +| N20 | FixedSizeList and Map as well-known extensions | research, low priority | Needs requested layouts for the offset-free FixedSizeList form | +| N21 | Duration and Interval extensions (PRs #10467, #10468) | impl, in flight | Both PRs leave edition membership open | +| N22 | Execution layouts (RFC 0062) | research | Kernels request acceptable layouts, for example a limb split or 128-bit lanes | +| N23 | Engine type plugins for DuckDB and DataFusion | impl | Arrow already has plugin registries (`vortex-arrow/src/session.rs`) | +| N24 | Refinement types (RFC 0055) | research only | UUID within FixedSizeBinary(16), Utf8 within Binary | +| N26 | Scalar value model (RFC 0056) | research, then impl | Decides which types keep a host `Scalar` | +| N27 | DuckDB HUGEINT and UHUGEINT import and export | impl | First user-visible wide-integer columns | +| N28 | One `Narrow` encoding | impl, in flight | D4. Integers first (PRs #10223, #10332, #10333, issue #10020), then decimals. D8 settles one encoding with one wire ID, `vortex.narrow`. D10 and D15 settle the decimal child and its kernels. The wire format waits for the wire change (D18) | + +(There is no N25. The Variant interaction is an edge on N10.) + +Not in the graph: Union completion. Union casts and several aggregates are still TODOs +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:223-228`). That work is independent of this graph. + +### 7.2 Graph + +```mermaid +flowchart TD + classDef urgent fill:#f8d7da,stroke:#842029,color:#000 + classDef critical fill:#fff3cd,stroke:#997404,color:#000 + classDef impl fill:#d1e7dd,stroke:#0f5132,color:#000 + classDef research fill:#e2e3e5,stroke:#41464b,color:#000,stroke-dasharray: 4 3 + classDef optional fill:#ffffff,stroke:#41464b,color:#000,stroke-dasharray: 2 2 + + N1["N1 Extension opacity fixes"]:::urgent + N11["N11 One canonical decimal width"]:::urgent + N28["N28 One Narrow encoding"]:::critical + N2["N2 Rewrite rules owned by the type"]:::research + N3["N3 Derives for one dtype"]:::impl + N4["N4 Cast rules #8570"]:::impl + N5["N5 Value checks in try_new"]:::impl + N6["N6 Nullability restructure"]:::research + N7["N7 Fixed-width structural layer"]:::impl + N8["N8 Scaled-integer kernels"]:::critical + N9["N9 Wide integers I128 I256 U128 U256"]:::critical + N10["N10 Wire safety and core dtype gating"]:::impl + N12["N12 Zero-copy integer view"]:::critical + N13["N13 Decimal semantics as rewrite rules"]:::critical + N14["N14 Limb split for wide integers"]:::optional + N15["N15 Decimal as extension type"]:::optional + N16["N16 Wire translation layer"]:::research + N17["N17 FixedSizeBinary core type"]:::impl + N18["N18 UUID on FixedSizeBinary"]:::impl + N19["N19 Utf8 as extension"]:::research + N20["N20 FixedSizeList and Map as extensions"]:::research + N21["N21 Duration and Interval"]:::impl + N22["N22 Execution layouts"]:::research + N23["N23 Engine type plugins"]:::impl + N24["N24 Refinement types"]:::research + N26["N26 Scalar value model"]:::research + N27["N27 DuckDB HUGEINT and UHUGEINT"]:::impl + + N1 --> N15 + N1 --> N21 + N1 -.-> N18 + N2 --> N3 + N2 -.-> N13 + N3 --> N15 + N3 -.-> N21 + N4 --> N15 + N4 --> N21 + N8 -.-> N4 + N8 --> N13 + N7 -.-> N9 + N7 -.-> N17 + N28 --> N11 + N11 -.-> N12 + N9 --> N12 + N9 --> N14 + N9 --> N27 + N10 --> N27 + N10 -.-> N9 + N10 -.-> N17 + N12 --> N13 + N28 -.-> N13 + N13 --> N15 + N16 --> N15 + N16 -.-> N18 + N16 --> N19 + N16 --> N20 + N17 --> N18 + N22 --> N19 + N22 --> N20 + N23 -.-> N21 + N23 -.-> N27 + N23 -.-> N18 + N5 -.-> N13 + N5 -.-> N17 + N26 -.-> N17 + N24 -.-> N18 + N24 -.-> N19 + N6 -.-> N15 +``` + +Solid arrows are hard dependencies: the target must not land before the source. Dashed arrows are +soft: the target works without the source, but it is cheaper, safer, or faster with it. + +### 7.3 Edges with reasons + +1. N1 to N15: as an extension on `develop`, Decimal inherits storage casts and lone-side storage + compares (`arrays/extension/compute/cast.rs:17-22`, + `scalar_fn/fns/binary/compare/mod.rs:177-194`). +2. N1 to N21: without N1, a `Duration` cast across units reinterprets the storage on the row path + (EXT-C1). On the array path, a comparison with a raw `i64` literal compares storage (EXT-Q1). + New types must not ship with a known leak that users can come to depend on. +3. N1 to N18 (soft): after N1, a UUID column no longer compares with a storage literal. The + engine literal paths then change once, before the storage changes. PR #9886 shows the + current workaround. +4. N2 to N3: a derive is one kind of rule that the type owns. It runs the storage operation for + operands of one dtype. +5. N2 to N13 (soft): the decimal rules can reuse the rule mechanism of extension types. With + recommendation XQ2, rules over `ReduceNode` also rewrite bound expressions for pruning. The + built-in type can own its rules without N2. +6. N3 to N15: after N1 and N2, a decimal extension type without derives loses compare, hash, + `min`, `max` and sort within one `(p, s)`. Comparisons across scales stay with the engine (X3). +7. N3 to N21 (soft): `Duration` declares `eq` and `ord` for one unit. Before N3, the shared storage + compare kernel gives the same answer for one unit. Across units, the engine casts first (X3). +8. N4 to N15: rescaling casts and casts to and from integers and floats must be rules of the type + (C1). +9. N4 to N21: unit conversions are casts. +10. N8 to N4 (soft): Timestamp, Time, and Duration unit casts are powers of ten, except Days. +11. N8 to N13: decimal rules need `rescale` and `check_range`. +12. N7 to N9 (soft): width-keyed structural kernels give wide integers take, filter, run-end, + sparse, dict, and constant support without new per-type code. +13. N7 to N17 (soft): width-keyed structural kernels give FixedSizeBinary run-end, sparse, dict, + and constant support without new per-type code. Take and filter already come from + `FixedWidthArray`. +14. N28 to N11: the last step of N11 makes `DecimalArray::try_new` accept only the canonical + width. Before that step, the read plugin needs decimal `Narrow` to convert old narrow nodes + (D9). Without `Narrow`, D3 forces the full width on every column (D4). +15. N11 to N12 (soft): after N11, all arrays of one decimal dtype have one view dtype. Before N11, + a delegated binary kernel must widen one side first. +16. N9 to N12: the view for precision 19 to 76 is `Primitive(I128)` or `Primitive(I256)`. The + view for precision 1 to 18 does not wait for N9. +17. N9 to N14: the limb split is an encoding of wide integers. +18. N9 to N27: HUGEINT maps to `Primitive(I128)`, and UHUGEINT maps to `Primitive(U128)`. +19. N10 to N27: a top-level wide column writes new dtype values. An edition must gate them, and + the protobuf encoding must fail loudly in old readers. +20. N10 to N9 (soft): ship the strict enum decoding in readers before any writer can emit wide + PTypes. Variant shredding stores protobuf dtypes on disk + (`vortex-array/src/arrays/variant/vtable/mod.rs:54`). +21. N10 to N17 (soft): needed only if FixedSizeBinary gets a new core wire tag. DQ7 recommends + one, and then this edge becomes hard. +22. N12 to N13: the rules run over the integer view. +23. N28 to N13 (soft): decimal `Narrow` kernels pass work to the decimal kernels of the child + (D15). Once N13 moves those kernels onto integer kernels, a decimal `Narrow` gets the speed of + the integer kernels without its own code. +24. N13 to N15: once decimal semantics are rules over the view, the optional move to an extension + changes the API, not the behavior. +25. N16 to N15: the `Decimal` wire tag must translate to and from the extension, recursively. +26. N16 to N18 (soft): UUID storage is `FixedSizeBinary(16)` in memory and `FixedSizeList` on + the wire. +27. N16 to N19 and N16 to N20: the `Utf8`, `FixedSizeList`, and `Map` tags need the same + translation. +28. N17 to N18: UUID storage needs FixedSizeBinary. +29. N22 to N19 and N22 to N20: Utf8 views and the offset-free FixedSizeList layout must stay + available when those types become extensions. +30. N23 to N21, N27, and N18 (soft): with plugins, new types get DuckDB and DataFusion mappings + without edits to central `match` statements (`vortex-duckdb/src/convert/dtype.rs`, + `vortex-datafusion/src/convert/scalars.rs`). Edits to the central statements also work. +31. N5 to N13 (soft): the places where data enters establish invariant I2 (values within the + precision). The rules can then trust their inputs. +32. N5 to N17 (soft): a cast from `Binary` must check every length. FixedSizeBinary can check + lengths in its own constructors and casts, so the shared hook only avoids a second mechanism. +33. N26 to N17 (soft): a FixedSizeBinary scalar is a binary payload, which RFC 0056 treats as + scalar-compatible. +34. N24 to N18 and N24 to N19 (soft): a refinement framework can express these types, but + neither one depends on it. +35. N6 to N15 (soft): a wrapper repeats its nullability in the storage dtype, so every + nullability change rebuilds the extension dtype + (`vortex-array/src/dtype/extension/typed.rs:91-99`). + +### 7.4 Critical path and parallel work + +- **Critical path for one canonical form:** strict decoding of the decimal width and the integer + `Narrow` in memory, then the benchmarks and fixtures, then N11 as two changes (D11). The + in-memory change holds the width owner, N28 for decimals, DBP decoding, the read plugin, the DBP + scheme on the canonical array and the strict `try_new`, each as its own PR. The wire change holds + the `vortex.narrow` wire format, the preview edition with DBP v2, and the decimal `Narrow` + scheme. N11 does not wait for N9. +- **Critical path for delegated decimal kernels (future work, D20):** N9, N12, N13. N8 must land + before N13. N7 is soft, but it shrinks N9, so do it first. N12 for precision 1 to 18 can land at + any point. +- **The optional move (N15):** after N13, with N1, N2 then N3, N4, and N16 joining at N15. +- **Parallel tracks:** + - Extension correctness: N1, N2, N3, N4, N5. + - Decimal: N28, N11, N12, N7, N8, N9, N13, N14. + - Bytes: N17, N18. + - Temporal: N21, after N1 and N4. + - Wire and engines: N10, N16, N23, N27. +- **Research-only nodes:** N6, N22, N24. **Research before implementation:** N2, N16, N26, N19, + N20. + +### 7.5 Mapping to the extension-type research + +The extension-type notes in `../../extension-types/` use their own ids for urgent changes (U1 to +U8) and phases (0 to 6). The decisions file adds ids X1 to X9 and XQ1 to XQ6. This table maps them +onto the graph. + +| Extension research item | Node here | +| --- | --- | +| U1 to U8, the urgent changes | N1, item P3 in section 8.3. U5 and U6 are dtype and storage checks, not the value validation of N5 | +| U4 infallible nullability changes | Part of N1. It also removes the panic in the rebuild that edge 35 describes | +| Phase 1, contract and conformance harness | Not a separate node. Its decimal-like reference type over `i64` is an early acceptance test for N3 and N13 | +| Phase 2, typing, derives and rewrite rules (X2, X4) | N2 and N3. Part 5 of the extension notes lands them after phases 0 and 1, which the graph does not show | +| Phase 3, cast rules that the types own (C1) | N4 | +| Phase 4, the lift rule for structural arrays, and storage statistics (X5, XQ1) | The placement rule of section 3.3. DateTimeParts below the structural encodings is the temporal twin of DBP | +| Phase 5, value checks in `ExtensionArray::try_new`, and nullability (X7) | N5 and N6 | +| Phase 6, authoring and boundaries | N23 | + +The decimal fix (N28 and N11) does not wait for the extension phases. The built-in Decimal +owns its rules (D1, section 3.4). The optional move (N15) needs phases 0, 2 and 3 first (N1, N3 +and N4). + +## 8. Recommendations + +### 8.1 Decision + +The maintainer settled the main choices on 2026-10-19: + +- `DecimalArray` stays a leaf with one buffer and validity (D2). +- Each decimal dtype has one canonical encoding, and the precision fixes the width (D3). +- A `Narrow` encoding stores a column at a lower precision with the same scale (D4). +- Vortex adds `I128`, `I256`, `U128` and `U256` (D5). +- FixedSizeBinary is a core type for opaque bytes (D6). +- `Decimal` stays a built-in type (D1). + +On 2026-10-22 the maintainer settled the wire IDs: + +- `vortex.decimal` keeps its wire ID (D7). +- One `vortex.narrow` ID serves integers and decimals (D8). + +On 2026-10-23 the maintainer settled the rest of the canonical decimal fix, and revised D9 to D11: + +- Each precision gets the smallest signed width (D12). Imports copy into it and fail on a value + outside the precision (D13). `try_new` checks the width and every valid value (D14). +- A reader converts an old narrow `vortex.decimal` node by its max into a `Narrow` or a copy (D9). +- A decimal `Narrow` child has a strictly narrower canonical width, and the producer chooses its + precision (D10). `Narrow` kernels pass work to the child, and casts of a `Narrow` pass to the + child (D15). +- In DBP, the MSP is any signed integer up to 64 bits, and the lower parts are `u64`. DBP decodes + to the canonical array of its own dtype (D16). +- The compressor builds DBP from the canonical array, so file contents stay the same (D17). +- `vortex.narrow` stays in memory until the maintainer decides to write it. Then it enters one + preview edition together with DBP v2 (D18). +- The bar of no compute regression applies to the final state, and gap columns use the next + width in memory (D19). +- The work lands as an in-memory change and a wire change. PRs #10305 and #10314 are ignored for + now (D11). +- The integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and + scales, and casts that lose information are future work. The plan to always store min and max on + the MSP is dropped (D20). + +A second review on 2026-10-23 confirmed these decisions and added four: + +- Every narrowing copy writes zeros in null slots and fails on a value outside the precision + (D21). +- An integer `Narrow` child keeps the signedness of its parent (D22). +- `vortex.narrow` is only for columns with a fixed logical dtype (D23). +- The compat fixtures are published from a release before the in-memory change, and the error + cases are unit tests (D24). + +It also changed five decisions. The read plugin scans instead of using stored statistics (D9). +The `Narrow` metadata stores only the child precision (D8). Each in-memory step is its own PR +(D11). The DBP constructor and module doc require `u64` lower parts (D16). One DBP scheme serves +every edition, and default writers pay the `i256` cost until both formats enter `core` (D17). + +A planning review on 2026-10-23 split the work into 22 PRs, with one brief for each PR in +`agents/briefs/` (part 4 of the Decimal notes). It added D25 to D35: + +- Generic structural encodings always sit above `Narrow` (D25). +- A writer that cannot write `vortex.narrow` writes the canonical array, as a documented + workaround until `core` (D26). +- Which arrays the compressor narrows is deferred (D27). +- `encode` builds a `Narrow`, and the deprecated `encode_values` returns only the child. The + existing narrowing call sites move to `encode_values` (D28). +- The DBP constructor rejects lower parts that assemble wider than the canonical width (D29). +- `try_new` takes an `ExecutionCtx`, and imports check their own values (D30). +- Only the PRs that change how files are read wait for the compat fixtures (D31). +- The `vortex.narrow` metadata is a prost message with a typed `oneof` (D32). +- The compressor uses a `Narrow` scheme at first (D33). +- Strict decoding covers eleven more `PType` fields (D34). +- The integer `Narrow` lands as five stacked PRs, then the call-site PR (D35). +- An operation on a `Narrow` fails only where the parent would fail, and aggregates use the + accumulator of the parent dtype (D36). +- The host converts old narrow nodes, also on CUDA read paths (D37). +- The integer PRs are written from scratch, with the local rewrite as a reference (D38). + +These notes add the integer view of section 3.2, so that decimal kernels delegate to integer +kernels. D20 makes it future work. Section 9 lists the open questions with the recommendations of +these notes. + +Reasons for integer lanes and the integer view, in order of weight. They support the future work +of D20, and they are not reviewed: + +1. Every decimal operation is an integer operation plus scale bookkeeping. The view turns the + decimal kernels into rules over shared integer kernels. FixedSizeBinary values keep every + decimal kernel and add a reinterpretation step. +2. Byte order is not integer order for the layout that Arrow and DuckDB use (section 4.2). +3. Integer statistics and integer compression work unchanged. Exchange with Arrow and DuckDB + stays zero-copy where the widths match. +4. The decimal wire format needs no change, because the leaf is the `vortex.decimal` node. +5. The cost is a large but mechanical PType migration that the compiler can drive. It does not + block the one-canonical-form fix. + +### 8.2 Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| The PType migration touches 36 crates (counting each encoding crate) and several languages | Typed sub-enums and macros, compiler-driven steps (section 5.3), one crate per PR | +| A leftover `other => panic!` arm panics on a wide type | Retire the panic macros. Fuzz decimals and wide integers at every width | +| Old readers silently read a protobuf wide PType as `U8` | Use a new protobuf oneof. Ship strict readers early (N10) | +| Rewritten arithmetic makes several passes | Fused scaled-integer kernels (N8). Compare before and after with the existing `vortex-array/benches/binary_ops.rs`, `cast_decimal.rs`, and `compare.rs` | +| D3 without `Narrow` forces the full width on every column | Land `Narrow` for decimals in the in-memory change, before `try_new` rejects other widths | +| A narrow buffer does not give a precision, because precision and integer width do not line up | Every producer finds the precision from the values. Readers convert old nodes by their max (D9). DBP decoding widens a narrow MSP (D16) | +| Compute on DBP regresses when its decoding stops returning narrow leaves | In the final state, the compressor narrows the decimal type first, so decoding reuses the buffer. Until then, the maintainer accepts the cost (D19). The max pass of D16 can come later. New benchmarks measure the cost (section 8.3, B1) | +| Gap columns use the next width in memory in the final state | Accepted (D19). Pushdown of sum, min and max into the MSP can come later | +| The conversion of old `vortex.decimal` nodes scans their buffer, because the plugin does not see stored statistics | Old narrow nodes are rare, and the speed of this path matters little (D9) | +| Arrow import at a width other than the canonical width copies the buffer | Accepted (D12, D13). Arrow export already copies for widths below 128 bits | +| A writer without an edition for `Narrow` or DBP v2 stores precision 39 to 76 at 32 bytes, even when values fit 16 bytes. Default writers use `core`, so this lasts until both formats enter `core` | Accepted (D17). The wire change puts `vortex.narrow` and DBP v2 into one preview edition (D18). If it ships in the same release as the in-memory change, writers that select that edition never pay the cost (D11) | +| Public API breaks: `DecimalArray::new::` with a `T` other than the canonical type, and `DecimalBuilder::new::` | Constructors that convert, and errors that name the canonical width. Keep the in-memory id `vortex.decimal`, which the Python bindings key on (`vortex-python/src/arrays/native.rs:174-175`) | +| Wide integers become visible to users before engines can export them | Writer gating (N10) and an explicit engine policy (section 5.6) | + +### 8.3 Sequence + +[Part 4](../04-implementation.md) of the Decimal notes replaces this sequence with PR IDs, waves and +briefs. The list below keeps the reasons for the order. + +PR #10230 gave decimals an integer child, which D2 replaces. It closed on 2026-10-19. Its fix to the +DBP cast rule becomes P2 below. The plan below replaces its other steps. + +Independent changes, in any order. On 2026-10-23, none of these is on `develop`: + +- **P1.** Strict decoding: `DecimalType::try_from(raw)` in the decimal deserializer + (`vortex-array/src/arrays/decimal/vtable/mod.rs:170`) and `PType::try_from(raw)` in the + protobuf dtype reader (`vortex-array/src/dtype/serde/proto.rs:36`). Part of N10. The decimal + part must land before step 4 of the in-memory change. +- **P2.** The DBP nullability fix (section 3.3). It adds the guard of #10230 to the cast rule + (`encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`), a cast kernel for + the case that needs execution, and tests for a nullable-to-non-nullable cast with nulls, without + nulls, and with only nulls. It is its own PR, with no issue. +- **P3.** N1: the urgent extension changes U1 to U8. +- **P4.** The integer `Narrow` in memory. #10223, #10332 and #10333 closed on 2026-10-23. The + new PRs are written from scratch, with the local rewrite of #10223 and #10332 as a reference + (D38). NR1 adds `NarrowArray::encode_values`, and NR6 moves the call sites to it (D28). The fix + for dictionary fill codes is its own PR, and it ports the rewrite's fix as it is. + +Benchmarks and fixtures, before the producers change: + +- **B1.** Benchmarks for decimal compare, filter, sum, min, max and the Arrow and DuckDB exports, + on DBP and on `Narrow`. They include a gap column and the decoding of DBP with a narrow MSP. + Record a CodSpeed base. The related benchmarks of today are + `vortex-array/benches/{compare,binary_ops,cast_decimal,filter_fixed_width}.rs`, + `encodings/decimal-byte-parts/benches/dbp_assemble.rs` and `vortex-arrow/benches/to_arrow.rs`. +- **B2.** The compat fixtures of section 3.5, published from a release before step 1 (D24). The + error cases are unit tests. + +The in-memory change (D11). Each step is its own PR, because none of them needs temporary code: + +1. Add `DecimalDType::canonical_values_type()` and route the 25 `smallest_decimal_value_type` calls + through it, so that the width function has one owner (D12). Producers and imports take the width + from the precision. Imports copy, write zeros in null slots, and fail on a value outside the + precision (D13). +2. `Narrow` over decimals (N28): the child rules of D10, the kernels and casts of D15, and the rule + that joins a `Narrow` over a `Narrow`. When `validate_dtypes` opens for decimals, the compare + kernel and the sum pushdown get decimal code in the same change. +3. DBP: require `u64` lower parts, and correct the module doc (`mod.rs:16`). Make decoding produce + the canonical array of the DBP dtype, with the `TODO(connor)` of D16 in `assemble.rs`. Decoding + widens a narrower MSP, and it narrows a wider one with the copy of D21. +4. The `vortex.decimal` read plugin converts old nodes (D9, D21, section 3.3). It scans the values, + because it cannot use stored statistics. P1 must be on `develop` first. +5. The DBP scheme takes the canonical array, and `narrowed_decimal` goes away (D17). The scheme + stays after the wire change, for writers whose edition does not permit `vortex.narrow`. File + contents stay the same. The exception is precision 39 to 76 with values above 64 bits, which + becomes a plain `i256` array. +6. `DecimalArray::try_new` accepts only the canonical width and checks every valid value, with + `new_unchecked` for kernels (D14). The `patch` panic, the fill-null upcast, and the "does not + fit" constant tests become unreachable and can go. Update tests that build narrow arrays on + purpose, for example `decimal_narrow_storage` + (`vortex-array/src/arrays/decimal/compute/mod.rs:82-86`). Builders follow, including + `DecimalBuilder::extend`, which accepts wider buffers (PR #10155). + +The wire change, in lockstep (D18), and if possible in the same release as the in-memory change +(D11): + +7. The `vortex.narrow` wire format for integers and decimals (D8), which is the second part of PR + #10333. +8. One new preview edition that permits `vortex.narrow` and DBP v2, together with the other wire + improvements. +9. A decimal `Narrow` scheme that narrows the decimal type first, for writers that select that + edition (D17). It runs before the DBP scheme, which then compresses the child. + +Later: + +10. `vortex.narrow` and DBP v2 enter `core` together (D18). +11. The max pass in DBP decoding (D16), and the pushdown of sum, min and max into the MSP for gap + columns (D19), if benchmarks show that they are needed. + +The waves below are future work for decimals (D20), and their recommendations are not reviewed. +Wave 2 also holds extension-type work, which D20 does not cover. + +Wave 2, foundations, in parallel with the canonical decimal fix: + +12. N7: width-keyed structural kernels for run-end, sparse, dict, constant, and builders. +13. N8: `vortex.int.rescale`, `vortex.int.check_range`, and the fused scaled kernels. Use them for + the Timestamp unit casts of N4. +14. N4: cast rules (#8570). Land N21 (Duration first, then Interval) on top of N1 and N4. +15. N2, then N3: the rewrite rules that types own, then derives for one dtype. Use the decimal + rules of section 3.4 as acceptance tests. +16. N23: plugin registries for DuckDB and DataFusion type mapping. + +Wave 3, new types: + +17. N9 in the order of section 5.3, with all four wide types. Recommendation (DQ5): user-visible + columns only behind an edition. +18. N10: the core dtype edition kind and the protobuf oneof. +19. N17: FixedSizeBinary, after DQ7. +20. N5: the value check in `ExtensionArray::try_new`. The decimal check is step 6 of the in-memory + change. + +Wave 4, decimal kernels: + +21. N12 for all precisions. Precision 19 to 76 needs N9. +22. N13: rules over the view. Delete `compare/decimal.rs`, most of `numeric/decimal.rs`, + `decimal/utils.rs`, and the decimal aggregates. +23. N14, which is optional. +24. N18 and N27. + +Wave 5, optional: + +25. N16, then N15 if the built-in type and an extension type become identical. Revisit N19 and + N20 with the results of N22. + +## 9. Open questions + +### 9.1 Questions in the decisions file + +The decisions file lists these questions as open. Each answer below is a recommendation of these +notes, which the maintainer has not confirmed. DQ5, DQ6, DQ7 and CQ1 belong to future work (D20). + +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. First compare, arithmetic, cast, sum, min, max, hash and structural operations | +| DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes (X3) | +| DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | +| XQ1 | Which arrays does the rule of X5 pull out, and what stays under the wrapper? | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. For `Decimal`, the wire forms already have this shape (section 3.3) | +| XQ2 | Where do the rules of X4 live? | On `ExtVTable`, written over `ReduceNode`. The built-in `Decimal` owns the same kind of rules | +| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, and never panic. For decimals, D14 settles this: file reads trust the precision. A read that makes a narrowing copy reads the values anyway, so it checks the precision (D21) | +| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet. The answer affects N18 | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error. Lossy steps such as `trunc` and `round` are separate expressions. The decimal-to-integer rule of section 3.4 keeps the current truncation until CQ1 is settled | + +### 9.2 Other open questions + +1. What is the lower bound of a negative scale? The docs name a `MIN_SCALE` that does not exist + (`vortex-array/src/dtype/decimal/mod.rs:47`). Tests use -128. DuckDB rejects negative scales, + and Arrow allows them. This question is future work (D20). +2. PRs #10467 and #10468 leave two decisions open. One is the order of intervals: field-wise as + in Arrow, or normalized as in DuckDB. The other is the edition membership of + `vortex.duration` and `vortex.interval`. +3. Does `is_int()` mean any integer or a lane integer? Section 5.3 proposes to remove it, so that + each of the 53 call sites chooses. + +## 10. Evidence log and limits + +Code read at commit `5fc5cb8`: the files cited in sections 2 to 6, plus the supporting +inventories. Every count in this report comes from `grep` over the workspace, excluding `target`. + +The maintainer decisions of 2026-10-19, 2026-10-22 and 2026-10-23 are recorded in +[`decisions.md`](../../decisions.md). Sections 1 and 3 to 9 follow them. Section 3 replaced a +wrapper design with one integer child, which D2 rejects. + +The review of 2026-10-23 read these files in the worktree of this branch, which has the code of +`5fc5cb8`: + +- `vortex-array/src/dtype/decimal/mod.rs:68`: a positive scale larger than the precision is + rejected, which limits how far a column can narrow (section 3.3). +- `encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs:16-35`, + `vortex-array/src/validity.rs:435-511` and `rules.rs:14`: the DBP cast rule and the contract + that it skips. The diff of PR #10230 for `cast.rs` adds the guard. +- `encodings/decimal-byte-parts/src/decimal_byte_parts/array.rs:111-118` and `split.rs:68-71`: + the constructor accepts any unsigned lower part, but the split makes only `u64` parts. +- The statistics trace of section 3.3: `vortex-compressor/src/compressor/cascade.rs:82-102` and + `:302-327`, `vortex-btrblocks/src/schemes/integer/{bitpacking,for_,pco}.rs`, + `vortex-layout/src/layouts/compressed.rs:73` and `:114-127`, + `vortex-layout/src/layouts/flat/reader.rs:150-152`, `vortex-array/src/array/erased.rs:238-249` + and `encodings/fastlanes/src/lib.rs:77-91`. A subagent read them, and the main review checked + the cited lines. +- `vortex-btrblocks/src/builder.rs:46-63` and `:190-193`: the compact mode reuses the + `DecimalScheme`. +- `vortex-arrow/src/dtype.rs:337-346` and `vortex-duckdb/src/exporter/decimal.rs:124`: the export + widths. +- The integer `Narrow` check in the local rewrite of #10223 + (`vortex-array/src/arrays/narrow/mod.rs:73-88` in the worktree `vortex3`). + +The second review of 2026-10-23 read these files in the same worktree: + +- `vortex-array/src/serde.rs:413-424`: the encoding assertion and the stored statistics come after + the plugin returns, so the plugin cannot use them (D9). +- `encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs:16`, `array.rs:97-118` and + `split.rs:98-118`: the module doc and the constructor allow narrower lower parts, and the split + keeps a buffer of `i8` to `i64` as the MSP. +- `vortex-btrblocks/src/schemes/decimal.rs:100-166`: the DBP scheme narrows first, and it returns + the narrowed leaf for `I128` and `I256` in v1 mode. +- Every call of `PrimitiveArrayExt::narrow` in `vortex-btrblocks/src` and + `vortex-compressor/src`: none narrows a DBP part (D23). +- `vortex-test/compat-gen/README.md`, `src/fixtures/arrays/synthetic/arrays/decimal.rs`, + `encodings/decimal_byte_parts.rs` and `encodings/mod.rs:47`: how the harness publishes and + checks fixtures, the decimal fixture at canonical `i32` and `i64` widths, the DBP fixtures with + `i64` MSPs, and the disabled v2 fixture (D24). +- `docs/specs/versioning/using-editions.md:47`: the `core` family holds the formats of the default + writer (D17). + +The review of 2026-10-22 read these files at `5fc5cb8`. They are the same on `develop` at +`dbaf5f8`, except that the `min_max` short-circuit moved from line 86 to line 89: + +- `docs/specs/versioning/design.md` and `docs/specs/versioning/editions.md`: the rule for new + wire IDs, the format selection rule, the compatibility invariants, and the process for a new + component. +- `vortex-array/src/array/plugin.rs`, `vortex-array/src/session/mod.rs:68` and + `encodings/alp/src/alp/plugin.rs:90`: how a plugin reads several wire forms and returns another + encoding. +- `vortex-array/src/serde.rs:283`, `:413` and `:419-424`: which statistics a node writes, the + encoding assertion, and when the reader attaches stored statistics. +- `vortex-array/src/aggregate_fn/fns/min_max/mod.rs:86`: `min_max` returns cached statistics + first. +- Every file under `encodings/decimal-byte-parts/src/decimal_byte_parts/`: the DBP semantics of + section 3.3. +- `vortex-btrblocks/src/schemes/decimal.rs:122-137`, and the history of `narrowed_decimal` + (commits `1b89b2f950` and `0036b1566d`): where narrow decimal nodes come from. +- `encodings/fastlanes/src/{for_,bitpacking}/plugin/mod.rs`: what the existing v2 IDs add. +- The `Narrow` compare kernel and cast rule on the `Narrow` stack + (`vortex-array/src/arrays/narrow/{compare,rules}.rs`): the rules that trust the child dtype. + +RFC drafts read from the `vortex-data/rfcs` repository: + +- Branch `ngates/types`: `rfcs/0062-value-types-and-representations.md`, read in full. +- Branch `ct/types`: `rfcs/0055-type-system.md`, read in full. +- Branch `ngates/rfc-0029-types`: `rfcs/0029-types.md`, read in full. It is an earlier draft of + RFC 0055 with the same decision framework and the same open FixedSizeBinary question. It frames + `DType` as a quotient type and canonicalization as a section, where RFC 0055 talks about + canonical execution. +- Branch `ngates/rfc-0005-extension`: `rfcs/0005-extension.md`, read in full. +- Branch `ngates/scalar-values`: `rfcs/0056-scalar-values.md`, read in part: the design sections + and every mention of decimals, extensions, and binary values. +- Branch `adamg/map-type-rfc`: `rfcs/0063-map-type.md` is an empty template. +- Branch `ngates/execution`: `rfcs/0019-array-execution.md`, skimmed. It is not relevant here. + +GitHub, read only: + +| Reference | Topic, and the depth of reading where it was recorded | +| --- | --- | +| [Issue #5820](https://github.com/vortex-data/vortex/issues/5820) | Decimal logical and physical mismatch. The body and all five human comments in full, and the 21 automated crash reports at the level of their summaries and errors | +| [Epic #7706](https://github.com/vortex-data/vortex/issues/7706) | Type system | +| [Issue #8570](https://github.com/vortex-data/vortex/issues/8570) | Cast hooks for extension types | +| [Issue #9902](https://github.com/vortex-data/vortex/issues/9902) | FixedSizeBinary, with its five comments | +| [Epic #10171](https://github.com/vortex-data/vortex/issues/10171) | Time series | +| [PR #8864](https://github.com/vortex-data/vortex/pull/8864) | Fixed-width sharing, merged | +| [PR #8865](https://github.com/vortex-data/vortex/pull/8865) | FixedSizeBinary prototype, closed. The description, the comments, and the first 100 of its 118 changed files | +| [PR #9886](https://github.com/vortex-data/vortex/pull/9886) | DataFusion literal for a UUID column, closed | +| [PR #10467](https://github.com/vortex-data/vortex/pull/10467) | Duration extension type. The files and the comments | +| [PR #10468](https://github.com/vortex-data/vortex/pull/10468) | Interval extension type. The file list and the new `interval.rs` | +| [PR #10223](https://github.com/vortex-data/vortex/pull/10223) | `Narrow` integer encoding, draft. The description, the diff, and the comments, read on 2026-10-19 | +| [PR #10332](https://github.com/vortex-data/vortex/pull/10332) | Arithmetic and selection through `Narrow`, draft, stacked on #10223. Same depth, same date | +| [PR #10333](https://github.com/vortex-data/vortex/pull/10333) | `Narrow` through compression, and `vortex.narrow` in `preview2026.10.0`, draft, stacked on #10332. Same depth, same date. On 2026-10-22, the review read the CodSpeed reports of #10223, #10332 and #10333, and the diff of #10333 again | +| [PR #10230](https://github.com/vortex-data/vortex/pull/10230) | Canonical decimals with one integer child, stacked on #10333. The description, the diff, and the comments, read on 2026-10-19. It closed on 2026-10-19. On 2026-10-23, the review read its diff of the DBP cast rule again | +| [PR #10305](https://github.com/vortex-data/vortex/pull/10305) | Decimal storage width at most the canonical width, open. The description, read on 2026-10-22 and 2026-10-23 | +| [PR #10314](https://github.com/vortex-data/vortex/pull/10314) | Bounded decimal dispatch macros, draft, stacked on #10305. The description, read on 2026-10-22 and 2026-10-23 | +| [PR #10681](https://github.com/vortex-data/vortex/pull/10681) | Removal of `array_session()`, merged on 2026-10-23. The commit message and its diff of `dict/compute/fill_null.rs`, read on 2026-10-23 | +| [PR #10155](https://github.com/vortex-data/vortex/pull/10155) | `DecimalBuilder::extend` from wider buffers for DuckDB imports, merged. The commit message, read on 2026-10-22 | +| [Issue #10701](https://github.com/vortex-data/vortex/issues/10701) | Tracking issue for the canonical decimal fix, opened by the maintainer on 2026-10-22. The body, read on 2026-10-23. Its question about the gap of section 3.3 was answered on 2026-10-23 | +| [Issue #10733](https://github.com/vortex-data/vortex/issues/10733) | Tracking issue for `Narrow` over integers and decimals, opened by the maintainer on 2026-10-23. The body, read on 2026-10-23. It keeps the question about `List` and `ListView` offsets open | +| [Epic #10730](https://github.com/vortex-data/vortex/issues/10730) | Hardening of the type system, opened by the maintainer on 2026-10-23. The body, read on 2026-10-23 | +| [Epic #10020](https://github.com/vortex-data/vortex/issues/10020) | Narrow integer encoding. The title and the first part of the body, read on 2026-10-19 | + +Verified outside the repository: + +- prost-derive 0.14.4 enum getters fall back to the first variant on unknown values + (`src/field/scalar.rs:286-300`, `src/lib.rs:308` in the cargo registry). +- arrow-buffer 59.3.0 `i256` layout and order (`src/bigint/mod.rs:59-66`, `:171-183`). +- arrow-rs 59.3, the version of the workspace: the precision limits of the four decimal types + (`arrow-data-59.3.1/src/decimal.rs`), the scale rule (`arrow-array-59.3.0/src/types.rs:1436`), + the construction without a value check (`arrow-array-59.3.0/src/array/primitive_array.rs:1642-1671`), + and the format comment on decimal widths (`arrow-ipc-59.3.1/src/gen/Schema.rs:3100-3102`). +- A search of the GitHub issues on 2026-10-23 found no issue for the DBP cast rule. +- Byte order against integer order, with the script in appendix A. + +Verified with a run: `../../extension-types/agents/probe_followup.rs` ran in a debug build of +`vortex-btrblocks` on `5fc5cb8`. + +- Probe R1 checks the decimal to `f32` cast of section 1.1. The array path fails with `No + CastKernel to cast canonical array vortex.decimal from decimal(10,2) to f32`. The row path + returns `[1f32, 2.5f32]`, and the scalar cast returns `1f32`. +- Probe R6 checks the `ext_storage` pushdown rules that section 3.3 cites. Existing rules push + `ext_storage` into dictionary values and chunks. `ext_storage` decodes a `Masked` input and + fails on a `Sparse` input with extension values. + +A script checked that every `file:line` reference in this report points at an existing range, and +the first line of each range was reviewed. The external crate references above were checked by +hand. A second review checked the main claims of sections 1 and 2 by hand again. + +Not verified, because no run covered them: + +- Any performance statement, including the cost of 128-bit and 256-bit kernels and the effect of + rewrites on fused arithmetic. +- The exact set of compile errors from the PType migration. The counts are a `grep` proxy. +- The alignment of `i128` on targets other than x86_64 and aarch64, for example wasm32. +- Whether the fuzzer reports in issue #5820 that end in a null decimal sum come from narrow + storage. This report does not rely on that. +- Whether a decimal `Narrow` reaches the speed of today's narrow leaves. No benchmark covers it. + The benchmark comment on PR #10230 measures a different design. +- The cost of the copies at Arrow import and in readers that convert old buffers. +- The in-memory cost of the full-width DBP decoding between the in-memory change and the decimal + `Narrow` scheme (D19). +- The size of the `i256` cost of D17 for default writers. No data shows how many columns have + precision 39 to 76 and values above 64 bits. +- How often columns hold gap values. The gap is a large part of each integer range, so these notes + do not assume that it is rare. +- Which integer scheme usually wins for a real MSP, and so how often an MSP has stored min and + max. +- Whether the DBP cast rule gives a wrong validity before execution. No test covers it. +- The cause of the `take_map` regressions in the CodSpeed report of PR #10333. The benchmark + does not call the narrowing code that the PR changes. + +## Appendix A: byte order script + +Run it with `python3 -I byte_order.py`. In this run, little-endian memcmp agreed with integer +order for 10,060 of 19,999 random pairs. Big-endian memcmp with the sign bit flipped agreed for all +19,999 pairs. + +```python +"""Compare memcmp order of 128-bit two's complement bytes with integer order.""" + +import random + + +def le(v: int) -> bytes: + return v.to_bytes(16, "little", signed=True) + + +def be_flip(v: int) -> bytes: + b = bytearray(v.to_bytes(16, "big", signed=True)) + b[0] ^= 0x80 + return bytes(b) + + +random.seed(7) +values = [random.randint(-(2**127), 2**127 - 1) for _ in range(20000)] +pairs = list(zip(values, values[1:])) +ok_le = sum((le(a) < le(b)) == (a < b) for a, b in pairs) +ok_flip = sum((be_flip(a) < be_flip(b)) == (a < b) for a, b in pairs) +print(f"little-endian: {ok_le} / {len(pairs)}") +print(f"big-endian, sign bit flipped: {ok_flip} / {len(pairs)}") +for a, b in [(1, 256), (-1, 0)]: + print(f"{a} < {b}: little-endian memcmp says {le(a) < le(b)}") +``` diff --git a/research/decisions.md b/research/decisions.md new file mode 100644 index 00000000000..af4741fb7c3 --- /dev/null +++ b/research/decisions.md @@ -0,0 +1,167 @@ + + +# Decisions + +This file records the decisions that the maintainer made while reviewing these notes on +2026-10-19, 2026-10-22 and 2026-10-23. It also lists the questions that stay open. The other +documents follow this file. If a document disagrees with it, this file wins. + +"Settled" marks a choice of the maintainer. "Recommendation" marks a proposal of these notes that +the maintainer has not confirmed yet. + +## Extension types + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| X1 | Extension types are opaque by default. No code applies storage semantics to an extension value unless the type opts in | Storage semantics give wrong results for most types (EXT-Q2, EXT-C2) | +| X2 | A derive (`eq` or `ord`) is a property of one extension dtype. It never applies across dtypes | Two dtypes with different metadata are different types | +| X3 | Engines convert operands to a common dtype. Vortex runs only operations whose operand dtypes already match | Engines already do this in their planners, and one owner avoids two rule sets that disagree | +| X4 | A type implements its behavior as rewrite rules on arrays. The rules are registered with the type, together with its compression scheme and the other parts of the type. There is no `ext_wrap` expression | A rule builds its result array directly, so a separate wrap expression adds nothing | +| X5 | A rewrite rule pulls generic structural arrays, such as `Constant` and `Dict`, out from under the extension wrapper | Code that looks for a constant or a dictionary then finds it at the top, without any knowledge of extension types | +| X6 | Extension values inside a struct, list or map are handled by removing extension layers at every depth. Nested kernels get no extension code | Kernels then see storage dtypes only | +| X7 | `ExtensionArray::try_new` checks every non-null value. A new `ExtensionArray::new_unchecked` skips the check for kernels that keep legal values | Today arrays are never checked (EXT-V1). This is a bug | +| X8 | API changes are acceptable where correctness needs them, for example fallible constructors and private enum variants | Correctness and clear abstractions matter more than the current API | +| X9 | Whether a validity query raises the errors of values that it skips is out of scope here | Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns this question, and it is not specific to extension types | + +### Open + +| ID | Question | Recommendation | +| --- | --- | --- | +| XQ1 | Which arrays does the rule of X5 pull out, and what stays under the wrapper? | Pull out the persistent structural encodings: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. Slices, filters and scalar functions stay under the wrapper. A canonical extension array is a wrapper over canonical storage | +| XQ2 | Where do the rules of X4 live? | On the `ExtVTable`, so that they travel with the dtype and need no session lookup. Write them over `ReduceNode`, so that they also rewrite bound expressions for pruning | +| XQ3 | Do file reads check extension values, or trust the file? | Check only what memory safety needs, trust semantic constraints, and never panic on an illegal value. Offer a strict read mode that checks values | +| XQ4 | Which values can sit in null slots? | Any value. Accessors that take a child out of its parent mask it with the validity of the parent. This also allows a storage encoding to store nulls as a reserved value | +| XQ5 | How does an aggregate declare its result? | As a value of its input, as a bound on its input, or as neither. Only a value gets wrapped again | +| XQ6 | Does `Uuid` order by its bytes? | No recommendation yet | + +## Decimal and new physical types + +Issue [#10701](https://github.com/vortex-data/vortex/issues/10701) tracks the fix of the canonical +decimal encoding. + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| D1 | `Decimal` stays a built-in type. It behaves like a hardcoded extension type: opaque by default, with derives for one dtype and rules that the type owns. If the two become identical, a later change can remove the built-in type | A move to an extension type adds no capability, and the built-in type can follow the same rules | +| D2 | A canonical decimal array is a leaf with one buffer and validity. It has no data child | A leaf is simpler than a wrapper over an integer child, and it is the same as the `vortex.decimal` wire node | +| D3 | Each decimal dtype has exactly one canonical encoding. The precision fixes the width of the buffer | Today one dtype has six physical forms, which causes the crashes in issue [#5820](https://github.com/vortex-data/vortex/issues/5820) | +| D4 | A `Narrow` encoding stores a decimal column as a child with a lower precision and the same scale | Without it, D3 forces the full width on every column, which costs performance | +| D5 | Vortex adds `I128`, `I256`, `U128` and `U256` as primitive integer types | Wide decimals need an integer view, and wide integers are useful as columns | +| D6 | FixedSizeBinary becomes a core type for opaque bytes of a fixed length, such as UUIDs and hashes | Its defining operations (equality, hashing, byte order, cheap scalars) are core operations. Arrow and Parquet have it natively | +| D7 | `vortex.decimal` keeps its wire ID. A reader finds an old node when its stored width differs from the canonical width of its dtype | D3 removes forms and adds none. The versioning spec asks for a new ID only when readers must understand something new | +| D8 | One `Narrow` encoding, with one `vortex.narrow` wire ID, covers integers and decimals. Integers narrow by width (issue [#10020](https://github.com/vortex-data/vortex/issues/10020)), and decimals narrow by precision. The metadata stores only the child `PType` of an integer child, or only the child precision of a decimal child, in a prost message with a typed `oneof` (D32). It stores no scale and no nullability. This settles DQ2 and DQ3 | A release that permits an ID must read every valid instance of it. So the decimal reader ships in the same release as the first edition that permits `vortex.narrow`. The child must have the scale and nullability of the parent, so metadata that stored them would only let a file hold an invalid child. The format is in no edition yet, so it can still change | +| D9 | A reader converts an old `vortex.decimal` node. For a narrower node, it scans the non-null values for their max, and it chooses the best precision that the values and the scale allow, or a larger one. The result is a `Narrow` over a child at the width of that precision, or the canonical array when that width is the canonical width of the column. The result copies the buffer when its width differs from the stored width. For a wider node, the reader makes a narrowing copy (D21) | A width does not give a precision. The read plugin cannot use stored statistics, because the deserializer attaches them after the plugin returns. Old narrow nodes are rare, so the speed of this path matters little | +| D10 | A decimal `Narrow` child has the same scale and nullability as its parent, and a canonical width strictly narrower than the canonical width of the parent. The producer chooses the child precision: the best precision that the values and the scale allow, or a larger one | A `Narrow` that does not reduce the width has no use. The integer `Narrow` has the same rule. The rule can be relaxed later | +| D11 | The canonical-width work lands as an in-memory change, and then as a wire change (D18). Each step of the in-memory change is its own PR, because none of the steps needs temporary code. The wire change lands in lockstep, if possible in the same release as the in-memory change. PRs [#10305](https://github.com/vortex-data/vortex/pull/10305) and [#10314](https://github.com/vortex-data/vortex/pull/10314) stay open, and the work ignores them for now | Small PRs are easier to review, but not when a split needs temporary code or changes file contents twice. In one release, writers that select the preview edition never pay the `i256` cost of D17 | +| D12 | Each precision has the smallest signed width that holds its values: `i8` for precision 1 to 2, `i16` for 3 to 4, `i32` for 5 to 9, `i64` for 10 to 18, `i128` for 19 to 38 and `i256` for 39 to 76. `DecimalDType::canonical_values_type()` owns the mapping. Its name differs from the stored-width accessor `DecimalArray::values_type()`, so code cannot mix the two up. This settles DQ1 | It is the existing `smallest_decimal_value_type`. From precision 5, it is the width of the smallest Arrow decimal type that holds the precision | +| D13 | Imports copy into the canonical width. They write zeros in null slots, and they fail on a value outside the precision. The Arrow import logic can be configured to handle such values in another way. To widen an array, for example for an export, Vortex executes a `Narrow` with a wider dtype over it | arrow-rs does not check decimal values when it builds an array. An error is better than a silent change of the data | +| D14 | `DecimalArray::try_new` checks the width and every valid value, with an `ExecutionCtx` (D30). `new_unchecked` skips the value check, for kernels that keep values legal. File reads trust the precision and never panic | This is the rule of X7 for decimals, and the recommendation XQ3 for file reads | +| D15 | `Narrow` kernels pass work to the child, with canonicalization as the fallback. A `Narrow` over a `Narrow` becomes one `Narrow`. A cast of a `Narrow` passes to the child with the same target dtype, except that a wider integer target returns a `Narrow` over the child. Another decimal cast to a precision with the same scale returns a `Narrow` when the canonical width grows, changes only the dtype when the width stays, and checks every value when the precision drops | A `Narrow` holds exactly the values of its child, so work on the child gives the same result on less data | +| D16 | In DecimalByteParts, the MSP is any signed integer up to 64 bits, and every lower part has the dtype `u64`. The constructor and the module doc become stricter, and the constructor rejects lower parts that assemble wider than the canonical width (D29). This settles DQ4. Operations run on the parts wherever they can. Decoding produces the canonical array of the DecimalByteParts dtype. It widens a narrower MSP, and it narrows a wider MSP with the copy of D21. A `TODO(connor)` marks a later optimization: find the max of a narrow MSP, and return a `Narrow` that reuses its buffer | A narrow MSP needs no conversion at read time. The stricter constructor breaks no file, because no file has lower parts: no edition permits v2, and its compat fixture is disabled. The compressor never narrows the dtype of a part. Simple decoding is correct, and the optimization waits until benchmarks show large regressions | +| D17 | The compressor has one DecimalByteParts scheme. The scheme takes the canonical array, uses the value range to choose the number of lower parts, and narrows the MSP dtype as the compressor narrows every internal child. This replaces `narrowed_decimal` and keeps file contents, except that large values in null slots can give a narrower MSP. The scheme stays after the wire change, for writers whose edition does not permit `vortex.narrow`. When an edition permits `vortex.narrow`, a `Narrow` scheme runs before it and narrows the decimal type first (D33). In an edition without the lower-parts format, precision 39 to 76 with values that need more than 64 bits is written as a plain `i256` array. The `core` family holds the formats of the default writer, so default writers pay this cost until both formats enter `core` | The canonical fix makes a narrow decimal array illegal in memory, so the scheme cannot call `narrowed_decimal`. File contents do not change before the wire change. The `i256` cost affects only very wide decimals with values above 64 bits. The only way around it writes the old narrow form into new files, and it needs a special serializer | +| D18 | `vortex.narrow` stays in memory only until the maintainer decides to write it to files. Every new wire format enters a preview edition before `core`. `vortex.narrow` and DecimalByteParts with lower parts enter one preview edition together, and later `core` together. Other IDs can join that edition, but this plan adds only these two. This settles DQ9 | In-memory changes land first, so the edition change stays small | +| D19 | The bar of no compute regression applies to the final state, not to the steps in between. Missing benchmarks land before the changes that they cover. A gap column holds values that fit a width but need the precision of the next width. In the final state, such a column uses the wider width in memory, and that cost is accepted. Pushdown of sum, min and max into the MSP can come later | A true child label needs the next width for such values. Disk size and exports do not change | +| D20 | The canonical decimal work does not include the integer view, wide integers, FixedSizeBinary, decimal arithmetic across precisions and scales, casts that lose information, or the lower bound of a negative scale. Their recommendations are unreviewed future work. Epic [#10730](https://github.com/vortex-data/vortex/issues/10730) tracks wide integers and FixedSizeBinary. Issue [#9902](https://github.com/vortex-data/vortex/issues/9902) is its sub-issue for FixedSizeBinary, and wide integers get a tracking issue when that work is scheduled. The recommendation to always store min and max on the MSP (DQ8) is dropped | The canonical fix needs none of them. Stored min and max exist only on a BitPacked MSP, and a slice drops them | +| D21 | Every narrowing copy writes zeros in null slots, and it fails on a valid value outside the precision of the target dtype. Three paths make such a copy: an old node wider than its canonical width, an old narrow node whose best precision has a smaller width than the stored width, and DecimalByteParts decoding with an MSP wider than the canonical width. A read without a copy trusts the precision and does not scan | The copy reads every value anyway, so the precision check costs nothing extra, and it matches the import rule of D13. A check without a copy needs a scan that the read does not otherwise make | +| D22 | An integer `Narrow` child has the same signedness and nullability as its parent, and a strictly smaller width. So an `I64` column with values from 0 to 200 gets an `I16` child, not a `U8` child | Mixed signedness needs more code in casts, in the compare shortcut and in the arithmetic bounds, and it saves at most one width. Part 1 of PR [#10333](https://github.com/vortex-data/vortex/pull/10333) also keeps the signedness of internal buffers | +| D23 | `vortex.narrow` is only for columns with a fixed logical dtype: decimal columns and top-level integer columns. Which arrays the compressor wraps in a `Narrow` is deferred (D27). Internal integer children keep changing their own dtype, through the deprecated `encode_values` (D28). These include list offsets, dictionary codes, run ends, patch and sparse indices, `DateTimeParts`, VarBin and FSST offsets, and the DecimalByteParts MSP. Whether `List` and `ListView` offsets get a fixed `I64` dtype with `Narrow` is an open question on issue [#10733](https://github.com/vortex-data/vortex/issues/10733), and the decimal work does not depend on it | The parent chooses the dtype of an internal child, so the child can narrow its dtype without a `Narrow`. A column has a dtype that the user chose | +| D24 | The compat fixtures are published from a release before the PRs that change how files are read (D31), because only the old writer can make narrow leaves. They cover only reads that succeed: narrow, wide and Arrow-style widths, gap values, high scales at narrow widths, nonzero values in null slots, and DecimalByteParts with a narrow MSP. Error cases, such as a value outside the precision in a narrowing copy, are unit tests | The compat harness (`vortex-test/compat-gen`) expects the current reader to read every published fixture | +| D25 | Generic structural encodings always sit above `Narrow`. Reduce rules move `Dict`, `RunEnd`, `Sparse`, `Constant`, `Masked`, `Chunked` and `Interleave` above a `Narrow`. The chunked and interleave merge rules of the local rewrite, which put `Narrow` on top, are dropped. A follow-up issue covers the same rule for every encoding | The Arrow and DuckDB exporters and the dict layout look for these encodings at the root. One direction for every rule cannot loop | +| D26 | When a writer cannot write `vortex.narrow`, because no wire form exists yet or the selected editions do not permit it, the serializer writes the executed canonical array. The code documents this as a workaround. It stays until `vortex.narrow` is in `core` | Decimal `Narrow` appears in ordinary pipelines through casts and old files, and IPC or a writer without compression would fail on it. A plugin cannot return the wire form of another encoding, so the fallback lives in the serializer | +| D27 | Which arrays the compressor narrows is deferred to the design pass of WR3 | The candidates include struct fields, list elements, extension storage, and arrays that layouts create, such as dict-layout codes and zone maps | +| D28 | `Narrow` has `encode`, which builds a `Narrow`, and `encode_values`, which returns only the narrowed child with the same signedness. `encode_values` is `#[deprecated]`. It replaces `PrimitiveArrayExt::narrow` at every call site, and each call carries `#[expect(deprecated)]` with a `TODO(connor)` reason. The PR that moves the call sites checks that the latest release reads the files that it writes | Every narrowing that changes a dtype is then marked for a later decision. Readers on `develop` accept signed internal children | +| D29 | The DecimalByteParts constructor rejects a lower-part count whose assembled width is wider than the canonical width. Precision up to 18 allows no lower parts, 19 to 38 at most one, and 39 to 76 any count | Decoding then never needs a narrowing copy after assembly. No file breaks, because no edition permits lower parts | +| D30 | `DecimalArray::try_new` takes an `ExecutionCtx`, because the value check reads the validity, which can be lazy. Imports check their own values against the Arrow null buffer and then use the unchecked constructors, so a zero-copy import scans once | This follows `VarBinViewArray::try_new`, and it keeps the context out of the import API | +| D31 | Library changes land first. Only the PRs that change how files are read (DecimalByteParts decoding and the read plugin) wait for the compat fixtures, which the next regular release publishes. This changes D24 | The fixtures must be in the store when CI first runs the changed reader. Nothing else needs them | +| D32 | The `vortex.narrow` metadata is a prost message with a typed `oneof`: the child `PType` of an integer parent, or the child precision of a decimal parent. A missing variant is an error, and the `PType` decodes strictly | The typed `oneof` keeps the two kinds apart, and a missing variant cannot decode silently as `U8` | +| D33 | The compressor uses a real `Narrow` scheme at first. If the compressor turns out to always want it, it can become a hard-coded step. The design pass of WR3 settles its precedence with the DecimalByteParts scheme and the cascade level that it uses | A scheme fits the existing framework. The `AlwaysUse` contract and the cascade budget are the two costs to settle | +| D34 | Strict decoding covers the decimal width, the protobuf `PType` and the eleven `PType` fields of array and layout metadata, in one PR with three commits | It is the same bug with the same fix | +| D35 | The integer `Narrow` lands as five stacked PRs: the encoding, benchmarks, aggregates, arithmetic and casts, and the structural rules. The PR that moves the call sites to `encode_values` comes after them. The dictionary fill-code fix lands before the fifth PR | The benchmarks exist before the kernels change, and the fill-code fix guards the dictionary-codes rule | +| D36 | An operation on a `Narrow` fails only where the same operation on the parent dtype fails, and it gives the same result. Aggregates on a `Narrow` use the accumulator of the parent dtype and feed it the child, for integers and decimals alike, so `Sum` and `SumV2` need no `Narrow` code | `Narrow` is a physical encoding, so it must not change behavior. A child accumulator overflows at a lower precision than the parent | +| D37 | The `vortex.decimal` read plugin converts old narrow nodes on the host, also on CUDA read paths. A GPU scan of such a node copies its buffer to the host | Old narrow nodes are rare, and CUDA has no `Narrow` kernel | +| D38 | The integer `Narrow` PRs are written from scratch from their briefs. The local rewrite of #10223 and #10332 is a reference for tests and edge cases, not a base. FX3 ports the rewrite's dictionary fill fix as it is | The rewrite predates D25, D26, D28 and D36, so a cherry-pick would carry its old structure into the new PRs | + +D1 to D6 date from 2026-10-19, D7 and D8 from 2026-10-22, and D12 to D38 from 2026-10-23. On +2026-10-23 the maintainer also revised D9, D10 and D11. Their 2026-10-22 versions used `P(W)`, the +largest precision whose canonical width is `W`, as the child precision of old data, and they planned +the work as one complete change. A second review on 2026-10-23 added D21 to D24, and it changed D8, +D9, D11, D16, D17 and D20. A planning review on the same day added D25 to D38, and it changed D8, +D12, D14, D15, D16, D17, D18, D23 and D24. [Part 2](decimal/02-design.md) of the Decimal notes +explains the design decisions from D7 on, and [part 4](decimal/04-implementation.md) splits the work +into PRs. + +### Open + +These questions belong to future work (D20). Their recommendations are not reviewed. + +| ID | Question | Recommendation | +| --- | --- | --- | +| DQ5 | Are wide integers user-visible columns, and which kernels do they get first? | User-visible behind an edition. A small kernel set first: compare, arithmetic, cast, sum, min, max, hash and structural operations | +| DQ6 | Does decimal arithmetic accept operands with different precision and scale? | Yes, with the Arrow rules for the result type. Comparisons keep the rule of equal dtypes | +| DQ7 | How does FixedSizeBinary appear on the wire? | As a new core dtype tag behind an edition | + +## Casts + +### Settled + +| ID | Decision | Reason | +| --- | --- | --- | +| C1 | A cast between two dtypes exists only if a rule for that pair exists. Vortex ships rules for common pairs of built-in types, as the Rust standard library ships `From` implementations. Reading storage as another type is never a cast | Opt-in conversions with good defaults. Today casts reinterpret storage and disagree between paths | + +### Open + +| ID | Question | Recommendation | +| --- | --- | --- | +| CQ1 | What does the default cast do with a value that it cannot represent exactly? | It returns an error, like `TryFrom`. Lossy steps such as `trunc` and `round` are separate, named expressions. For decimals, this is future work (D20) | +| CQ2 | How do struct casts match fields? | By name, with unique field names (issue [#6900](https://github.com/vortex-data/vortex/issues/6900)). Extra source fields return an error | + +## The `Narrow` pull requests + +This review read the draft stack on 2026-10-19 and again on 2026-10-22. The state below is from +2026-10-23. + +- The draft PRs [#10223](https://github.com/vortex-data/vortex/pull/10223), + [#10332](https://github.com/vortex-data/vortex/pull/10332) and + [#10333](https://github.com/vortex-data/vortex/pull/10333) date from 2026-10-06. The maintainer + closed all three on 2026-10-23, and new PRs replace them. Issue + [#10733](https://github.com/vortex-data/vortex/issues/10733) tracks the new PRs. +- A rewrite of #10223 and #10332 exists in five local commits, but it is not pushed. It makes + `Narrow` an in-memory encoding with no serialized form. The new PRs are written from scratch, and + the rewrite is only a reference for tests and edge cases (D38). It predates + [#10681](https://github.com/vortex-data/vortex/pull/10681), which removed `array_session()`. The + local branch has the name `ct/narrow-pushdown`, which is the head branch of the closed #10332, so + the new PRs use new branch names. +- The rewrite also fixes a fill code that the dictionary `fill_null` kernel truncates. That bug + exists on `develop` without `Narrow`, so the fix becomes its own PR. +- The only integer-only check of #10223 and #10332 is the private `validate_dtypes`, so decimal + support stays additive. When that check opens for decimals, the compare kernel and the sum + pushdown need decimal code in the same change. +- The CodSpeed reports on #10223 and #10332 compare against `3c87da0`, because `cc018ee` had no + run. Their gains, such as those of the FSST benchmarks, do not come from `Narrow`. The new PRs + need new reports. +- The work of PR #10333 comes back in new PRs: + 1. NR1 adds `NarrowArray::encode_values`, and NR6 replaces `PrimitiveArrayExt::narrow` with it at + every call site (D28). Internal buffers then keep their signedness in every preset. Golden + snapshots and file contents change, but no wire format changes. + 2. The serialization of D8, the edition entry and the `Narrow` scheme in the compressor are part + of the wire change, WR1 to WR3 (D11, D18). +- [Part 4](decimal/04-implementation.md) of the Decimal notes lists every new PR, and the briefs in + `decimal/agents/briefs/` describe each one. +- The CodSpeed report on #10333 shows `take_map` regressions of 22% and 11% in Simulation mode. + The `take_map` benchmark builds `u64` patch indices directly and does not call the narrowing + code. The cause is not known. The new PRs get fresh CodSpeed runs, and the investigation + happens there. +- PR [#10230](https://github.com/vortex-data/vortex/pull/10230) gave decimals an integer child, + which D2 replaces. It closed on 2026-10-19. Its fix to the DecimalByteParts cast rule becomes its + own PR, with no issue. The PR adds the guard of #10230, a cast kernel for the case that needs + execution, and tests. [Part 3](decimal/03-roadmap.md) of the Decimal notes replaces the other + steps of #10230. +- PR #10305 limits the decimal storage width to the canonical width, and PR #10314 adds bounded + dispatch macros on top of it. Another contributor owns both, and both are open. The work + ignores them for now (D11). diff --git a/research/extension-types/01-why-opaque.md b/research/extension-types/01-why-opaque.md new file mode 100644 index 00000000000..47eeac136ab --- /dev/null +++ b/research/extension-types/01-why-opaque.md @@ -0,0 +1,99 @@ + + +# 1. Why extension types must be opaque + +## Terms + +An **extension type** gives new meaning to the values of a **storage dtype**. `Timestamp` stores an +`i64`, and the extension type says that the `i64` counts time units since the Unix epoch. `Json` +stores `Utf8`, and the extension type says that the text is a JSON document. + +A **structural operation** does not read values. Slice, take, filter, mask, validity, reading a row, +serialization and compression of the storage are structural. A **semantic operation** depends on +what the values mean. Equality, ordering, arithmetic, casts, aggregates over values, zero values and +export to other systems are semantic. + +## The argument + +A storage operation computes a result for the storage meaning. A semantic operation on an +extension type can run on storage. The result is then correct only if the storage result and the +extension result agree for every input. Agreement depends on the extension type, on the operation, +and often on the metadata: + +| Extension type | Operation | What storage computes | Agrees | +| --- | --- | --- | --- | +| `Timestamp(ms)` | `<` with `Timestamp(ms)` | `i64` order | Yes | +| `Timestamp(ms)` | `<` with `Timestamp(ns)` | Order of counts in different units | No | +| `Decimal(10, 2)` | `<` with `Decimal(10, 2)` | Order of unscaled integers | Yes | +| `Decimal(10, 2)` | `*` with `Decimal(10, 2)` | A product with scale 4 | No, the result type changes | +| `Json` | `=` | Text equality | No, key order and spaces differ | +| WKB geometry | `=` | Byte equality | Only in one direction (see below) | +| Unit vector | `+` | Element sums | No, the sum is not a unit vector | +| `Uuid` | `<` | Byte order | A design decision | + +`Decimal` is a built-in dtype, but it has the same shape: an integer storage value plus a scale. The +[Decimal notes](../decimal/README.md) use it as a test of this design. + +The meaning lives in the extension type, so the core cannot know which rows of this table are +"yes". The core has two choices: + +- **Transparent:** inherit storage behavior unless the type opts out. When an author forgets to opt + out, the result is a silent wrong value. +- **Opaque:** inherit no semantic behavior unless the type opts in. When an author forgets to opt + in, the result is an "unsupported" error. + +Only the second failure is safe. It also covers types that nobody can audit: third-party types and +unknown types read from a file. Structural operations are the exception. They never read values, so +they are correct for every extension type and always stay available. + +Opaque does not mean inaccessible. `ext_storage` remains the explicit way to read the storage, like +`.0` on a public newtype field. + +## The newtype model + +A Rust newtype has the same split: + +| Rust | Vortex extension type | +| --- | --- | +| `struct Meters(f64)` is a separate type | Identity is the ID, the metadata and the storage dtype | +| `#[repr(transparent)]` costs nothing | One storage child, compressed as storage | +| `Vec` can slice, filter and index | Structural operations work on the storage child | +| `m.0` reads the field | `ext_storage` | +| `Meters::new(x)` can check invariants | Building an extension value checks the rules of the type | +| Nothing is inherited without `#[derive]` | No semantic operation is inherited without an opt-in | +| `impl From for Meters` converts on purpose | The extension type owns its casts | + +## Equality needs care + +Within one extension dtype, equal storage values are equal extension values. The reverse holds only +when each value has exactly one storage form. This document calls that **canonical storage**. +`Timestamp` and `Decimal` storage are canonical. JSON text and WKB bytes are not, because one +document or geometry has many encodings. + +So storage equality is safe for compression and for constant detection. `!=`, `DISTINCT`, `GROUP BY` +and joins are safe only when the type declares that its storage is canonical. + +## Refinement types are a special case + +A **refinement type** restricts a parent type to a subset of its values. Examples are UTF-8 text +over binary, and a unit vector over a vector. Every value of the refinement is a valid value of the +parent, so reads and predicates of the parent stay correct. Operations that produce new values +return the parent type, because the result can leave the subset. In the opaque model, a refinement +type opts in to all of the behavior of its parent and adds a value check. + +## Prior art + +- **Postgres** has both models. A base type made with `CREATE TYPE` gets ordering only from an + operator class. A `DOMAIN` is a refinement: it inherits the operators of its base type and adds + `CHECK` constraints. The `json` type stores text and has no `=` operator. The `jsonb` type stores + a canonical form, and it defines `=`, ordering and hashing on that form. +- **Parquet** first wrote min and max statistics in the physical sort order. That order was wrong + for some logical types, and the format later added a declared column order. +- **Haskell** newtypes derive nothing by default. `deriving newtype` and `DerivingVia` opt in. + Type roles decide when a value can be converted between a newtype and its representation. +- The draft [RFC 0062](https://github.com/vortex-data/rfcs/blob/ngates/types/rfcs/0062-value-types-and-representations.md) + states the same rule: "An extension does not inherit storage comparison, hashing, arithmetic, or + parsing by default." diff --git a/research/extension-types/02-current-problems.md b/research/extension-types/02-current-problems.md new file mode 100644 index 00000000000..ebda0758587 --- /dev/null +++ b/research/extension-types/02-current-problems.md @@ -0,0 +1,120 @@ + + +# 2. Current problems on `develop` + +All findings are for `develop` at `5fc5cb8`. Each ID refers to an entry in +[`agents/findings.md`](agents/findings.md), which has file and line references, reproductions and +fix sketches. "Confirmed" means that a throwaway test ([`agents/probe.rs`](agents/probe.rs) or +[`agents/probe_followup.rs`](agents/probe_followup.rs)) reproduced the behavior. + +## Results depend on the encoding and the read path (EXT-P1) + +The same cast or comparison returns a correct value, a wrong value, or an error. The outcome depends +on the encoding of the input, and on whether one row or the whole array is read. Confirmed: + +| Expression | Whole array | One row (`execute_scalar`) | +| --- | --- | --- | +| `cast(Timestamp(ms) to Timestamp(ns))` | Error | Wrong: one second becomes one microsecond | +| Same cast, constant input | Wrong: one second becomes one microsecond | Wrong | +| `cast(i64 to Timestamp(ms))` | Error | Wraps the raw `i64` | +| `Timestamp(ms) < i64` | Compares raw storage | Error | + +Cast and compare each have two implementations. Constant inputs use the scalar implementation, and +other encodings use the array kernels. The two implementations apply different rules to extension +types. Reading one row wraps each input row in a one-row constant, so it always takes the scalar +implementation. + +## Wrong results + +- **Casts reinterpret storage (EXT-C1, EXT-C2).** A scalar cast into an extension type casts to the + storage dtype and wraps the result. The source type plays no part. An array cast out of an + extension type returns the storage cast, so `Timestamp(ms)` casts to `i64`, `f64` and `u8`. A + scalar cast out of an extension type accepts the exact storage dtype as a target. Issue + [#7504](https://github.com/vortex-data/vortex/issues/7504) decided to remove the array cast out of + an extension type, but it is still on `develop`. +- **Comparisons against raw storage (EXT-Q1).** The compare type check is skipped when either side + is an extension. A lone extension side then compares its storage, so `ts < 1500` compares raw + `i64` values. `between` checks types strictly, so `ts BETWEEN 0 AND 1500` fails while + `ts >= 0 AND ts <= 1500` runs. +- **Storage equality and order for every type (EXT-Q2).** Any two values of one extension dtype + compare by storage. This happens in the compare kernel, the nested comparator and `ExtScalar`. A + test type with no ordering still supports `<`. The built-in kernel runs first, and the registry + cannot override it, so an extension type has no way to replace it. +- **`DateTimeParts` compare (EXT-Q3).** The kernel calls `as_extension()` on its constant operand + before it checks the type, and it does not compare time units. Confirmed: against an `i64` + constant, the array path panics. Against a `Timestamp(ns)` constant, a `Timestamp(ms)` column + gets a wrong result. +- **Engine boundaries (EXT-B1 to EXT-B5).** DataFusion converts non-temporal extension scalars to + storage scalars. Arrow export to a storage-like target type exports storage through the cast + fallback. Python `as_py()` returns the storage value. DuckDB decides aggregate pushdown from the + storage type. JNI write summaries report storage values. + +## Panics and invalid values + +- **Arrays are never value-checked (EXT-V1).** `ExtensionArray::new` and `ExtensionBuilder` accept + values that the type rejects. Reading such a row panics. Confirmed with a type that rejects zero. + `ext_storage` over the same array returns the zero on the array path and panics on the row path. +- **Scalars skip the storage check (EXT-V7).** An extension scalar can hold a value of another + storage type, for example a `Timestamp` with an `f64` value. Then `==` panics in one direction + and returns `false` in the other. Confirmed. +- **Nullability changes can panic (EXT-N1).** `ExtDType::with_nullability` validates the type again + and panics if validation fails. A type that accepts only non-nullable storage panics in `take` + with nullable indices. Confirmed. +- **Zero values skip validation (EXT-V3).** `Scalar::zero_value` uses the storage zero. For a type + that rejects zero, debug builds panic and release builds return an invalid scalar. Confirmed. +- **Built-in temporal types (EXT-V4, EXT-V5).** `Date` and `Time` panic on non-primitive storage, so + a malformed file panics instead of returning an error. `Timestamp` accepts the `Days` unit and + unknown time zones, and then rejects every value. +- **Arrays skip the storage dtype check (EXT-V2).** `Extension::validate` checks the length, but + it compares the dtype with itself. It accepts `Utf8` storage under a `Timestamp` dtype. + Confirmed. + +## Disagreements + +- **Ordering (EXT-A1).** `min_max` uses storage order for every extension type. `is_sorted` returns + `false` for every extension type, so a sorted timestamp column reports "not sorted". `vortex-row` + rejects extension types. `sum` rejects them. Three open PRs add three different opt-in mechanisms: + [#10246](https://github.com/vortex-data/vortex/pull/10246), + [#10566](https://github.com/vortex-data/vortex/pull/10566) and + [#10408](https://github.com/vortex-data/vortex/pull/10408). +- **Zone maps (EXT-A2).** When a statistic rejects an extension dtype, the zone map uses the storage + dtype. JSON and WKB columns store exact, unbounded bounds, so WKB zone maps hold whole geometry + blobs. +- **Two shapes for one constant (EXT-S1).** A constant extension column is either + `Constant(extension scalar)` or `Extension(Constant(storage))`. Code must handle both. +- **Encodings with an extension dtype (EXT-S2).** `DateTimeParts` carries the `Timestamp` dtype + itself, so an extension-typed array is not always the wrapper. +- **Unknown types (EXT-R3).** A reader without the type gets a placeholder that compares and orders + values by storage. + +## Root causes + +1. **One dispatch key.** Every extension type shares the `vortex.ext` encoding, so a kernel written + for one extension type runs for all of them. +2. **No declaration of inherited behavior.** `ExtVTable` cannot say which semantic operations a type + shares with its storage, so each subsystem guesses. +3. **Two implementations.** Cast and compare each have a scalar and an array implementation with + different extension rules. +4. **Trusted arrays.** Values are checked only when a scalar is built, and that check panics. +5. **Nullability in the storage dtype.** Any nullability change validates the type again. +6. **Manual authoring.** The core has no typed array view, no explicit wrap expression, and no + export hooks for DuckDB, DataFusion or Python. Extension crates fill the gaps by hand. + `vortex-spatial` has seven typed array wrappers, and its Arrow code is 4.7 times the size of its + type definitions. + +
+Smaller problems + +- A same-type extension cast hides the storage error behind "no `CastKernel`" (EXT-C3). +- Eight extension types never check values, because `unpack_native` returns the raw storage + (EXT-V6). +- `Debug for ExtDTypeRef` prints the metadata before the struct, then `metadata: Ok(())` (EXT-R1). +- `DTypeSession::register` silently replaces a type with the same ID, and Arrow plugins register + twice if `initialize` runs twice (EXT-R2). +- The developer guide page for extension dtypes is a stub, and the vtables page shows an old API + (EXT-R4). + +
diff --git a/research/extension-types/03-design.md b/research/extension-types/03-design.md new file mode 100644 index 00000000000..ddd5bfdc041 --- /dev/null +++ b/research/extension-types/03-design.md @@ -0,0 +1,282 @@ + + +# 3. Design + +This part follows [`../decisions.md`](../decisions.md), which records the decisions of the +maintainer. IDs such as X4 and XQ1 refer to that file. + +## Goal + +Extension types behave like Rust newtypes. Structural operations always work. A semantic operation +works only where the extension type opts in or provides its own rule. A function that declares that +it understands the type can also allow it. No code path applies storage semantics to an extension +value without that opt-in. + +## Rules + +1. A semantic operation on an extension type is rejected unless the type or a type-aware function + opts in (X1). +2. Structural operations are always allowed. +3. Two kinds of rules decide how generic operations treat extension values: the derive rules of the + core and the rules that each type owns. Every node meets a typing check when code builds it, and + the rules run on every path. +4. After the rules run, the kernels of generic semantic operations, statistics and pruning see + storage dtypes only. A function that declares an extension type can see that type. +5. The extension type owns every conversion that changes meaning. There are no implicit casts. +6. An unknown extension type supports structural operations and `ext_storage` only. + +## Models considered + +| Model | Summary | Verdict | +| --- | --- | --- | +| Transparent | Every operation runs on storage | Rejected: wrong results for most types | +| Refinement for every type | Inherit reads and predicates, return the parent type for new values | Correct only for true subtypes, kept as an opt-in | +| Opaque, flags in each subsystem | Each consumer reads a declaration and decides | Every consumer must check, and new consumers drift | +| Opaque, kernels per extension ID | Look up `(function, extension ID)` before the generic kernel | Computes results, but statistics and pruning learn nothing | +| Type-owned encodings | Each extension type gets its own array encoding | Mixes logical type and encoding, and generic encodings still carry the type | +| Full dtype plugin | The type defines its own arrays, scalars and compute | The most power and the most work, and storage becomes optional | +| **Opaque, rewrite rules** | Derive rules and rules that the type owns rewrite semantic operations into storage operations | **Chosen (X1, X4)** | + +The deciding property is the number of places that must understand extension semantics. With +flags, every consumer must read the flag correctly. Today, three groups of consumers give three +different answers to "is this type ordered?" (EXT-A1, EXT-B4): + +- `compare` and `min_max` treat every extension type as ordered. +- `is_sorted` and `vortex-row` treat no extension type as ordered. +- DuckDB pushdown treats every extension type except `Uuid` as ordered. + +With rewrite rules, only the rules need to understand extension semantics. Unlike kernels, rules +also rewrite predicates, so statistics and pruning keep working on storage. + +## Proposal + +### Typing and rewrite rules + +A call with an extension operand passes two steps: + +1. Typing checks the call when code builds the node. It accepts four kinds of calls. They are + structural calls, calls of a type-aware function, calls that a derive covers, and calls that a + rule of the type covers. Everything else is an error. +2. Rewrite rules replace an accepted semantic call with operations on storage. + +```text +ts_ms < lit(Timestamp(ms) 1500) => ext_storage(ts_ms) < 1500i64 +min(ts_ms) => min(ext_storage(ts_ms)), and the rule builds a Timestamp(ms) + scalar from the result +cast(ts_ms, Timestamp(ns)) => Extension(Timestamp(ns), + checked_mul(ext_storage(ts_ms), 1_000_000)) +json_a = json_b => rejected, because Json derives no equality +``` + +The rules run in the optimizer and in the executor, so the row path and the array path apply the +same rewrite. A missing rule gives an error, never a wrong result. Pushdown, pruning and +compression keep working, because the rewritten operations are ordinary storage operations. + +### Derives for one dtype + +A derive says that a semantic operation on values of one extension dtype is the same operation on +their storage values. It is a property of one dtype (X2): + +```rust +/// Storage behavior that values of `dtype` inherit, like `#[derive]` on a newtype. +/// +/// The default inherits nothing, so every semantic operation on this dtype is rejected. +fn derives(dtype: &ExtDType) -> Derives { + Derives::default() +} +``` + +`Derives` has two flags. `eq` means that storage equality is value equality, which also makes the +storage hash valid. `ord` means that storage order is value order. + +A derive applies only when every operand in a derived position has the same extension dtype: + +- A comparison needs two such operands, and `between` needs three. +- A unary aggregate, such as `min` or `is_sorted`, reads the flags of its one input. +- A struct or list that holds extension values inherits a flag only if every nested extension + type declares it. The derive rule then removes every extension layer at every depth, so the + kernel sees only storage (X6). + +So `ts < 5i64` stays rejected, because only one side is an extension. + +| Type | Derives | +| --- | --- | +| `Timestamp` | `eq`, `ord` | +| `Date`, `Time` | `eq`, `ord` | +| `Decimal`, which stays built-in but follows the same rules (D1) | `eq`, `ord` | +| `Uuid` | `eq`, plus `ord` if byte order is the chosen order | +| `Vector`, `FixedShapeTensor` | `eq` | +| `Json`, WKB, native geometry | None | +| Unknown type | None | + +### Operations across dtypes + +Two dtypes with different metadata are different dtypes, for example `Timestamp(ms)` and +`Timestamp(ns)`. A derive never applies across them, and Vortex rejects such an operation (X3). +An engine converts the operands to one dtype first, as its planner already does for SQL. A user of +the Rust or Python API writes an explicit cast. + +### Function declarations + +Each function declares how it uses an operand that contains an extension type: + +- Structural: it moves or selects values, for example `fill_null` or `case_when`. +- Derive: it needs `eq` or `ord`, for example `=` or `<`. +- Type-aware: it implements the semantics of named extension types itself, for example + `replace_time_zone` for `Timestamp`. +- Nothing: the default. A rule of the type covers the call, or typing rejects it. + +Only the function owner declares. Typing never infers a declaration from the input types that a +function accepts, because `Binary` accepts extension operands today. + +An aggregate with a derive also declares whether its result is a value of its input, a bound on its +input, or neither (XQ5, open). Only a value is wrapped again. For example, `min` returns a value, +and `bounded_max` over `Utf8` storage returns a truncated bound, which stays a storage value. + +### Rules that the type owns + +A type implements its own semantics as rewrite rules on arrays (X4). The rules are registered with +the type, together with its compression scheme and the other parts of the type. Cast rules convert +the type to another dtype, or another dtype into the type. Call rules cover other functions and +aggregates, such as `sum` over a decimal-like type. + +A cast exists only if a rule covers it (C1). This is the design of issue +[#8570](https://github.com/vortex-data/vortex/issues/8570), with rules in place of closures. + +No `ext_wrap` expression exists. A rule that produces extension values builds the extension array +itself. It uses `ExtensionArray::new_unchecked` when legal input gives legal output, and +`ExtensionArray::try_new` otherwise. So no user expression can build a wrapper without a check. + +Two explicit expressions remain: + +- `ext_storage(x)` reads the storage. It exists today. It never checks values. +- `ext_storage_deep(x)` removes every extension layer at every depth. It is new. The derive rules + use it for operands that hold nested extension values, for example in a comparison of + `struct{ts: Timestamp}` values. + +Where the rules live is open (XQ2). These notes recommend a method on `ExtVTable`, so that the +rules travel with the dtype and need no session lookup. The existing session registry for array +kernels is keyed by encoding, and every extension array has the same encoding id. Also, 83 plain +`.optimize()` calls outside tests never consult that registry. + +### Scalar identity + +`Scalar` implements `PartialEq`, `Eq` and `Hash` as identity: the same dtype and the same stored +value. These traits serve Rust code, such as maps and tests, not SQL semantics. `PartialOrd` +returns `Equal` for identical scalars, and an order only for ordered dtypes. An extension dtype is +ordered only if it declares `ord`. This keeps the Rust rule that `==` and `partial_cmp` agree. +Expression comparisons follow the derive rules, not these traits. + +### Physical form + +A rewrite rule pulls generic structural arrays, such as `Constant` and `Dict`, out from under the +extension wrapper (X5). Code that looks for a constant or a dictionary then finds it at the top, +without any knowledge of extension types. + +Which arrays the rule pulls out is open (XQ1). These notes recommend this split: + +- Above the wrapper: `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked` and `Interleave`. +- Below the wrapper: the canonical storage, storage encodings such as BitPacked or FSST, and slices, + filters and scalar functions that wait to run. +- A canonical extension array is a wrapper over canonical storage. So a canonical array never hides + a pending computation or a compressed child. + +Each structural encoding gets one method that rebuilds it with its value children passed through a +function. That method moves the wrapper, pushes `ext_storage` down, and decodes a structural +encoding that holds extension values. `RunEnd` and `Sparse`, which fail on extension values today, +get support from the same method. + +Type-specific encodings are a separate question. `DateTimeParts` moves under the wrapper as an +`i64` encoding, so its compare kernel works on storage. The compressor chooses it from the +extension metadata. + +[`agents/design-notes.md`](agents/design-notes.md), section 6, has the details. + +### Statistics + +Zone maps and file statistics store storage statistics for extension columns. The derive rules +produce predicates on `ext_storage(col)`, and pruning uses the storage statistics for them. +`vortex-spatial` already prunes in this way, by hand. + +### Validation + +Three kinds of validation exist: + +- Dtype validation checks the metadata and the storage dtype, once for each dtype. +- Storage-shape validation checks that storage has the storage dtype, for arrays and scalars. +- Value validation (`validate_array`) checks that each non-null value is a legal value of the + type. + +Every extension array that Vortex builds holds only legal values (X7). `ExtensionArray::try_new` +checks the values in one batch with `validate_array`. Today it checks only the storage dtype, which +is a bug (EXT-V1). Kernels that keep legal values use a new `ExtensionArray::new_unchecked`, so they +pay nothing. + +The default `validate_array` applies the existing scalar check of the type to each value. A type +can override it with a faster check, for example a range check on the minimum and maximum of a +`Timestamp` column. + +Two questions stay open: + +- Do file reads check values (XQ3)? These notes recommend that decoding checks only what memory + safety needs, as it already does for UTF-8, and trusts the rest. A strict read mode checks values. + Because files are trusted, building a scalar from an array always checks the value and returns an + error for an illegal one. No read panics. +- Which values can sit in null slots (XQ4)? These notes recommend any value. A child that leaves + its parent is masked with the validity of the parent. This keeps Arrow imports free of copies, + and it also allows a storage encoding to store nulls as a reserved value. + +### Nullability + +The short-term rule: the result of `validate_dtype` must not depend on nullability. Building a dtype +checks both nullabilities, so a later nullability change never validates the type again. The +long-term option: store nullability on the extension dtype, like every other `DType` variant, and +keep the storage dtype non-nullable. + +### Authoring and boundaries + +The core provides a typed array view, a typed accessor for native values, metadata helpers, and a +conformance test harness for `ExtVTable` implementations. DuckDB, DataFusion and Python get export +plugins like the Arrow registry. Then a new extension type does not need code in each engine crate. + +## Costs and risks + +- Typing must cover every path that builds a node. The types enforce it: `BoundExpression` loses + its public variants, and `Cast::new` becomes fallible in the same step as typing. The design + accepts these API changes for correctness (X8). +- The rules must run on every path: the static optimizer, the session-aware optimizer, and the + executor. A test checks that all paths give the same result. +- `DateTimeParts` changes from a `Timestamp` encoding to an `i64` encoding. That is a file format + change for the encoding, and it needs an edition. +- Old readers cannot decode `RunEnd` and `Sparse` over extension values. So writers keep the old + shape of those two until an edition allows the new one. +- Engines must send extension literals for extension columns. Today the lone-extension compare rule + can hide literals that have the storage type. +- After the rules run, plans show storage operations, so plans are harder to read. +- The new scalar order must land with or after the change that rejects `min` and `max` without + `ord`. Otherwise `min` and `max` can panic, for example on a chunked `Json` column. This risk + comes from reading the code, not from a run. + +## Open questions + +The full list with recommendations is in [`../decisions.md`](../decisions.md). The questions for +extension types are: + +- XQ1: which arrays the rule of X5 pulls out, and whether canonical form requires canonical storage. +- XQ2: where the rules of a type live. A related question is whether a crate can register a rule + for a function and a type that it does not own. +- XQ3: whether file reads check values. +- XQ4: which values can sit in null slots. +- XQ5: how an aggregate declares its result. +- XQ6: whether `Uuid` orders by bytes, or has no order. + +Older questions also stay open: + +- Are `eq` and `ord` enough? A refinement over a storage dtype can need a third flag that allows + comparisons with plain storage values. +- Existing files store min and max for extension columns as extension scalars. Readers can convert + them to storage values, but the switch needs a compatibility decision. +- Is the nullability contract enough, or does nullability move onto the extension dtype? diff --git a/research/extension-types/04-urgent-changes.md b/research/extension-types/04-urgent-changes.md new file mode 100644 index 00000000000..492c6a05702 --- /dev/null +++ b/research/extension-types/04-urgent-changes.md @@ -0,0 +1,133 @@ + + +# 4. Urgent changes + +These changes fix wrong results and panics. They do not depend on the design in part 3, and each one +fits in a small PR with a regression test. Finding IDs refer to +[`agents/findings.md`](agents/findings.md). + +| ID | Change | Fixes | Visible effect | +| --- | --- | --- | --- | +| U1 | Give cast one rule on both paths | EXT-P1, EXT-C1, EXT-C2, EXT-C3 | A cast that changes meaning returns an error | +| U2 | Give compare one type rule | EXT-P1, EXT-Q1, EXT-Q3 | `ts < 1500` fails when it is built | +| U3 | Return errors from reads | EXT-V1 | No panic when a row is read | +| U4 | Make nullability changes infallible | EXT-N1 | No panic in `take`, `mask` or `min_max` | +| U5 | Fix the built-in temporal validation | EXT-V4, EXT-V5, EXT-V8 | A malformed dtype or an out-of-range value returns an error | +| U6 | Check the storage dtype of arrays and scalars | EXT-V2, EXT-V7 | An array or a scalar with the wrong storage is rejected | +| U7 | Stop invalid zero values | EXT-V3 | No invalid scalars | +| U8 | Fix `Debug for ExtDTypeRef` | EXT-R1 | Correct debug output | + +## Test requirement + +U1 and U2 need a shared test helper. It evaluates each case three ways: on the whole array, on one +row with `execute_scalar`, and with constant inputs. The three results must be equal, or all three +must be errors. The table in EXT-P1 lists the first cases. + +## U1. One cast rule + +- Remove the wrap in `Scalar::cast` for extension targets (`vortex-array/src/scalar/cast.rs:56-61`). + Keep only casts between one extension dtype and the same dtype with another nullability. +- Remove the storage fallback in the extension cast kernel + (`vortex-array/src/arrays/extension/compute/cast.rs:18-22`). +- Remove the storage target case in `ExtScalar::cast` + (`vortex-array/src/scalar/typed_view/extension/mod.rs:87-96`). `Scalar::cast` sends extension + sources there (`vortex-array/src/scalar/cast.rs:84`). Without this change, constants and the row + path still cast `Timestamp(ms)` to `i64` after the array path stops. +- Return the storage error instead of "no `CastKernel`" (EXT-C3). +- Change the Arrow exporters that call `array.cast(target)` (EXT-B2). An exporter that exports + storage on purpose calls `ext_storage`. Every other exporter returns an error. + +Visible changes: `cast(Timestamp, i64)` returns an error, and `ext_storage` replaces it. Python +`vx.scalar(5, dtype=timestamp)` returns an error. The dedicated temporal constructors keep working. + +## U2. One compare type rule + +- `Binary::return_dtype` requires equal dtypes (ignoring nullability) for every comparison, with no + exception for extension types (`vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`). +- Remove the lone-extension case in `compare_arrays` + (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`). +- `CompareExecuteAdaptor` checks that the operand dtypes match before it calls an encoding kernel + (`compare/mod.rs:73-121`). With the stricter `return_dtype`, this also fixes the `DateTimeParts` + compare (EXT-Q3). +- The `DateTimeParts` compare kernel checks that the constant has the dtype of the column, + including the time unit, before it splits the constant + (`encodings/datetime-parts/src/compute/compare.rs:31-52`). + +Coupling: an engine converter that sends a storage-typed literal for an extension column must send +an extension literal. DuckDB and DataFusion already convert temporal literals to extension scalars. +A test for each engine confirms the other cases. + +## U3. Errors from reads + +- Row reads of an extension array build the extension scalar with `try_new` and return its error + (`vortex-array/src/arrays/extension/vtable/operations.rs:29`). +- `ExtensionConstantRule` builds the extension scalar with `try_new` + (`vortex-array/src/arrays/extension/compute/rules.rs:39`). If the check fails, the rule returns + `Ok(None)`. The array stays `Extension(Constant)`, and a read of a row returns the error. A rule + that returns the error makes optimization fail, so `ext_storage` and structural operations on + the array also fail. +- `Scalar::extension_ref` gets a fallible variant, and read paths use it. + +## U4. Nullability changes cannot fail + +- `ExtDType::try_with_vtable` validates the dtype with both nullabilities. After that, + `with_nullability` cannot fail and does not validate again. +- `with_nullability` builds the new dtype directly. Today it calls `try_with_vtable` + (`vortex-array/src/dtype/extension/typed.rs:91-98`). With the first change, that call validates + both nullabilities on every nullability change. +- The documentation of `ExtVTable::validate_dtype` states that its result must not depend on + nullability. + +A fallible `with_nullability` is the alternative. It moves the error into every caller, so a check +when the dtype is built is the better place. + +## U5. Built-in temporal validation + +- `Date` and `Time` match on `DType::Primitive` and return an error for any other storage + (`vortex-array/src/extension/datetime/date.rs:104`, `time.rs:105`). +- `Timestamp::validate_dtype` rejects the `Days` unit and unknown time zones + (`vortex-array/src/extension/datetime/timestamp.rs:198-204`). +- `Timestamp::unpack_native` uses the `try_` setters of `jiff::Span` and returns their error + (`vortex-array/src/extension/datetime/timestamp.rs:214-229`). Today an out-of-range value + panics (EXT-V8). + +## U6. Storage checks for arrays and scalars + +- `Extension::validate` compares the dtype of the storage child with `ext_dtype.storage_dtype()` + (`vortex-array/src/arrays/extension/vtable/mod.rs:96-123`). +- Scalar validation checks that an extension value matches the storage dtype, before it calls the + check of the type (`vortex-array/src/scalar/validate.rs:100`, `:294`). This fixes EXT-V7. +- `Scalar::try_new` converts the `U64` form of an `F16` value to `F16` + (`vortex-array/src/scalar/scalar_impl.rs:73`). Validation then rejects the `U64` form. Today the + fast path accepts it (`vortex-array/src/scalar/validate.rs:46-50`), and so does the detailed + check (`:147-157`). The proto decoder already converts the old form + (`vortex-array/src/scalar/proto.rs:343`), so only direct construction makes the `U64` form. Then + `==` stays symmetric for `F16` scalars (probe R10). This case is not specific to extension types. + +## U7. Zero values + +- `Scalar::zero_value` validates extension values and does not use `new_unchecked` + (`vortex-array/src/scalar/scalar_impl.rs:165-169`). An extension type that rejects the storage + zero has no zero value. + +## U8. Debug output + +- Format the metadata through a `Debug` adapter, like the existing `MetadataDisplay` + (`vortex-array/src/dtype/extension/erased.rs:181-191`). + +## Coordination with open work + +Three open PRs add mechanisms for extension semantics: + +- [#10246](https://github.com/vortex-data/vortex/pull/10246) adds a per-type aggregate kernel + registry. +- [#10566](https://github.com/vortex-data/vortex/pull/10566) adds an aggregate adapter with a + composite ID. +- [#10408](https://github.com/vortex-data/vortex/pull/10408) adds `between` pushdown for + `AnyTemporal`. + +Each one is a reasonable local fix, but each one fixes a different mechanism in place. Settle the +derive model in part 3 first, or merge each PR with a follow-up that moves it onto derives. diff --git a/research/extension-types/05-implementation.md b/research/extension-types/05-implementation.md new file mode 100644 index 00000000000..2e34640277d --- /dev/null +++ b/research/extension-types/05-implementation.md @@ -0,0 +1,176 @@ + + +# 5. Implementation + +This plan follows [`../decisions.md`](../decisions.md). IDs such as X4 and XQ1 refer to that file. + +## Phases + +| Phase | Content | Depends on | +| --- | --- | --- | +| 0 | The urgent changes in part 4, and the path-consistency test helper | Nothing | +| 1 | The `ExtVTable` contract with the default `validate_array`, a conformance harness, and three reference test types | Nothing | +| 2 | Typing, derives and rewrite rules: `Derives`, the derive rules for compare, min, max and sort, `ext_storage_deep`, and rules that the type owns | 0, 1 | +| 3 | Cast rules that the types own, and an audit of engine literals | 2 | +| 4 | The lift rule for structural arrays, and storage statistics for extension columns | 2 | +| 5 | Value checks in `ExtensionArray::try_new`, and nullability on the extension dtype | 0, 1 | +| 6 | Authoring helpers, engine export plugins, and documentation | 2, 3 | + +```mermaid +flowchart LR + P0["0. Urgent changes"] --> P2["2. Typing and rules"] + P1["1. Contract and harness"] --> P2 + P2 --> P3["3. Cast rules"] + P2 --> P4["4. Lift rule and statistics"] + P0 --> P5["5. Value checks and nullability"] + P1 --> P5 + P2 --> P6["6. Authoring and boundaries"] + P3 --> P6 +``` + +Phases 0 and 1 can run in parallel. Phase 5 can run in parallel with phase 2. Phase 2 is the +critical step, because phases 3, 4 and 6 build on the rules. + +## Phase 0: urgent changes + +- Write the path-consistency helper first, because U1 and U2 use it. +- Land U1 to U8 as separate PRs. +- Some existing code depends on a leak, for example a comparison of an extension column with a + storage literal. The full test suite finds these callers. Fix each one with an explicit + `ext_storage`. + +## Phase 1: contract and harness + +- Document the contract on `ExtVTable`: structural and semantic operations, the derive rules, + validation, and nullability independence. +- Add `validate_array` to `ExtVTable`, with a default that checks each value through + `validate_scalar_value`. Phase 5 builds on it. +- Add a conformance harness for `ExtVTable` implementations. It checks the metadata round trip, + that `validate_dtype` ignores nullability, and that a `validate_array` override agrees with the + default. +- Add three reference test types: + - An opaque type that derives nothing. Typing must reject every semantic operation on it. + - A decimal-like type over `i64` that derives `eq` and `ord` for each dtype. A differential test + compares its results with the built-in `Decimal`. + - A refinement type that derives everything and adds a value check. +- The harness and the reference types grow with later phases. Phase 2 adds the derive rules and + the derives. Phase 3 adds cast round trips. + +## Phase 2: typing, derives and rules + +- Add `Derives`, `ExtVTable::derives`, and the erased forwarder on `ExtDTypeRef`. +- Add extension declarations to scalar and aggregate functions. An aggregate also declares whether + its result is a value of its input, a bound, or neither (XQ5). +- Extend the typing check in its four places: `BoundExpression::try_new_vec`, + `ScalarFnArray::try_new_from_slots`, `AggregateDTypes::try_new` and `ScalarFn::validate`. Make + the variants of `BoundExpression` private. Make `Cast::new` fallible, because typing rejects + casts that no rule covers. +- Add the derive rules for compare, `between`, min, max, `is_sorted` and sort keys. They rewrite + derived operands to `ext_storage`, or to the new `ext_storage_deep` for nested values. A rule + that returns a value of the input builds the extension scalar itself. There is no `ext_wrap` + expression (X4). +- Add the mechanism for rules that a type owns, and register them with the type. These notes + recommend a method on `ExtVTable`, so the static optimizer, the session-aware optimizer and the + executor all find the rules through the dtype (XQ2). Write the rules over `ReduceNode`, so that + they also rewrite bound expressions for pruning. +- Add the rule `ext_storage(Extension(s))` to `s`, so the row path of `ext_storage` reads storage + without a value check. +- Remove the code that applies storage semantics: the extension compare kernel, the extension + branch of the nested comparator and the extension path of `min_max`. `ExtScalar` keeps the + storage order only for types with `ord`. +- Change `PartialOrd` for `Scalar` in the same step as `min` and `max`. An earlier change of + `PartialOrd` can make `max` panic on a chunked `Json` column, or in DuckDB aggregate pushdown. + This risk comes from reading the code. +- Give each type in the repository its derives (the table in part 3). +- Move the open PRs onto derives and rules. #10408 becomes a derive check. #10246 and #10566 + become rules that the types own. + +## Phase 3: cast rules + +- Implement the casts of #8570 as rules that the types own (C1). A cast exists only if a rule + covers it. +- Add `Timestamp` unit casts with overflow checks, and unit casts for `Date` and `Time`. A unit + cast builds its result with `ExtensionArray::new_unchecked`, because legal input gives legal + output. +- Audit the DuckDB and DataFusion converters: every literal for an extension column must be an + extension scalar. + +## Phase 4: lift rule and statistics + +- Add the static rule that pulls structural arrays out from under the extension wrapper (X5). The + set of arrays is open (XQ1). These notes recommend `Constant`, `Dict`, `RunEnd`, `Sparse`, + `Chunked`, `Masked` and `Interleave`. +- Give each structural encoding one method that rebuilds it with its value children passed through + a function. Use it for the lift rule, for the pushdown of `ext_storage`, and to decode structural + encodings that hold extension values. `RunEnd` and `Sparse` get extension support this way. +- Remove the extension take rule, which builds `Extension(Dict(indices, storage))`. +- Make the canonical form of an extension array a wrapper over canonical storage, if maintainers + accept that recommendation (XQ1). +- Keep the old wire shape for `RunEnd` and `Sparse` over extension values until an edition allows + the new one. Readers convert old shapes with the lift rule. +- Store storage statistics for extension columns. Readers convert existing extension-typed + statistics to storage values. +- Move `DateTimeParts` under the wrapper as an `i64` encoding. An edition gates the new form, and + readers keep the old form. + +## Phase 5: value checks and nullability + +- Make `ExtensionArray::try_new` check values with `validate_array`, and add + `ExtensionArray::new_unchecked`. Move kernels, builders and rules that keep legal values onto + `new_unchecked` ([`agents/design-notes.md`](agents/design-notes.md), section 8). +- Imports from Arrow, DuckDB, FFI and Python call `try_new`. File reads follow the answer to XQ3. +- Add faster overrides of `validate_array` where a type has one, for example `Timestamp`. +- Building a scalar from an array keeps the check of U3, because file data can hold illegal values + under the recommendation of XQ3. +- Apply the answer to XQ4 to the rows under a null parent. These notes recommend that any value is + allowed there, and that accessors that take a child out of its parent mask it. +- Optionally move nullability onto the extension dtype. + +## Phase 6: authoring and boundaries + +- Add a typed array view, a native value accessor, and metadata helpers. +- Add export plugin traits for DuckDB, DataFusion and Python, modeled on the Arrow registry. Move + the hard-coded type lists into plugins. +- Write the developer guide page for extension types. + +## Test strategy + +- **Path consistency.** Every semantic operation gives the same answer on the whole array, on one + row, and with constant inputs. +- **Rule coverage.** The same expression gives the same result through `optimize`, `optimize_ctx`, + execution without optimization, and the row path. +- **Opaque rejection.** Typing rejects every semantic operation on the opaque reference type. +- **Scalar contract.** For generated scalars, `a == b` holds exactly when `partial_cmp(a, b)` + returns `Some(Equal)`, and equal scalars have equal hashes. +- **Differential parity.** The decimal-like reference type matches the built-in `Decimal` for every + operation that both support, including errors. +- **Engines.** DuckDB and DataFusion pushdown tests use extension literals for temporal, `Uuid` and + `Json` columns. + +## Compatibility + +- Phases 0 to 3 change behavior, not the file format. The release notes must list the behavior + changes from part 4. +- Phase 4 changes the `DateTimeParts` format, the statistics dtype of extension columns, and the + shape of `RunEnd` and `Sparse` over extension values. All three need editions and read + compatibility. +- Phase 5 can add a value check when a file is read, depending on XQ3. It does not change the + format. + +## Relation to the Decimal and type roadmap + +The [Decimal notes](../decimal/README.md) place these phases in a larger graph of type work. Three +points affect the order: + +- `Decimal` stays a built-in type (D1). Its fixes do not wait for this plan, but it follows the same + rules: derives for one dtype and rules that the type owns. +- `Decimal` and extension types share the lift rule of phase 4. A constant decimal column is + already `vortex.constant` with a decimal scalar, so the old decimal wire forms fit it. +- Phase 3 and the Decimal work share one integer kernel family, `rescale` and `check_range`. + `Timestamp` unit casts and decimal rescales both use it. + +The decimal-like reference type of phase 1 compares the rules of phase 2 with the built-in +`Decimal`. diff --git a/research/extension-types/README.md b/research/extension-types/README.md new file mode 100644 index 00000000000..e0f5ce2d18c --- /dev/null +++ b/research/extension-types/README.md @@ -0,0 +1,55 @@ +# Extension types + +Extension types are meant to work like Rust newtypes: an opaque wrapper over a storage type, with +behavior added on purpose. On `develop`, extension types inherit storage behavior in many places. +That causes wrong results, panics, and code paths that disagree with each other. + +| Part | Content | +| --- | --- | +| [1. Why extension types must be opaque](01-why-opaque.md) | The argument for an opaque default | +| [2. Current problems](02-current-problems.md) | What is wrong on `develop` | +| [3. Design](03-design.md) | The proposed model, the alternatives, and the open questions | +| [4. Urgent changes](04-urgent-changes.md) | Fixes to make now, before the larger design | +| [5. Implementation](05-implementation.md) | Phases, order, tests and compatibility | + +## Summary + +- An extension type gives new meaning to storage values. A storage operation is correct for an + extension type only when it agrees with that meaning, and only the extension type knows when it + agrees. So the safe default is to inherit no semantic behavior. +- On `develop`, the same cast or comparison can return a correct value, a wrong value, or an error. + The outcome depends on the encoding of the input, and on whether one row or the whole array is + read. +- The proposal: extension types are opaque by default. A typing check rejects every semantic + operation that no declaration covers. Rewrite rules turn the allowed ones into storage + operations. The kernels of generic operations, statistics and pruning see only storage. +- Each extension dtype declares the storage behavior that it inherits, such as equality and order. + Each type also owns rewrite rules for its own operations, such as casts. Engines convert + operands of different dtypes before Vortex sees them. +- A rewrite rule keeps generic structural arrays such as `Constant` and `Dict` above the extension + wrapper. So code finds them without any knowledge of extension types. +- [`../decisions.md`](../decisions.md) records which of these points are decided and which are + still open. +- Some fixes are urgent, because they cause wrong results or panics. They cover implicit casts + through storage, comparisons against raw storage, storage checks, and panics on reads and on + nullability changes. + +## Agent documents + +- [`agents/findings.md`](agents/findings.md): every finding, with file and line references, + reproductions, observed and expected behavior, and fix sketches. +- [`agents/design-notes.md`](agents/design-notes.md): API sketches, invariants, rewrite rules, the + derive table for each type, the validation model, and file lists for each phase. +- [`agents/probe.rs`](agents/probe.rs) and [`agents/probe_followup.rs`](agents/probe_followup.rs): + the throwaway tests that reproduced the confirmed findings. + +## Evidence + +The findings come from reading `develop` at `5fc5cb8`, and from two throwaway tests in debug +builds. Findings marked "lead" in the agent catalog are not confirmed. No performance claim is +measured. + +## Related + +The [Decimal notes](../decimal/README.md) use `Decimal` as a test of this design. They also place +this plan in a larger graph of type work. diff --git a/research/extension-types/agents/design-notes.md b/research/extension-types/agents/design-notes.md new file mode 100644 index 00000000000..cb62149ca3f --- /dev/null +++ b/research/extension-types/agents/design-notes.md @@ -0,0 +1,828 @@ + + +# Extension type design notes (agent reference) + +This file is the detailed design behind `../03-design.md` and `../05-implementation.md`. It +follows `../../decisions.md`, which records the decisions of the maintainer. IDs such as X4 and XQ1 +refer to that file. "Open" marks a decision that is not made. Finding IDs (`EXT-*`) refer to +`findings.md`, which covers current code only. Probe IDs (R1 to R10) refer to appendix A of +`findings.md`. Paths and line numbers are for `develop` at `5fc5cb8`. + +## 1. Invariants + +The design holds these invariants. Every change in the plan must keep them. + +- **I1.** A semantic operation on an operand that contains an extension dtype is rejected by + default. A declared derive, a rule that the type owns, or a declaration of the function can allow + it (sections 3 and 4). +- **I2.** A structural operation on an extension value is always allowed and keeps the extension + dtype. +- **I3.** Two kinds of rules decide how a generic operation treats extension values. They are the + derive rules of the core and the rules that each extension type owns (section 4). No other code + applies storage semantics to an extension value. Every node meets a typing check when code + builds it (section 4.1). The rules run on every path that optimizes or executes arrays and bound + expressions (section 4.5). +- **I4.** After the rules run, a generic operation applies storage semantics to an extension value + only where a derive covers it (sections 3.4 and 4.3). The kernels of generic semantic + operations, statistics and pruning see storage dtypes only, at every depth. Generic structural + encodings sit above the extension wrapper, and storage encodings sit below it (section 6). A + type-aware function sees the extension types that it declares. `DateTimeParts` stays an + exception until phase 4 (EXT-S2). +- **I5.** The only ways to move between an extension dtype and another dtype are `ext_storage`, + `ext_storage_deep` and the rules that the extension type owns. A rule that produces extension + values builds the extension array itself (section 5). No `ext_wrap` expression exists (X4). +- **I6.** The row path and the array path apply the same rules, so they return the same result + (EXT-P1). Issue [#10313](https://github.com/vortex-data/vortex/issues/10313) owns the separate + question of whether a validity query raises errors from values that it skips (X9). +- **I7.** Reading a value never panics. Building an extension scalar from an array checks the value + and returns an error for an illegal value (section 8). +- **I8.** A nullability change of a dtype never fails and never validates the type again. +- **I9.** An unknown extension type (`ForeignExtDType`) declares no derives and owns no rules, and + no `TypeAware` declaration matches it. + +## 2. Operation classes + +Structural operations do not read values. Semantic operations depend on what values mean. A +type-aware function implements the semantics of the extension types that it declares. + +| Operation | Class | Note | +| --- | --- | --- | +| slice, take, filter, mask | Structural | Run on storage, keep the wrapper | +| `is_null`, `is_not_null`, validity, `null_count`, `count` | Structural | | +| `fill_null` with a value of the same dtype | Structural | Selects between existing values | +| `zip`, `case_when` with branches of the same dtype | Structural | Selects between existing values | +| `first`, `last` | Structural | Selects an existing value | +| `uncompressed_size_in_bytes` | Structural | Size of the storage | +| `pack`, `merge`, `select`, `get_item` on a struct that holds extension fields | Structural | The extension field stays opaque | +| `cast` to the same dtype with another nullability | Structural | | +| `scalar_at`, serialization, compression of storage | Structural | Constant detection for compression compares storage | +| `ext_storage`, `ext_storage_deep` | Structural | Explicit unwrap of the top level, or of every level | +| `=`, `!=` | Semantic | Needs `eq` or a rule of the type | +| `<`, `<=`, `>`, `>=`, `between` | Semantic | Needs `ord` or a rule of the type | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | Semantic | Needs `ord` or a rule of the type | +| `is_constant`, `all_non_distinct`, `list_contains`, distinct, group by, join keys | Semantic | Needs `eq` or a rule of the type | +| Arithmetic, `sum`, `sum_v2`, `mean` | Semantic | Needs a rule of the type, because the result type usually changes | +| `cast` other than a nullability change | Semantic | Needs a cast rule of the type (section 4.4) | +| `is_nan`, `nan_count`, `like`, `byte_length` | Semantic | Needs a rule of the type | +| Zero and default values | Semantic | Needs a `zero_value` hook | +| Export to Arrow, DuckDB, DataFusion, Python, JNI | Semantic | Needs an export plugin | +| `replace_time_zone`, tensor functions, spatial functions | Type-aware | The function owner declares the types (section 4.2) | + +## 3. Derives + +### 3.1 Definition + +A derive is a declaration on one extension dtype (X2). It says that a semantic operation on values +of that dtype is the same operation on their storage values. + +```rust +/// Storage behavior that values of one extension dtype inherit, like `#[derive]` on a newtype. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Derives { + /// Storage equality is value equality for values of this dtype. The storage hash is then a + /// valid hash for value equality. + pub eq: bool, + /// Storage order is value order for values of this dtype. `ord` requires `eq`. + pub ord: bool, +} + +pub trait ExtVTable { + // The existing methods stay. + + /// Returns the storage behavior that values of `ext` inherit. + /// + /// The default inherits nothing, so every semantic operation on this dtype is rejected. + fn derives(ext: &ExtDType) -> Derives { + Derives::default() + } +} +``` + +`ExtDTypeRef` gets an erased forwarder: `fn derives(&self) -> Derives`. Typing and the derive rules +read the flags from the non-nullable form of the dtype. So a type that breaks D4 still gives one +answer. + +### 3.2 Rules + +- **D1.** A derive applies only when every operand in a derived position has the same extension + dtype, ignoring nullability. A lone extension operand never gets a derive, so `ts < 5i64` stays + rejected. Operations across dtypes are rejected (section 3.5). +- **D2.** `eq` requires canonical storage: each value of the dtype has exactly one storage form. +- **D3.** `ord` requires `eq`, and storage order must equal value order. +- **D4.** The flags do not depend on nullability. + +The conformance harness checks D4, and that `ord` implies `eq`. It checks D2 and D3 only when a +type supplies an optional reference order on native values for tests. `NativeValue` requires only +`Display` (`vortex-array/src/dtype/extension/vtable.rs:29-32`), so the harness has no other oracle. + +### 3.3 Use by operation + +| Operation | Operands in derived positions | Flag | +| --- | --- | --- | +| `=`, `!=` | Both operands | `eq` | +| `<`, `<=`, `>`, `>=` | Both operands | `ord` | +| `between` | All three operands. `Between::return_dtype` already requires one dtype (`vortex-array/src/scalar_fn/fns/between/mod.rs:282-305`) | `ord` | +| `min`, `max`, `min_max`, `bounded_min`, `bounded_max`, `is_sorted`, sort keys | The one input | `ord` | +| `is_constant`, `all_non_distinct`, distinct, group by | The one input | `eq` | +| `list_contains` | The list element and the needle | `eq` | +| Join keys, hash partitioning | The key columns on both sides | `eq` | + +An aggregate binds to one input dtype (`AggregateFnVTable::return_dtype`, +`vortex-array/src/aggregate_fn/vtable.rs:170`). So a unary operation reads the flags of its one +input, and it never needs a second dtype. A function outside this table gets no derive. It uses a +declaration or a rule that the type owns (section 4). + +### 3.4 Containers + +A struct, list, fixed-size list or map dtype that holds extension values has `eq` or `ord` only if +every nested extension dtype declares it. An extension dtype whose storage holds another extension +dtype follows the same rule. An order comparison of two `struct{ts: Timestamp(ms)}` values needs +`ord` on `Timestamp(ms)`. Any comparison of two `struct{doc: Json}` values is rejected. + +After the check, the derive rule removes every extension layer at every depth (section 4.3, X6). +The kernel then sees only storage dtypes. So the extension branch of the nested comparator +(`vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`) goes away in phase 2. + +### 3.5 Operations across dtypes + +Derives never apply across dtypes. Two dtypes with different metadata are different dtypes, for +example `Timestamp(ms)` and `Timestamp(ns)`, or two time zones. Vortex rejects such an operation +(X3). An engine converts the operands to one dtype first, as its planner already does for SQL. A +user of the Rust or Python API writes an explicit cast. Rust and the built-in `Decimal` reject such +operations today as well. + +### 3.6 Table for the types in the repository + +| Type | `eq` | `ord` | Reason | +| --- | --- | --- | --- | +| `Timestamp` | Yes | Yes | One dtype fixes the unit and the time zone, so storage counts one unit from one epoch | +| `Date`, `Time` | Yes | Yes | | +| `Uuid` | Yes | Open | Byte order is the Postgres order (XQ6) | +| `Json` | No | No | Text is not canonical | +| `Vector` | Yes | No | No natural order | +| `FixedShapeTensor` | Yes | No | No natural order | +| WKB and native geometry types | No | No | Two encodings can hold one geometry | +| `ForeignExtDType` | No | No | I9 | + +Vortex compares floats with a total order. Equality is bitwise, so `NaN` equals a `NaN` with the +same bits, and `-0.0` differs from `+0.0` +(`vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs:31-34`, `nested.rs:258`). Probe R3 +shows this for primitive and fixed-size list compares. A `Vector` derive inherits these storage +semantics on purpose. + +## 4. Typing and rewrite rules + +A call with an extension operand passes two steps. Typing checks the call when code builds the +node. Rewrite rules then replace an accepted semantic call with operations on storage. Typing never +rewrites, and the rules never accept a call that typing rejected. + +### 4.1 Typing + +Every node of a compute graph meets a typing check when code builds it. Today that check is +`return_dtype`, and it runs in four places: + +- `BoundExpression::try_new_vec` (`vortex-array/src/expr/bound_expression.rs:129-148`). + `Expression::bind`, `with_children` in the optimizer and the reduce rules of bound expressions + all reach it. +- `ScalarFnArray::try_new_from_slots` (`vortex-array/src/arrays/scalar_fn/array.rs:102-114`). + `try_new`, `try_new_with_len`, `ArrayRef::apply_bound` and the reduce rules of arrays all reach + it. +- `AggregateDTypes::try_new` (`vortex-array/src/aggregate_fn/vtable.rs:55-75`), where an aggregate + meets an input dtype. `Accumulator::try_new` reaches it, and DuckDB aggregate pushdown calls + `Accumulator::try_new` directly (`vortex-duckdb/src/convert/expr.rs:543-552`). + `Accumulator::from_dtypes` takes dtypes that are already resolved, so it accepts only checked + dtypes. `return_dtype` and `state_dtype` of `AggregateFnRef` answer dtype questions without an + accumulator (`vortex-array/src/aggregate_fn/erased.rs:86-95`), so they apply the same check. +- `ScalarFn::validate` (`vortex-array/src/arrays/scalar_fn/vtable/mod.rs:79-121`) checks a scalar + function array that code builds from parts. + +Typing accepts a call with an extension operand only in these cases, tried in this order: + +1. The function declares the call structural for these operands. +2. The function declares itself type-aware for every extension type in the operands. +3. The function declares a derive, and the operands meet D1 and section 3.4. +4. A rule of an extension type in the operands covers the call. Typing asks the types in argument + order. + +Otherwise typing returns an error that names the function, the extension ID and the missing +declaration. For a cast, typing tries three cases in order. The first is a nullability change of +the same dtype. The second is a rule of the source type, and the third is a rule of the target type. + +### 4.2 Function declarations + +```rust +/// How a function uses an operand whose dtype contains an extension dtype. +pub enum ExtensionUse { + /// The function moves or selects values and never reads them. + Structural, + /// The function applies storage equality or order. Typing checks these flags (D1). + Derive(Derives), + /// The function implements the semantics of the matched extension type itself. + TypeAware, + /// The function has no semantics for extension values. A rule of the type covers the call, + /// or typing rejects it. + None, +} + +pub trait ScalarFnVTable { + /// Declares how this function uses the operand at `arg`, which contains `ext`. + /// + /// The default is `ExtensionUse::None`. + fn extension_use( + &self, + options: &Self::Options, + arg: usize, + ext: &ExtDTypeRef, + ) -> ExtensionUse { + ExtensionUse::None + } +} +``` + +`AggregateFnVTable` gets the same method without `arg`, because an aggregate has one input. + +An aggregate with a `Derive` declaration also declares how its result relates to its input (XQ5, +open). These notes recommend three kinds: + +- A value of the input is one of its stored values. The derive rule builds an extension scalar of + the input dtype from the storage result. Examples are the results of `min` and `max`, and each + field of the struct that `min_max` returns. +- A bound on the input stays a storage value (S5). `bounded_min` and `bounded_max` declare a bound + when they truncate the storage, which happens for `Utf8` and `Binary`. For other storage dtypes, + they declare a value. +- Any other result stays as it is, for example the `Bool` of `is_sorted`. + +A value result can only get its top-level layer back. So if a value result holds extension values +inside a container, for example `min` over `List`, typing rejects the call. + +More rules for declarations: + +- Only the function owner declares, in the definition of the function. The core grants no + type-aware declaration by default. +- A declaration can depend on the options. `Binary` declares `Derive` with `eq` or `ord` by + operator. `cast` is structural only for a nullability change. `stat` depends on its aggregate + (`vortex-array/src/scalar_fn/fns/stat.rs:112-114`). +- A `TypeAware` declaration matches an extension type by its vtable, for example + `ext.is::()`, never by ID string. A `ForeignExtDType` keeps the real ID + (`vortex-array/src/dtype/extension/foreign.rs:28-56`). So a match by ID also covers a + placeholder, which breaks I9. +- `replace_time_zone` declares `TypeAware` for `Timestamp`. Today its `return_dtype` accepts only + timestamps (`vortex-array/src/scalar_fn/fns/replace_time_zone.rs:131-148`). The tensor and + spatial functions declare their own types in the same way. +- Typing never infers a declaration from `return_dtype`. `Binary::return_dtype` accepts any + extension operand in a comparison today (`vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`, + EXT-Q1). +- A value that a function keeps in its options counts as an operand. `DynamicComparison` keeps its + right operand in its options and calls `Binary` directly + (`vortex-array/src/scalar_fn/fns/dynamic.rs:80-111`). Its declaration covers that value. The + value is a function that returns a new `ScalarValue` at each execution (`dynamic.rs:181-186`). + So the derive rule changes the dtype of the value to the storage dtype, and the values stay as + they are. After that change, nothing checks a new value against the extension type. So the rule + keeps one check: each new value goes through `validate_scalar_value`, and an illegal value is an + error. Today `Rhs::scalar` panics on such a value (`dynamic.rs:189-193`). + +### 4.3 Nested extension values + +An operand counts as an extension operand if its dtype contains an extension dtype at any depth. +Examples are `list_contains` on `List` +(`vortex-array/src/scalar_fn/fns/list_contains/mod.rs:178-194`) and the spatial `collect` on +`List` (`vortex-spatial/src/scalar_fn/collect.rs:60-96`). `ext_storage` unwraps only the +top level, and it rejects an operand whose top-level dtype is not an extension +(`vortex-array/src/scalar_fn/fns/ext_storage.rs:58-63`). + +So a derive rule uses a new structural expression, `ext_storage_deep` (X6). It replaces every +extension dtype in the operand with its storage dtype, at every depth. It repeats this while a +storage dtype holds another extension dtype. Typing has already checked section 3.4, so every layer +that it removes declares the flag. + +- For an array, it rebuilds each container around the storage of its children. It copies no + values. +- For a scalar, it changes only the dtype, because an extension scalar already holds its storage + value. +- For a top-level extension whose storage holds no extension dtype, it equals `ext_storage`. The + derive rule then emits `ext_storage` itself. So the statistics rule S2, the simplification rules + of section 4.8 and the pushdown rules of section 6 apply unchanged to the common case. Those + rules gain a matching case for `ext_storage_deep`, which appears only for nested extension + values. + +For example, `list_contains` over `List` and a `Timestamp(ms)` needle becomes +`list_contains` over `List` and an `i64` needle. Its kernel sees only storage. The spatial +`collect` declares `TypeAware` for `Point`, so no layer is removed there. + +### 4.4 Rules that the type owns + +A type implements its own semantics as rewrite rules (X4). The rules are registered with the type, +together with its compression scheme and the other parts of the type. A rule matches a call that +has an operand of the type. It replaces the call with operations on storage, or it declines. + +- **Cast rules.** A rule converts the type to another dtype, or another dtype into the type. Cast + rules implement the design of issue [#8570](https://github.com/vortex-data/vortex/issues/8570), + with rules in place of closures. A cast exists only if a rule covers it (C1). +- **Call rules.** A rule covers a scalar function or an aggregate with an operand of the type. An + example is `sum` over a decimal-like type, which needs a wider result dtype. + +A rule reaches storage through `ext_storage` or `ext_storage_deep`. When its result holds values of +an extension type, it builds the extension array or scalar itself (section 5). + +Where the rules live is open (XQ2). These notes recommend a method on `ExtVTable` that returns the +rules of the type. `ExtDTypeRef` holds the vtable, so the rules travel with the dtype and need no +session lookup. Registering the type in a session then registers its rules too. + +The existing session registry, `ArrayKernels` (`vortex-array/src/optimizer/kernels.rs`), is the +alternative. It has two problems for extension types: + +- It is keyed by encoding ids, and every extension array has the id `vortex.ext`. So each rule + checks the dtype itself, and the rules of every registered type run on every extension column. +- Only the session-aware optimizer consults it. A grep counts 83 plain `.optimize()` calls outside + tests, and they use static rules only. Two of them are `ArrayRef::cast` + (`vortex-array/src/builtins.rs:176`) and `ArrayRef::apply_bound` + (`vortex-array/src/expression.rs:41`). + +Rules written over `ReduceNode` serve both arrays and bound expressions, as `reduce_null` already +does (`vortex-array/src/scalar_fn/fns/is_not_null.rs:50`). Pruning works on bound expressions +(`vortex-file/src/pruning.rs:35-49`). So a rule that serves only arrays cannot rewrite a predicate +onto storage statistics. + +Open: whether a crate can register a rule for a function and a type that it does not own. + +### 4.5 Where the rules run + +- The derive rules belong to the functions that declare a derive, such as `Binary`, `Between` and + the order aggregates. So the static optimizer runs them. +- Under the recommendation of XQ2, the type rules come from the dtype. So the static optimizer + finds them as well. +- The executor applies the same rules before it dispatches a kernel. A lazy array that nobody + optimized still gets the rewrite, so the row path and the array path agree (I6). +- After phase 2, no generic kernel applies storage semantics. So a semantic call that reaches a + generic kernel without a rewrite fails. A missing rule gives an error, never a wrong result. + +### 4.6 Examples + +```text +Timestamp(ms) a < Timestamp(ms) b => ext_storage(a) < ext_storage(b) +Timestamp(ms) a < lit(Timestamp(ms) 5) => ext_storage(a) < lit(5i64) +Timestamp(ms) a < lit(5i64) => rejected by typing (D1) +Timestamp(ms) a < Timestamp(ns) b => rejected by typing (X3) +min(Timestamp(ms) a) => min(ext_storage(a)), and the rule builds a + Timestamp(ms) scalar from the result +cast(Timestamp(ms) a, Timestamp(ns)) => Extension(Timestamp(ns), + checked_mul(ext_storage(a), 1_000_000)) +cast(Timestamp(ns) a, Timestamp(ms)) => rejected unless the type defines rounding +Json a = Json b => rejected by typing +``` + +The cast rule builds its result with `ExtensionArray::new_unchecked`. Legal input gives legal +output, and `checked_mul` fails on overflow. The result is a wrapper over a pending computation, +which section 6 allows. + +### 4.7 Array API and enforcement + +Typing runs in `try_new_from_slots` (`vortex-array/src/arrays/scalar_fn/array.rs:102-114`), which +`try_new` and `try_new_with_len` both call. `ArrayRef::apply_bound` uses `try_new_with_len` +(`vortex-array/src/expression.rs:19-42`). Typing never changes the node, so a typed constructor +keeps its return type. + +The types make the typing entry points of section 4.1 the only paths. The design accepts the API +changes that this needs (X8): + +- `BoundExpression` keeps its variants private. Today they are public, so any code can build a + node with any dtype (`vortex-array/src/expr/bound_expression.rs:42-59`). +- A constructor for a call that typing can reject becomes fallible. `Cast::new` is the main case. + It calls `vortex_expect` on `try_new` (`vortex-array/src/scalar_fn/fns/cast/mod.rs:62-65`), and + typing rejects every cast that no rule covers. So `Cast::new` becomes fallible in phase 2. A + constructor for a structural call, such as `IsNull::new` + (`vortex-array/src/scalar_fn/fns/is_null.rs:35-37`), can stay infallible, because typing never + rejects it. +- Code that runs a function on raw arguments applies typing and the rules first. + `DynamicComparison::execute` calls `Binary` directly (section 4.2). The row encoder of sort keys + runs on raw columns, and today it rejects extension dtypes at every depth + (`vortex-row/src/encoder.rs:112`, `:126-144`). After the derive rule, it receives the output of + `ext_storage_deep`. + +### 4.8 Simplification + +`ext_storage` is infallible today (`vortex-array/src/scalar_fn/fns/ext_storage.rs:87-93`), so these +rules never remove or add a failure: + +| Rule | Valid when | Reason | +| --- | --- | --- | +| `ext_storage(lit(s))` to `lit(s.to_storage_scalar())` | Always | Both read the storage of one valid scalar, and the dtypes match | +| `ext_storage_deep(lit(s))` to a literal of the storage dtype | Always | The same as the first rule, at every depth | +| `ext_storage(Extension(s))` to `s` | Always | The wrapper holds `s` as its storage. The row path then reads storage directly (section 8.3). No such rule exists today (probe R9) | +| `ext_storage(Constant(extension scalar))` to `Constant(storage scalar)` | Always | The same as the first rule, for a constant array | + +`Expression::bind` folds nothing (`vortex-array/src/expr/bound_expression.rs:309-323`), but +simplification turns `cast(lit)` into a literal +(`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`). When such a fold fails, the cast stays in +the tree. This behavior needs no change. + +## 5. Wrapping storage + +No `ext_wrap` expression exists (X4). A rule that produces extension values builds the extension +array or scalar itself. + +- A rule uses `ExtensionArray::new_unchecked` when legal input gives legal output. Examples are the + unit cast of section 4.6 and the result of `min`, which is a stored value. +- A rule uses `ExtensionArray::try_new` for storage values that no check has seen. +- A rule builds an extension scalar with `Scalar::try_new`, which always checks (section 8). +- Wrapping is a Rust call, not an expression. So no user expression and no serialized plan can + build a wrapper without a check. +- A lazy computation that produces extension values is a wrapper over a pending computation, for + example the result of the cast in section 4.6. Section 6 keeps pending computations under the + wrapper. + +## 6. Physical form + +### 6.1 Decision + +A rewrite rule pulls generic structural arrays out from under the extension wrapper (X5). Code +that looks for a constant, a dictionary or runs then finds them at the top, without any knowledge +of extension types. The compressor, the exporters, pruning and `as_constant()` all depend on this. + +Which arrays the rule pulls out is open (XQ1). These notes recommend three roles: + +| Role | Encodings | Place | +| --- | --- | --- | +| Structural | `Constant`, `Dict`, `RunEnd`, `Sparse`, `Chunked`, `Masked`, `Interleave` | Above the wrapper | +| Storage | The canonical storage arrays. The FastLanes encodings (BitPacked, FoR, Delta, RLE), ALP, ALP-RD, ByteBool, ZigZag, Pco, Zstd, FSST, OnPair, Sequence, PiecewiseSequence and Patched. Also DecimalByteParts, and `DateTimeParts` after phase 4 | Below the wrapper | +| Pending | Slices, filters and scalar functions that wait to run | Below the wrapper, with the storage that they apply to | + +A storage encoding computes values from a layout that only built-in dtypes have, so it can never +carry an extension dtype. Patched, for example, accepts only primitive dtypes +(`vortex-array/src/arrays/patched/array.rs:187`). A structural encoding holds values of the column +dtype, whatever it is. + +These notes also recommend that a canonical extension array is a wrapper over a canonical storage +array. Then a canonical array never hides a pending computation or a compressed child. Today +`Extension::execute` returns the array unchanged +(`vortex-array/src/arrays/extension/vtable/mod.rs:189-191`), so a canonical extension array can +hold any child. + +### 6.2 Current code + +- The compressor writes `Constant(extension scalar)` for a constant extension column, and + `Masked(Constant(extension scalar))` when some rows are null + (`vortex-compressor/src/compressor/cascade.rs:272-273`, + `vortex-compressor/src/compressor/constant/mod.rs:175-193`). For other columns, it compares two + results and keeps the smaller one (`cascade.rs:175-205`). The first comes from a scheme that + matches the extension type. For example, `TemporalScheme` writes `DateTimeParts` for a + `Timestamp` (`vortex-btrblocks/src/schemes/temporal.rs:45-56`, `:124`). When no scheme matches, + the first result is the input itself (`cascade.rs:288-290`). The second is `vortex.ext` over the + compressed storage, and it wins a tie (`cascade.rs:201`). Probe R8 shows the constant, masked + and `vortex.ext` shapes. Its compressor had no temporal scheme. +- `ExtensionConstantRule` rewrites `Extension(Constant(storage))` to `Constant(extension scalar)` + (`vortex-array/src/arrays/extension/compute/rules.rs:28-46`). Probe R5 shows the rewrite. This + rule is the first case of the decision in section 6.1. +- `ArrayRef::as_constant()` matches only the `Constant` encoding + (`vortex-array/src/array/erased.rs:485-487`). Probe R5 shows that it misses + `Extension(Constant)`. +- `ArrayRef::take` wraps its input in a lazy `DictArray` + (`vortex-array/src/array/erased.rs:262-264`). The extension take rule applies `take` to the + storage, so it builds `Extension(Dict(indices, storage))` + (`vortex-array/src/arrays/extension/compute/take.rs`). Under the decision, `take` of an extension + array gives `Dict(indices, Extension(storage))`, so this rule goes. +- The extension mask rule builds a lazy mask expression over the storage + (`vortex-array/src/arrays/extension/compute/mask.rs`). That is a pending computation, so it stays + under the wrapper. +- `ext_storage` reads a constant input directly. For other inputs, it executes the input to a + canonical `ExtensionArray` (`vortex-array/src/scalar_fn/fns/ext_storage.rs:78-84`). +- Generic rules push scalar functions into dictionary values and into chunks. The dictionary rule + pushes an infallible function, or any function when every value is referenced + (`vortex-array/src/arrays/dict/compute/rules.rs:96-208`). The chunked rules push a unary + function, or a function with constant siblings + (`vortex-array/src/arrays/chunked/compute/rules.rs:34-115`). Probe R6 shows that + `ext_storage` keeps both encodings. Over `Masked`, it decodes the input. Over `Sparse` with + extension values, it fails, because sparse canonicalization rejects extension arrays + (`encodings/sparse/src/canonical.rs:360-362`). +- Constant canonicalization and masked execution have extension arms + (`vortex-array/src/arrays/constant/vtable/canonical.rs:204-219`, + `vortex-array/src/arrays/masked/execute.rs:178-193`). So does dictionary execution + (`vortex-array/src/arrays/dict/execute.rs:182-193`). Run-end decoding rejects extension values + (`encodings/runend/src/array.rs:509`). +- `vortex.ext`, `vortex.constant`, `vortex.dict` and `vortex.chunked` are in core edition 2025.05 + (`vortex-edition/src/declarations/core/v2025_05.rs:27-33`). `vortex.masked` is in 2025.10 + (`v2025_10.rs:24`). + +### 6.3 Mechanism and costs + +- One static rule on the `Extension` encoding moves a structural child above the wrapper. For + example, `Extension(Dict(codes, values))` becomes `Dict(codes, Extension(values))`. A static rule + runs in every `.optimize()` call, unlike a session rule (section 4.4). +- Each structural encoding implements one method that rebuilds it with its value children and value + scalars passed through a function. That method gives three things: the lift rule, the pushdown + of `ext_storage`, and decoding of a structural encoding with an extension dtype. Decoding strips + the wrapper, decodes the storage form, and wraps the result again. +- So the extension code that `Constant`, `Masked` and `Dict` have today becomes one shared path. + `RunEnd` and `Sparse`, which fail on extension values today, get support through the same path. +- `ext_storage` then never computes. It passes through the structural encodings and removes the + wrapper. +- Readers apply the lift rule, so old files with `vortex.ext` over `vortex.dict` change shape when + they are read. Old readers decode `Constant`, `Dict`, `Masked` and `Chunked` with extension + values, but not `RunEnd` or `Sparse`. So writers keep the old shape for those two until an + edition allows the new one. +- `Constant(extension scalar)` holds an extension scalar. Readers of statistics convert it to a + storage scalar (S3). + +### 6.4 Type-specific encodings + +Type-specific encodings are a separate question. `DateTimeParts` carries the `Timestamp` dtype +today (`encodings/datetime-parts/src/canonical.rs:28`, EXT-S2). After the derive rule, a compare +reaches it as `ext_storage(column) < lit(i64)`. To keep its compare kernel, `DateTimeParts` +becomes an `i64` encoding with the time unit in its metadata. Readers convert the old form on +read. An edition gates writes of the new form. `vortex-btrblocks/src/schemes/temporal.rs` chooses +the encoding from the extension metadata, and compresses the storage child. The same applies to +any future encoding for one extension type. + +## 7. Statistics + +- **S1.** Zone maps and file statistics for an extension column store statistics of the storage, + with the storage dtype. The storage fallback in `aggregate_state_dtype` + (`vortex-layout/src/layouts/zoned/schema.rs:177-188`) becomes the rule. Today it applies only + when an aggregate rejects the extension dtype. +- **S2.** A new stats rewrite rule maps `stat(ext_storage(x), s)` to the storage statistic `s` of + `x`. After the derive rules, bound predicates reference `ext_storage(col)`, so pruning needs only + this rule. +- **S3.** Readers convert existing extension-typed min and max to storage scalars with + `to_storage_scalar()`. The values are already storage values. +- **S4.** Engine statistics export (DuckDB `column_statistics.rs`, DataFusion `convert/stats`, JNI + write summaries) goes through the export plugin of the type. A type without `ord` exports no min + and max. +- **S5.** Bounded min and max (truncation) apply to the storage dtype, so they also apply to + extensions over `Utf8` and `Binary`. + +## 8. Value validation + +### 8.1 Three kinds of validation + +| Kind | What it checks | When | Cost | +| --- | --- | --- | --- | +| Dtype validation | `validate_dtype`: the metadata and the storage dtype | When a dtype is built or read, for both nullabilities (section 9) | Once for each dtype | +| Storage-shape validation | The storage child has the length of the array, which `Extension::validate` checks today, and the storage dtype (U6). A scalar value has the storage dtype (EXT-V7) | When an array or a scalar is built or read | Constant | +| Value validation | `validate_array`: each non-null value is a legal value of the type | When `ExtensionArray::try_new` builds an array, and at the other places in section 8.3 | One pass over the values | + +### 8.2 The invariant + +Every extension array that Vortex builds holds only legal values in its non-null rows (X7). A +legal value is a value that `validate_scalar_value` accepts. Data from a file holds legal values +only if the read checks them (XQ3). + +- `ExtensionArray::try_new` checks every non-null value with `validate_array`, and it returns an + error for an illegal value. It takes an `ExecutionCtx`, because the check can execute compressed + storage. `ExtensionArray::new` calls `try_new`. Today `try_new` checks only the storage dtype + (`vortex-array/src/arrays/extension/array.rs:52-73`, EXT-V1). +- `ExtensionArray::new_unchecked` skips the check. Its caller promises that every non-null value + is legal, as with other `new_unchecked` constructors. A debug build checks the promise. +- Kernels that keep values, such as slice, take, filter and mask, use `new_unchecked`. So do rules + that make legal values from legal input, such as a unit cast of `Timestamp`. So does code that + wraps storage it already holds again, such as canonicalization and the compressor + (`vortex-compressor/src/compressor/cascade.rs:198`). A grep counts about 100 call sites of + `ExtensionArray::new` and `try_new` outside test files. Each one moves to the right constructor. +- A `Scalar` is always legal, because `Scalar::try_new` runs the value check. Today two gaps + exist. A scalar can hold a value of the wrong storage type (EXT-V7, fixed by U6). + `Scalar::zero_value` skips the check (EXT-V3, fixed by U7). +- Building a scalar from an array checks the value with `Scalar::try_new`, because data from a file + can hold illegal values (XQ3). The check returns an error, never a panic (I7). + +A value check in every call of `ExtensionArray::new` slows common paths. In this design, kernels +use `new_unchecked`, so only the places in section 8.3 pay for the check. Type overrides of +`validate_array` make that check fast. + +### 8.3 Where value validation runs + +- `ExtensionArray::try_new`, for data that no check has seen. +- Rules that wrap storage values that no check has seen (section 5). +- Imports from Arrow, DuckDB, FFI and Python. They call `try_new`. +- `Scalar::try_new`, for every extension scalar, including a read of one row. +- File reads, if the answer to XQ3 asks for it. + +`ext_storage` never checks values, on either path. Today the row path checks them. It reads each +child as an extension scalar (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`), and +that read runs the per-row check (`vortex-array/src/arrays/extension/vtable/operations.rs:19-30`). +The array path reads the storage without a check +(`vortex-array/src/scalar_fn/fns/ext_storage.rs:83-84`). Probe R9 shows the difference: the array +path returns `[0, 1]` for `NonZero [0, 1]`, and the row path panics. Under X7, +`ExtensionArray::new` rejects that array, but a file can still hold it. So the row path of +`ext_storage` reads the storage child directly, through the rule `ext_storage(Extension(s))` to +`s` (section 4.8). + +### 8.4 The array validator and the scalar validator + +```rust +pub trait ExtVTable { + /// Checks every non-null value of `storage`. + /// + /// The default applies `validate_scalar_value` to each of those values. An override must + /// accept exactly the arrays that the default accepts. + fn validate_array( + ext: &ExtDType, + storage: &ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult<()> { + validate_each_value::(ext, storage, ctx) + } +} +``` + +- `validate_scalar_value` stays the rule for one value. Its default calls `unpack_native` + (`vortex-array/src/dtype/extension/vtable.rs:53-66`). +- The default `validate_array` is not a no-op. It applies `validate_scalar_value` to each non-null + value. +- A check returns an error for an illegal value and never panics. Today the `Timestamp` check + panics on values outside the range of `jiff::Span`, because it calls the panicking setters + (`vortex-array/src/extension/datetime/timestamp.rs:214-229`, EXT-V8). U5 fixes it. +- A type can override `validate_array` with a faster check. The conformance harness compares the + override with the default on generated inputs. +- Examples of overrides: `Timestamp` checks the storage minimum and maximum against its valid + range. `Uuid` checks the version nibble of each value against the metadata. A hypothetical + `NormalizedVector` type checks the norm of each row within a tolerance. + +### 8.5 Open questions + +- **File reads (XQ3).** Decoding builds arrays through the encoding vtable, not through `try_new`. + A reader can check values, or it can trust the file. These notes recommend a general rule for + Vortex: decoding checks only what memory safety needs, and it trusts semantic constraints. + Decoding already checks lengths, offsets and UTF-8, because Rust requires valid UTF-8 for `str` + (`vortex-array/src/arrays/varbinview/array.rs:234-245`). A strict read mode runs + `validate_array` on every chunk. No operation panics on an illegal value (I7). + `Constant(extension scalar)` already checks its value on decode + (`vortex-array/src/arrays/constant/vtable/mod.rs:160-161`). +- **Values in null slots (XQ4).** Builders fill a non-nullable child under a null parent with + placeholder values (`vortex-array/src/builders/fixed_size_list.rs:294-305`, + `vortex-array/src/builders/mod.rs:156-166`). Arrow leaves the child values under a null parent + undefined. A type such as `NonZero` rejects the placeholder 0. + - These notes recommend that a null slot can hold any value. The value check skips null slots + and rows under a null parent. Every public accessor that takes a child out of its parent masks + the child with the validity of the parent, as `get_item` already does + (`vortex-array/src/scalar_fn/fns/get_item.rs:205-209`). + - This recommendation keeps Arrow imports free of copies. It also allows a storage encoding to + store nulls as a reserved value, such as 0 for `NonZero`. + - The alternative requires a legal value in every slot. Then imports with nulls must copy, and a + type with no zero needs a nullable child. + +## 9. Nullability + +- **N1, short term.** `ExtDType::try_with_vtable` calls `validate_dtype` for both nullabilities of + the storage dtype, and fails if either call fails. The documentation states that the result of + `validate_dtype` must not depend on nullability. `with_nullability` then builds the new dtype + directly and never validates again. Today it calls `try_with_vtable` + (`vortex-array/src/dtype/extension/typed.rs:91-98`). A type that breaks the rule fails when its + dtype is built, not later in a kernel. Calling `validate_dtype` only with the non-nullable form + is the alternative, but it hides such a type. +- **N2, long term.** `ExtDType` stores a non-nullable storage dtype and a `Nullability`, like every + other `DType` variant. The storage child has the dtype `storage_dtype.with_nullability(n)`. The + wire format does not change, because the serialized storage dtype already carries nullability. + +## 10. Zero values + +```rust +pub trait ExtVTable { + /// Returns the zero of this type, or `None` if the type has no zero. + fn zero_value(ext: &ExtDType) -> Option { + None + } +} +``` + +- `Scalar::zero_value` and `ScalarValue::try_zero_value` use this hook for extension dtypes. +- Builders fill a non-nullable child under a null parent + (`vortex-array/src/builders/fixed_size_list.rs:303`, `vortex-array/src/builders/mod.rs:156-166`). + Under the recommendation of XQ4, any value is allowed there. So the builder can keep the storage + zero, and the hook is not needed for padding. + +## 11. Scalar identity and order + +### 11.1 Current code + +- `Scalar` derives `Eq` (`vortex-array/src/scalar/mod.rs:41`). `PartialEq` and `Hash` compare the + dtype without nullability and the value (`vortex-array/src/scalar/scalar_impl.rs:336-356`). +- `PartialOrd` returns `None` for different dtypes. Otherwise it compares the stored values + (`scalar_impl.rs:358-474`), so extension values compare by storage for every type (EXT-Q2). + Probe R2 shows `Some(Less)` for two `Opaque` values. +- Floats use bitwise equality and `total_cmp` (`vortex-array/src/dtype/ptype.rs:328-336`). +- `scalar_cmp` computes every comparison operator from `partial_cmp` + (`vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:241-267`). Constant inputs and the row + path use it. +- A `Variant` value has no order, so `partial_cmp` returns `None` even when `==` holds + (`scalar_impl.rs:449-451`). +- An extension over `Union` storage also gets `None` even when `==` holds. The union arm needs + `DType::Union`, but it receives the extension dtype (`scalar_impl.rs:441-443`). The tuple arm + unwraps extensions (`scalar_impl.rs:469-471`). +- An extension scalar can hold a value of the wrong storage type. Then `==` can panic (EXT-V7, + probe R2). +- A valid `F16` scalar can hold a `U64` value, for compatibility with an old serialized form + (`vortex-array/src/scalar/validate.rs:46-50`, `:147-157`). Then `==` panics in one direction + and returns `false` in the other (probe R10). + +### 11.2 The Rust rule + +`PartialOrd` must agree with `PartialEq`: `a == b` holds exactly when `partial_cmp(a, b)` returns +`Some(Equal)`. Suppose `PartialEq` stays structural and `PartialOrd` follows the derives. Then the +rule breaks. For an `Opaque` scalar `x` without `ord`, `x == x` is true, but `partial_cmp(x, x)` +returns `None`. + +### 11.3 Contract + +- `PartialEq`, `Eq` and `Hash` describe identity, as today: the same dtype without nullability, + and equal stored values. Decimal values of different widths compare by value + (`vortex-array/src/scalar/typed_view/decimal/dvalue.rs:255-281`). Primitive values of different + widths compare by value when one converts to the other. Equality panics when the conversion + fails, for example `U8(5)` against `I8(-1)` + (`vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-138`). A valid primitive value has + the `PType` of its dtype, except the `U64` form of an `F16` value + (`vortex-array/src/scalar/validate.rs:46-50`). So valid scalars of one dtype never reach the + panic after U6. +- `PartialOrd` follows three steps: + 1. If `a == b`, return `Some(Equal)`. + 2. If both have one ordered dtype, ignoring nullability, return the order of the stored values. + Nulls sort first. + 3. Otherwise, return `None`. +- A built-in dtype keeps its current order. An extension dtype is ordered only if it declares + `ord`. A container is ordered only if all its members are ordered. The flags come from the + non-nullable dtype. +- `Ord` is not implemented, as today. +- A valid `Eq` needs values whose variant matches the dtype. So U6 checks the storage dtype of + extension scalars (EXT-V7). Scalar construction also converts the `U64` form of an `F16` value + to `F16` (probe R10). + +These steps keep the Rust rule. Step 1 returns `Some(Equal)` exactly when `==` holds. Under the +stored-value order, two values that are not equal never compare as `Equal`, because float +equality is bitwise and float order is `total_cmp`. The steps also fix the `Variant` and `Union` +cases. + +### 11.4 Relation to expression semantics + +- Identity serves Rust code: maps of constants, deduplication, tests and caches. It never decides + an expression result for extension values. +- Identity implies semantic equality, because one dtype and one storage value give one value. + The reverse does not hold. Without `eq`, two values that are not identical can be equal. +- Expression comparisons follow the derive rules. After the rules run, `scalar_cmp` and the nested + comparator see storage dtypes at every depth (section 4.3). So the order of extension scalars + never decides an expression result, and `scalar_cmp` keeps its current operators. For example, + `=` on `struct{v: Vector}` compares `struct{v: FixedSizeList}` storage, where the built-in + order agrees with `==`. + +### 11.5 Order of work + +Change `PartialOrd` only together with, or after, the phase 2 change that rejects `min` and `max` +for extension types without `ord`. Today `min`, `max` and their variants merge partial results with +`vortex_expect` on `partial_min` or `partial_max`. The sites are +`vortex-array/src/aggregate_fn/fns/min/mod.rs:59`, `max/mod.rs:59`, `bounded_min/mod.rs:71`, +`bounded_max/mod.rs:86` and `min_max/mod.rs:247`, `:252`. With the new order, a merge of two +different `Json` maxima panics. + +- `ChunkedArrayAggregate` merges one partial for each chunk + (`vortex-array/src/arrays/chunked/compute/aggregate.rs:28-32`). So `max` over a chunked `Json` + column panics. +- DuckDB aggregate pushdown accumulates every split into one partial + (`vortex-duckdb/src/file_reader.rs:298-300`). It does not exclude `Json` + (`vortex-duckdb/src/table_function.rs:525-537`). +- The default zone-map writer is not affected. Each zone is one canonical chunk with its own + accumulator (`vortex-layout/src/layouts/repartition.rs:158-161`, + `vortex-layout/src/layouts/zoned/builder.rs:106-118`). + +These results come from reading the code, not from a run. + +## 12. Files to change, by phase + +| Phase | Files | +| --- | --- | +| 0, U1 | `vortex-array/src/scalar/cast.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-array/src/arrays/extension/compute/cast.rs`, `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | +| 0, U2 | `vortex-array/src/scalar_fn/fns/binary/mod.rs`, `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs`, `encodings/datetime-parts/src/compute/compare.rs` | +| 0, U3 | `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/rules.rs`, `vortex-array/src/scalar/constructor.rs` | +| 0, U4 | `vortex-array/src/dtype/extension/typed.rs`, `vortex-array/src/dtype/extension/vtable.rs` | +| 0, U5 | `vortex-array/src/extension/datetime/{date,time,timestamp}.rs` | +| 0, U6 | `vortex-array/src/arrays/extension/vtable/mod.rs`, `vortex-array/src/scalar/validate.rs`, `vortex-array/src/scalar/scalar_impl.rs` | +| 0, U7 | `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/scalar_value.rs` | +| 0, U8 | `vortex-array/src/dtype/extension/erased.rs` | +| 1 | `vortex-array/src/dtype/extension/vtable.rs`, `vortex-array/src/compute/conformance/`, `vortex-array/src/extension/tests/` | +| 2 | `vortex-array/src/dtype/extension/{vtable,typed,erased}.rs`, `vortex-array/src/scalar_fn/vtable.rs`, `vortex-array/src/aggregate_fn/{vtable,erased}.rs`, `vortex-array/src/expr/bound_expression.rs`, `vortex-array/src/arrays/scalar_fn/array.rs`, `vortex-array/src/optimizer/`, `vortex-array/src/scalar_fn/fns/{binary,between,cast,dynamic.rs,ext_storage.rs}`, `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs`, `vortex-array/src/aggregate_fn/fns/{min,max,min_max,bounded_min,bounded_max,is_sorted}/`, `vortex-array/src/arrays/extension/compute/compare.rs`, `vortex-array/src/scalar/scalar_impl.rs`, `vortex-array/src/scalar/typed_view/extension/mod.rs`, `vortex-layout/src/layouts/zoned/writer.rs`, `vortex-row/src/encoder.rs`, `vortex-duckdb/src/table_function.rs` | +| 3 | `vortex-array/src/scalar_fn/fns/cast/mod.rs`, `vortex-array/src/extension/datetime/`, `vortex-duckdb/src/convert/expr.rs`, `vortex-datafusion/src/convert/exprs/mod.rs` | +| 4 | `vortex-array/src/arrays/extension/compute/{rules,take}.rs`, `vortex-array/src/arrays/extension/vtable/mod.rs`, `vortex-array/src/arrays/{constant,chunked,dict,masked}/`, `encodings/sparse/`, `encodings/runend/`, `encodings/datetime-parts/`, `vortex-btrblocks/src/schemes/temporal.rs`, `vortex-layout/src/layouts/zoned/`, `vortex-layout/src/layouts/file_stats.rs`, `vortex-file/src/pruning.rs`, `vortex-array/src/stats/rewrite/`, `vortex-edition/` | +| 5 | `vortex-array/src/arrays/extension/array.rs`, `vortex-array/src/arrays/extension/vtable/operations.rs`, `vortex-array/src/arrays/extension/compute/`, `vortex-array/src/builders/{extension,fixed_size_list,mod}.rs`, `vortex-array/src/dtype/extension/`, `vortex-array/src/extension/datetime/timestamp.rs`, the Arrow, DuckDB, FFI and Python import paths | +| 6 | `vortex-arrow/src/session.rs`, `vortex-duckdb/src/convert/`, `vortex-duckdb/src/exporter/`, `vortex-datafusion/src/convert/`, `vortex-python/src/scalar/`, `docs/developer-guide/extending/extension-dtypes.md` | + +## 13. Tests + +- **Path consistency.** A helper in `vortex-array/src/compute/conformance/` takes an expression + builder and inputs. It evaluates the expression on the array path, on the row path, and with + constant-encoded inputs. It asserts equal values or an error on all three. +- **Rule coverage.** The same expression gives the same result through `optimize`, + `optimize_ctx`, execution without optimization, and the row path. This test catches a rule that + one path does not run. +- **Reference types.** `vortex-array/src/extension/tests/` already has `DivisibleInt`. Add: + - `Opaque` over `i64`, with no derives. Typing rejects every semantic operation. + - `DecimalLike` over `i64`, with `eq` and `ord` for each dtype. A differential test runs each + operation on `DecimalLike` and on the built-in `Decimal`, and compares values or errors. + - `Refined` over `i64` (for example, positive values only), with every derive and a + `validate_array` override. +- **Conformance harness for `ExtVTable`.** It checks the metadata round trip, that + `validate_dtype` ignores nullability, rules D1 to D4, cast round trips, and path consistency. D2 + and D3 need the optional reference order. The harness also compares a `validate_array` override + with the default. +- **Scalar contract.** For generated scalars, `a == b` holds exactly when `partial_cmp(a, b)` + returns `Some(Equal)`, and equal scalars have equal hashes. +- **Engines.** DuckDB and DataFusion pushdown tests with extension literals for `Timestamp`, `Date`, + `Uuid` and `Json` columns. A `Json` comparison must not be pushed down. diff --git a/research/extension-types/agents/findings.md b/research/extension-types/agents/findings.md new file mode 100644 index 00000000000..fee5e0fed61 --- /dev/null +++ b/research/extension-types/agents/findings.md @@ -0,0 +1,640 @@ + + +# Extension type findings (agent reference) + +This file is the detailed catalog behind `../02-current-problems.md`. Each finding has an ID that +the human documents use. All paths are relative to the repository root. All line numbers are for +`develop` at commit `5fc5cb8` (2026-10-17). + +This catalog covers current code only. Problems in the proposed design appear as design rationale +in `design-notes.md`. + +## Conventions + +Severity: + +- **S1, wrong result.** An operation returns a value without an error, and the value is wrong for + the extension type. +- **S2, panic or invalid state.** An operation panics, or an invalid value enters an array or a + scalar. +- **S3, inconsistency.** Two parts of the system give different answers to the same question. +- **S4, hygiene.** Debug output, registration, documentation. + +Status: + +- **probe** means that a throwaway test reproduced the behavior. `probe.rs` ran against + `vortex-array`, and `probe_followup.rs` ran in `vortex-btrblocks`. Both ran in debug builds. +- **code** means that the behavior follows directly from the cited code. Nothing ran. +- **lead** means that the code suggests a problem, but the problem is not confirmed. A lead names + the check that confirms or rejects it. + +Terms: + +- **Row path:** `ArrayRef::execute_scalar(i)` on a lazy array. For a `ScalarFn` array, this + evaluates the function on one-row constant inputs. +- **Array path:** `execute::` on the lazy array, then read the stored values. +- **Wrapper:** the `Extension` array (encoding ID `vortex.ext`) that holds the storage array as its + only child. + +## Summary + +| ID | Title | Severity | Status | +| --- | --- | --- | --- | +| EXT-P1 | Encoding and read path change the result of cast and compare | S1 | probe | +| EXT-C1 | Scalar cast into an extension reinterprets the storage value | S1 | probe | +| EXT-C2 | Array cast out of an extension returns the storage cast | S1 | probe | +| EXT-C3 | Same-type extension cast hides the storage error | S3 | code | +| EXT-Q1 | Compare skips the type check when one side is an extension | S1 | probe | +| EXT-Q2 | Every extension type gets storage equality and order, with no override | S1 | probe | +| EXT-Q3 | `DateTimeParts` compare kernel trusts its operand types | S1 | probe | +| EXT-A1 | Ordering-based operations disagree about extension types | S3 | probe | +| EXT-A2 | Zone maps fall back to storage statistics | S3 | code | +| EXT-V1 | Arrays are never value-checked, and reads panic on bad values | S2 | probe | +| EXT-V2 | `Extension::validate` never checks the storage dtype | S2 | probe | +| EXT-V3 | Zero values come from storage and skip extension validation | S2 | probe | +| EXT-V4 | `Date` and `Time` panic on non-primitive storage | S2 | code | +| EXT-V5 | `Timestamp` accepts dtypes that reject every value | S2 | code | +| EXT-V6 | Eight extension types never check values | S3 | code | +| EXT-V7 | Extension scalars skip the storage dtype check | S2 | probe | +| EXT-V8 | The `Timestamp` value check panics on out-of-range values | S2 | code | +| EXT-N1 | A nullability change can panic | S2 | probe | +| EXT-S1 | A constant extension column has two shapes | S3 | code | +| EXT-S2 | Some encodings carry the extension dtype themselves | S3 | code | +| EXT-B1 | DataFusion converts extension values to storage values | S1 | code | +| EXT-B2 | Arrow export can silently export storage | S1 | code | +| EXT-B3 | Python returns and builds raw storage values | S1 | code | +| EXT-B4 | DuckDB decides pushdown from the storage type | S3 | code | +| EXT-B5 | JNI write summaries report storage values | S3 | code | +| EXT-B6 | Pruning looks through casts out of extensions | S3 | lead | +| EXT-R1 | `Debug for ExtDTypeRef` prints the wrong output | S4 | probe | +| EXT-R2 | Registration overwrites and duplicates silently | S4 | code | +| EXT-R3 | Unknown extension IDs inherit storage semantics | S3 | code | +| EXT-R4 | Documentation is a stub or stale | S4 | code | + +## Findings + +### EXT-P1: Encoding and read path change the result of cast and compare + +- **Locations:** + - Row path: `vortex-array/src/arrays/scalar_fn/vtable/operations.rs:30-42` wraps each input row in + a one-row `ConstantArray` and executes the function on those constants. + - Constant path: `vortex-array/src/arrays/constant/compute/cast.rs:14-20` calls `Scalar::cast`. + `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:158-163` calls `scalar_cmp` when both + sides are constant. + - Array path: `vortex-array/src/arrays/extension/compute/cast.rs:17-43` and + `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`. +- **Mechanism:** Cast and compare each have a scalar implementation and an array implementation. + Constant inputs use the scalar implementation. Other encodings use the array implementation. The + two implementations apply different rules to extension types. The row path converts every input + to a constant, so it always takes the scalar implementation. +- **Observed** (probe IDs in brackets): + + | Expression | Array path | Row path | + | --- | --- | --- | + | `cast(Timestamp(ms) to Timestamp(ns))`, primitive storage | Error: no `CastKernel` [D1'] | `1970-01-01T00:00:01Z` becomes `00:00:00.000001Z` [D1] | + | Same cast, constant storage | `00:00:01Z` becomes `00:00:00.000001Z` [D2'] | Same wrong value [D2] | + | `cast(i64 to Timestamp(ms))` | Error: no `CastKernel` [B'] | Wraps the raw `i64` [B] | + | `cast(Opaque to NonZero)`, two unrelated test types | Error: no `CastKernel` [E4'] | Wraps the raw value [E4] | + | `Timestamp(ms) < i64` | Compares raw storage: `[true, false]` [F1'] | Error: incompatible types [F1] | + +- **Expected:** One answer per expression, whatever the encoding and the read path. The type + system RFC draft (`vortex-data/rfcs#55`) states this as a rule for every operation. +- **Fix:** Give cast and compare one rule. Short term, make both implementations reject the cases + in EXT-C1, EXT-C2 and EXT-Q1. Long term, rewrite semantic operations into storage operations + with rules that both paths apply (see `design-notes.md`). +- **Regression test:** Add a helper that evaluates an expression three ways: on the array path, on + the row path, and with constant-encoded inputs. The helper asserts equal values or an error in all + three. Run it for every case in the table above. + +### EXT-C1: Scalar cast into an extension reinterprets the storage value + +- **Location:** `vortex-array/src/scalar/cast.rs:56-61`, which has a `TODO(connor)` that says the + code is not correct for extension types. +- **Mechanism:** When the target dtype is an extension, `Scalar::cast` casts the value to the + storage dtype of the target. It then wraps the result in the target dtype. The source type and its + metadata play no part. +- **Observed:** `Timestamp(ms)=1000` casts to `Timestamp(ns)=1000`, so one second becomes one + microsecond [A]. A plain `i64` casts to `Timestamp` [C]. A value of one extension type casts to an + unrelated extension type [E4]. +- **Reach:** All constant inputs (EXT-P1), expression simplification of literals + (`vortex-array/src/scalar_fn/fns/cast/mod.rs:170-181`), and Python + `vx.scalar(v, dtype=ext)` (`vortex-python/src/scalar/factory.rs:106`). +- **Expected:** An error, unless the source and target are the same extension dtype with a + different nullability. A conversion that changes meaning must go through a cast that the + extension type owns (issue #8570), or through an explicit `ext_storage`. +- **Fix:** Remove the wrap. Keep only the nullability case. +- **Regression test:** The three cases above must return an error on the row path, the array path, + and with constant inputs. + +### EXT-C2: Array cast out of an extension returns the storage cast + +- **Location:** `vortex-array/src/arrays/extension/compute/cast.rs:18-22`. +- **Mechanism:** If the target is not the same extension dtype (ignoring nullability), the kernel + returns `storage_array().cast(target)`. +- **Scalar path:** `ExtScalar::cast` accepts the exact storage dtype as a target, ignoring + nullability (`vortex-array/src/scalar/typed_view/extension/mod.rs:87-96`). `Scalar::cast` sends + extension sources there (`vortex-array/src/scalar/cast.rs:84`). So a constant `Timestamp(ms)` + casts to `i64`, even if the kernel stops doing it. +- **Observed:** `Timestamp(ms)` casts to `i64`, `f64` and `u8` [E1, E2, E3, E1']. +- **History:** PR #7469 added this behavior. Issue #7504 decided to remove it, and was closed as a + duplicate of #8570, which is still open. The conformance suite says the opposite of the kernel: + `vortex-array/src/compute/conformance/consistency.rs:1051` ("Extension types typically only cast + to themselves"). +- **Coupling:** The Arrow exporters call `array.cast(target)` for primitive, byte-view and struct + targets (`vortex-arrow/src/executor/primitive.rs:45`, `byte_view.rs:64`, `struct_.rs:115`). When + they receive an extension array, they export its storage through this fallback (EXT-B2). Removing + the fallback without changing these callers turns a silent export into an error. +- **Expected:** An error, unless the extension type defines the cast. +- **Fix:** Remove the fallback in the kernel and the storage target case in `ExtScalar::cast`. + Callers that intend to read storage must call `ext_storage`. +- **Regression test:** Casting `Timestamp(ms)` to `i64` returns an error on the row path, the + array path, and with constant inputs. `ext_storage` on the same input returns the `i64` storage. + +### EXT-C3: Same-type extension cast hides the storage error + +- **Location:** `vortex-array/src/arrays/extension/compute/cast.rs:28-36`. +- **Mechanism:** When the target is the same extension dtype, the kernel casts the storage. If that + cast fails, the kernel logs `tracing::warn!` and returns `Ok(None)`. The caller then reports "no + CastKernel", and the real error is lost. +- **Fix:** Return the error. + +### EXT-Q1: Compare skips the type check when one side is an extension + +- **Locations:** + - `vortex-array/src/scalar_fn/fns/binary/mod.rs:162-168`: `Binary::return_dtype` rejects + different dtypes for comparisons only when neither side is an extension. + - `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193`: `compare_arrays` replaces a + lone extension side with its storage (`extension_storage`, line 226) and compares. + - `vortex-array/src/scalar_fn/fns/between/mod.rs:282-305`: `Between::return_dtype` requires the + same dtype for all three operands. +- **Observed:** On the array path, `Timestamp(ms) < 1500i64` compares the raw `i64` storage + [F1', F2']. On the row path, the same expression is an error [F1, F2]. `Timestamp(ms) < 'a'` is + accepted when the expression is built, and fails at execution with `Cannot compare different + DTypes i64 and utf8` [F3']. `between(ts, 0i64, 1500i64)` is rejected when it is built [F4]. So + `ts >= 0 AND ts <= 1500` runs while `ts BETWEEN 0 AND 1500` does not. +- **Expected:** Comparison requires the same dtype, ignoring nullability, like every other dtype. +- **Fix:** Remove the extension exception in `Binary::return_dtype`. Remove the lone extension case + in `compare_arrays`. Engines that send storage-typed literals for extension columns must send + extension literals. +- **Regression test:** All three comparisons are rejected when the expression is built. + +### EXT-Q2: Every extension type gets storage equality and order, with no override + +- **Locations:** + - `vortex-array/src/arrays/extension/compute/compare.rs:18-55`: the compare kernel compares + storage whenever both sides have the same extension dtype. + - `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229-233`: the comparator for nested + values compares storage. + - `vortex-array/src/scalar/typed_view/extension/mod.rs:107-137`: `ExtScalar` `PartialEq`, + `PartialOrd` and `Hash` use storage (`TODO(connor)` at line 107). + - `vortex-array/src/arrays/extension/vtable/kernel.rs:15-22`: the compare kernel is registered on + the shared `Extension` encoding, so it applies to every extension type. + - `vortex-array/src/optimizer/kernels.rs:262`: kernels registered later for the same key run + after the built-in kernel and cannot override it. +- **Observed:** A test type with no ordering semantics (`Opaque`) supports `<` and returns storage + order [G]. +- **Impact:** The result is wrong for any type whose storage order or storage equality differs from + its value order or value equality. Examples in the repository: JSON (key order and whitespace), + WKB geometry (two encodings of one geometry), vectors and tensors (no natural order). +- **Fix:** See `design-notes.md` (derives for one dtype, which typing and the derive rules read). + +### EXT-Q3: `DateTimeParts` compare kernel trusts its operand types + +- **Locations:** + - `encodings/datetime-parts/src/compute/compare.rs:24-61`: the kernel calls + `rhs_const.as_extension()` before it checks the dtype. It then splits the constant with the time + unit of the constant. It does not compare that unit with the time unit of the column. + - `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:73-121`: `CompareExecuteAdaptor` calls + the kernel of the encoding without checking that the operand dtypes match. + - `vortex-array/src/scalar/downcast.rs:161-164`: `Scalar::as_extension` panics if the scalar is + not an extension. +- **Observed** [R7]: the probe encodes a `Timestamp(ms)` column `[1.6 s, 1.4 s]` as + `DateTimeParts`. The correct answer to `< 1.5 s` is `[false, true]`. + - Against the `i64` constant `1500`, the array path panics with `Failed to convert scalar to + extension`. The row path returns `Cannot compare scalars with incompatible types`. + - Against a `Timestamp(ns)` constant of 1.5 s, the array path returns `[true, true]`, which is + wrong. The row path returns an error. + - Against a `Timestamp(ms)` constant of 1.5 s, the array path returns `[false, true]`. +- **Earlier counter-evidence:** `vortex-file/src/tests.rs:2369` and `:2410` assert that a unit + mismatch returns an error in a file scan. The probe shows that the kernel itself raises no + error. So another layer raises it in those tests. This catalog does not identify that layer. +- **Fix:** Make `CompareExecuteAdaptor` require equal dtypes (ignoring nullability) before it calls + any kernel. Make the kernel check the unit. U2 covers both. + +### EXT-A1: Ordering-based operations disagree about extension types + +- **Locations:** + - `vortex-array/src/aggregate_fn/fns/min_max/mod.rs:284-313` accepts every extension dtype. + `min_max/extension.rs:17-35` computes min and max on storage. + - `vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs:255-283` rejects every extension dtype, and + `is_sorted_impl` then returns `false` (`mod.rs:104-109`). The extension branch that calls + `check_extension_sorted` can never run. + - `vortex-row/src/lib.rs:24-28` and `vortex-row/src/encoder.rs:128` reject extension types + because the crate "does not define a total ordering for them". + - `vortex-array/src/aggregate_fn/fns/sum/mod.rs:131` rejects every extension dtype. +- **Observed:** `min_max` on the `Opaque` test type returns storage min and max [H]. `is_sorted` on + a sorted `Timestamp` array returns `false` [I]. +- **Open work that touches this.** Each of these PRs adds a different opt-in mechanism: + - PR #10246 adds a per-type aggregate kernel registry and `StorageAggregateKernel`. + - PR #10566 adds an `ExtensionAggregate` adapter with a composite aggregate ID. + - PR #10408 adds a `between` pushdown that matches `AnyTemporal` inside the rule. +- **Fix:** Make every ordering-based operation read one declaration (see `design-notes.md`). + +### EXT-A2: Zone maps fall back to storage statistics + +- **Locations:** + - `vortex-layout/src/layouts/zoned/schema.rs:177-188`: when an aggregate rejects the extension + dtype, `aggregate_state_dtype` uses the storage dtype. `legacy_stats_table_dtype` does the + same for old files (lines 112-121). + - `vortex-layout/src/layouts/zoned/writer.rs:217-231`: bounded min and max apply only to + top-level `Utf8` and `Binary`. An extension over `Utf8` or `Binary` (JSON, WKB) stores exact, + unbounded bounds. +- **Impact:** WKB zone maps store whole geometry blobs as their min and max, ordered by bytes. +- **Fix:** Store storage statistics explicitly for every extension column, and have pruning use them + through `ext_storage` (see `design-notes.md`). + +### EXT-V1: Arrays are never value-checked, and reads panic on bad values + +- **Locations:** + - `vortex-array/src/arrays/extension/array.rs:52-57`: `try_new` checks the storage dtype only. + - `vortex-array/src/builders/extension.rs:110-112`: `append_zeros` writes storage zeros. + - `vortex-array/src/arrays/extension/vtable/operations.rs:29`: reading a row calls + `Scalar::extension_ref`. + - `vortex-array/src/arrays/extension/compute/rules.rs:39`: `ExtensionConstantRule` calls + `Scalar::extension_ref`. + - `vortex-array/src/scalar/constructor.rs:255-260`: `Scalar::extension_ref` panics with + `vortex_expect` if the value is invalid. `Scalar::extension::` (line 243) also panics. +- **Observed:** With a `NonZero` test type, `ExtensionArray::new(non_zero, [0])` succeeds [L3]. + `append_zeros(1)` succeeds [L1]. Reading the row panics with `unable to construct an extension + Scalar` [L2]. `ext_storage` over `NonZero [0, 1]` returns `[0, 1]` on the array path and panics + on the row path [R9]. The row path reads the child as an extension scalar + (`vortex-array/src/arrays/scalar_fn/vtable/operations.rs:35-39`). +- **Fix:** Return an error from these read paths (U3). Make `ExtensionArray::try_new` check values + with a batch `validate_array`, and add `ExtensionArray::new_unchecked` for kernels + (`design-notes.md`, section 8). Make the row path of `ext_storage` read the storage without a + value check (phase 2). + +### EXT-V2: `Extension::validate` never checks the storage dtype + +- **Location:** `vortex-array/src/arrays/extension/vtable/mod.rs:96-123`. Line 116 builds + `DType::Extension(ext_dtype.clone())` from `dtype` and compares the result with `dtype`. The two + are always equal. +- **Observed:** `ExtensionArray::try_from_parts` accepts a `Utf8` storage child under a + `Timestamp(ms)` dtype [O]. +- **Fix:** Compare the dtype of the storage child with `ext_dtype.storage_dtype()`. + +### EXT-V3: Zero values come from storage and skip extension validation + +- **Locations:** + - `vortex-array/src/scalar/scalar_value.rs:90-95`: the zero of an extension is the zero of its + storage. + - `vortex-array/src/scalar/scalar_impl.rs:165-169`: `Scalar::zero_value` calls `new_unchecked`. + Its `SAFETY` comment says that the value is valid for the dtype. + - `vortex-array/src/scalar/scalar_impl.rs:97-104`: `new_unchecked` validates only in debug + builds. + - `vortex-array/src/scalar/downcast.rs:172`: `as_extension_opt` assumes that every `Scalar` is + valid. + - `vortex-array/src/scalar/scalar_impl.rs:254`: `is_zero` on an extension is `is_zero` on its + storage. +- **Observed:** For the `NonZero` test type, `Scalar::zero_value` panics in a debug build [K1]. + In a release build, it returns an invalid scalar (from code). +- **Fix:** Make the zero of an extension type an opt-in. Without it, `try_zero_value` returns + `None`. + +### EXT-V4: `Date` and `Time` panic on non-primitive storage + +- **Locations:** `vortex-array/src/extension/datetime/date.rs:104` and + `vortex-array/src/extension/datetime/time.rs:105` call `storage_dtype().as_ptype()`. + `vortex-array/src/dtype/dtype_impl.rs:209-215` panics if the dtype is not primitive. +- **Impact:** A file whose `vortex.date` or `vortex.time` dtype has non-primitive storage panics + while its dtype is read. It does not return an error. +- **Fix:** Match on `DType::Primitive` and return an error otherwise. + +### EXT-V5: `Timestamp` accepts dtypes that reject every value + +- **Locations:** `vortex-array/src/extension/datetime/timestamp.rs:198-204` checks only for `i64` + storage. `unpack_native` rejects the `Days` unit (line 231) and unknown time zones (line 241). +- **Impact:** `Timestamp(Days)` and a timestamp with an unknown time zone are valid dtypes, but + every non-null value fails. `vortex-array/src/dtype/extension/vtable.rs:20-24` says that + `validate_dtype` must prevent this. +- **Fix:** Reject the `Days` unit and unknown time zones in `validate_dtype`. + +### EXT-V6: Eight extension types never check values + +- **Locations:** These types return the raw `&ScalarValue` from `unpack_native` and do not + override `validate_scalar_value`: + - `vortex-tensor/src/types/vector/vtable.rs` + - `vortex-tensor/src/types/fixed_shape_tensor/vtable.rs` + - `vortex-spatial/src/extension/linestring.rs`, `multipoint.rs`, `polygon.rs`, + `multilinestring.rs`, `multipolygon.rs` and `rect.rs` +- **Impact:** These types check nothing at the value level, and `Display` shows raw storage. This is + allowed by the trait, but it means that "scalars are checked" is not true for these types. + +### EXT-V7: Extension scalars skip the storage dtype check + +- **Locations:** + - `vortex-array/src/scalar/validate.rs:100` and `:294`: for an extension dtype, scalar validation + calls only `validate_storage_value`. + - `vortex-array/src/dtype/extension/erased.rs:107-109`: `validate_storage_value` calls only the + `validate_scalar_value` of the type. + - `vortex-array/src/scalar/validate.rs:46-50`: a primitive dtype requires a value of the same + `PType`, except the `U64` form of an `F16` value. `Scalar::try_new` does not apply this check + to extension storage. + - `vortex-array/src/scalar/constructor.rs:256`: `Scalar::extension_ref` asserts the storage + dtype. So the gap is a raw `ScalarValue` passed to `Scalar::new` or `Scalar::try_new`. + - `vortex-array/src/extension/datetime/timestamp.rs:211`: `Timestamp` casts any primitive value + to `i64` before it checks the range. + - `vortex-array/src/scalar/typed_view/primitive/pvalue.rs:122-160`: `PValue` equality and order + upcast the other value with `vortex_expect`. The upcast of a float to an integer fails + (`pvalue.rs:197-203`, `:397-418`). +- **Observed** [R2]: `Scalar::try_new(Timestamp(ms), F64(5.0))` succeeds and displays as + `1970-01-01T00:00:00.005Z`. Call that scalar `f`, and call the `Timestamp(ms)` scalar with + `I64(5)` `i`. Then `i == f` panics with `upcast`, and `f == i` returns `false`. So `==` is not + symmetric. `i.partial_cmp(&f)` also panics. +- **Related primitive case** [R10]: a valid `F16` scalar can hold a `U64` value, for compatibility + with an old serialized form (`vortex-array/src/scalar/validate.rs:46-50`, `:147-157`). `==` + between the `U64` form and the `F16` form of one value panics in one direction and returns + `false` in the other. The proto decoder already converts the old form + (`vortex-array/src/scalar/proto.rs:343`), so only direct construction makes the `U64` form. +- **Reach:** Not measured. The probe builds the scalars with `Scalar::try_new` directly. +- **Fix:** Check that an extension value matches the storage dtype before the check of the type. + U6 covers it. Convert the `U64` form of an `F16` value to `F16` when the scalar is built. + +### EXT-V8: The `Timestamp` value check panics on out-of-range values + +- **Location:** `vortex-array/src/extension/datetime/timestamp.rs:214-229`: `unpack_native` builds + a `jiff::Span` with `seconds`, `milliseconds`, `microseconds` or `nanoseconds`. These setters + call `expect` on their `try_` versions, so they panic on a value outside the range of a `Span`. + For seconds, the limit is 631,107,417,600. +- **Expected:** An error. `Scalar::try_new` and the value check of an array must not panic on + illegal data, for example an Arrow `timestamp[s]` column that holds `i64::MAX`. +- **Status:** code. Nothing ran. +- **Fix:** Use `Span::try_seconds` and the other `try_` setters, and return their error. U5 + covers it. + +### EXT-N1: A nullability change can panic + +- **Locations:** + - `vortex-array/src/dtype/extension/typed.rs:91-99`: `ExtDType::with_nullability` runs + `validate_dtype` again and panics if it fails. + - Callers: `take` (`vortex-array/src/arrays/extension/compute/take.rs:23` and `:42`), `mask` + (`compute/mask.rs:23`), masked execution (`vortex-array/src/arrays/masked/execute.rs:187-190`), + `min_max` (`vortex-array/src/aggregate_fn/fns/min_max/extension.rs:23`), and + `DType::with_nullability` (`vortex-array/src/dtype/dtype_impl.rs:108`), which + `DType::as_nullable` uses. +- **Observed:** A test type that accepts only non-nullable storage panics in `take` with nullable + indices [M1]. +- **Cause:** Nullability is part of the storage dtype, and nothing says that `validate_dtype` must + ignore it. +- **Fix:** Short term, validate both nullabilities when the dtype is built, so `with_nullability` + cannot fail later. Long term, see `design-notes.md` ("Nullability"). + +### EXT-S1: A constant extension column has two shapes + +- **Locations:** `vortex-array/src/arrays/extension/compute/rules.rs:28-46` rewrites + `Extension(Constant(storage))` to `Constant(extension scalar)`. + `vortex-array/src/scalar_fn/unstable/row/types/element/tuple/element_tuple.rs:126` has + `TODO(connor): This is maybe incorrect unless this is a refinement type?`. +- **Impact:** Kernels and extension crates must handle both shapes. `vortex-tensor` handles both by + hand. The same choice exists for `Dict` and `Chunked`. +- **Fix:** A rewrite rule pulls structural arrays such as `Constant` and `Dict` out from under the + wrapper (decision X5 in `../../decisions.md`). `design-notes.md`, section 6, has the details. + +### EXT-S2: Some encodings carry the extension dtype themselves + +- **Locations:** `encodings/datetime-parts/src/canonical.rs:28` requires a `Timestamp` dtype on the + `DateTimeParts` array. `encodings/parquet-variant/src/kernel.rs:196` registers a kernel on the + shared `Extension` encoding, and the kernel checks `is::()` itself. +- **Impact:** An array with an extension dtype is not always the wrapper. Extension-specific kernels + cannot be keyed on the wrapper encoding alone. + +### EXT-B1: DataFusion converts extension values to storage values + +- **Locations:** + - `vortex-datafusion/src/convert/scalars.rs:138-145`: a non-temporal extension scalar converts as + its storage scalar. + - `vortex-datafusion/src/convert/scalars.rs:324`: `struct_to_df` converts extension children to + storage values. + - `vortex-datafusion/src/convert/schema.rs:105`: schema conversion walks through extension dtypes + to the storage struct. + - `vortex-datafusion/src/convert/stats/mod.rs:66`: `is_nan_free` checks `dtype.is_float()`, which + is false for an extension with float storage. +- **Guard to keep:** `vortex-datafusion/src/convert/exprs/mod.rs:355-361` refuses to push down a + cast that involves an extension type. + +### EXT-B2: Arrow export can silently export storage + +- **Locations:** + - `vortex-arrow/src/executor/primitive.rs:45`, `byte_view.rs:64`, `struct_.rs:115`: exporters + call `array.cast(target)`, which reaches EXT-C2 for an extension input. + - `vortex-arrow/src/session.rs:329`: `to_arrow_field` handles `AnyTemporal` by itself and needs a + plugin for every other extension type. + - `vortex-arrow/src/session.rs:873-876`: export falls back to `execute_arrow_naive` with the bare + Arrow data type. + - `vortex-arrow/src/scalar.rs:242`: scalar export supports temporal extension types only. +- **Impact:** A caller can ask for a storage-like Arrow type, for example `Int64` for a timestamp + column through `to_arrow_array(arrow_type=...)` in Python. The caller then gets the storage + values with no error. + +### EXT-B3: Python returns and builds raw storage values + +- **Locations:** `vortex-python/src/scalar/into_py.rs:93-95` (`as_py()` returns the storage value). + `vortex-python/src/scalar/factory.rs:106` (`vx.scalar(v, dtype=ext)` calls `Scalar::cast`, so + EXT-C1 applies). + +### EXT-B4: DuckDB decides pushdown from the storage type + +- **Locations:** + - `vortex-duckdb/src/table_function.rs:532-537`: aggregate pushdown excludes `Uuid` only, because + its storage is a `FixedSizeList`. Every other extension type gets min and max pushed down in + storage order. + - `vortex-duckdb/src/column_statistics.rs:37`: the NaN check uses `dtype.is_float()`. + - `vortex-duckdb/src/convert/dtype.rs:260-284` and `vortex-duckdb/src/exporter/extension.rs:36`: + a hard-coded list of extension types. There is no plugin hook. + +### EXT-B5: JNI write summaries report storage values + +- **Location:** `vortex-jni/src/writer.rs:217` converts extension bounds with + `to_storage_scalar()`. Java receives a raw `i64` for a timestamp bound, with no unit. + +### EXT-B6: Pruning looks through casts out of extensions (lead) + +- **Locations:** `vortex-file/src/pruning.rs:105-116` (`direct_field_path` looks through `Cast`) + and `vortex-array/src/stats/rewrite/builtins.rs:797-803` (`cast_stat` casts min and max). +- **Suspected behavior:** With EXT-C2, `cast(ext_col, i64)` reuses the statistics of the extension + column through a storage cast. This is correct only for casts that keep order. +- **Check:** Build a pruning predicate on `cast(timestamp_col, u8)` and compare the pruning result + with a full scan. + +### EXT-R1: `Debug for ExtDTypeRef` prints the wrong output + +- **Location:** `vortex-array/src/dtype/extension/erased.rs:181-191`. Line 183 writes the metadata + into the formatter and keeps the returned `fmt::Result` as the metadata field. +- **Observed** [N]: + + ```text + TimestampOptions { unit: Milliseconds, tz: None }ExtDType { id: Id("vortex.timestamp"), + metadata: Ok(()), storage_dtype: Primitive(I64, NonNullable) } + ``` + + The output is one line. It is split here to fit the page. + +### EXT-R2: Registration overwrites and duplicates silently + +- **Locations:** `vortex-array/src/dtype/session.rs:57-60` (`register` replaces an existing ID with + no error). `vortex-arrow/src/session.rs:257-269` (`register_exporter` and `register_importer` + push onto a list, so a second `initialize` call registers each plugin twice). + +### EXT-R3: Unknown extension IDs inherit storage semantics + +- **Locations:** + - `vortex-array/src/dtype/serde/flatbuffers.rs:321`, `proto.rs:145` and `serde.rs:806`: three + copies of the lookup, with three different error messages. + - `vortex-array/src/dtype/extension/foreign.rs:28-75`: with `allow_unknown()`, an unknown ID + becomes a `ForeignExtDType` that accepts any storage and any value. + - `vortex-array/src/dtype/extension/mod.rs:31-32`: `ForeignExtDType` is crate-private, so other + crates cannot detect a placeholder. +- **Impact:** A placeholder gets all of EXT-Q2 and EXT-A1, so a reader without the type compares + and orders values by storage. + +### EXT-R4: Documentation is a stub or stale + +- `docs/developer-guide/extending/extension-dtypes.md` is a stub. +- `docs/developer-guide/internals/vtables.md:84-113` shows the old `ExtDTypeVTable` shape. +- `vortex-array/src/arrays/extension/vtable/mod.rs:39-74` says that extension arrays delegate most + operations to storage. That statement conflicts with an opaque design. + +## Appendix A: probe results + +`probe.rs` defines three test extension types: `NonZero` (`u64`, rejects zero), `NonNullableOnly` +(rejects nullable storage), and `Opaque` (`i64`, no meaning for order). The probe ran in a debug +build of `vortex-array`. "Row" means `execute_scalar` per row on the lazy result. "Array" means +`execute::`. + +| Probe | Expression | Path | Result | +| --- | --- | --- | --- | +| A | scalar `cast(Timestamp(ms)=1000, Timestamp(ns))` | scalar | `1970-01-01T00:00:01Z => 1970-01-01T00:00:00.000001Z` | +| B | `cast(i64 [1000, 2000], Timestamp(ms))` | row | wraps: `[00:00:01Z, 00:00:02Z]` | +| B' | same | array | `No CastKernel to cast canonical array vortex.primitive from i64 to vortex.timestamp[ms](i64)` | +| C | `cast(Constant(1000i64), Timestamp(ms))` | row | wraps | +| D1 | `cast(Timestamp(ms) [1000, 2000], Timestamp(ns))` | row | `[00:00:00.000001Z, 00:00:00.000002Z]` | +| D1' | same | array | no `CastKernel` error | +| D2 | same cast, constant storage | row | `00:00:00.000001Z` | +| D2' | same | array | `00:00:00.000001Z` | +| E1, E1' | `cast(Timestamp(ms), i64)` | both | `[1000, 2000]` | +| E2 | `cast(Timestamp(ms), f64)` | row | `[1000.0, 2000.0]` | +| E3 | `cast(Timestamp(ms) [1, 2], u8)` | row | `[1, 2]` | +| E4 | `cast(Opaque, NonZero)` | row | `scratch.non_zero(u64) [1, 2]` | +| E4' | same | array | no `CastKernel` error | +| F1, F2 | `Timestamp(ms) < i64` (array, then constant) | row | `Cannot compare scalars with incompatible types` | +| F1', F2' | same | array | `[true, false]` | +| F3' | `Timestamp(ms) < utf8` | array | `Cannot compare different DTypes i64 and utf8` | +| F4 | `between(Timestamp(ms), 0i64, 1500i64)` | build | `Array dtype ... does not match lower dtype i64` | +| F5' | `Timestamp(ms) < Timestamp(ns)` | array | `Cannot compare extension dtypes ...` | +| G | `Opaque < Opaque` | row | `[true, false]` | +| H | `min_max(Opaque [5, 1, 3])` | call | `min=1 max=5` | +| I | `is_sorted(Timestamp(ms) [1, 2, 3])` | call | `false` | +| K1 | `Scalar::zero_value(NonZero)` | call | panic: `Scalar::new_unchecked called with incompatible dtype and value` | +| L1 | `ExtensionBuilder(NonZero).append_zeros(1).finish()` | call | succeeds | +| L2 | then `execute_scalar(0)` | row | panic: ``unable to construct an extension `Scalar` `` | +| L3 | `ExtensionArray::new(NonZero, [0])` | call | succeeds | +| M1 | `take(NonNullableOnly, nullable indices)` | call | panic: `Extension DType should not fail validation ...` | +| N | `format!("{:?}", Timestamp(ms))` | call | see EXT-R1 | +| O | `try_from_parts(Timestamp(ms), utf8 storage)` | call | succeeds | +| P | `Timestamp(ms) + Timestamp(ms)` | build | `incompatible types for arithmetic operation` | +| Q | `Scalar::is_zero(Timestamp(ms)=0)` | call | `Some(true)` | + +### Follow-up probes + +`probe_followup.rs` ran in a debug build of `vortex-btrblocks`, on `5fc5cb8`. It reuses `NonZero` +and `Opaque`. Its session registers the `DateTimeParts` and `Sparse` encodings. R8 uses a cascading +compressor with the integer dictionary scheme only. With the default schemes, `TemporalScheme` can +also write `DateTimeParts` at the top of a `Timestamp` column +(`vortex-btrblocks/src/schemes/temporal.rs:45-56`). + +| Probe | Expression | Path | Result | +| --- | --- | --- | --- | +| R1 | `cast(Decimal(10, 2) [1.00, 2.50], f32)` | array | `No CastKernel to cast canonical array vortex.decimal from decimal(10,2) to f32` | +| R1 | same | row | `[1f32, 2.5f32]` | +| R1 | `cast(Decimal(10, 2) 1.00, f32)` on a scalar | scalar | `1f32` | +| R2 | `Opaque(1).partial_cmp(Opaque(2))` | call | `Some(Less)` | +| R2 | `Opaque(1) == Opaque(1)`, then `partial_cmp` | call | `true`, `Some(Equal)` | +| R2 | `Scalar::try_new(Timestamp(ms), F64(5.0))` | call | succeeds: `1970-01-01T00:00:00.005Z` | +| R2 | `Timestamp(ms) I64(5) == Timestamp(ms) F64(5.0)` | call | panic: `upcast` | +| R2 | `Timestamp(ms) F64(5.0) == Timestamp(ms) I64(5)` | call | `false` | +| R2 | `Timestamp(ms) I64(5)` `partial_cmp` `F64(5.0)` | call | panic: `upcast` | +| R3 | `NaN == NaN`, then `-0.0 == +0.0`, on scalars | call | `true`, `false` | +| R3 | `[NaN, -0.0] = [NaN, +0.0]` | array | `[true, false]` | +| R3 | `FixedSizeList`: `[[NaN, -0.0], [NaN, 1.0]] = [[NaN, +0.0], [NaN, 1.0]]` | array | `[false, true]` | +| R4 | `Scalar::try_new(NonZero, 0u64)` | call | `zero is not a valid NonZero value` | +| R5 | `Extension(Constant(1000i64))`: tree, then `as_constant()` | call | `vortex.ext` over `vortex.constant`, `None` | +| R5 | same, after `optimize()` | call | `vortex.constant(vortex.timestamp[ms](i64))`, `Some(1970-01-01T00:00:01Z)` | +| R6 | `ext_storage(Extension(Dict))`, one step | call | `vortex.dict(i64)` | +| R6 | `ext_storage(Dict(codes, Extension(values)))` | call | The lazy tree pushes `ext_storage` into the values. One step gives `vortex.dict(i64)` | +| R6 | `ext_storage(Chunked[Extension, Extension])` | call | The lazy tree pushes `ext_storage` into each chunk | +| R6 | `ext_storage(Masked(Extension))`, one step | call | No pushdown. One step decodes to `vortex.primitive(i64?)` | +| R6 | `ext_storage(Sparse)` with extension values, one step | call | `Sparse canonicalization does not support Extension arrays yet` | +| R7 | `DateTimeParts` of `Timestamp(ms) [1.6 s, 1.4 s]` `< 1500i64` | array | panic: `Failed to convert scalar to extension` | +| R7 | same | row | `Cannot compare scalars with incompatible types` | +| R7 | same column `< Timestamp(ns)` 1.5 s | array | `[true, true]`, which is wrong. Expected: `[false, true]` | +| R7 | same | row | `Cannot compare scalars with incompatible types` | +| R7 | same column `< Timestamp(ms)` 1.5 s, the control | array | `[false, true]` | +| R8 | Compress a constant `Timestamp(ms)` column | call | `vortex.constant(vortex.timestamp[ms](i64))` | +| R8 | Compress a low-cardinality `Opaque` column | call | `vortex.ext` over `vortex.dict(i64)` | +| R8 | Compress a constant `Timestamp(ms)` column with some nulls | call | `vortex.masked` over `vortex.constant(vortex.timestamp[ms](i64?))` | +| R9 | `ext_storage(NonZero [0, 1])` | array | `[0u64, 1u64]` | +| R9 | same | row | panic: ``unable to construct an extension `Scalar` `` | +| R10 | `Scalar::try_new(F16, U64 bits of 1.0)` | call | succeeds: `1f16` | +| R10 | `U64` form `==` `F16` form of 1.0 | call | panic: `upcast` | +| R10 | `F16` form `==` `U64` form of 1.0 | call | `false` | + +## Appendix B: where storage semantics are applied + +This table lists every place where code outside an extension crate applies storage semantics to an +extension value. "Structural" means that the operation does not read values, so it is correct for +any extension type. + +| Location | Behavior | Kind | +| --- | --- | --- | +| `vortex-array/src/arrays/extension/compute/{filter,slice,take,mask}.rs` | Run on storage and wrap again | Structural | +| `vortex-array/src/arrays/dict/execute.rs:182`, `arrays/masked/execute.rs:178` | Run on storage and wrap again | Structural | +| `vortex-compressor/src/compressor/cascade.rs:175-205` | Compress storage and wrap again | Structural | +| `vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes/extension.rs` | Size of storage | Structural | +| `vortex-array/src/aggregate_fn/fns/is_constant/extension.rs` | Storage equality | Same storage implies same value. A `false` can be wrong for types with non-canonical storage. | +| `vortex-array/src/aggregate_fn/fns/all_non_distinct/extension.rs` | Storage equality | Same note as `is_constant` | +| `vortex-array/src/arrays/extension/compute/compare.rs` | Storage compare | Semantic (EXT-Q2) | +| `vortex-array/src/scalar_fn/fns/binary/compare/mod.rs:177-193` | Storage compare, lone side | Semantic (EXT-Q1) | +| `vortex-array/src/scalar_fn/fns/binary/compare/nested.rs:229` | Storage compare | Semantic (EXT-Q2) | +| `vortex-array/src/scalar/typed_view/extension/mod.rs:107-137` | Storage eq, order, hash | Semantic (EXT-Q2) | +| `vortex-array/src/aggregate_fn/fns/min_max/extension.rs` | Storage min and max | Semantic (EXT-A1) | +| `vortex-array/src/arrays/extension/compute/cast.rs:18-22` | Storage cast | Semantic (EXT-C2) | +| `vortex-array/src/scalar/typed_view/extension/mod.rs:87-96` | Storage cast of a scalar, exact storage target only | Semantic (EXT-C2) | +| `vortex-array/src/scalar/cast.rs:56-61` | Storage cast and wrap | Semantic (EXT-C1) | +| `vortex-array/src/scalar/scalar_value.rs:90-95`, `builders/extension.rs:110` | Storage zero | Semantic (EXT-V3) | +| `vortex-layout/src/layouts/zoned/schema.rs:112-121`, `:177-188` | Storage stats dtype | Semantic (EXT-A2) | +| `vortex-arrow/src/executor/{primitive,byte_view,struct_}.rs` | Storage export through cast | Semantic (EXT-B2) | +| `vortex-datafusion/src/convert/scalars.rs:138-145`, `schema.rs:105` | Storage conversion | Semantic (EXT-B1) | +| `vortex-python/src/scalar/into_py.rs:93-95` | Storage value | Semantic (EXT-B3) | +| `vortex-duckdb/src/table_function.rs:532-537` | Storage-based pushdown decision | Semantic (EXT-B4) | +| `vortex-jni/src/writer.rs:217` | Storage bounds | Semantic (EXT-B5) | +| `vortex-spatial/src/prune/mod.rs:100-104` | `ext_storage` on a statistic, in the owning crate | Correct use (explicit unwrap by the type owner) | + +## Appendix C: authoring cost + +These numbers come from a survey of every `ExtVTable` implementation. They are evidence for the +design, not defects. + +- The repository has 15 extension types outside tests (plus `ForeignExtDType`). +- `vortex-spatial` has about 5,800 lines of non-test source. Its eight `ExtVTable` implementations + are 241 lines (about 4%). Its Arrow import and export plugins are 1,132 lines. +- Authors wrote eight typed wrappers around `ExtensionArray` (`TemporalData` in + `vortex-array/src/arrays/datetime/mod.rs:44`, and `WellKnownBinaryData`, `PointData`, + `LineStringData`, `MultiPointData`, `PolygonData`, `MultiLineStringData`, `MultiPolygonData` in + `vortex-spatial`). +- Authors wrote several matchers (`AnyTemporal`, `AnyTensor`, `AnyVector`, `AnyFixedShapeTensor`) + and row input elements (`TensorRow`, `GeometryRow`). +- DuckDB, DataFusion and Python have no plugin hooks for extension types, so each new type needs + code in those crates. diff --git a/research/extension-types/agents/probe.rs b/research/extension-types/agents/probe.rs new file mode 100644 index 00000000000..35588c2233a --- /dev/null +++ b/research/extension-types/agents/probe.rs @@ -0,0 +1,520 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Throwaway probes for extension type behavior on `develop` at `5fc5cb8`. +//! +//! This file is not part of any crate, so nothing compiles it. To run it, copy it into +//! `vortex-array/tests/` and run: +//! +//! ```text +//! cargo test -p vortex-array --test probe -- --nocapture +//! ``` +//! +//! The build needs several GB of disk space. Appendix A of `findings.md` records the output of the +//! last run. The `probes` test reads rows with `execute_scalar` (the row path). The +//! `probes_array_path` test executes whole arrays with `RecursiveCanonical` (the array path). + +#![allow(clippy::all, clippy::disallowed_methods, unused)] + +use std::panic::AssertUnwindSafe; +use std::panic::catch_unwind; + +use vortex_array::ArrayParts; +use vortex_array::ArrayRef; +use vortex_array::EmptyArrayData; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::aggregate_fn::NumericalAggregateOpts; +use vortex_array::aggregate_fn::fns::is_sorted::is_sorted; +use vortex_array::aggregate_fn::fns::min_max::min_max; +use vortex_array::array_session; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::Extension; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::VarBinViewArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::extension::ExtensionSlots; +use vortex_array::builders::ArrayBuilder; +use vortex_array::builders::ExtensionBuilder; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::dtype::extension::ExtDTypeRef; +use vortex_array::dtype::extension::ExtId; +use vortex_array::dtype::extension::ExtVTable; +use vortex_array::extension::datetime::TimeUnit; +use vortex_array::extension::datetime::Timestamp; +use vortex_array::scalar::Scalar; +use vortex_array::scalar::ScalarValue; +use vortex_array::scalar_fn::fns::between::BetweenOptions; +use vortex_array::scalar_fn::fns::between::StrictComparison; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_buffer::BufferAllocatorRef; +use vortex_buffer::buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; + +/// u64 values where zero is not a valid value. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonZero; + +impl ExtVTable for NonZero { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_zero") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::U64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + let v = storage_value.as_primitive().cast::()?; + if v == 0 { + vortex_bail!("zero is not a valid NonZero value"); + } + Ok(v) + } +} + +/// An extension type that only accepts non-nullable storage. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonNullableOnly; + +impl ExtVTable for NonNullableOnly { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_nullable_only") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!( + ext_dtype.storage_dtype() == &DType::Primitive(PType::U64, Nullability::NonNullable) + ); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +/// An extension type with no meaningful order (think "opaque handle"). +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct Opaque; + +impl ExtVTable for Opaque { + type Metadata = EmptyMetadata; + type NativeValue<'a> = i64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.opaque") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::I64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +fn report(name: &str, f: impl FnOnce() -> VortexResult) { + match catch_unwind(AssertUnwindSafe(f)) { + Ok(Ok(v)) => println!("[{name}] OK: {v:?}"), + Ok(Err(e)) => println!( + "[{name}] ERR: {}", + e.to_string().lines().next().unwrap_or_default() + ), + Err(p) => { + let msg = p + .downcast_ref::() + .cloned() + .or_else(|| p.downcast_ref::<&str>().map(|s| s.to_string())) + .unwrap_or_default(); + println!( + "[{name}] PANIC: {}", + msg.lines().next().unwrap_or_default() + ) + } + } +} + +fn ts(unit: TimeUnit) -> ExtDTypeRef { + Timestamp::new(unit, Nullability::NonNullable).erased() +} + +fn ts_array(unit: TimeUnit, values: ArrayRef) -> ArrayRef { + ExtensionArray::new(ts(unit), values).into_array() +} + +fn display(a: &ArrayRef) -> VortexResult { + let mut ctx = array_session().create_execution_ctx(); + let mut out = vec![]; + for i in 0..a.len() { + out.push(a.execute_scalar(i, &mut ctx)?.to_string()); + } + Ok(format!("{} [{}]", a.dtype(), out.join(", "))) +} + +/// Fully materialize through the array kernels, then read the stored values. +fn display_array(a: &ArrayRef) -> VortexResult { + let mut ctx = array_session().create_execution_ctx(); + let canonical = a + .clone() + .execute::(&mut ctx)? + .0 + .into_array(); + display(&canonical) +} + +#[test] +fn probes_array_path() { + report("B' array path: PrimitiveArray i64 -> ts(ms)", || { + let a = buffer![1000i64, 2000].into_array(); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?) + }); + + report("D1' array path: ts(ms) -> ts(ns)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?) + }); + + report("D2' array path: constant ts(ms) -> ts(ns)", || { + let a = ConstantArray::new( + Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)), + 2, + ) + .into_array(); + display_array(&a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?) + }); + + report("E1' array path: ts(ms) -> i64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display_array(&a.cast(DType::Primitive(PType::I64, Nullability::NonNullable))?) + }); + + report("E4' array path: Opaque -> NonZero", || { + let opaque = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(opaque, buffer![1i64, 2].into_array()).into_array(); + display_array(&a.cast(DType::Extension(nz))?) + }); + + report("E5' array path: i64 [0] -> NonZero (invalid value)", || { + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = buffer![0i64, 1].into_array(); + display_array(&a.cast(DType::Extension(nz))?) + }); + + report("F1' array path: ts(ms) array < i64 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = buffer![1500i64, 1500].into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F2' array path: ts(ms) array < i64 constant", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F3' array path: ts(ms) array < utf8 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + display_array(&a.binary(b, Operator::Lt)?) + }); + + report("F5' array path: ts(ms) array < ts(ns) array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ts_array(TimeUnit::Nanoseconds, buffer![1500i64, 1500].into_array()); + display_array(&a.binary(b, Operator::Lt)?) + }); +} + +#[test] +fn probes() { + let session = array_session(); + + // --- Casts -------------------------------------------------------------------------------- + + report("A scalar cast ts(ms)=1000 -> ts(ns)", || { + let s = Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)); + let c = s.cast(&DType::Extension(ts(TimeUnit::Nanoseconds)))?; + Ok(format!("{s} => {c}")) + }); + + report("B array cast PrimitiveArray i64 -> ts(ms)", || { + let a = buffer![1000i64, 2000].into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?; + display(&c) + }); + + report("C constant cast i64 -> ts(ms)", || { + let a = ConstantArray::new(Scalar::from(1000i64), 2).into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Milliseconds)))?; + display(&c) + }); + + report("D1 array cast ts(ms) -> ts(ns) (non-constant)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let c = a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?; + display(&c) + }); + + report("D2 array cast ts(ms) -> ts(ns) (constant)", || { + let a = ConstantArray::new( + Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(1000i64)), + 2, + ) + .into_array(); + let c = a.cast(DType::Extension(ts(TimeUnit::Nanoseconds)))?; + display(&c) + }); + + report("E1 array cast ts(ms) -> i64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display(&a.cast(DType::Primitive(PType::I64, Nullability::NonNullable))?) + }); + + report("E2 array cast ts(ms) -> f64", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + display(&a.cast(DType::Primitive(PType::F64, Nullability::NonNullable))?) + }); + + report("E3 array cast ts(ms) -> u8 (narrowing)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + display(&a.cast(DType::Primitive(PType::U8, Nullability::NonNullable))?) + }); + + report("E4 array cast Opaque -> NonZero (different ext ids)", || { + let opaque = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(opaque, buffer![1i64, 2].into_array()).into_array(); + display(&a.cast(DType::Extension(nz))?) + }); + + // --- Comparisons -------------------------------------------------------------------------- + + report("F1 compare ts(ms) array < i64 array", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = buffer![1500i64, 1500].into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + report("F2 compare ts(ms) array < i64 constant", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + report("F3 compare ts(ms) array < utf8 array (typing)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let b = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + // Building the expression succeeds or fails here; executing it is in display(). + let r = a.binary(b, Operator::Lt)?; + display(&r) + }); + + report("F4 between ts(ms) with i64 bounds", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1000i64, 2000].into_array()); + let lo = ConstantArray::new(Scalar::from(0i64), 2).into_array(); + let hi = ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + display(&a.between( + lo, + hi, + BetweenOptions { + lower_strict: StrictComparison::NonStrict, + upper_strict: StrictComparison::NonStrict, + }, + )?) + }); + + report("G compare Opaque < Opaque (no opt-in needed)", || { + let dt = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(dt.clone(), buffer![1i64, 5].into_array()).into_array(); + let b = ExtensionArray::new(dt, buffer![3i64, 3].into_array()).into_array(); + display(&a.binary(b, Operator::Lt)?) + }); + + // --- Aggregates --------------------------------------------------------------------------- + + report("H min_max Opaque (no opt-in needed)", || { + let dt = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + )? + .erased(); + let a = ExtensionArray::new(dt, buffer![5i64, 1, 3].into_array()).into_array(); + let mut ctx = session.create_execution_ctx(); + let r = min_max(&a, &mut ctx, NumericalAggregateOpts::default())?; + Ok(r.map(|r| format!("min={} max={}", r.min, r.max))) + }); + + report("I is_sorted on sorted ts(ms)", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2, 3].into_array()); + let mut ctx = session.create_execution_ctx(); + is_sorted(&a, &mut ctx) + }); + + // --- Invalid values from "zero" / "default" paths ----------------------------------------- + + let nz = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased(); + + report("K1 Scalar::zero_value(NonZero)", || { + let z = Scalar::zero_value(&DType::Extension(nz.clone())); + Ok(format!("created {z}")) + }); + + report("K2 Scalar::try_new(NonZero, zero_value)", || { + let z = Scalar::zero_value(&DType::Extension(nz.clone())); + let (dtype, value) = z.into_parts(); + Ok(Scalar::try_new(dtype, value)?.to_string()) + }); + + report("L1 ExtensionBuilder(NonZero).append_zeros(1).finish()", || { + let mut b = ExtensionBuilder::new_in(nz.clone(), BufferAllocatorRef::static_ref()); + b.append_zeros(1); + let a = b.finish(); + Ok(format!("built array of len {}", a.len())) + }); + + report("L2 ... then execute_scalar(0)", || { + let mut b = ExtensionBuilder::new_in(nz.clone(), BufferAllocatorRef::static_ref()); + b.append_zeros(1); + let a = b.finish(); + display(&a) + }); + + report("L3 ExtensionArray::new(NonZero, [0]) (no value validation)", || { + let a = ExtensionArray::new(nz.clone(), buffer![0u64].into_array()).into_array(); + Ok(format!("built array of len {}", a.len())) + }); + + // --- Nullability changes ------------------------------------------------------------------ + + let nn = ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased(); + + report("M1 take(NonNullableOnly, nullable indices)", || { + let a = ExtensionArray::new(nn.clone(), buffer![1u64, 2].into_array()).into_array(); + let idx = PrimitiveArray::from_option_iter([Some(0u32), None]).into_array(); + display(&a.take(idx)?) + }); + + report("M2 with_nullability(NonNullableOnly, Nullable)", || { + Ok(nn.with_nullability(Nullability::Nullable).to_string()) + }); + + // --- Misc --------------------------------------------------------------------------------- + + report("N Debug of ExtDTypeRef", || Ok(format!("{:?}", ts(TimeUnit::Milliseconds)))); + + report("O try_from_parts with utf8 storage under ts(ms)", || { + let storage = VarBinViewArray::from_iter_str(["a", "b"]).into_array(); + let parts = ArrayParts::new( + Extension, + DType::Extension(ts(TimeUnit::Milliseconds)), + 2, + EmptyArrayData, + ExtensionSlots { storage }.into_slots(), + ); + let a = ExtensionArray::try_from_parts(parts)?; + Ok(format!("accepted: dtype={} storage dtype={}", a.dtype(), a.storage_array().dtype())) + }); + + report("P arithmetic ts + ts", || { + let a = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + let b = ts_array(TimeUnit::Milliseconds, buffer![1i64, 2].into_array()); + display(&a.binary(b, Operator::Add)?) + }); + + report("Q Scalar::is_zero on ts(ms)=0", || { + let s = Scalar::extension_ref(ts(TimeUnit::Milliseconds), Scalar::from(0i64)); + Ok(s.is_zero()) + }); +} diff --git a/research/extension-types/agents/probe_followup.rs b/research/extension-types/agents/probe_followup.rs new file mode 100644 index 00000000000..c4d154e836c --- /dev/null +++ b/research/extension-types/agents/probe_followup.rs @@ -0,0 +1,522 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Throwaway follow-up probes for the type research on `develop` at `5fc5cb8`. +//! +//! This file is not part of any crate, so nothing compiles it. To run it, copy it to +//! `vortex-btrblocks/tests/research_probe.rs` and run: +//! +//! ```text +//! cargo test -p vortex-btrblocks --test research_probe -- --nocapture --test-threads=1 +//! ``` +//! +//! Delete the copy afterwards. Appendix A of `findings.md` records the output of the last run. +//! "Array path" means `execute::` on the lazy array, then a read of each row. +//! "Row path" means `execute_scalar` on each row of the lazy array. + +#![allow(clippy::all, clippy::disallowed_methods, unused)] + +use std::panic::AssertUnwindSafe; +use std::panic::catch_unwind; +use std::sync::LazyLock; + +use vortex_array::ArrayRef; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::RecursiveCanonical; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::ChunkedArray; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::DecimalArray; +use vortex_array::arrays::DictArray; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::TemporalArray; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::DecimalDType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::dtype::extension::ExtDTypeRef; +use vortex_array::dtype::extension::ExtId; +use vortex_array::dtype::extension::ExtVTable; +use vortex_array::expr::ext_storage; +use vortex_array::expr::root; +use vortex_array::extension::datetime::TimeUnit; +use vortex_array::extension::datetime::Timestamp; +use vortex_array::optimizer::ArrayOptimizer; +use vortex_array::scalar::DecimalValue; +use vortex_array::scalar::PValue; +use vortex_array::scalar::Scalar; +use vortex_array::scalar::ScalarValue; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_array::validity::Validity; +use vortex_btrblocks::CascadingCompressor; +use vortex_btrblocks::Scheme; +use vortex_btrblocks::schemes::integer::IntDictScheme; +use vortex_buffer::buffer; +use vortex_datetime_parts::DateTimeParts; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_session::VortexSession; +use vortex_sparse::Sparse; + +/// The array session with the `DateTimeParts` and `Sparse` encodings and their kernels registered. +static SESSION: LazyLock = LazyLock::new(|| { + let session = vortex_array::array_session(); + vortex_datetime_parts::initialize(&session); + vortex_sparse::initialize(&session); + session +}); + +/// u64 values where zero is not a valid value. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct NonZero; + +impl ExtVTable for NonZero { + type Metadata = EmptyMetadata; + type NativeValue<'a> = u64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.non_zero") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::U64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + let v = storage_value.as_primitive().cast::()?; + if v == 0 { + vortex_bail!("zero is not a valid NonZero value"); + } + Ok(v) + } +} + +/// An extension type with no meaningful order (think "opaque handle"). +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +struct Opaque; + +impl ExtVTable for Opaque { + type Metadata = EmptyMetadata; + type NativeValue<'a> = i64; + + fn id(&self) -> ExtId { + ExtId::new("scratch.opaque") + } + + fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { + Ok(vec![]) + } + + fn deserialize_metadata(&self, _metadata: &[u8]) -> VortexResult { + Ok(EmptyMetadata) + } + + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + vortex_ensure!(matches!( + ext_dtype.storage_dtype(), + DType::Primitive(PType::I64, _) + )); + Ok(()) + } + + fn unpack_native<'a>( + _ext_dtype: &'a ExtDType, + storage_value: &'a ScalarValue, + ) -> VortexResult> { + storage_value.as_primitive().cast::() + } +} + +fn report(name: &str, f: impl FnOnce() -> VortexResult) { + match catch_unwind(AssertUnwindSafe(f)) { + Ok(Ok(v)) => println!("[{name}] OK: {v:?}"), + Ok(Err(e)) => println!( + "[{name}] ERR: {}", + e.to_string().lines().next().unwrap_or_default() + ), + Err(p) => { + let msg = p + .downcast_ref::() + .cloned() + .or_else(|| p.downcast_ref::<&str>().map(|s| s.to_string())) + .unwrap_or_default(); + println!( + "[{name}] PANIC: {}", + msg.lines().next().unwrap_or_default() + ) + } + } +} + +fn ts(unit: TimeUnit, nullability: Nullability) -> ExtDTypeRef { + Timestamp::new(unit, nullability).erased() +} + +fn opaque() -> ExtDTypeRef { + ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::I64, Nullability::NonNullable), + ) + .unwrap() + .erased() +} + +fn non_zero() -> ExtDTypeRef { + ExtDType::::try_new( + EmptyMetadata, + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .erased() +} + +/// Reads every row with `execute_scalar` (the row path). +fn rows(a: &ArrayRef) -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let mut out = vec![]; + for i in 0..a.len() { + out.push(a.execute_scalar(i, &mut ctx)?.to_string()); + } + Ok(format!("{} [{}]", a.dtype(), out.join(", "))) +} + +/// Executes the whole array through the array kernels, then reads the stored values. +fn array(a: &ArrayRef) -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let canonical = a + .clone() + .execute::(&mut ctx)? + .0 + .into_array(); + rows(&canonical) +} + +/// The encoding tree of an array, on one line. +fn tree(a: &ArrayRef) -> String { + a.display_tree_encodings_only() + .to_string() + .lines() + .map(str::trim) + .collect::>() + .join(" | ") +} + +#[test] +fn r1_decimal_to_f32() { + let dec = || { + DecimalArray::new( + buffer![100i64, 250], + DecimalDType::new(10, 2), + Validity::NonNullable, + ) + .into_array() + }; + let f32_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); + + report("R1 array path: Decimal(10, 2) [1.00, 2.50] -> f32", || { + array(&dec().cast(f32_dtype.clone())?) + }); + report("R1 row path: Decimal(10, 2) [1.00, 2.50] -> f32", || { + rows(&dec().cast(f32_dtype.clone())?) + }); + report("R1 scalar path: Decimal(10, 2) 1.00 -> f32", || { + let s = Scalar::decimal( + DecimalValue::I64(100), + DecimalDType::new(10, 2), + Nullability::NonNullable, + ); + Ok(s.cast(&f32_dtype)?.to_string()) + }); +} + +#[test] +fn r2_scalar_traits() { + let ms = ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let a = Scalar::extension_ref(opaque(), Scalar::from(1i64)); + let b = Scalar::extension_ref(opaque(), Scalar::from(2i64)); + + report("R2 Opaque(1).partial_cmp(Opaque(2))", || Ok(a.partial_cmp(&b))); + report("R2 Opaque(1) == Opaque(1), and partial_cmp", || { + Ok((a == a.clone(), a.partial_cmp(&a))) + }); + + let f64_ts = || { + Scalar::try_new( + DType::Extension(ms.clone()), + Some(ScalarValue::Primitive(PValue::F64(5.0))), + ) + }; + let i64_ts = Scalar::extension_ref(ms.clone(), Scalar::from(5i64)); + + report("R2 try_new(Timestamp(ms), F64 5.0)", || { + Ok(f64_ts()?.to_string()) + }); + report("R2 Timestamp(ms) I64 5 == Timestamp(ms) F64 5.0", || { + Ok(i64_ts == f64_ts()?) + }); + report("R2 Timestamp(ms) F64 5.0 == Timestamp(ms) I64 5", || { + Ok(f64_ts()? == i64_ts) + }); + report("R2 Timestamp(ms) I64 5 partial_cmp F64 5.0", || { + Ok(i64_ts.partial_cmp(&f64_ts()?)) + }); +} + +#[test] +fn r3_float_identity() { + report("R3 Scalar NaN == NaN", || { + Ok(Scalar::from(f64::NAN) == Scalar::from(f64::NAN)) + }); + report("R3 Scalar -0.0 == +0.0", || { + Ok(Scalar::from(-0.0f64) == Scalar::from(0.0f64)) + }); + report("R3 array path: [NaN, -0.0] = [NaN, +0.0]", || { + let l = buffer![f64::NAN, -0.0].into_array(); + let r = buffer![f64::NAN, 0.0].into_array(); + array(&l.binary(r, Operator::Eq)?) + }); + report( + "R3 array path: FSL [[NaN, -0.0], [NaN, 1.0]] = [[NaN, +0.0], [NaN, 1.0]]", + || { + let l = FixedSizeListArray::try_new( + buffer![f32::NAN, -0.0, f32::NAN, 1.0].into_array(), + 2, + Validity::NonNullable, + 2, + )? + .into_array(); + let r = FixedSizeListArray::try_new( + buffer![f32::NAN, 0.0, f32::NAN, 1.0].into_array(), + 2, + Validity::NonNullable, + 2, + )? + .into_array(); + array(&l.binary(r, Operator::Eq)?) + }, + ); +} + +#[test] +fn r4_non_zero_check() { + report("R4 Scalar::try_new(NonZero, 0u64)", || { + Ok(Scalar::try_new( + DType::Extension(non_zero()), + Some(ScalarValue::Primitive(PValue::U64(0))), + )? + .to_string()) + }); +} + +#[test] +fn r5_constant_detection() { + let ms = ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let ext_const = ExtensionArray::new( + ms, + ConstantArray::new(Scalar::from(1000i64), 3).into_array(), + ) + .into_array(); + + report("R5 Extension(Constant): tree and as_constant()", || { + Ok((tree(&ext_const), ext_const.as_constant().map(|s| s.to_string()))) + }); + report("R5 after optimize(): tree and as_constant()", || { + let optimized = ext_const.optimize()?; + Ok(( + tree(&optimized), + optimized.as_constant().map(|s| s.to_string()), + )) + }); +} + +#[test] +fn r6_storage_extraction() { + let codes = || buffer![0u8, 1, 0, 1].into_array(); + let values = || buffer![10i64, 20].into_array(); + + let one_step = |input: ArrayRef| -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let lazy = input.apply(&ext_storage(root()))?; + let stepped = lazy.clone().execute::(&mut ctx)?; + Ok(format!("lazy: {} || one step: {}", tree(&lazy), tree(&stepped))) + }; + + report("R6 ext_storage(Extension(Dict))", || { + let input = ExtensionArray::new( + opaque(), + DictArray::try_new(codes(), values())?.into_array(), + ) + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Dict(codes, Extension(values)))", || { + let input = DictArray::try_new( + codes(), + ExtensionArray::new(opaque(), values()).into_array(), + )? + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Chunked[Extension, Extension])", || { + let chunk = || ExtensionArray::new(opaque(), values()).into_array(); + let input = ChunkedArray::try_new([chunk(), chunk()], DType::Extension(opaque()))? + .into_array(); + one_step(input) + }); + report("R6 ext_storage(Masked(Extension))", || { + let child = ExtensionArray::new(opaque(), buffer![10i64, 20, 10, 20].into_array()) + .into_array(); + let validity = + Validity::Array(BoolArray::from_iter([true, false, true, false]).into_array()); + one_step(MaskedArray::try_new(child, validity)?.into_array()) + }); + report("R6 ext_storage(Sparse with extension values)", || { + let input = Sparse::try_new( + buffer![1u64, 3].into_array(), + ExtensionArray::new(opaque(), values()).into_array(), + 4, + Scalar::extension_ref(opaque(), Scalar::from(0i64)), + )? + .into_array(); + one_step(input) + }); +} + +#[test] +fn r7_datetime_parts_compare() { + let ms = || ts(TimeUnit::Milliseconds, Nullability::NonNullable); + let ns = ts(TimeUnit::Nanoseconds, Nullability::NonNullable); + + // 1.6 s and 1.4 s, so `< 1.5 s` is [false, true]. + let column = || -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + let temporal = TemporalArray::new_timestamp( + buffer![1600i64, 1400].into_array(), + TimeUnit::Milliseconds, + None, + ); + Ok(DateTimeParts::try_from_temporal(temporal, &mut ctx)?.into_array()) + }; + let i64_const = || ConstantArray::new(Scalar::from(1500i64), 2).into_array(); + let ns_const = || { + ConstantArray::new( + Scalar::extension_ref(ns.clone(), Scalar::from(1_500_000_000i64)), + 2, + ) + .into_array() + }; + let ms_const = || { + ConstantArray::new(Scalar::extension_ref(ms(), Scalar::from(1500i64)), 2).into_array() + }; + + report("R7 column tree", || Ok(tree(&column()?))); + report("R7 array path: DateTimeParts(ms) < i64 1500", || { + array(&column()?.binary(i64_const(), Operator::Lt)?) + }); + report("R7 row path: DateTimeParts(ms) < i64 1500", || { + rows(&column()?.binary(i64_const(), Operator::Lt)?) + }); + report("R7 array path: DateTimeParts(ms) < Timestamp(ns) 1.5 s", || { + array(&column()?.binary(ns_const(), Operator::Lt)?) + }); + report("R7 row path: DateTimeParts(ms) < Timestamp(ns) 1.5 s", || { + rows(&column()?.binary(ns_const(), Operator::Lt)?) + }); + report("R7 array path (control): DateTimeParts(ms) < Timestamp(ms) 1500", || { + array(&column()?.binary(ms_const(), Operator::Lt)?) + }); +} + +#[test] +fn r8_compressor_shapes() { + let compressor = CascadingCompressor::new(vec![&IntDictScheme as &'static dyn Scheme]); + let compress = |input: ArrayRef| -> VortexResult { + let mut ctx = SESSION.create_execution_ctx(); + Ok(tree(&compressor.compress(&input, &mut ctx)?)) + }; + + report("R8 constant Timestamp(ms) column", || { + compress( + ExtensionArray::new( + ts(TimeUnit::Milliseconds, Nullability::NonNullable), + buffer![1000i64; 64].into_array(), + ) + .into_array(), + ) + }); + report("R8 low-cardinality Opaque column", || { + let values: Vec = (0..300).map(|i| [10i64, 20, 30][i % 3]).collect(); + compress( + ExtensionArray::new(opaque(), PrimitiveArray::from_iter(values).into_array()) + .into_array(), + ) + }); + report("R8 partly null constant Timestamp(ms) column", || { + let values = (0..64).map(|i| (i % 4 != 0).then_some(1000i64)); + compress( + ExtensionArray::new( + ts(TimeUnit::Milliseconds, Nullability::Nullable), + PrimitiveArray::from_option_iter(values).into_array(), + ) + .into_array(), + ) + }); +} + +#[test] +fn r10_f16_backcompat() { + let f16_dtype = DType::Primitive(PType::F16, Nullability::NonNullable); + let one = vortex_array::dtype::half::f16::from_f32(1.0); + let bits_form = || { + Scalar::try_new( + f16_dtype.clone(), + Some(ScalarValue::Primitive(PValue::U64(u64::from(one.to_bits())))), + ) + }; + let native_form = Scalar::primitive(one, Nullability::NonNullable); + + report("R10 try_new(F16, U64 bits of 1.0)", || { + Ok(bits_form()?.to_string()) + }); + report("R10 U64 form == F16 form", || Ok(bits_form()? == native_form)); + report("R10 F16 form == U64 form", || Ok(native_form == bits_form()?)); +} + +#[test] +fn r9_ext_storage_paths() { + let input = + ExtensionArray::new(non_zero(), buffer![0u64, 1].into_array()).into_array(); + + report("R9 lazy ext_storage(NonZero [0, 1]) tree", || { + Ok(tree(&input.clone().apply(&ext_storage(root()))?)) + }); + report("R9 array path: ext_storage(NonZero [0, 1])", || { + array(&input.clone().apply(&ext_storage(root()))?) + }); + report("R9 row path: ext_storage(NonZero [0, 1])", || { + rows(&input.clone().apply(&ext_storage(root()))?) + }); +}