Skip to content

FoR scheme bit-packs its child outside the allowed schemes #10751

Open

Description

@connortsui20

What happened?

FoRScheme::compress bit-packs its encoded child by calling BITPACKING_V1.compress directly (vortex-btrblocks/src/schemes/integer/for_.rs:220). The child skips the compressor's scheme selection, and FoRScheme::produced_encodings declares only fastlanes.for (and fastlanes.for.v2), not the bit-packing IDs that the child uses. The compressor keeps a scheme only when every ID that it declares is allowed (vortex-btrblocks/src/builder.rs, allowed_schemes), so this check misses the bit-packed child.

From reading the code, this has two effects:

  • If fastlanes.bitpacked is not allowed, or the bit-packing scheme is excluded with exclude_schemes, the FoR scheme still writes a bit-packed child. The writer then rejects the forbidden ID, and the write fails, although the compressor could have picked another scheme. With experimental patches enabled, the same applies to vortex.patched.
  • When fastlanes.bitpacked.v2 is allowed, the FoR child still uses the v1 bit-packing variant.

Expected: FoR declares the IDs that its child can write, or compresses the child through the selected schemes.

Environment

develop at 3974ee069c. Found by reading the code, not reproduced by a run.

Additional context

Found while updating the compression section of the versioning docs (#10747).

Activity

  1. connortsui20 commented on Oct 23, 2026

    @connortsui20
    MemberAuthor

    Talked offline with @mhk197, we likely need stateful schemes that have options, otherwise we cannot refine schemes correctly.

    So it seems like there is more work to be done here.

  2. mhk197 commented on Oct 23, 2026

    @mhk197
    Contributor

    @connortsui20 correction to this -- we should instantiate our schemes rather than having them be static for API design reasons rather than correctness. We can still refine schemes correctly within the status quo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugA bug issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions