[ilya-zlobintsev]

For columns with many repeating values, lance uses dictionary encoding, storing a dictionary per page within each fragment.

Previously, if you did a simple matching filter like id = "a" or id in ("a", "b"), it would go through the whole read path of materializing the dictionary into an array, only to then hash the values of the array to compare them with the filter.

This PR adds a pruning optimization where it checks the dictionaries right away to exclude pages which are known to not have the filtered values, and only does the whole decode/materialization step otherwise. This is only applied for string/binary columns with dictionary encoding, anything else falls back to the normal path. Fragments with a pre 2.1 format, or ones with overlays, are also skipped.

[Copilot]

🟡 Changes recommended

Sequential metadata loading introduces a cold-scan startup regression for multi-column filters.

1 open issue

• Avoid serializing metadata loads and initializing non-prunable columns · New

What changed in this PR

Adds dictionary-based page pruning to Lance’s filtered scans, avoiding full decoding when pages cannot match string or binary equality filters.

Changes:

• Propagates dictionary checks through decoder, file, and fragment readers.
• Applies pruning before filtered reads, with legacy and overlay fallbacks.
• Adds coverage for matching values, nulls, caching, and unsupported cases.

File	Description
rust/​lance/​src/​io/​exec/​filtered_read.rs	Applies pruning to scan ranges.
rust/​lance/​src/​dataset/​fragment.rs	Exposes overlay-aware fragment checks.
rust/​lance-file/​src/​reader.rs	Connects file metadata to pruning.
rust/​lance-encoding/​src/​encodings/​logical/​primitive.rs	Checks dictionaries and builds retained ranges.
rust/​lance-encoding/​src/​decoder.rs	Coordinates supported-column pruning.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Malformed dictionary offsets can panic during pruning instead of following normal corruption-error handling.

2 open issues

• Validate dictionary offsets before slicing to prevent pruning panics · New
• Avoid serializing metadata loads and initializing non-prunable columns

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-layer pruning affects query-result correctness and needs runtime validation and human review.

1 open issue

• Avoid serializing metadata loads and initializing non-prunable columns

1 resolved since last review

• Validate dictionary offsets before slicing to prevent pruning panics

_{Review effort: Balanced}

[amunra]

Claude says that hashing here might be needlessly expensive and suggests structural equality instead and provided an example impl:

---

/// Above this many sought values, hashing each dictionary entry once beats comparing it
/// against every value.
const LINEAR_SCAN_MAX_VALUES: usize = 8;

/// The values an equality filter looks for, in the form that is fastest to probe.
pub(crate) enum SoughtValues<'a> {
    /// Slice equality compares lengths first, so a mismatched entry costs one compare
    /// and no hashing.
    Few(&'a [&'a [u8]]),
    Many(HashSet<&'a [u8]>),
}

impl<'a> SoughtValues<'a> {
    pub(crate) fn new(values: &'a [&'a [u8]]) -> Self {
        if values.len() <= LINEAR_SCAN_MAX_VALUES {
            Self::Few(values)
        } else {
            Self::Many(values.iter().copied().collect())
        }
    }

    fn contains(&self, entry: &[u8]) -> bool {
        match self {
            Self::Few(values) => values.iter().any(|value| *value == entry),
            Self::Many(values) => values.contains(entry),
        }
    }
}

fn variable_width_contains_any<O: ArrowNativeType>(
    data: &[u8],
    offsets: &[O],
    values: &SoughtValues<'_>,
) -> bool {
    offsets.windows(2).any(|bounds| {
        data.get(bounds[0].as_usize()..bounds[1].as_usize())
            .is_none_or(|entry| values.contains(entry))
    })
}

What changes around it:

• StructuralPageScheduler::may_contain_any takes &SoughtValues<'_> instead of &HashSet<&[u8]>. Eight implementors, all mechanical, and the seven that ignore the argument do not care about its type.
• CoreFieldDecoderStrategy::rows_that_may_contain_any at decoder.rs:654 builds SoughtValues::new(values) where it builds the set today. For the common case that is now zero allocations per call. For a long
  IN list it is the same HashSet as before.
• Behaviour is identical, including the empty-set case, which still prunes every dictionary page, and the corrupt-offset case, which still answers "may contain".

Expected effect on a 100,000-entry page with a single-literal miss: from roughly one SipHash per entry, a few milliseconds, to one length compare per entry, well under a millisecond. Honest framing: that is
small against the tens of milliseconds a pruned page saves in decode, and most dictionary pages are far smaller than the cap. It is worth doing because it removes the only per-entry loop the PR adds and
costs nothing in complexity, not because it will show up in a benchmark.

One optional extra while in there: reading the offsets with chunks_exact(8) and u64::from_le_bytes instead of borrow_to_typed_slice:: removes the misaligned-copy branch for LargeUtf8 entirely. Separate
change, same function.

✻ Cooked for 1m 4s · done 12:27 PM

[ilya-zlobintsev]

Initially I wanted to be smarter about this and move out the HashSet construction to happen once per scan, so this cost isn't an issue in the first place, but that requires more logic on the planning stage and a lot more plumbing to get it in. Maybe upstream will point me in the right direction once I submit the PR there.

So In the end, I implemented this suggestion 🙂

[amunra]

Ditto, I think this should not be a default trait. There's only three impls anyhow (v2_adapter::Reader, V1Reader and NullReader).

• The v2 reader logic exists already
• The V1 reader should return futures::future::ready(Ok(None)).boxed()
• And arguably the null reader should return Some(vec![]).

[amunra]

Possibly this or a follow-up PR

I think there's a more elegant and more generic way to express this PR. One that would help with additional query types, rather than just exact matches.

The current impl is tied to one operator and if we wanted to support BETWEEN or <, the approach would bloat a lot of methods.
Instead the logic in rust/lance/src/io/exec/filtered_read.rs that deals with evaluating the predicate could just ask the page what values it knows about.

pub enum PageValues<'a> {
    /// Every non-null value on the page is in this array. No nulls in the array.
    /// Dictionary entries, or the single scalar of a constant page.
    Exactly(&'a dyn Array),
    /// The page has no non-null values.
    Empty,
    /// Nothing is known. The page may hold anything.
    Unknown,
}

trait StructuralPageScheduler {
    fn known_values(&self) -> PageValues<'_>;
}

This would expose things at the lowest level.

Then a generalisation of rows_that_may_contain_any could express the match:

pub trait ValuePredicate: Send + Sync {
    fn any_may_match(&self, values: &dyn Array) -> Result<bool>;
}

Then, glue with DataFusion:

use datafusion_physical_expr::PhysicalExpr;

struct PhysicalExprColumnPredicate {
    expr: Arc<dyn PhysicalExpr>,
    schema: SchemaRef,
}

impl ValuePredicate for PhysicalExprColumnPredicate {
    fn any_may_match(&self, values: &dyn Array) -> Result<bool> {
        // RecordBatch wants an owned ArrayRef. to_data() is a shallow clone, the
        // buffers are refcounted, so this is one small allocation and no copy.
        let values = make_array(values.to_data());
        let batch = RecordBatch::try_new(self.schema.clone(), vec![values])?;
        let mask = self.expr.evaluate(&batch)?.into_array(batch.num_rows())?;
        Ok(mask.as_boolean().true_count() > 0)
    }
}