Skip to content

Add ReturnFieldsRetriever: concurrent scatter/gather utility for per-hit return fields - #16572

Open
zihanx wants to merge 4 commits into
apache:mainfrom
zihanx:return-fields-retriever
Open

Add ReturnFieldsRetriever: concurrent scatter/gather utility for per-hit return fields#16572
zihanx wants to merge 4 commits into
apache:mainfrom
zihanx:return-fields-retriever

Conversation

@zihanx

@zihanx zihanx commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a reusable utility for fetching "return fields" — one value per hit for a set of
global doc IDs — concurrently across leaves, returning results 1:1 in the caller's input
order. This is the follow-up to the ReaderUtil.partitionByLeaf scatter/gather API
#16496; it builds the retriever the ordinal tracking was designed for.

Motivation

Fetching field values for a page of hits (in ranking order) is a common need, and getting
the leaf partitioning + concurrency + input-order reassembly right is tricky and easy to
get subtly wrong. Today callers hand-roll this. This packages it once.

What's included

Two classes:

  • ReturnFieldsRetriever — the generic engine. Owns only the reusable machinery:
    scatter doc IDs to leaves via ReaderUtil.partitionByLeaf, run each leaf as an
    independent task on a supplied Executor (via TaskExecutor), and gather per-leaf
    results back into input order. It is agnostic about what is produced per hit: the
    caller supplies a LeafVisitorFactory<T> that turns a document into a value of type T.

    public static <T> T[] retrieve(
        IndexReader reader, int[] globalDocIds,
        LeafVisitorFactory<T> factory, IntFunction<T[]> arrayFactory, Executor executor)
  • DoubleValuesRetriever — a thin DoubleValuesSource-specific specialization built
    on the engine. Adapts each source to a LeafVisitor<double[]>:

    public static double[][] retrieve(
        IndexReader reader, int[] globalDocIds, DoubleValuesSource[] sources, Executor executor)

Design notes

  • Concurrency contract: leaves run concurrently, so newLeafVisitor is called once per
    leaf and must return a fresh, single-threaded visitor. Each leaf writes disjoint slots of the result array,
    so no synchronization is needed.
  • Input order preserved: result[i] is the value for globalDocIds[i]. The input
    array is not mutated.

Zihan Xu added 2 commits August 27, 2026 15:13
…-hit return fields, plus a DoubleValuesSource specialization (DoubleValuesRetriever).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant