[Feature](lance) Expand file TVF path and multi-dataset support

Parent

Tracked by #66340. The baseline s3() and local() Lance TVFs were introduced by #65730.

Scope

Expand glob patterns to multiple Lance datasets.
Read multiple Lance datasets in one TVF invocation.
Support additional file TVF entry points and broader path handling.
Preserve dataset boundaries, version pinning, schema compatibility, and secure path handling.

Completion criteria

FE split planning and BE execution support every selected dataset.
Tests cover glob expansion, mixed paths, multiple datasets, and incompatible schemas.
User documentation describes supported TVFs, paths, and limitations.
Representative local and object-storage paths are verified.

@yy782

Hi @yy782, thanks for offering to help with Lance support in #66340! Would you be interested in taking on this issue to expand Lance file TVF path handling and support reading multiple datasets, including glob expansion? Please let us know if this looks like a good fit for you.

Hi Gabriel, thanks for the invitation. I'd be happy to take on this issue.

Here's my rough plan: I'll first spend a few days thinking through and deciding on an approach, then post a design proposal in the issue outlining the goals and how I plan to break down the implementation, and ask you to review it. Once the approach is settled, I'll split it into small PRs and move forward step by step.

If this direction sounds good, I'll get started.

@yy782

Hi @yy782, that sounds like a great plan! Please go ahead and share your design proposal when it's ready—I'd be happy to review it. Breaking the implementation into small PRs works well. Thanks for taking this on!

Design Proposal — Lance file TVF path expansion and multi-dataset support

0. Goals and Phase-1 boundaries

0.1 Goals

Support glob wildcard expansion to multiple Lance datasets.
Read multiple datasets in a single TVF call (explicit multi-path / glob / mixed).
Support this on the s3() entry point, and automatically cover OSS / COS / TOS / GCS / Azure (whose URIs are normalized to s3://).

0.2 Explicitly out of scope (boundary shrink)

Multi-dataset local() is out of scope for this phase; single-dataset keeps the existing capability from baseline [feature](lance) support lance catalog in doris #65730 and stays untouched this phase. local() pins the BE that holds the file via backend_id: the FE never touches the local disk, it only produces one whole-dataset split (version=0 → latest) routed back to that BE, which opens it with local FS permissions. The FE therefore cannot enumerate multiple local datasets — multi-dataset would require an architecture change (BE-side enumeration or explicit multi-path, with more complex backend_id constraints), so it is out of scope for this phase (see §7 Q1).
hdfs() blocked upstream: lance-c does not support the hdfs backend yet; revisit once upstream is ready.

1. TVF surface and glob syntax

1.1 Supported entry points

TVF / backend	Status in this phase
`s3()`	Main landing point; multi-dataset Lance reading is supported here first
OSS / COS / TOS / GCS / Azure	URI is normalized to `s3://`, goes through the same function, covered automatically
`file()`	Pure delegation to the TVFs above, covered automatically
`local()`	excluded (not in Phase-1), see §0.2 / §7 Q1
`hdfs()`	Blocked upstream (lance-c does not support it yet)
`http()` / `http_stream()` / `cdc_stream()` / `group_commit()`	Do not support the Lance format, not involved

1.2 glob syntax scope

Syntax	Verdict	Basis
`*`	Supported	Both DuckDB and ClickHouse support it
`?`	Supported	Present in DuckDB's official glob table; also in ClickHouse's official list
`[]` (character class)	Supported	DuckDB's official glob table has `[abc]`, `[a-z]`
`{}` (enumeration)	Supported	ClickHouse documents `{abc,def}` for url parameters
`**` (recursive across levels)	Not supported	With level-by-level expansion (§1.3), every level costs one list call and the request count grows with the total number of directories traversed; dataset directories also have no uniform naming suffix, so they can only be determined one by one (§1.4) and cannot be pruned up front

The syntax comparison above all comes from the plain-file (Parquet / CSV) reading scenario of the two systems: neither ClickHouse nor DuckDB supports glob expansion for Lance.

1.3 Directory expansion

Expand level by level with fileSystem.listDirectories, then filter once and keep the paths that qualify.

1.4 Determining whether an expanded result is a dataset

The candidate directories produced by expansion are not all Lance datasets (the same level may be mixed with ordinary directories such as _tmp/, xxx_bak/), so each one has to be determined individually.

My current approach: just try Dataset.open() — if it opens, it is a valid dataset; if it fails, skip it.

Dataset.open()

This matches DuckDB's thinking: after expanding a glob, DuckDB also performs no format pre-check and simply opens the file — "whether it can be opened" is itself the determination.

There is one difference from DuckDB, however, and it affects the failure semantics in §3: DuckDB fails the whole query when an open fails, whereas here a candidate directory that cannot be opened is skipped. The reason is that expanded directories may legitimately be non-dataset neighbours, so "cannot open" is an expected, normal case and should not take down the entire query.

1.5 Expansion boundaries and failure handling

Zero matches: fail directly, but distinguish the two cases — "the directory listing produced nothing" and "things were listed but none is a dataset" — so the user can diagnose it themselves.
Count cap: cap the number of results from one expansion at 1000. The concern is a pattern like s3://bucket/* expanding into tens of thousands of datasets and dragging down the FE; exceeding the cap fails directly and asks the user to write a narrower glob.

2. Dataset boundaries and the multi-dataset model

2.1 Ways to specify multiple datasets

Means	Example	Scope of this phase
glob wildcard	`s3://bucket/datasets/*`	Supported
explicit multiple paths	`"uri" = "s3://bucket/a, s3://bucket/b"`	Supported (comma separation has an ambiguity risk, see §2.1.1)
mixed form	`"uri" = "s3://bucket/a, s3://bucket/20*"`	Supported (see §2.1.1)

2.1.1 Multi-path separation: ambiguity caused by S3 keys that may contain commas

S3 keys may contain commas, so "uri" = "s3://bucket/a, s3://bucket/b" is inherently ambiguous — it may be two datasets, or it may be a single key that contains a comma.

Both other systems deliberately avoid this form: DuckDB uses an array literal (read_parquet(['a.parquet','b.parquet']), internally a LIST(VARCHAR)), ClickHouse uses {} enumeration (§1.2). Neither is ambiguous.

But Doris TVF parameters are Map<String, String> string pairs with no array literal available, so my plan is:

Keep comma separation for now, but state explicitly in the documentation that paths containing commas are not supported, consistent with the existing glob syntax.

2.2 version pinning

A fixed version is required by snapshot semantics: all splits of one query must read the same version. The problem is that the latest versions of different datasets are simply not the same number — there is no way to derive a single common version.

So I lean towards keeping the (uri, version, fragments) triple separately per dataset: each dataset reads along its own version axis rather than forcing a merge.

2.3 Schema compatibility: merge rules

After expanding N datasets, one question cannot be avoided: how do N schemas become 1.

The basic convention I have settled on is union of columns, missing columns filled with NULL — this is exactly the semantics of DuckDB's union_by_name=true, quoted from the official docs:

The union_by_name option can be used to unify the schema of files that have different or missing columns. For files that do not have certain columns, NULL values are filled in

There is a prerequisite here: for a missing column to be filled with NULL, the merged column must be nullable. If some dataset declares that column NOT NULL, the merge has to relax it to nullable — otherwise "this dataset does not have the column at all" simply cannot be represented.

2.4 Conflicts that need a ruling

This section (including §2.4.1–§2.4.3) refers to DuckDB's and ClickHouse's multi-file merge for "plain files" — the rules they apply when reading plain files such as Parquet / JSON / CSV and combining several schemas into one.

Conflict	Concrete case	Handling
Column name case	`AGE` on one side, `age` on the other	Merge into one column (Doris column names are case-insensitive, both refer to the same column); the types of the two are then decided by the whitelist rules in §2.4.1, and anything outside the whitelist errors out
Different column types	`int32` vs `int64`, `int64` vs `uint64`, `int` vs `string`	Unify via named whitelist rules, error out outside the whitelist (ClickHouse style, see §2.4.1). For example `int32` vs `int64` can be unified; `int64` vs `uint64` and `int` vs `string` are not supported as column type conversions under Lance semantics, so they error out
Nullability	nullable on one side, non-nullable on the other	Merge to nullable (ClickHouse style, see §2.4.2). Nested types are pushed down recursively, but the `map` key is not relaxed
Different nested fields	`struct<a:int>` vs `struct<a:int,b:string>`	Union by field name: merge into `struct<a:int,b:string>`, with the missing sub-field filled with NULL; same-named sub-fields recursively look for a common type and only error out if none is found (see §2.4.3)

2.4.1 Type conflict resolution: I prefer the ClickHouse style over the DuckDB style

DuckDB's union_by_name is "pick the higher-scored / wider type, never throw" — int and string are silently merged into string, with integers converted to strings and no hint whatsoever. That suits us poorly: the merged type is eventually pushed down to the BE to materialize Arrow data (§4), so "picking a wider type" does not really solve anything; it only moves the failure from FE to BE and makes it harder to diagnose.

ClickHouse is named and controlled: it only unifies under a few whitelisted rules (promotion within the numeric family, integer↔float, and so on), and for nested types it recursively looks for a common type and gives up when none is found — it is not a general implicit cast. When unification is impossible it errors out directly.

So my plan for landing this:

Make the merge rules a whitelist of named rules, not a general "take the wider one": only allow a few cases such as promotion within the numeric family (int8/16/32 → int64, float → double); cross-family cases (int vs string, signed vs unsigned) always error out.
Include both sides of the conflict in the error message: which dataset, and the column name and type on each side, so the user can locate the offending dataset.
With this, the BE does not need a general type adaptation layer — incompatibilities are rejected at the FE, and the BE only needs to support the narrowing/widening materialization corresponding to the few whitelisted conversions (see §4).

One more note on column name case: ClickHouse treats age / AGE as two independent columns because its identifiers are case-sensitive. That route relies on the precondition of "case-sensitive identifiers", which Doris does not have (Doris column names are case-insensitive); copying it would produce ambiguous references, so I do not adopt it and instead merge into one column per Doris semantics. But the two cases must be distinguished: if within one single dataset there really are both age and AGE columns, then the data genuinely has two fields and merging would lose data — that should error out explicitly with an explanation.

2.4.2 Nullability merge: mixed means nullable (ClickHouse style)

When one side is nullable and the other is not, the merged result is relaxed to nullable.

The basis is ClickHouse's merge strategy: as soon as any Nullable appears among the types, all other types that can be wrapped in Nullable are relaxed as well. This is consistent with the basic convention in §2.3 ("union of columns, missing filled with NULL") — if dataset A declares a column NOT NULL while dataset B does not have that column at all, the merged column must be able to represent "the value does not exist".

Nested types are relaxed by the same recursive push-down rather than only at the top level; but one constraint has to be honoured along with it — a Map key can never be nullable (only the value is relaxed).

2.4.3 Nested type merge: union by field name

Example A: struct missing a field → union, missing sub-field filled with NULL

Suppose s3://bucket/db* expands into two datasets, db1 and db2.

db1's schema:

id:   int64
addr: struct<city: string, zip: string>

db2's schema:

id:   int64
addr: struct<city: string, country: string>

In addr, db1 has zip and db2 has country. After merging (union):

id:   int64
addr: struct<city: string, zip: string, country: string>

The rows actually read:

Source	id	addr.city	addr.zip	addr.country
row from db1	1	Beijing	100000	NULL
row from db2	2	Shanghai	NULL	CN

That is "missing sub-field filled with NULL" — the same thing as filling NULL for a missing top-level column.

Example B: two levels of nesting, recursing level by level

db1: addr: struct<geo: struct<lat: float>>
db2: addr: struct<geo: struct<lat: double, lng: double>>

Level 1 addr: both sides have geo → keep recursing.
Level 2 geo: lat exists on both sides → float vs double → take double; lng exists only in db2 → keep it as is.
Merged result: addr: struct<geo: struct<lat: double, lng: double>>

When reading db1, geo.lng is filled with NULL.

Two kinds of "inconsistency" are easy to confuse, and they are handled differently:

Case	Example	Result
Field name differs (the focus of this section)	`addr<zip>` vs `addr<country>`	Union, no error
Same field name but incompatible type	`addr<zip:int32>` vs `addr<zip:string>`	Error (recursive common-type search fails, decided by the whitelist in §2.4.1)

2.4.4 Multi-dataset constraints

Cross-provider / cross-credential multi-dataset queries are not supported in this phase.

3. Execution model (FE planning cost / failure semantics / memory)

3.1 FE planning cost

Every dataset needs one Dataset.open() for metadata loading (determination plus fetching version / fragments / schema, see §1.4). The planning cost grows linearly with N datasets, so allocator / memory usage has to be evaluated on the FE side and tied to the cap of 1000 in §1.5, so that planning does not drag the FE down.

I previously considered using Dataset.listManifestLocations to only list manifests and save a full open, but the lance-core:12.0.0 pinned by Doris does not have that method yet (it arrives in a newer version). So for this phase we stick to Dataset.open() and rely on the count cap as the safety net. If the lance dependency is upgraded later, this optimization can be revisited.

3.2 Shape of the scan tasks dispatched by the FE

Add a multi-dataset branch to the existing TVFScanNode#getLanceSplits:

doris/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java

Line 195 in 78aa9c9

private List<Split> getLanceSplits() throws UserException {

fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java
  → TVFScanNode#getLanceSplits()   (multi-dataset branch)

Roughly:

private List<Split> getLanceSplits() throws UserException {
        if (!sessionVariable.enableFileScannerV2) {
        throw new UserException("Lance TVF requires enable_file_scanner_v2=true");
        }
        if (tableValuedFunction.getTFileType() == TFileType.FILE_LOCAL) {
        // A local dataset is visible to its selected BE, not to FE. Keep exactly one
        // whole-dataset split; BE resolves version zero to latest when it opens the dataset.
        return Collections.singletonList(
                LanceSplit.scanLatestDataset(tableValuedFunction.getFilePath()));
        }

        List<LanceTableMetadata> datasets = tableValuedFunction.getLanceDatasets();// newly added API, returns multiple datasets
        if (datasets.isEmpty()) {
        long version = tableValuedFunction.getLanceDatasetVersion();
        if (version <= 0) {
                throw new UserException(
                        "S3 Lance TVF metadata was not initialized with a fixed dataset version");
        }
        return LanceSplitBuilder.buildFragmentSplits(tableValuedFunction.getFilePath(), version,
                tableValuedFunction.getLanceFragments(), 1);
        }
        List<Split> splits = Lists.newArrayList();
        for (LanceTableMetadata dataset : datasets) {
        splits.addAll(LanceSplitBuilder.buildFragmentSplits(
                dataset.getDatasetUri(), dataset.getVersion(), dataset.getFragments(), 1));
        }
        return splits;
}

4. BE-side relaxation (execution side of schema compatibility)

doris/be/src/format_v2/lance/lance_record_batch_converter.cpp

Line 124 in 78aa9c9

    
           Status LanceRecordBatchConverter::_bind_schema(const std::shared_ptr<arrow::Schema>& schema) {

be/src/format_v2/lance/lance_record_batch_converter.cpp
  → LanceRecordBatchConverter::_bind_schema(...)

It currently requires strict two-way agreement between the columns returned by lance-c and the FE projected columns: one column short → Lance did not return requested column; one column extra → Lance returned unknown column.

With a single dataset the two are naturally identical and this never causes trouble, but after schema merging the FE pushes down a union, and when reading any one dataset it necessarily lacks part of that union. So it has to be relaxed to: columns absent from the projection are filled with NULL rather than raising an error.

Nested fields need attention here: nested struct also takes a union of fields (§2.4.3), so "fill missing with NULL" must recurse into nested fields and not only handle top-level columns — if a dataset's struct is missing a sub-field, that sub-field must be filled with NULL too. Correspondingly, the type merge side must also recursively find a common type for same-named sub-fields (both DuckDB and ClickHouse merge nested types recursively, see §2.4.3). This means the BE-side NULL filling and type unification both have to support nesting, which is more work than "top-level columns only" and should be evaluated together.

5. Security and path handling

Prefix constraint: glob expansion results must still fall inside the user's authorized path prefix, preventing ../ or an out-of-range prefix from escaping to unauthorized datasets.
local constraint: local() multi-dataset involves backend_id pinning and local permissions; it is not extended in this phase and needs to be documented as unsupported for local() (see §0.2).

6. Testing

Scope 1 (glob): expansion of */?/[]/{}; ** not supported; non-dataset directories skipped; both error wordings for "zero matches" and "all failed"; error when the cap of 1000 is exceeded.
Scope 2 (multi-dataset): explicit multi-path / glob / mixed; schema union with missing columns filled with NULL; case merging; unification within the type whitelist and errors outside it (the error carries dataset + column information); nullability relaxation; nested recursive union / NULL filling; error when same-named sub-field types are incompatible
Scope 3 (entry points): s3() as the main target; OSS/COS/TOS/GCS/Azure normalized to s3:// and covered automatically; file() delegation; local() single dataset keeps the existing behavior; hdfs() unsupported.

7. Open Questions for Reviewers

Q1: Should local() be included in this phase, or strictly excluded from Phase-1? (see §0.2)
Q2: Are the two error wordings for "zero matches" diagnostic enough? (see §1.5)

Following up on the design proposal, I suggest narrowing Phase 1 to multiple datasets with identical schemas, with any Dataset.open() failure failing the query. Schema union can be implemented in a subsequent PR.

The following are design risks and proposed requirements, rather than claims that every item is an existing implementation bug.

Fail the query on any dataset-open failure. An open failure may mean a permission error, timeout, corrupt metadata, or unsupported format; skipping it can silently return incomplete results. Fail for both explicit paths and glob matches, including ordinary directories matched by a glob. Report the failing path, operation, and underlying cause, without exposing credentials. Distinguish “no paths matched” from “a matched path could not be opened.” Update the proposal and tests to remove the skip-on-failure behavior.
Define strict schema equality for Phase 1. Validate all datasets in FE before dispatching scans. Compare the logical source schema recursively: exact field names/case, order, types and type parameters, nullability, and nested structure. Include decimal precision/scale, timestamp unit/timezone, and fixed-size list dimensions where applicable; comparing only converted Doris types can hide source differences. Dataset-local internal field IDs need not be equal. On mismatch, report both datasets, the field path, and the conflicting definitions. Preserve the existing strict BE converter checks; do not add NULL filling, type promotion, case merging, or nested schema union in this PR.
Audit dataset-specific state and cache identities. Identical logical schemas do not imply identical field IDs, fragment IDs, or index state. Keep each fragment associated with its dataset and version. Check whether metadata, reader, runtime-filter, and index-related caches are safe to share; state tied to a snapshot must distinguish dataset URI/version and any other relevant context. Verify how pushed expressions bind fields rather than assuming they are interchangeable. If a reader is reused across datasets, reset dataset-specific scanner/schema/binding state. This is an implementation audit requirement, not an assertion that the current caches are incorrect.
Bound planning resources before materializing all results. The proposed 1000-dataset cap is not a memory bound. In the referenced baseline, S3ObjStorage.listDirectories() traverses all pages and accumulates directory entries before returning. A post-listing cap is too late for a large prefix. Apply incremental limits to candidate enumeration, listing requests, and elapsed time. Apply the dataset cap across the entire TVF invocation, not separately to each input glob. Also budget total fragments/splits, plan size, and schema size: even one dataset can have many fragments. Bound metadata-open concurrency and account for heap, off-heap, and native allocations. Define precisely what each limit counts.
Make cancellation and resource cleanup part of the execution model. Listing, metadata opening, and retries should respect the query deadline/cancellation, with finite request timeouts. On one failure, cancel remaining work and close already-opened handles/allocators. Cover success, failure, timeout, and cancellation paths. Avoid a thread pool per dataset or an unbounded task queue; bound concurrent metadata work and release its resources promptly.
Define deduplication for overlapping inputs. For example, an explicit dataset URI and a glob may identify the same dataset. I suggest dataset-set semantics: safely normalize identities, deduplicate before opening/pinning versions, and read each dataset once. Do not deduplicate records or distinct datasets with identical contents. Use deterministic traversal order so reference-schema selection and diagnostics do not depend on listing order.
Use an unambiguous multi-path representation. Comma splitting conflicts with both object keys and brace patterns such as s3://example-bucket/{a,b}. Consider preserving the existing single-path uri behavior and adding a uris parameter containing a JSON array string; Map<String, String> does not prevent this representation. Define how uri and uris interact, parse glob syntax separately for each element, and specify literal wildcard escaping and brace-enumeration scope. Reject unsupported ** and malformed patterns explicitly. Document path-level and trailing-slash behavior without breaking existing single-path inputs.
Pin one coherent metadata snapshot per dataset. Schema, version, and fragments must come from the same snapshot, rather than separate opens of latest. Resolve the dataset set once per query and reuse the resolved identities/versions for execution retries. If a pinned version is removed or becomes unreadable, fail rather than falling back to latest. Document that per-dataset snapshot consistency does not provide a transactionally consistent snapshot across datasets.
Align path parsing, authorization, and actual access. Listing, deduplication, authorization checks, and Lance open must agree on the object identity. Define the source of the authorized prefix and check scheme, bucket, and path-segment boundaries rather than a raw string prefix. Do not apply local-filesystem ../ normalization to object keys if it changes the addressed object. Keep credentials and signed URL secrets out of diagnostics.
Validate each storage provider instead of assuming automatic coverage. Normalizing a URI to s3:// does not establish that FE listing and native Lance open agree on endpoint, authentication, addressing, and pagination. Preserve the provider information needed to build storage options. Claim support only for combinations validated through the full listing → metadata open → BE scan path. Reject unsupported cross-provider/cross-credential combinations during planning.
Preserve existing single-dataset behavior. Keep single-path parsing and routing compatible, retain the existing single-dataset local() path, and explicitly reject multi-dataset local() in this phase. Cover s3(), file() delegation, empty datasets/no fragments, COUNT(*), LIMIT, and cancellation. With strict schema equality, no relaxation of the existing BE column checks should be necessary for schema union.

Suggested minimum acceptance coverage:

Same-schema datasets produce the same results as separate scans combined with UNION ALL, after input dataset deduplication.
Any open failure fails the query and identifies the path; no candidate is silently skipped.
Schema mismatches fail in FE with precise differences, including nested/type-parameter differences.
Overlapping explicit/glob inputs read each dataset once; parsing tests cover commas, braces, escaping, and invalid patterns.
Large listings with few matches still hit enumeration budgets; few datasets with many fragments still hit planning budgets.
Cancellation and partial metadata-open failure release resources and stop outstanding work.
Concurrent dataset updates preserve pinned-version reads; deleted pinned versions fail explicitly.
Results remain equivalent with applicable filter pushdown enabled/disabled and with runtime filters, including datasets with different internal field IDs or index layouts.
Existing single-dataset and local() behavior remains compatible; each claimed provider passes the end-to-end path.

This keeps the first implementation focused on path expansion, deduplication, strict schema validation, per-dataset version pinning, and bounded scan planning. NULL filling, type promotion, case reconciliation, and nested schema evolution can be designed and tested separately in the schema-union PR.

Revised Design — Lance File TVF Path Expansion and Multi-Dataset Support (Phase 1)

Responding to the review comments from Gabriel39 in #66492.

Thanks for the thorough review. The comments go well beyond what I had considered — failure semantics, planning budgets, cancellation and cleanup, cache identities, and the multi-path representation in particular — and they materially tightened this design. Several items below exist only because you raised them. The revision follows your comments point by point; where I have taken a different route, the reasoning is stated inline so you can push back on it.

1. Phase 1 Scope

In scope:

Read multiple Lance datasets in a single TVF invocation: glob expansion, an explicit path list, or a mix of both. Limited to datasets whose schemas are exactly identical.
Require strict schema equality validation across all datasets.
One self-consistent metadata snapshot per dataset; the version is pinned per dataset.
Preserve the existing single-dataset behavior.
Phase 1 lands only on s3://, and claims support only for combinations that have been verified end to end. At present that is MinIO, as demonstrated by test_lance_s3_tvf.groovy，

doris/regression-test/suites/external_table_p0/lance/test_lance_s3_tvf.groovy

Line 27 in 3d78b6b

String lanceTvf = """

. OSS / COS / TOS / GCS / Azure, and any other S3-compatible endpoint that has not been verified, are not claimed as supported.
One provider per invocation. Reject multi-provider / cross-credential combinations: multi-dataset mode accepts a single provider and a single credential set. Preserve the provider information needed to build storage options through both expansion and open.

2. Path Input: `uri` and `uris`

uri keeps its current single-path semantics, so the behavior of existing single-dataset input is unchanged.
Multiple paths are expressed through uris, not through comma separation.
uri and uris are mutually exclusive. Supplying both is a parse error, reported with an explicit message naming both keys.
Each element of uris is parsed independently under the glob grammar. Literal elements and pattern elements may coexist in the same array.
Multi-path and glob take effect only on uris; uri retains single-dataset semantics.
The element count of uris is itself bounded. An array of ten thousand elements must be rejected at parse time, not discovered later during expansion.

-- Explicit list
SELECT * FROM s3(
  "uris" = '["s3://bucket/a", "s3://bucket/b"]',
  "format" = "lance");

-- Literal and glob mixed
SELECT * FROM s3(
  "uris" = '["s3://bucket/b", "s3://bucket/20*"]',
  "format" = "lance");

Report an explicit parse error for each of the following: malformed JSON, a non-string element, an empty element, an empty array, an invalid element scheme, an element ending in /, and an invalid pattern.

3. Glob Grammar and Expansion

Supported: *, ?, [], {}.
Not supported: ** (cross-level recursion). ** is rejected with an explicit error rather than being silently treated as *.

Per-element semantics:

A pattern is split into a root (scheme://authority) and a sequence of path segments.
A literal segment performs no storage access; it is appended to every prefix already reached.
A wildcard segment produces one bounded directory listing (§4) per parent prefix already reached.
Expansion proceeds breadth first, level by level.

Escaping and brace scope:

A backslash escapes the next character, so \* and \? are literals. This is the spelling to use when a path genuinely contains a wildcard character.
{} applies only within a single path segment, enumerating alternatives for that segment: in s3://bucket/{a,b}/ds, the second segment is expanded. A {...} that spans / is an invalid pattern, not a cross-segment union.
Nesting inside {} and [] is not supported and is rejected as an invalid pattern in Phase 1.
An unclosed [ or {, and an empty [], are invalid patterns, rejected at parse time before any listing is issued.
[] supports character sets, ranges, negated ranges, and negated sets. It does not support sequences, steps, or nesting. {} supports only enumerating alternatives; no nesting, sequences, or steps.

Trailing slash: elements of uris may not end with /. This is rejected at parse time with a message asking for the trailing slash to be removed, so that a given directory has exactly one spelling. The existing behavior of uri is unaffected.

4. Bounded Enumeration

(a) A bounded, pageable directory-listing primitive. With delimiter="/", maxKeys, and a continuation token, it returns one page per call so the caller can stop midway. Budgets below are checked as each page arrives, so enumeration can be aborted in the middle.

(b) Incremental budgets, counted across the entire TVF invocation.

Budget	What it counts	Default
Directory candidates	Total directory entries listed, counted before deciding whether each is a dataset	1 000
Listing requests	Individual listing calls issued, summed across all storage backends, cumulative for the invocation	1 000
Listing time	Wall-clock time spent listing, cumulative for the invocation	One third of the deadline (20s by default)
Datasets	Datasets actually read, after deduplication	500
Fragments / splits	Sum of fragments / splits over all datasets	100 000
Fields per schema	Fields in one dataset's schema	1 000
Total planned fields	Sum of fields over all datasets in the plan	No separate cap; bounded by the dataset cap and the per-schema field cap
Concurrent opens	Dataset handles open at the same time	4

The default value of each budget needs to be confirmed in the PR.

Memory bound. Planning-time memory falls into two parts, each with its own constraint:

Retained after loading: the version / schema / fragments of each dataset stay in the plan until the query ends, and grow with the dataset count and the fragment count. They are constrained jointly by the dataset cap, the per-schema field cap, and the fragments/splits cap.
Transient cost while open: the off-heap and JNI-side cost a dataset occupies while open is released on close and does not accumulate with the dataset count. It is constrained by concurrent opens, so the peak is fixed and does not grow with the dataset count.

5. Dataset Identity, Deduplication, and Determinism

Dataset-set semantics. An explicit URI and a glob may identify the same dataset, and a dataset must be read only once; deduplicate with a set. Deduplication happens after expansion and normalization and before opening and version pinning, so no dataset is opened twice and two different pinned versions of the same dataset cannot both enter the plan.

Identity is the only deduplication key. Deduplicate by normalized dataset identity only, independent of data content: two datasets with identical contents but different URIs remain two datasets, each read once, with the rows doubled. Results are not deduplicated by row.

Normalization used for identity, in this order:

The bucket is preserved verbatim; no case folding.
The path is split on / and compared verbatim; no folding of any kind. A repeated slash is a valid character in an object key — a//b and a/b are two distinct keys — so // is rejected explicitly and never folded into a single /. Folding would change the object being addressed.
.. and . in path segments are treated as literals and are not folded: a/../b and b are two distinct keys in object storage.

Deterministic order. Candidates returned by listing are sorted before use. The sort result determines both the traversal order and the selection of the reference schema (§8).

6. Version Pinning per Dataset

One open takes version, schema, and fragments from the same handle. No common version is synthesized; each dataset keeps its own (uri, version, fragments).

The dataset set is resolved during TVF construction and is held by an object that is not rebuilt within a plan.

Therefore, within one analyzed plan — including any execution retry — the dataset set, versions, and fragments all come from that single resolved result; latest is not resolved a second time.

Therefore, if a pinned version is deleted or becomes unreadable, that is an open failure: the query fails (§7) with an explicit error, and there is no fallback to latest.

7. Any Open Failure Fails the Query

Regardless of explicit multi-path, mixed input, or glob matching: if any candidate — including a path finally determined not to be a dataset — cannot be opened, the whole query fails.

Each failure reports:

the failing path;
the failing operation (listing, dataset open, …);
no sensitive information: credentials, signed-URL secrets, and storage-option values must never appear in the message.

Two zero-result cases are reported separately:

No path matched: enumeration produced zero candidates. The message states that the pattern matched nothing and reports the pattern itself together with the number of listing requests issued.
Matched but could not be opened: candidates were produced and all failed. The message states how many candidates were found and reports the first failure in full.

8. Strict Schema Equality

Validation is performed entirely in FE, before any scan task is dispatched.

The comparison is recursive and requires exact equality of:

field names;
types and type parameters: decimal precision and scale, timestamp unit and timezone, fixed-size list dimension, and comparable attributes;
nullability;
nested structure, compared recursively;
field order, which is part of the equality definition. Any pushdown later placed on the multi-dataset path must be designed against this premise and must not assume that fields of different datasets are interchangeable.
Lance internal field IDs need not be equal.

Reference schema: the first dataset under the deterministic order (§5); all others are compared against it. The TVF output columns continue to be derived from a single schema, exactly as today.

9. Dataset State and Cache Audit

This section is an audit. It does not assert that the existing caches are defective.

BE side — changes required in Phase 1

Today LanceTableReader keeps only one open dataset and errors when the dataset key changes, while a single scanner processes several scan ranges and reuses the same reader.

Chosen approach: when the dataset key (uri + version + storage options) changes because of a new split, close and reset the dataset-scoped state instead of erroring. This covers the opened dataset handle and the recorded dataset key, the Arrow schema bound in the record-batch converter, the cached dataset schema, and the full-text search query context. Every split carries the identity of its own dataset, so which dataset to open is explicitly determined and requires no implicit context.
Profile accounting: the two data-cache byte counters take the absolute snapshot of the current dataset handle and replace rather than accumulate. Accumulate into a local running total before switching datasets and write the accumulated value back; otherwise only the numbers of the last closed dataset survive.
Pushed expression binding: that cache implicitly assumes the schema is invariant within one scan local state and that the query conditions are unique. No change is needed in this phase; ambiguity arises only once multiple schemas are merged.

Shared session cache. The BE holds one process-global Lance session owning the metadata and index caches. Both tiers are a "single global pool plus key prefix" structure: the metadata cache is prefixed by dataset URI and carries the version in the entry key, and the index cache adds the index identity on top of it. Cross-dataset and cross-version false hits are therefore already ruled out, and Phase 1 does not modify this cache.

10. Cancellation and Resource Cleanup

A planning-time timeout already exists, but its checkpoints do not cover the scan-node finalize segment, which is exactly where listing and open happen.

The planning path therefore has to compute its own deadline and place its own checkpoints.

Deadline. min(query_timeout, 60s), consistent with the existing Lance metadata read. It is not disabled together with the planning-timeout switch.
Per-operation timeout. A deadline only helps between checkpoints; a real stall happens inside one listing or one open, where no checkpoint ever runs. In addition to the overall deadline, each listing and each open therefore needs a timeout of its own.
Checkpoints. Before each listing, between pages of one listing, before each open, and inside the submitted tasks — not only at the points where results are awaited.
Concurrency. One shared bounded thread pool: fixed thread count, bounded queue, and rejection with an error rather than an unbounded backlog or blocking the submitting thread. No thread pool per dataset and no unbounded task queue.
Failure handling. On the first failure: stop issuing new listing requests and new opens, let in-flight work finish but start nothing new, then close every handle and allocator already created. Tasks already inside a native call are not interrupted; wait for them to return.
Paths to cover: success, listing failure, open failure, timeout.

11. Path Parsing and Identity Consistency

Listing, deduplication, and the Lance open all operate on the same normalized URI string.
Do not fold .. and . for object keys: doing so changes the object that is addressed (§5).

12. Preserving Existing Single-Dataset Behavior

uri parsing and routing are unchanged, so uri semantics stay exactly as they are today.
local() single dataset is unchanged, and multi-dataset queries are explicitly rejected with a clear error.
file() delegation is unchanged: it forwards to the same TVF, so it gains new behavior only where the underlying TVF supports it.
The following must keep working and are covered by the test plan (§13): empty datasets and datasets with zero fragments, COUNT(*), LIMIT, cancellation, file() delegation, and the existing single-dataset s3() and local() paths.

13. Test Plan

Acceptance item	Cases covered
Multi-dataset results are equivalent to `UNION ALL`	N datasets read in one call, compared row by row against the `UNION ALL` of N single-dataset scans
Overlapping datasets are read once	An explicit path and a glob identifying the same dataset
Query behavior when a dataset fails to open	Permission error, timeout, corrupt manifest, non-dataset directory; covered separately for explicit paths and glob matches
Schema mismatch is rejected	Swapped field order, different name case, different decimal precision and scale, different timestamp unit / timezone, and comparable differences
Path parsing and glob grammar	Keys containing commas, `{a,b}` spanning segments, `\{` and `\`, unclosed `[` and `{`, empty `[]`, `*`, `//`, trailing `/`, malformed JSON, `uri` and `uris` used together
Enumeration and planning budgets	A large listing with few matches; few datasets with very many fragments
Cleanup on cancellation and partial failure	Other datasets still queued when one fails; timeout; cancellation in the middle of listing
No regression in existing single-dataset behavior	Single-dataset `s3()` and `local()`, `file()` delegation, empty dataset, zero fragments, `COUNT(*)`, `LIMIT`, cancellation, multi-dataset `local()`
Traversal order and the two zero-result cases	Order consistency across runs; the "no path matched" and "matched but could not be opened" errors

@Gabriel39

Hi @Gabriel39, gentle ping on the design proposal — it has been up for a few days now, and I'd appreciate a review whenever you have time.

@yy782

Hi @yy782, thanks for revising the design. I have no further concerns with the updated proposal. Please go ahead with implementation and split the work into small PRs as planned.

[feature](lance) Support multi-path uris parsing in Lance TVF

[Feature](lance) Expand file TVF path and multi-dataset support #66492

Description