Skip to content

feat(namespace): support Lance Namespace across table operations#35

Open
FANNG1 wants to merge 14 commits into
daft-engine:mainfrom
FANNG1:namespace
Open

feat(namespace): support Lance Namespace across table operations#35
FANNG1 wants to merge 14 commits into
daft-engine:mainfrom
FANNG1:namespace

Conversation

@FANNG1

@FANNG1 FANNG1 commented Jun 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #50 (motivating use case: Gravitino Lance REST server managed tables + Daft multimodal processing).

Summary

  • add a shared Lance namespace resolver for URI-or-namespace validation, namespace client construction, table resolution, and storage option merging
  • wire namespace table support through daft_lance.read_lance, the new daft_lance.write_lance, LanceDataSink, and worker scan reconstruction
  • add dir namespace coverage plus a gated Gravitino Lance REST namespace integration test
  • tighten untyped namespace table-not-found detection to fail closed unless the error is clearly a 404 table miss
  • scope the Lance native teardown workaround to marked namespace tests instead of every lancedb test

Validation

  • uv run pytest tests/io/lancedb/test_namespace.py -q
  • uv run pytest tests/io/lancedb/test_blob_v2_policy.py -q
  • env DAFT_RUNNER=ray uv run --with 'ray[data,client]==2.55.1' python -m pytest tests/io/lancedb/test_namespace.py -q
  • git diff --check origin/main...HEAD
  • uv run python -m compileall -q daft_lance

Cross-repo plan

daft_lance.write_lance in this PR looks like a copy of DataFrame.write_lance, but the
direction is the opposite: Eventual-Inc/Daft#6957 already migrated all of daft.io.lance to
this package, and DataFrame.write_lance's inline body is the last piece left behind
upstream. Eventual-Inc/Daft#7282 proposes reducing it to a thin delegation to
daft_lance.write_lance, making this module the single canonical implementation.

Until Eventual-Inc/Daft#7282 lands, namespace users call daft_lance.read_lance /
daft_lance.write_lance directly. (An earlier revision of this PR shipped a patch_daft()
monkeypatch as interim glue for Daft's native entry points; it was removed in favor of the
direct entry points since it had never been released.)

@FANNG1 FANNG1 marked this pull request as draft June 19, 2026 12:17
This was referenced Jul 16, 2026
Open
Open
@FANNG1 FANNG1 changed the title Add Lance namespace read/write support feat(namespace): support Lance Namespace across table operations Jul 16, 2026
fanng added 13 commits July 16, 2026 19:02
feat: add lance namespace read write support
403e4b6
feat(namespace): extend Lance Namespace support and harden implementa…
aa34c09 
…tion

Build on the initial namespace read/write support:

- Extend namespace params (namespace_impl/namespace_properties/table_id)
  to merge_columns, merge_columns_df, create_scalar_index, compact_files,
  threading namespace kwargs into their commit sites.
- Add an explicit daft_lance.write_lance() and make the daft monkeypatch
  opt-in via patch_daft(); wire write_lance(mode="merge") through namespace.
- Fix _declare_table fallback that always raised ImportError on the
  CreateEmptyTableRequest import; keep only declare_table.
- Narrow overwrite resolution to only declare on TableNotFound instead of
  swallowing every describe_table exception.
- Drop dead code (open_lance_dataset, pylance<5 compat) and dedupe the
  worker dataset-reopen logic into open_dataset_from_open_kwargs.
- Cover all entry points with dir-namespace tests plus a Gravitino REST
  namespace e2e roundtrip.
- Work around an upstream daft+lance native teardown SIGSEGV (unrelated to
  test outcomes) via a pytest_unconfigure hard-exit after reporting.
fix(namespace): tighten missing table handling
7b1a7c4
refactor(sink): resolve namespace tables at execution time via DataSi…
b2e28c5 
…nk.start()

Constructing a LanceDataSink no longer talks to the namespace or opens the
dataset; all resolution moves to start(), which Daft runs once on the driver
before the sink is serialized to workers. This keeps declare_table from
firing during plan construction and from leaving orphan declared tables
behind when local parameter validation fails.

Create-mode resolution is now describe-first (check_declared=True): a real
existing table fails fast with a clear error suggesting overwrite/append, a
declared-only stub is reused as a placeholder (write_fragments switches to
overwrite when the stub materialized a dataset), and a lost declare race is
recovered by re-describing and classifying. resolve_namespace_table returns
a ResolvedNamespaceTable carrying uri/storage_options/placeholder state so
the sink does not have to re-describe.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
fix(namespace): thread io_config credentials into namespace-resolved …
37b0256 
…locations

Previously io_config was converted to storage options only when a plain uri
was passed, so a namespace that resolves to s3://... without vending
credentials silently ignored the user's io_config. construct_lance_dataset
and LanceDataSink now derive storage options from io_config against the
resolved location and layer them as: io_config-derived < user-provided
storage_options < namespace-vended. Plain-uri behavior is unchanged
(user-provided options still replace io_config-derived ones).

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
chore: zero out mypy errors in daft_lance sources and namespace tests
64501fb 
daft_lance/ sources and the namespace test files now pass mypy strict. The
bulk came from LanceDataSink._namespace_kwargs being annotated dict[str,
object], which poisoned every **-expansion into typed lance APIs. Also
widens read_lance's default_scan_options to dict[str, Any] to match the
documented usage ({"with_row_address": True}) and the other entry points.

Pre-existing errors in the rest of tests/ are out of scope for this branch.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
docs(namespace): document namespace params everywhere and validate or…
0feda98 
…phan properties

- read_lance / merge_columns(_df) / create_scalar_index / compact_files
  docstrings now describe table_id / namespace_impl / namespace_properties
- namespace_properties without namespace_impl now fails fast instead of
  being silently ignored
- README documents io_config fallback for namespace locations and the
  DAFT_LANCE_NAMESPACE_CACHE_SIZE environment variable

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
fix(namespace): address independent review findings
2e71956 
- overwrite-mode resolution now describes with check_declared=True (initial
  and declare-race recovery): strict namespace impls 404 a plain describe on
  declared-only stubs, so the previous recovery path failed in exactly the
  case it existed for. A declared-only stub is a valid overwrite target.
- mode=create over a namespace placeholder that materialized a dataset is
  rejected with a clear error when use_mem_wal=True; the MemWAL path cannot
  apply the overwrite-placeholder semantics.
- plain-uri entry points treat storage_options={} as unset again (falsy
  fallthrough to io_config-derived options), matching pre-branch behavior.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
test(namespace): lock in sink pickle round-trip and compaction fragme…
6fb5437 
…nt count

- pickle a started sink and run write()/finalize() on the copy, codifying
  the driver-resolves/worker-writes contract instead of relying on manual
  verification
- compact_files test now asserts the fragment count actually shrinks, not
  just data correctness

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
fix(namespace): request vended credentials explicitly and unify empty…
7833d58 
… storage_options handling

- describe_table/declare_table now pass vend_credentials=True: per the
  lance-namespace spec, whether credentials are returned is
  implementation-defined when the flag is unset, so the documented
  "namespace vends storage_options" behavior was not guaranteed
- LanceDataSink now treats storage_options={} the same as the read entry
  points (falsy fallthrough to io_config-derived options); previously
  write_lance dropped io_config credentials for that input while read_lance
  kept them

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
docs(namespace): point rest:// uri error at the namespace parameters
72cb1c7 
The old message predated namespace support and directed users to "the
daft-lance package" — which is where they already are. Show the actual
read_lance(namespace_impl="rest", ...) usage instead.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
docs(namespace): explain the integration layer's design for reviewers
8627210 
Module docstring covers the three load-bearing decisions (serialize the
triple + per-process client cache, describe-first resolution, defensive
error classification), and resolve_namespace_table documents the full
create/overwrite/read state machine including the declare-race legs.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
refactor(namespace): drop patch_daft in favor of daft_lance entry points
3bf5943 
patch_daft() was introduced on this branch and never released, so it can be
removed without a deprecation cycle. Until Eventual-Inc/Daft#7282 delegates
DataFrame.write_lance / read_lance natively, namespace users call
daft_lance.read_lance / daft_lance.write_lance directly — which the README
documents as the primary usage anyway. This also removes the only monkeypatch
in the package and avoids having to keep the patch compatible with the
upstream signature change coming in #7282.

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E
@FANNG1 FANNG1 marked this pull request as ready for review July 16, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support Lance Namespace table addressing (catalog-managed Lance tables, e.g. Gravitino Lance REST server)

1 participant

@FANNG1