Skip to content

[docs](lance) Document automatic vector and BM25 SQL search - #4441

Merged
morningman merged 1 commit into
apache:masterfrom
Gabriel39:dev/lance-auto-search-docs
Oct 23, 2026
Merged

morningman merged 1 commit into
apache:masterfrom
Gabriel39:dev/lance-auto-search-docs

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

Ordinary SQL search over Lance is not described in the current 4.x reference. This documents the branch-4.2 behavior introduced by apache/doris#69346 (implementation PR is still open; this documentation depends on it).

For example, an eligible ORDER BY cosine_similarity_approximate(embedding, [0.1,0.2,0.3,0.4]) DESC LIMIT 10 becomes native Lance cosine retrieval, with 1 - _distance projected as similarity. Exact functions and ACL/EXISTS queries retain scalar distance evaluation and global Top-N.

  • Add matching English/Chinese guidance for five approximate vector functions, MATCH/BM25, ordering and result conversion, native versus scalar NULL/NaN/zero-vector semantics, UInt8 Hamming validation, bounded row-ID prefilters, and JOIN/EXISTS boundaries.
  • Add English/Chinese SQL references for cosine_distance_approximate, cosine_similarity_approximate, hamming_distance, and hamming_distance_approximate, including normal, zero/empty, NULL, and out-of-range examples.
  • Add navigation entries, link from Lance Catalog, and clarify that ordinary SQL rewritten to search TVFs can qualify for two-phase output reads.

Version scope: this change intentionally updates 4.x English and Chinese only. The implementation targets branch-4.2, not Doris master; current/development and older-version pages are not being advertised as supporting it. This explains the governance version-counterpart warnings. Availability wording requires a Doris 4.2 build containing this capability, rather than implying every build with the explicit TVFs already supports rewriting.

Validation: all 12 changed documentation pages compiled with the Docusaurus MDX processor; changed-page link checks found no broken links; SQL-function structure checks passed; paired English/Chinese SQL blocks match; all six added sidebar references resolve; whitespace and added-content disclosure checks passed. SEO descriptions satisfy the configured length range. Remaining SEO duplicate-title warnings reflect shared SQL identifiers across locales and existing versioned titles. Full-repository sidebar lint reports unrelated existing findings. Full site build and live SQL execution were not run.

@morningman morningman closed this Oct 23, 2026
@morningman morningman reopened this Oct 23, 2026
@morningman morningman closed this Oct 23, 2026
@morningman morningman reopened this Oct 23, 2026
@morningman
morningman merged commit a0312e4 into apache:master Oct 23, 2026
9 of 17 checks passed
morningman added a commit that referenced this pull request Oct 23, 2026
## Summary

The Russian planner keeps reopening four deletion tasks from its fixed
T03 inventory after the Binlog-to-Changelog renames in #4426. Record the
reviewed deletions so subsequent plans stop requesting cleanup that is
already complete.

The ledger binds each task to its original ID, source path and source
hash, with a timestamp, explicit machine-review identity and evidence
from #4426/#4431. At Apache master
`a0312e41f13f47ff7da7855759d0d6f7ed3b5fc1`, all four replacement Russian
documents are current, obsolete sources/targets/state records are
absent, affected inbound links and sidebars are clean, and the actual
Docusaurus `createRedirects` callback maps all four old Russian URLs to
their replacements. The remaining references to the old image filename
point to an existing retained asset.

## Validation

- Existing planner tests: 19 passed, including reviewed-deletion
suppression and evidence validation.
- Configuration checks pass for all three managed locales; the four
ledger records pass `validateResolved`.
- Native audit and planner comparison: deletion tasks drop from 4 to 0.
Coverage and all six unrelated source-update tasks remain unchanged:
required/current/pending = 3354/3348/6; no sidebar gaps or planner
exceptions.
- Shared Russian PR preflight and `git diff --check` pass. No document
bodies or page-state records change, so preflight checks zero changed
documents.
- All four Russian redirect mappings verified through the loaded
Docusaurus configuration. No website build or public deployment was
performed locally; CI selects Russian 4.x.

The six remaining Russian 4.x tasks come from the later source changes
in #4441: four new distance-function documents and updates to Lance
Catalog and hybrid search. This ledger change preserves those tasks and
does not claim complete current coverage.

## Self-review

| Checkpoint | Conclusion |
| --- | --- |
| Goal | Four completed historical cleanup tasks disappear from planning
while new source work remains visible. |
| Scope and clarity | One locale-specific ledger file; original
inventory and translation identities are preserved. |
| Information architecture | Current docs, active versions, community
pages and other locales retain their existing content and navigation. |
| Paths, links and redirects | No paths or anchors change here; all four
existing replacement routes, cleanup and redirects were checked. |
| Frontend and build conventions | No frontend, style, theme, workflow
or build changes. Responsive/theme behavior is unaffected. |
| Verification | Native audit/planner comparison, existing planner
tests, configuration/schema validation, actual redirect callback and
shared admission checks cover the metadata change. CI/public deployment
results are not asserted. |
| Other correctness, accessibility, SEO and maintenance | No additional
issue found in this change. Ledger evidence and machine identity remain
explicit; six new source-update tasks are reported separately. |

## Versions

- [x] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)

## Languages

- [ ] Chinese
- [ ] English
- [x] Russian translation-state ledger

## Docs Checklist

- [x] Checked by AI
- [ ] Test Cases Built (metadata-only change; existing tests and native
checks run, no new tests or website build)
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync

The ledger belongs only to Russian. The existing English/Chinese
sources, Russian replacement documents and shared redirects are already
merged and need no counterpart edits for this bookkeeping change.

Co-authored-by: morningman <moringman@apache.org>

This branch was previously deployed

1 inactive deployment
Production — 023b928c Deployed Oct 23, 2026 by morningman via Build Check #8985
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants