Skip to content

perf(scanner): renmove backtics from column names#7782

Open
yuvalif wants to merge 1 commit into
lance-format:mainfrom
yuvalif:remove-backticks-on-column-search
Open

perf(scanner): renmove backtics from column names#7782
yuvalif wants to merge 1 commit into
lance-format:mainfrom
yuvalif:remove-backticks-on-column-search

Conversation

@yuvalif

@yuvalif yuvalif commented Jul 14, 2026

Copy link
Copy Markdown

backtics are added around column names to support dots and other cases. however, when compare to the column names in the table schema, they never match (as the table schema columns dont have backtics). this means that we always go through the slower, sql parser code path. removing the backtics before the comparisson ensure that if the column expression matches a column name the sql path will not be needed.

Fixes: #7781

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of backtick-quoted column names in SQL expressions.
    • Column references now resolve correctly regardless of letter casing.
    • More complex expressions, such as arithmetic involving columns, continue to be interpreted correctly.

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Planner::parse_expr now strips surrounding backticks before resolving column names. Tests cover quoted identifiers, case-insensitive matching, and SQL parsing fallback for compound expressions.

Changes

Column expression resolution

Layer / File(s) Summary
Normalize column references and validate parsing
rust/lance-datafusion/src/planner.rs
Column lookup removes surrounding backticks, with tests covering quoted identifiers, case-insensitive matching, and expressions such as id + 1 falling back to SQL parsing.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: wjones127, westonpace, xuanwo

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly refers to removing backticks from column names, which matches the main change in the PR.
Linked Issues check ✅ Passed The code strips backticks before column-name resolution, avoiding the SQL fallback described in issue #7781.
Out of Scope Changes check ✅ Passed The changes stay focused on backtick handling and add a targeted test without unrelated scope.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-datafusion/src/planner.rs`:
- Around line 936-943: The public Planner::parse_expr API needs documentation
covering its backtick handling and expected column-resolution result. Add
rustdoc to parse_expr with links to the relevant Planner, Expr, and column types
or methods, plus a compilable example demonstrating that parsing `id` resolves
to Expr::Column; keep the implementation unchanged.
- Around line 1185-1199: The repeated identifier assertions in the planner test
should be parameterized with rstest. Update the test function containing the
parse_expr checks to use readable #[case::...] variants for each identifier
input and expected “id” column result, while keeping the separate “id + 1”
fallback assertion unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 36f3ade2-68ec-46a1-bb15-c7bf501083ae

📥 Commits

Reviewing files that changed from the base of the PR and between fff336c and 88f0520.

📒 Files selected for processing (1)
  • rust/lance-datafusion/src/planner.rs

Comment thread rust/lance-datafusion/src/planner.rs
Comment thread rust/lance-datafusion/src/planner.rs Outdated
backtics are added around column names to support dots and other cases.
however, when compare to the column names in the table schema, they
never match (as the table schema columns dont have backtics).
this means that we always go through the slower, sql parser code path.
removing the backtics before the comparisson ensure that if the column
expression matches a column name the sql path will not be needed.

Fixes: lance-format#7781

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@yuvalif yuvalif force-pushed the remove-backticks-on-column-search branch from 88f0520 to e45f26b Compare July 15, 2026 08:13
@github-actions github-actions Bot added the A-deps Dependency updates label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

column expression parsing always use sql

1 participant