[pepijnve]

Which issue does this PR close?

• None logged yet

This PR builds on #25887 so that should be merged first.

Rationale for this change

LogicalPlan has a representation for the DESCRIBE TABLE statement, but not for the various SHOW ... statements. This makes it necessary to lower SHOW statements to equivalent SELECT statements against the information_schema tables during SQL statement to logical plan conversion. The code in question is located in the sql module's SqlToRel::sql_statement_to_plan function.

As an example, show_tables_to_plan is implemented as

fn show_tables_to_plan(&self) -> Result<LogicalPlan> {
    if self.has_table("information_schema", "tables") {
        let query = "SELECT * FROM information_schema.tables;";
        let mut rewrite = DFParser::parse_sql(query)?;
        assert_eq!(rewrite.len(), 1);
        self.statement_to_plan(rewrite.pop_front().unwrap()) // length of rewrite is 1
    } else {
        plan_err!("SHOW TABLES is not supported unless information_schema is enabled")
    }
}

The implementation is simple enough, but has the downside that it hardcodes references to tables and their columns using magic strings in a module that should be independent of specific schemas, tables, and views. The exact set of columns (and hence the output format) of the show statements is being determined in the sql module which doesn't seem to be where it belongs. This is a logical-to-physical translation aspect, not a sql-to-logical one.

To move the translation to a more appropriate location, this MR extends the logical plan data model to support show statements as well. Rather than introducing a new plan type, the existing describe table plan type is extended to support all statements belonging to the family of introspection statements.

The logical-to-physical translation of show statements is move over to the physical_planner module next to the existing implementation for describe table.

What changes are included in this PR?

LogicalPlan::DescribeTable has been made been renamed to LogicalPlan::Show and extended to support all show statements.

Handling of show statements has been moved from sql::statement to core::physical_planner.

The implementations themselves have been decoupled from the information_schema tables. This allows SHOW statements to work even if the information schema provider is not enabled.

A new common::information_schema module has been added to avoid string based duplication of the information_schema schema. In other words, duplicate magic strings have been removed.

Are these changes tested?

Covered by SLTs

Are there any user-facing changes?

Yes

• LogicalPlan::DescribeTable has been replaced with LogicalPlan::Show
• Availability of SHOW statements is no longer tied to the information_schema schema being accessible in the catalog
• information_schema.views no longer contains base tables

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning 6a280f15786f59b7cca9fcef1a0ddb3c3e931a07
    Building datafusion v55.1.0 (current)
       Built [  63.626s] (current)
     Parsing datafusion v55.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion v55.1.0 (baseline)
       Built [  18.666s] (baseline)
     Parsing datafusion v55.1.0 (baseline)
      Parsed [   0.038s] (baseline)
    Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.599s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  84.525s] datafusion
    Building datafusion-catalog v55.1.0 (current)
       Built [  19.735s] (current)
     Parsing datafusion-catalog v55.1.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-catalog v55.1.0 (baseline)
       Built [  15.390s] (baseline)
     Parsing datafusion-catalog v55.1.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-catalog v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.116s] 229 checks: 227 pass, 2 fail, 0 warn, 31 skip

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/function_missing.ron

Failed in:
  function datafusion_catalog::information_schema::schemata_schema, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/catalog/src/information_schema.rs:1009

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_catalog::information_schema::InformationSchemataBuilder, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/catalog/src/information_schema.rs:1052

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  36.313s] datafusion-catalog
    Building datafusion-cli v55.1.0 (current)
       Built [  65.860s] (current)
     Parsing datafusion-cli v55.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-cli v55.1.0 (baseline)
       Built [  22.224s] (baseline)
     Parsing datafusion-cli v55.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-cli v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.132s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  90.848s] datafusion-cli
    Building datafusion-common v55.1.0 (current)
       Built [  19.281s] (current)
     Parsing datafusion-common v55.1.0 (current)
      Parsed [   0.079s] (current)
    Building datafusion-common v55.1.0 (baseline)
       Built [   1.198s] (baseline)
     Parsing datafusion-common v55.1.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.858s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CatalogOptions.system_catalog in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:225

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  22.768s] datafusion-common
    Building datafusion-execution v55.1.0 (current)
       Built [  10.506s] (current)
     Parsing datafusion-execution v55.1.0 (current)
      Parsed [   0.029s] (current)
    Building datafusion-execution v55.1.0 (baseline)
       Built [   6.052s] (baseline)
     Parsing datafusion-execution v55.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-execution v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.311s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  17.904s] datafusion-execution
    Building datafusion-expr v55.1.0 (current)
       Built [  16.376s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.112s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [   7.716s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.109s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   1.475s] 229 checks: 225 pass, 4 fail, 0 warn, 31 skip

--- failure enum_missing: pub enum removed or renamed ---

Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_missing.ron

Failed in:
  enum datafusion_expr::TableType, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/expr/src/table_source.rs:55

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:Show in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:360
  variant LogicalPlan:Show in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:360

--- failure enum_variant_missing: pub enum variant removed or renamed ---

Description:
A publicly-visible enum has at least one variant that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_missing.ron

Failed in:
  variant LogicalPlan::DescribeTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/expr/src/logical_plan/plan.rs:357
  variant LogicalPlan::DescribeTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/expr/src/logical_plan/plan.rs:357

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_expr::logical_plan::DescribeTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/expr/src/logical_plan/plan.rs:3763
  struct datafusion_expr::DescribeTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-6a280f15786f59b7cca9fcef1a0ddb3c3e931a07/268001d2abf9b252fb2405830357bc3000f82ea8/datafusion/expr/src/logical_plan/plan.rs:3763

     Summary semver requires new major version: 4 major and 0 minor checks failed
    Finished [  27.079s] datafusion-expr
    Building datafusion-optimizer v55.1.0 (current)
       Built [  17.805s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [   7.854s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.039s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.182s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  26.712s] datafusion-optimizer
    Building datafusion-proto v55.1.0 (current)
       Built [  27.118s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  22.574s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.118s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  51.000s] datafusion-proto
    Building datafusion-sql v55.1.0 (current)
       Built [  13.215s] (current)
     Parsing datafusion-sql v55.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion-sql v55.1.0 (baseline)
       Built [  13.300s] (baseline)
     Parsing datafusion-sql v55.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-sql v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.263s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  27.859s] datafusion-sql
    Building datafusion-substrait v55.1.0 (current)
       Built [ 305.081s] (current)
     Parsing datafusion-substrait v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-substrait v55.1.0 (baseline)
       Built [  19.765s] (baseline)
     Parsing datafusion-substrait v55.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-substrait v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.239s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [ 326.877s] datafusion-substrait