Skip to content

Optimize not running in python models after the first run #913

Closed

Description

@talperetz1

Describe the bug

When running dbt python model with workflow_job no optimize run after the operation

Steps To Reproduce

Run a python model with liquid cluster by

Expected behavior

Expected to see the following operations: merge, cluster by, optimiz

System information

The output of dbt --version:

Core:
  - installed: 1.9.1
  - latest:    1.9.1

Plugins:
  - databricks: 1.9.1 
  - spark:      1.9.0 

The operating system you're using:
linux

The output of python --version:
Python 3.11.2

Additional context

Issue Description:
I have many SQL models that utilize liquid_cluster_by. The behavior for these models includes merge, cluster by, and optimize operations.

However, when I run Python models, I noticed the following behavior:

On the first run (when creating the table), the optimize operation is executed.
On subsequent runs, only the merge operation is performed.
If liquid_cluster_by is present, the merge operation is followed by cluster by, which is pointless without optimize.
Additionally, I observed that the optimize operation does not execute for Python models, regardless of whether liquid_cluster_by is specified.

def model(dbt, session):

    dbt.config(submission_method='workflow_job')
    dbt.config(materialized='incremental')
    dbt.config(file_format='delta')
    dbt.config(unique_key=['x'])
    dbt.config(liquid_clustered_by=['x'])
    dbt.config(incremental_strategy='merge')
    dbt.config(on_schema_change='append_new_columns')
    dbt.config(location_root='s3://......')

Thanks,

Image

Activity

  1. benc-db commented on Jan 27, 2025

    @benc-db
    Collaborator

    Thanks for reporting

  2. talperetz1 commented on Jan 30, 2025

    @talperetz1
    Author

    @benc-db Thank you for your answer. I edited the bug info and added the dbt.config so it is clear that I am talking about workflow_job

  3. benc-db commented on Mar 17, 2025

    @benc-db
    Collaborator

    @talperetz1 can you try #963 with use_materialization_v2: true and let me know if you still have issues?

  4. sd-db commented on Oct 22, 2026

    @sd-db
    Collaborator

    Thanks for the report, and sorry for the long silence. We retested this on the current main branch using your exact config (workflow_job, incremental merge, liquid_clustered_by, external location_root) over 5 runs. Every incremental run now shows MERGE followed by OPTIMIZE in DESCRIBE HISTORY, so this is resolved in current versions. Closing as completed — feel free to reopen with your dbt.log if you still see it after upgrading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions