Repository navigation
Latest databricks-sql-connector causing failure with Configuration AUTOCOMMIT is not available. SQLSTATE: 42K0I #58386
Description
Apache Airflow Provider(s)
databricks
Versions of Apache Airflow Providers
apache-airflow-providers-databricks==7.7.4
Apache Airflow version
Tested on both latest AF2 and 3
Operating System
ubuntu
Deployment
Astronomer
Deployment details
No response
What happened
databricks-sql-connector==4.2.0 released Friday November 14th introduced changes to commit/autocommit properties on Databricks Connection object (databricks/databricks-sql-python#704). Because of that changes, when using Databricks operator like DatabricksSqlOperator I'm receiving an error: databricks.sql.exc.TransactionError: Failed to set autocommit to False: [CONFIG_NOT_AVAILABLE] Configuration AUTOCOMMIT is not available. SQLSTATE: 42K0I. Regardless of whether I try to use autocommit=False or True, the problem is that the Databricks is not accepting setting this config with SQL query. The Base DbApiHook set_autocomit method is setting the autocommit on dbx conn object, before it did not have this property so it passed and was ignored, but now that's implemented, it started to fail.
What you think should happen instead
I should be able to execute query on DBX without errors.
How to reproduce
Run any query with DatabricksSqlOperator and databricks-sql-connector==4.2.0.
Anything else
Not sure if the problem is within my DBX instance, if we should somehow work around this in Airflow or maybe the changes to databricks connection are to be adjusted.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
[-][databricks] Configuration AUTOCOMMIT is not available. SQLSTATE: 42K0I[/-][+]Latest `databricks-sql-connector` causing failure with `Configuration AUTOCOMMIT is not available. SQLSTATE: 42K0I`[/+]kmarq commentedon Nov 17, 2025
Being hit by this as well.
tkeller-moxe commentedon Nov 17, 2025
We are also hit by this, We are just pinning
databricks-sql-connectorto 4.1.4 to get around this for now.Seems like this feature is not generally available in Databricks yet?
kacpermuda commentedon Nov 18, 2025
Did the same for now.
Maybe that's the case. Could not even find a mention about this config in databricks docs
kacpermuda commentedon Nov 18, 2025
This is the response I got on the PR:
So it's a feature, not a bug. We can consider updating the Databricks hook code, but since it's probably going to be available soon for all, it may not be worth it. I'll close the ticket for now, temp solution is to pin the connector version and wait for this feature to be released. If somebody wants to modify the hook - feel free to reopen this ticket.
tkeller-moxe commentedon Nov 18, 2025
Curiosity would be do we expect the hook to support older DBX runtimes without pinning the package? Seems like that might continue to be a sore point.
Thanks for aggregating the details.
potiuk commentedon Nov 18, 2025
Gemerally you are supposed to use constraints where you install airflow and if you look at 3.1.3 constraintsL https://github.com/apache/airflow/blob/constraints-3.1.3/constraints-3.13.txt#L255 - you have 4.1.4 version of the
databricks-sql-connector.@kacpermuda -> do you think we should limit future provider to limit it unti they fix it ? It seems a bit odd to have it breaking by the client BTW. it seems still like a bug if they release client that is broken temporarily for some people - and it requries to
enroll. Maybe we should limit it temporarily and open an an issue with databrick that this is not fixed and so that they let us know when it's all generaly available so that we can remove it ? WDYT?I do not know how long it can take and how disruptive it is, but IMHO this is wrong to have something that fails if your account is not enrolled to private beta.
kmarq commentedon Nov 18, 2025
I've raised this with our Databricks account team as we were looking at enrolling in this preview. I don't know how they'll proceed, but agreed this seems like a broken release on their part.
potiuk commentedon Nov 18, 2025
Thanks @kmarq ! Let us know about the outcome. We are about 2 weeks from releasing new version of providers and likely weeks away from releasing 3.1.4, so we still have time to implement a mitigation of sorts
vikrantpuppala commentedon Nov 19, 2025
Hi, thank you all for the feedback, we've yanked the latest release of databricks-sql-connector to avoid breakages. We will relaunch the feature as a opt in until the feature is fully rolled out to all customers.
kacpermuda commentedon Nov 19, 2025
Thanks @vikrantpuppala , that's good to hear.
Fully agree @potiuk , should be an opt-in, but I can also understand that it was hard to detect - AFAIU there was no
.autocommitproperty on Databricks connection before, and Airflow was just doingconn.autocommit=Xin DbApiHook that suddenly started to use the newly implemented setter from Databricks connection. So the client/connection just rolled out a new feature under a new property and did not expect that somebody was already calling.autocommit. It should probably not fail for those not enrolled, but I understand those that are enrolled wanted to test it with a client.potiuk commentedon Nov 19, 2025
@vikrantpuppala -> thanks for being responsive :) . Community will appreciate it ! It saves time not only the users but also maintainers who have to deal with cases like that, so I am glad Databricks understands that!
jayantsing-db commentedon Nov 19, 2025
Looks like the databricks sql hook has
airflow/providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py
Line 105 in a32f94e
supports_autocommitset to True by default. This suggests that by default the connector should indeed make a set-autocommit call to Databricks. I believe it makes more sense to either make the default as false or user specify thesupports_autocommitas false rather than sql connector providing a knob for transaction which is by default OFF.kacpermuda commentedon Nov 19, 2025
That's right and DatabricksHook should probably also overwrite
set_autocommitmethod, as in base hook's method that is being called, theconn.autocommit = autocommitis executed regardless ofself.supports_autocommit(it merely logs a warning on conflict).jayantsing-db commentedon Nov 19, 2025
Agreed @kacpermuda, thanks
@potiuk what are your thoughts on this? I am a maintainer of databricks sql connector and we have been discussing this internally. It's really not ideal for the driver to have a default OFF knob around major specs like transactions. Should we modify the default behaviour or documentation/communication of supports_autocommit flag in airflow and possibly override the set_autocommit method too.
Additionally, @kacpermuda will you able to set supports_autocommit to false until your workspace is enrolled to the feature or the feature is generally available?
kacpermuda commentedon Nov 20, 2025
For my use case, I'll just pin to 4.1.4 until we figure out how to modify the databricks hook in Airflow and then will unpin + update to latest databricks provider. People using previous versions will hit that issue as well when unpinned connector is installed, but there is nothing we can do about it other than providing a new version of the provider that handles it.
potiuk commentedon Dec 16, 2025
No opinion, really - I think it would be better to use whatever databricks users will feell comfortable - and since you are the maintainer of sql connector - probably you are the best to make that decision