Repository navigation
materialization_v2: cache inconsistency detected in rename action for materialized = 'table' #1214
Description
Describe the bug
In materialized = table, with use_materialization_v2 and use_safer_relation_operations=True enabled, there is a low probability of the following issue occurring when models are run concurrently with multiple threads:
Internal Error
Cache inconsistency detected: in rename, new key ReferenceKey(database=‘’, schema='', identifier='') already in cache Error
Expected behavior
no cache error
Screenshots and log output
Error Log
Internal Error
Cache inconsistency detected: in rename, new key _ReferenceKey(database=‘’, schema='', identifier='') already in cache Error
System information
The output of dbt --version:
dbt=1.10.6
databricks=1.10.7
The output of python --version:
3.11.11
Activity
benc-db commentedon Oct 10, 2025
When you say 'run concurrently', do you mean within a single dbt CLI execution or between multiple CLI executions? If within a single, do you have any ideas as to what could be causing the cache conflict?
hongkai-start commentedon Oct 11, 2025
Within a single DBT CLI execution with multiple threads, I suspect the cause is related to thread locking. Assume there are two concurrent threads: Thread A and Thread B. Thread A acquires the lock, and Thread B is waiting to acquire it. However, Thread B has two operations (delete cache and rename cache) that are both waiting for the lock. These two operations have a specific execution order, but the operation that should be executed later (rename) acquires the lock first, resulting in execution failure.
billstark commentedon Feb 6, 2026
Hello @benc-db may I know whether there are any updates on this? I also encounter this issue. But I feel like this is due to when we renaming the
_stgtable to the original table name, the cache is not cleared in unity catalog?benc-db commentedon Feb 6, 2026
Is the error a dbt error or a UC error? If a dbt error, please cross post with dbt-core, because the cache code lives entirely upstream. If a UC error, I'm honestly not sure of how we can fix, other than wait for improved CTAS support so that we can deprecate materialization V2 and go back to atomic create/replace.
benc-db commentedon Feb 6, 2026
@sd-db viz
github-actions commentedon Aug 6, 2026
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.
sd-db commentedon Oct 17, 2026
We've reproduced this. The error comes from dbt's in-memory relation cache, not from Unity Catalog. It shows up when cache population is turned off (
--no-populate-cacheorDBT_POPULATE_CACHE=false) and many threads replace tables in the same schema.With population off, every thread lists the schema from the server when it starts and adds whatever it sees to dbt's cache. A slow listing can come back after a fast model has already renamed its table to
__dbt_backup. That listing puts the old name back in the cache, so the rename of__dbt_stgto the model name fails with "in rename, new key … already in cache".With 64 table models in one schema:
The race is in dbt-adapters, so we've filed it upstream: dbt-labs/dbt-adapters#2231
@hongkai-start @billstark, do you run with
--no-populate-cacheorDBT_POPULATE_CACHE=false, and how many threads do you use? Until there's a fix, lowering--threadsshould avoid the error. Re-enabling cache population avoids it too. If you already run with population on, please share the full error message without redactions, including the list of keys at the end.