You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The biggest trade off is that dbt really expects to be executing a DAG of models, and so some features don't make as much sense when dbt is only running a single model at a time. For example, dbt has the concept of on-run-start and on-run-end hooks which execute SQL at the beginning or end of a dbt run. With this approach, these hooks would run in every Airflow task.
Whenever we use the DbtTaskGroup operator, at least one model always fails with the error: could not complete because of conflict with concurrent transaction when running the on-run-end hook, making the downstream tasks fail. This makes it impossible to use Cosmos for us.
It's been almost 4 years since the limitation was originally pointed out, I was wondering if there has been any improvement / workaround to avoid the issue, or if there is anything that we are missing out.
The text was updated successfully, but these errors were encountered:
dosubotbot
added
the
area:execution
Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc
label
Nov 25, 2024
mttmtt31
changed the title
Cosmos and on-run-hooks
Cosmos and on-run-end hooks
Nov 26, 2024
Hi,
I would like to follow up one of the limitations pointed out in this article: https://www.astronomer.io/blog/airflow-dbt-2/
We have one on-run-end hook which grants usage on the schema (which, as of today, it's the only way to
grant usage
in dbt, as specified in the official documentation: https://docs.getdbt.com/blog/configuring-grants#granting-permissions-on-other-object-types).Whenever we use the
DbtTaskGroup
operator, at least one model always fails with the error:could not complete because of conflict with concurrent transaction
when running the on-run-end hook, making the downstream tasks fail. This makes it impossible to use Cosmos for us.It's been almost 4 years since the limitation was originally pointed out, I was wondering if there has been any improvement / workaround to avoid the issue, or if there is anything that we are missing out.
The text was updated successfully, but these errors were encountered: