Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cosmos and on-run-end hooks #1338

Open
mttmtt31 opened this issue Nov 25, 2024 · 0 comments
Open

Cosmos and on-run-end hooks #1338

mttmtt31 opened this issue Nov 25, 2024 · 0 comments
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc

Comments

@mttmtt31
Copy link

mttmtt31 commented Nov 25, 2024

Hi,

I would like to follow up one of the limitations pointed out in this article: https://www.astronomer.io/blog/airflow-dbt-2/

  1. The biggest trade off is that dbt really expects to be executing a DAG of models, and so some features don't make as much sense when dbt is only running a single model at a time. For example, dbt has the concept of on-run-start and on-run-end hooks which execute SQL at the beginning or end of a dbt run. With this approach, these hooks would run in every Airflow task.

We have one on-run-end hook which grants usage on the schema (which, as of today, it's the only way to grant usage in dbt, as specified in the official documentation: https://docs.getdbt.com/blog/configuring-grants#granting-permissions-on-other-object-types).

Whenever we use the DbtTaskGroup operator, at least one model always fails with the error:
could not complete because of conflict with concurrent transaction when running the on-run-end hook, making the downstream tasks fail. This makes it impossible to use Cosmos for us.

It's been almost 4 years since the limitation was originally pointed out, I was wondering if there has been any improvement / workaround to avoid the issue, or if there is anything that we are missing out.

@dosubot dosubot bot added the area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc label Nov 25, 2024
@mttmtt31 mttmtt31 changed the title Cosmos and on-run-hooks Cosmos and on-run-end hooks Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc
Projects
None yet
Development

No branches or pull requests

1 participant