Glue Table Already Exists for Temporary Tables

[GabrielEisenbergOlympus]

For context, I am running two Airflow tasks at the same time, for the same table. I have implemented idempotency, so each task is running for a different timespan. My understanding is that this should be fine for an incremental model. However, what happens is that an error gets thrown that the Glue Table already exists: GENERIC_INTERNAL_ERROR: Table already exists

My guess is that this is happening to the temporary tables being generated for each dbt run. Please correct me if my understanding is wrong.

Assuming this is correct, is there a way to concatenate a UUID to the temporary table names so that tasks can be run in parallel.

Thanks so much!

[nicor88]

@GabrielEisenbergOlympus that's the case for hive tables.
This PR introduce the possibility of using unique_tmp_table_suffix to achieve what you want, but it works only for hive tables and stategy='insert_overwrite' (I believe that we can add also append as a stategy).

To configure what you want you can use in your dbt model this config then:

• unique_tmp_table_suffix=true

In addition to that, you can consider to use temp_schemato store those unique tables in another glue database to isolate more access on underlying tables.

[GabrielEisenbergOlympus]

Currently, for the model I'm working on, the strategy is merge, but I'll have to look into the requirements and can confirm thereafter 😄

[nicor88]

merge it's not supported at the moment, because merge can be used only for iceberg tables.

[GabrielEisenbergOlympus]

We are using iceberg tables 🙃 I assume this is a problem?

[nicor88]

yup correct. the unique_tmp_table_suffix it's not supported for iceberg tables at all. See this issue: #688. To implement this issue the change it's easy as changing this condition, and add bunch of integration tests.

[GabrielEisenbergOlympus]

You are an absolute champion, thank you @nicor88! I'll give it a shot in the course of the day (or early next week) and will see how it goes 😄