Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable kubernetes tests for dbt-core-1.8.x #1344

Open
pankajastro opened this issue Nov 26, 2024 · 0 comments
Open

Enable kubernetes tests for dbt-core-1.8.x #1344

pankajastro opened this issue Nov 26, 2024 · 0 comments
Assignees
Labels
area:testing Related to testing, like unit tests, integration tests, etc execution:kubernetes Related to Kubernetes execution environment
Milestone

Comments

@pankajastro
Copy link
Contributor

pankajastro commented Nov 26, 2024

The Kubernetes test on dbt-core 1.8.x failing with error

  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/jaffle_shop_kubernetes.py", line 85, in <module>
    run_models = DbtTaskGroup(
                 ^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/airflow/task_group.py", line 28, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/converter.py", line 261, in __init__
    self.dbt_graph.load(method=render_config.load_method, execution_mode=execution_config.execution_mode)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 406, in load
    self.load_via_dbt_ls()
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 465, in load_via_dbt_ls
    self.load_via_dbt_ls_without_cache()
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 582, in load_via_dbt_ls_without_cache
    self.run_dbt_deps(dbt_cmd, tmpdir_path, env)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 515, in run_dbt_deps
    stdout = run_command(deps_command, dbt_project_path, env)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 156, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.10/bin/dbt', 'deps', '--project-dir', '/tmp/tmpmgl76aiy', '--profiles-dir', '/tmp/cosmos/profile/bac3236c5c54391c6c2161433c02c661510cb1db5380f8399bdf27c662a462b2', '--profile', '***_profile', '--target', 'dev'] due to the error:
Traceback (most recent call last):
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.10/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.10/lib/python3.11/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.10/lib/python3.11/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.10/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

In dbt codebase I can see module dbt.adapters.factory is available for only dbt < 1.8.x
CI: https://github.com/astronomer/astronomer-cosmos/actions/runs/12010604066/job/33477828079

@dosubot dosubot bot added area:testing Related to testing, like unit tests, integration tests, etc execution:kubernetes Related to Kubernetes execution environment labels Nov 26, 2024
pankajastro added a commit that referenced this issue Nov 26, 2024
…re Clone Test to Pass CI (#1337)

closes: #1335

- Disabled "Run-Integration-Tests-Sqlite" Job and created issue
#1341 to enable it
- Installed `eval_type_backport` in tests and created an issue
#1342 to remove it
- pin dbt version dbt-core<1.8.9 created issue to unpin it
#1343
- pin dbt version for Kubernetes tests and create issue to unpin it
#1344

CI success:
https://github.com/astronomer/astronomer-cosmos/actions/runs/12026097151/job/33524412817?pr=1337
@phanikumv phanikumv added this to the Cosmos 1.8.0 milestone Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:testing Related to testing, like unit tests, integration tests, etc execution:kubernetes Related to Kubernetes execution environment
Projects
None yet
Development

No branches or pull requests

2 participants