Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable tests for dbt-core-1.8.9 #1343

Closed
pankajastro opened this issue Nov 26, 2024 · 0 comments · Fixed by #1371
Closed

Enable tests for dbt-core-1.8.9 #1343

pankajastro opened this issue Nov 26, 2024 · 0 comments · Fixed by #1371
Assignees
@tatiana
Labels
area:testing Related to testing, like unit tests, integration tests, etc dbt:test Primarily related to dbt test command or functionality
Milestone
Cosmos 1.8.0

Comments

@pankajastro
Copy link
Contributor

pankajastro commented Nov 26, 2024 •
edited
Loading

Tests for dbt 1.8.9 failing with error

__________ ERROR collecting tests/test_example_dags_no_connections.py __________
tests/test_example_dags_no_connections.py:78: in <module>
    @pytest.mark.parametrize("dag_id", get_dag_ids())
tests/test_example_dags_no_connections.py:73: in get_dag_ids
    dag_bag = get_dag_bag()
tests/test_example_dags_no_connections.py:68: in get_dag_bag
    assert not db.import_errors
E   AssertionError: assert not {'/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py': 'Traceback (most re...virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/base/__init__.py)\n\n\n'}
E    +  where {'/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py': 'Traceback (most re...virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/base/__init__.py)\n\n\n'} = <airflow.models.dagbag.DagBag object at 0x7f0c091b39d0>.import_errors
------------------------------- Captured stdout --------------------------------
Adding ['cosmos_manifest_example.py'] to .airflowignore
Adding performance_dag.py to .airflowignore
Adding jaffle_shop_kubernetes.py to .airflowignore
Adding cosmos_profile_mapping.py to .airflowignore
.airflowignore contents: 
cosmos_manifest_example.py
performance_dag.py
jaffle_shop_kubernetes.py
example_cosmos_sources.py
cosmos_profile_mapping.py

[2024-11-25T20:31:32.327+0000] {dagbag.py:541} INFO - Filling up the DagBag from /home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags
[2024-11-25T20:31:32.360+0000] {graph.py:474} INFO - Trying to parse the dbt project using dbt ls cache cosmos_cache__example_model_version...
[2024-11-25T20:31:32.363+0000] {cache.py:320} INFO - Cosmos performance: time to calculate cache identifier cosmos_cache__example_model_version for current version: 0.00045867599999382946
[2024-11-25T20:31:32.363+0000] {graph.py:491} INFO - Cosmos performance [fv-az1198-442|3079]: The cache size for cosmos_cache__example_model_version is 26264
[2024-11-25T20:31:32.364+0000] {graph.py:499} INFO - Cosmos performance: Cache hit for cosmos_cache__example_model_version - 3aff3dd85b97095fe2b529[920](https://github.com/astronomer/astronomer-cosmos/actions/runs/12018236731/job/33502297498?pr=1340#step:7:921)2a7eaa4,d41d8cd98f00b204e9800998ecf8427e
[2024-11-25T20:31:32.364+0000] {graph.py:416} INFO - Total nodes: 34
[2024-11-25T20:31:32.365+0000] {graph.py:417} INFO - Total filtered nodes: 34
[2024-11-25T20:31:32.365+0000] {converter.py:265} INFO - Cosmos performance (example_model_version) -  [fv-az1198-442|3079]: It took 0.00494s to parse the dbt project for DAG using LoadMode.DBT_LS_CACHE
[2024-11-25T20:31:32.374+0000] {converter.py:309} INFO - Cosmos performance (example_model_version) - [fv-az1198-442|3079]: It took 0.00957s to build the Airflow DAG.
[2024-11-25T20:31:32.379+0000] {graph.py:474} INFO - Trying to parse the dbt project using dbt ls cache cosmos_cache__example_cosmos_python_models...
[2024-11-25T20:31:32.380+0000] {graph.py:480} INFO - Cosmos performance: Cache miss for cosmos_cache__example_cosmos_python_models
[2024-11-25T20:31:32.381+0000] {graph.py:532} INFO - Trying to parse the dbt project in `/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/dbt/jaffle_shop_python` using dbt ls...
[2024-11-25T20:31:32.381+0000] {config.py:343} INFO - Profile caching is enable.
[2024-11-25T20:31:32.381+0000] {config.py:323} INFO - Profile found in cache using profile: /tmp/cosmos/profile/963ca25a80b9edecb20ad7dc6198382b7b09dbe416456409f7421b410ac2b766/profiles.yml.
[2024-11-25T20:31:32.382+0000] {project.py:36} INFO - Project /home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/dbt/jaffle_shop_python does not have {'dependencies.yml', 'packages.yml'}
[2024-11-25T20:31:32.382+0000] {graph.py:136} INFO - Running command: `/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/bin/dbt ls --output json --project-dir /tmp/tmpe5u5_ugs --profiles-dir /tmp/cosmos/profile/963ca25a80b9edecb20ad7dc6198382b7b09dbe416456409f7421b410ac2b766 --profile default --target dev`
[2024-11-25T20:31:34.721+0000] {dagbag.py:350} ERROR - Failed to import: /home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py
Traceback (most recent call last):
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/airflow/models/dagbag.py", line 346, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py", line 37, in <module>
    example_cosmos_python_models = DbtDag(
                                   ^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/airflow/dag.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/converter.py", line 261, in __init__
    self.dbt_graph.load(method=render_config.load_method, execution_mode=execution_config.execution_mode)
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 406, in load
    self.load_via_dbt_ls()
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 465, in load_via_dbt_ls
    self.load_via_dbt_ls_without_cache()
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 584, in load_via_dbt_ls_without_cache
    nodes = self.run_dbt_ls(dbt_cmd, self.project_path, tmpdir_path, env)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 446, in run_dbt_ls
    stdout = run_command(ls_command, tmp_dir, env_vars)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 156, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/bin/dbt', 'ls', '--output', 'json', '--project-dir', '/tmp/tmpe5u5_ugs', '--profiles-dir', '/tmp/cosmos/profile/963ca25a80b9edecb20ad7dc6198382b7b09dbe416456409f7421b410ac2b766', '--profile', 'default', '--target', 'dev'] due to the error:
20:31:34  Running with dbt=1.8.9
20:31:34  Encountered an error:
cannot import name 'Credentials' from 'dbt.adapters.base' (/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/base/__init__.py)
20:31:34  Traceback (most recent call last):
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/cli/requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/cli/requires.py", line 101, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/cli/requires.py", line 215, in wrapper
    profile = load_profile(flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, threads)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/config/runtime.py", line 71, in load_profile
    profile = Profile.render(
              ^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/config/profile.py", line 403, in render
    return cls.from_raw_profiles(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/config/profile.py", line 369, in from_raw_profiles
    return cls.from_raw_profile_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/config/profile.py", line 325, in from_raw_profile_info
    credentials: Credentials = cls._credentials_from_profile(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/config/profile.py", line 149, in _credentials_from_profile
    cls = load_plugin(typename)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/factory.py", line 239, in load_plugin
    return FACTORY.load_plugin(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/factory.py", line 68, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line [940](https://github.com/astronomer/astronomer-cosmos/actions/runs/12018236731/job/33502297498?pr=1340#step:7:941), in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/databricks/__init__.py", line 1, in <module>
    from dbt.adapters.databricks.connections import DatabricksConnectionManager  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/databricks/connections.py", line 27, in <module>
    from dbt.adapters.base import Credentials
ImportError: cannot import name 'Credentials' from 'dbt.adapters.base' (/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.6/lib/python3.11/site-packages/dbt/adapters/base/__init__.py)

CI: https://github.com/astronomer/astronomer-cosmos/actions/runs/12018236731/job/33502297498?pr=1340
@dosubot dosubot bot added area:testing Related to testing, like unit tests, integration tests, etc dbt:test Primarily related to dbt test command or functionality labels Nov 26, 2024
@pankajastro pankajastro mentioned this issue Nov 26, 2024
Merged
pankajastro added a commit that referenced this issue Nov 26, 2024
@pankajastro
CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly Igno…
57ba69d 
…re Clone Test to Pass CI (#1337)

closes: #1335

- Disabled "Run-Integration-Tests-Sqlite" Job and created issue
#1341 to enable it
- Installed `eval_type_backport` in tests and created an issue
#1342 to remove it
- pin dbt version dbt-core<1.8.9 created issue to unpin it
#1343
- pin dbt version for Kubernetes tests and create issue to unpin it
#1344

CI success:
https://github.com/astronomer/astronomer-cosmos/actions/runs/12026097151/job/33524412817?pr=1337
@tatiana tatiana self-assigned this Nov 29, 2024
@phanikumv phanikumv added this to the Cosmos 1.8.0 milestone Dec 3, 2024
tatiana added a commit that referenced this issue Dec 9, 2024
@tatiana
Remove 'dbt-core<1.8.9' pin
782c5d6 
Closes: #1343
@tatiana tatiana mentioned this issue Dec 9, 2024
Merged
tatiana added a commit that referenced this issue Dec 9, 2024
@tatiana
Remove 'dbt-core<1.8.9' pin
29c3a58 
Closes: #1343
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tatiana tatiana

Labels
area:testing Related to testing, like unit tests, integration tests, etc dbt:test Primarily related to dbt test command or functionality
Projects
None yet
Milestone
Cosmos 1.8.0
Development

Successfully merging a pull request may close this issue.

Remove 'dbt-core<1.8.9' pin astronomer/astronomer-cosmos
3 participants
@tatiana @phanikumv @pankajastro