Print full log on dbt error

[fjmacagno]

I am trying to debug cosmos, which is not successfully creating the DAG. Unfortunately, the error message i am getting is useless, just using legacy validation callback. Could this line be changed so that the error prints out stdout as well, so that i have more context to go on?

As an example, this is the full error:

[2024-12-02T15:49:40.651+0000] {graph.py:136} INFO - Running command: `/home/astro/.local/bin/dbt ls --output json --project-dir /tmp/tmpme078_tw --profiles-dir /tmp/cosmos/profile/3cc5d8bec2a8c876092d2b790d2cde171ff387a024cbaa5d615bf88275988ecd --profile derived_eng_dbt --target dev`
Traceback (most recent call last):
  File "/usr/local/airflow/dags/dbt/cosmos_dag.py", line 21, in <module>
    my_cosmos_dag = DbtDag(
                    ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/airflow/dag.py", line 26, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/usr/local/lib/python3.12/site-packages/cosmos/converter.py", line 260, in __init__
    self.dbt_graph.load(method=render_config.load_method, execution_mode=execution_config.execution_mode)
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 402, in load
    self.load_via_dbt_ls()
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 461, in load_via_dbt_ls
    self.load_via_dbt_ls_without_cache()
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 580, in load_via_dbt_ls_without_cache
    nodes = self.run_dbt_ls(dbt_cmd, self.project_path, tmpdir_path, env)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 442, in run_dbt_ls
    stdout = run_command(ls_command, tmp_dir, env_vars)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/cosmos/dbt/graph.py", line 156, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/astro/.local/bin/dbt', 'ls', '--output', 'json', '--project-dir', '/tmp/tmpme078_tw', '--profiles-dir', '/tmp/cosmos/profile/3cc5d8bec2a8c876092d2b790d2cde171ff387a024cbaa5d615bf88275988ecd', '--profile', 'derived_eng_dbt', '--target', 'dev'] due to the error:
using legacy validation callback

If i edit the line referenced above to concat stdout + stderr i get

...

16:17:46  Running with dbt=1.8.7
16:17:47  Registered adapter: databricks=1.8.7
16:17:47  Unable to do partial parsing because config vars, config profile, or config target have changed
16:17:47  Unable to do partial parsing because profile has changed
16:17:48  Encountered an error:
Compilation Error
  Model 'model.derived_eng_dbt.testing_platform_test_results' (models/testing_platform/testing_platform_test_results.sql) depends on a source named 'silver_mysql_eng_testing.test_run' which was not found

which is exactly what i need.

[fjmacagno]

(also it seems that "create an issue from this line" both bypasses ticket templates and doesnt actually refer to the line. Let me know if you want me to recreate)

[pankajastro]

Hey @fjmacagno, Thank you so much for opening the issue. Could you please submit a PR with this improvement?

[KarolGongola]

I got issue with the same