Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wraps run_dbt_ function in sync_compatible #15740

Merged
merged 5 commits into from
Oct 21, 2024
Merged

Conversation

EmilRex
Copy link
Contributor

@EmilRex EmilRex commented Oct 17, 2024

Closes #15647

This PR wraps each of the run_dbt_ functions with sync_compatible. Not sure if that's a desirable fix, so open to suggestions.

Currently, the functions never execute (at least when called from a sync context):

from dbt.cli.main import dbtRunner
from prefect import flow, task

@task
async def trigger_dbt_cli_command():
    dbtRunner().invoke(["debug"])

@flow
def main():
    trigger_dbt_cli_command()

if __name__ == "__main__":
    main()
11:32:09.538 | INFO    | prefect.engine - Created flow run 'spectral-marmoset' for flow 'main'
11:32:09.539 | INFO    | prefect.engine - View at http://127.0.0.1:4200/runs/flow-run/ea66fd5c-f10d-415c-a181-0e347a83c217
11:32:09.589 | INFO    | Flow run 'spectral-marmoset' - Finished in state Completed()

Adding @sync_compatible to trigger_dbt_cli_command fixes it:

...
@task
@sync_compatible
async def trigger_dbt_cli_command():
...
11:32:40.622 | INFO    | prefect.engine - Created flow run 'great-zebra' for flow 'main'
11:32:40.623 | INFO    | prefect.engine - View at http://127.0.0.1:4200/runs/flow-run/2c60829a-57ed-4e80-885b-d82af2fde872
15:32:40  Running with dbt=1.8.7
15:32:40  dbt version: 1.8.7
15:32:40  python version: 3.12.7

...

11:32:41.017 | INFO    | Task run 'trigger_dbt_cli_command-211' - Finished in state Completed()
11:32:41.063 | INFO    | Flow run 'great-zebra' - Finished in state Completed()

@github-actions github-actions bot added the bug Something isn't working label Oct 17, 2024
@EmilRex
Copy link
Contributor Author

EmilRex commented Oct 17, 2024

So with the knowledge that we'd like to limit the use of sync_compatible in favor of more transparent sync and async implementation, I guess my question is what's the better approach here?

To me, the real problem is that a warning/error is suppressed, in that my flow completes successfully (without any warnings) but my task doesn't actually run. Is my understanding correct that #15008 would make is so that a warning is actually raised?

On a separate note, the prefect-dbt collection already makes use of sync_compatible (in trigger_dbt_cli_command) so we'll have to refactor it regardless. With that in mind, doesn't it make sense to fix the issue for now?

@desertaxle
Copy link
Member

@EmilRex I'm good with these changes as long as the tests pass.

@EmilRex EmilRex merged commit 4583539 into main Oct 21, 2024
13 checks passed
@EmilRex EmilRex deleted the run-dbt-sync-compatible branch October 21, 2024 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prefect-dbt task run_dbt_build not running but shows completed
2 participants