Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DBT Deploy support #74

Merged
merged 7 commits into from
Sep 24, 2024
Merged

Add DBT Deploy support #74

merged 7 commits into from
Sep 24, 2024

Conversation

neel-astro
Copy link
Collaborator

@neel-astro neel-astro commented Sep 3, 2024

Changes:

  • Add support for DBT Deploy for normal and preview deployments. This change includes adding deploy-type input to help distinguish between different forms of deploys, i.e. dags-only, dags-and-image, dbt and infer (to avoid introducing breaking change).
  • Add 'mount-path' as optional input to provide a mount path if the user wants to for dbt deploy
  • Add dev-ex team as the project code owners

Minor Fixes:

  • Fix the logic to install the custom CLI version, it was currently broken and mostly unnoticed because no one might be using it 🙂
  • Fix Github Action logging by grouping logs based on the steps, so that it is easy to navigate the logs for the users

Closes: astronomer/astro#23047

Testing:

Updated E2E tests to include sample DBT deploys

GHA Template I have been using for the tests:

name: Deploy (checkdeployaction)
on:
  push:
    branches:
      - main

jobs:
  prod-astronomer:
    env:
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
    runs-on: ubuntu-latest
    steps:
      - name: Print GITHUB_ACTIONS env var
        run: echo "GITHUB_ACTIONS=${GITHUB_ACTIONS}"
      - name: DBT Deploy
        uses: astronomer/deploy-action@dbt-deploy
        with:
          action: dbt-deploy
          deployment-name: neel-test-1
          root-folder: dbt/
          mount-path: /dbt
          workspace: ${{ secrets.WORKSPACE_ID }}
          cli-version: 1.28.1
      - name: Deploy DAGs/Image
        uses: astronomer/deploy-action@dbt-deploy
        with:
          root-folder: astro-project/
          deployment-name: neel-test-1
          workspace: ${{ secrets.WORKSPACE_ID }}
  • Image Deploy to Astro
Screenshot 2024-08-30 at 6 30 14 PM
  • DBT Deploy to Astro
Screenshot 2024-08-30 at 6 34 16 PM
  • DAGs Deploy to Astro
Screenshot 2024-08-30 at 7 52 07 PM
  • DBT Deploy to Preview Deployment in Astro
Screenshot 2024-09-03 at 2 51 36 PM
  • Case when someone uses CLI version < 1.28.1 for DBT Deploy
Screenshot 2024-08-30 at 3 59 10 PM

@@ -96,7 +97,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.4
uses: astronomer/deploy-action@v0.6
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will update these to 0.7 once this PR is merged and we cut a new release

shell: bash
id: deployment-preview
- name: Determine if DAG Deploy is enabled
run: |

echo ::group::Determine if DAG Deploy is enabled
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all the group & endgroup allows GHA to group these logs

skip_deploy=1

# This for loop checks for following cases:
# 1. If no file is part of the input root folder, then it skips deploy
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to accommodate for the case when dbt project and astro project are part of the same repo, we would need to ensure that we only trigger image deploy when any file is updated inside the astro project's root folder, this is slightly different from existing behaviour where if any file outside the dags folder is updated we trigger an image deploy

Copy link
Collaborator

@iancmoritz iancmoritz Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes sense! since these deploy-action releases are versioned, I'm comfortable with that and I wouldn't expect any breaking changes from it... in fact this might be better for users

action.yaml Outdated
exit 1
fi

# validate that mount path is provided
Copy link
Collaborator

@iancmoritz iancmoritz Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the mount path to be required here? Can we instead rely on the default mount path of the dbt Deploy (i.e. the directory name)?

Also noticed the mount-path input above not required (

mount-path:
) so if we do make it required I assume we should change

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mount path is not required by the feature and ideally we should be encouraging users not to set it because the CLI already provides a sensible default.

README.md Outdated
@@ -48,11 +48,11 @@ The following table lists the configuration options for the Deploy to Astro acti

| Name | Default | Description |
| ---|---|--- |
| `action` | `deploy` | Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either `create-deployment-preview`, `delete-deployment-preview` or `deploy-deployment-preview`. Don't sepcify anything if you are deploying to a regular deployment. |
| `action` | `deploy` | Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either `create-deployment-preview`, `delete-deployment-preview`, `deploy-deployment-preview` or `dbt-deploy`. |
Copy link
Contributor

@sunkickr sunkickr Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dbt-deploy-deployment-preview is also being added right? Instead of adding two new actions would it make sense to just add an option dbt-deploy that can be true/false?

README.md Outdated
with:
deployment-id: <deployment id>
root-folder: dbt/
mount-path: /dbt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between mount-path and root-folder? would they always be the same?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are both features of dbt deploys.

mount-path -> the directory where the dbt code is actually mounted in your container
root-folder -> the directory where the dbt code is retrieved from

action.yaml Outdated Show resolved Hide resolved
action.yaml Outdated Show resolved Hide resolved
action.yaml Show resolved Hide resolved
action.yaml Outdated Show resolved Hide resolved
action.yaml Outdated Show resolved Hide resolved
action.yaml Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
@iancmoritz iancmoritz mentioned this pull request Sep 4, 2024
@jaketf
Copy link

jaketf commented Sep 9, 2024

@neel-astro just dropping by to say kudos on a very well scoped PR with high quality description and leaving the codebase a little cleaner than you found it without distracting from the PR. I learned more about this feature by lurking here :)

@neel-astro neel-astro changed the base branch from main to e2e-tests September 12, 2024 08:16
Base automatically changed from e2e-tests to main September 18, 2024 03:43
@neel-astro neel-astro force-pushed the dbt-deploy branch 3 times, most recently from 3fb9bd2 to ff570e4 Compare September 18, 2024 12:38
Copy link
Collaborator

@jeremybeard jeremybeard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neel-astro, this is much simpler with the deploy type restructure. I left some comments, a few around all the output variables that I hope we can simplify even further.

dbt-deploy-test:
name: DBT Deploy Test
runs-on: ubuntu-latest
needs: [custom-docker-image-test, create-test-deployments]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this test depend on custom-docker-image-test?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does not depend on the custom-docker-image-test, but because these tests are running against a single deployment, to avoid any flakiness in test results, we are running a single test at a time, and hence we are required to use the needs clause to create the dependencies to avoid concurrent test scenario runs

action.yaml Show resolved Hide resolved
action.yaml Outdated
if [[ "${{ inputs.deploy-type }}" == "dbt" && $CLI_VERSION != "" ]]; then
REQUIRED_VERSION="1.28.1"
CURRENT_VERSION=$(astro version | awk '{print $4}')
if [[ $CURRENT_VERSION =~ ([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks robust. Another option might be to run npm install -g semver and use that but you've already done it this way so we don't have to change it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good callout 👍 , I moved it to semver which reduces the logic

action.yaml Outdated
@@ -257,62 +313,139 @@ runs:
echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT

# set action
ACTION=deploy
ACTION=${{ inputs.action }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this variable needed anymore? The only place I can see it used is at the end of the step to check the user provided a correct input, but we should have that at the start of the workflow.

Copy link
Collaborator Author

@neel-astro neel-astro Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropped the ACTION variable, instead moved the corresponding validation check to the top of the function

action.yaml Outdated
@@ -257,62 +313,139 @@ runs:
echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also still need this variable?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And SKIP_DEPLOY? That's only true if ${{ inputs.action }} == delete-deployment-preview but we could just look at that?

action.yaml Outdated
dags_only=0
fi

if [[ ${{ steps.deployment-preview.outputs.SKIP_DEPLOY }} == true ]]; then
# skip all deploy steps
dags_only=2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pre-existing use of "2" for a boolean variable is... not ideal. I know this is creeping the scope but can we change these to use "true"/"false" and if we need to express other things like skipping the deploy then can we just output a second variable?

Copy link
Collaborator Author

@neel-astro neel-astro Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense, I had an itch as well, so replaced the numbering with boolean for both dags_only and dbt_only

shell: bash
id: deployment-type
# If only DAGs changed and dag deploys is enabled, do a DAG-only deploy
- name: setup deploy options
- name: Setup image or dags deploy options
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this step need to be separate to the one before it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could merge them, but both are decently sized bash scripts, so we should avoid merging them unless we see a value addition by doing so otherwise it would be another very long script to manage. This function is well-scoped, and the other one isn't, and I agree with that.

action.yaml Outdated Show resolved Hide resolved
action.yaml Outdated Show resolved Hide resolved
.github/workflows/tests.yaml Outdated Show resolved Hide resolved
Copy link
Collaborator

@jeremybeard jeremybeard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, this is excellent. I've left one nitpick but approving ahead of that.

.github/workflows/tests.yaml Outdated Show resolved Hide resolved
Copy link
Collaborator

@iancmoritz iancmoritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work

@neel-astro neel-astro merged commit 09e2a1a into main Sep 24, 2024
1 check passed
@neelavarrao
Copy link

@neel-astro Do you know when you would be publishing a release with this feature? I was looking forward to using this.

@neel-astro
Copy link
Collaborator Author

@neelavarrao trying to get it out in the next couple of days, thanks for the interest 🙂

@neel-astro neel-astro deleted the dbt-deploy branch September 26, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants