Skip to content

Commit

Permalink
Address CR feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
tatiana committed Sep 14, 2023
1 parent 737de90 commit bed4612
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions docs/getting_started/dbt-airflow-concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,17 @@ who may be new to Airflow or dbt and are considering to use Cosmos.
| DAG | Workflow | Pipeline (Direct Acyclic Graph) that contains a group of steps | Airflow expects upstream tasks to have passed to run downstream tasks. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/dags.html |
| | | | dbt can run a subset of tasks assuming upstream tasks were run. | https://docs.getdbt.com/docs/introduction |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Task | Node | Step within a pipeline (DAG or workflow) | In dbt, users write mostly SQL and YML to define the steps of a pipeline. | https://docs.getdbt.com/reference/node-selection/syntax |
| | | | Airflow expects steps to be written in Python. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/tasks.html |
| Task | Node | Step within a pipeline (DAG or workflow) | In dbt, these are usually transformations that run on a remote database. | https://docs.getdbt.com/reference/node-selection/syntax |
| | | | In Airflow, steps can be anything, running locally in Airflow or remotely. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/tasks.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Language | Language | Programming or declarative language used to define pipelines and steps. | In dbt, users write SQL, YML and Python to define the steps of a pipeline. | https://docs.getdbt.com/docs/introduction#dbt-optimizes-your-workflow |
| | | | Airflow expects steps and pipelines are written in Python. | https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Variables | Variables | Key-value configuration that can be used in steps and avoids hard-coded values | | https://docs.getdbt.com/docs/build/project-variables |
| | | | | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/variables.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Templating | Macros | Jinja templating used to access variables, configuration and reference steps | dbt encourages using jinja templating for control structures (if and for). | https://docs.getdbt.com/docs/build/jinja-macros |
| | | | Airflow usage is limited to variables, macros and filters | https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html |
| | | | Native in Airflow/Python, used to define variables, macros and filters. | https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html |
+----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
| Connection | Profile | Configuration to connect to databases or other services | | https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html |
| | | | | https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles |
Expand Down

0 comments on commit bed4612

Please sign in to comment.