diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md index c11a4a77582..552c3a45ae0 100644 --- a/website/docs/docs/build/data-tests.md +++ b/website/docs/docs/build/data-tests.md @@ -125,7 +125,7 @@ You can find more information about these data tests, and additional configurati Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of tests—a good thing! You can also install generic data tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic tests](/best-practices/writing-custom-generic-tests) for more information. :::info -There are generic tests defined in some open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](/docs/build/packages) to learn more! +There are generic tests defined in some open-source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](/docs/build/packages) to learn more! ::: ### Example @@ -261,13 +261,23 @@ Note that, if you select to store test failures: * Test result tables are created in a schema suffixed or named `dbt_test__audit`, by default. It is possible to change this value by setting a `schema` config. (For more details on schema naming, see [using custom schemas](/docs/build/custom-schemas).) - A test's results will always **replace** previous failures for the same test. - + ## New `data_tests:` syntax -Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. As of v1.8, `tests:` is still supported in your YML configuration files as an alias but will be deprecated in the future in favor of `data_tests:`. + + +In dbt version 1.8, we updated the `tests` configuration to `data_tests`. For detailed information, select version v1.8 from the documentation navigation menu. + + + + + +Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. + +As of v1.8, `tests:` is still supported in your YML configuration files as an alias but will be deprecated in the future in favor of `data_tests:`. -As we progress towards this deprecation, the examples in our docs pages will be updated to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future. +As we progress towards this deprecation, we'll update the examples in our docs pages to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future. diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index 0d2f29d0962..8db172f6764 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -37,7 +37,6 @@ You should unit test a model: - Window functions - `case when` statements when there are many `when`s - Truncation - - Recursion - When you're writing custom logic to process input data, similar to creating a function. - We don't recommend conducting unit testing for functions like `min()` since these functions are tested extensively by the warehouse. If an unexpected issue arises, it's more likely a result of issues in the underlying data rather than the function itself. Therefore, fixture data in the unit test won't provide valuable information. - Logic for which you had bugs reported before. @@ -287,4 +286,4 @@ There is currently no way to unit test whether the dbt framework inserted/merged - [Unit testing versioned models](/reference/resource-properties/unit-testing-versions) - [Unit test inputs](/reference/resource-properties/unit-test-input) - [Unit test overrides](/reference/resource-properties/unit-test-overrides) -- [Platform-specific data types](/reference/resource-properties/data-types) \ No newline at end of file +- [Platform-specific data types](/reference/resource-properties/data-types) diff --git a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md index 232c7d4c1bc..f70793f3fee 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md @@ -37,11 +37,11 @@ The IDE streamlines your workflow, and features a popular user interface layout -4. **Command bar —** The Command bar, located in the lower left of the IDE, is used to invoke [dbt commands](/reference/dbt-commands). When a command is invoked, the associated logs are shown in the Invocation History Drawer. +5. **Command bar —** The Command bar, located in the lower left of the IDE, is used to invoke [dbt commands](/reference/dbt-commands). When a command is invoked, the associated logs are shown in the Invocation History Drawer. -5. **Defer to production —** The **Defer to production** toggle allows developers to only build and run and test models they've edited without having to first run and build all the models that come before them (upstream parents). Refer to [Using defer in dbt Cloud](/docs/cloud/about-cloud-develop-defer#defer-in-the-dbt-cloud-ide) for more info. +6. **Defer to production —** The **Defer to production** toggle allows developers to only build and run and test models they've edited without having to first run and build all the models that come before them (upstream parents). Refer to [Using defer in dbt Cloud](/docs/cloud/about-cloud-develop-defer#defer-in-the-dbt-cloud-ide) for more info. -6. **Status button —** The IDE Status button, located on the lower right of the IDE, displays the current IDE status. If there is an error in the status or in the dbt code that stops the project from parsing, the button will turn red and display "Error". If there aren't any errors, the button will display a green "Ready" status. To access the [IDE Status modal](#modals-and-menus), simply click on this button. +7. **Status button —** The IDE Status button, located on the lower right of the IDE, displays the current IDE status. If there is an error in the status or in the dbt code that stops the project from parsing, the button will turn red and display "Error". If there aren't any errors, the button will display a green "Ready" status. To access the [IDE Status modal](#modals-and-menus), simply click on this button. ## Editing features @@ -104,7 +104,7 @@ Starting from dbt v1.6 or higher, when you save changes to a model, you can comp 3. **Build button —** The build button allows users to quickly access dbt commands related to the active model in the File Editor. The available commands include dbt build, dbt test, and dbt run, with options to include only the current resource, the resource and its upstream dependencies, the resource, and its downstream dependencies, or the resource with all dependencies. This menu is available for all executable nodes. -3. **Format button —** The editor has a **Format** button that can reformat the contents of your files. For SQL files, it uses either `sqlfmt` or `sqlfluff`, and for Python files, it uses `black`. +4. **Format button —** The editor has a **Format** button that can reformat the contents of your files. For SQL files, it uses either `sqlfmt` or `sqlfluff`, and for Python files, it uses `black`. 5. **Results tab —** The Results console tab displays the most recent Preview results in tabular format. diff --git a/website/docs/docs/cloud/secure/snowflake-privatelink.md b/website/docs/docs/cloud/secure/snowflake-privatelink.md index 685092fc383..7e42275d2fa 100644 --- a/website/docs/docs/cloud/secure/snowflake-privatelink.md +++ b/website/docs/docs/cloud/secure/snowflake-privatelink.md @@ -56,3 +56,47 @@ Once dbt Cloud support completes the configuration, you can start creating new c 3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field). 4. Configure the remaining data platform details. 5. Test your connection and save it. + +## Configuring Network Policies +If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you will need to add a network rule for dbt Cloud. + +You can request the VPCE ID from [dbt Cloud Support](mailto:support@getdbt.com), that you can use to create a network policy. + +### Using the UI + +Open the Snowflake UI and take the following steps: +1. Go to the **Security** tab. +2. Click on **Network Rules**. +3. Click on **Add Rule**. +4. Give the rule a name. +5. Select a database and schema where the rule will be stored. These selections are for permission settings and organizational purposes; they do not affect the rule itself. +6. Set the type to `AWS VPCE ID` and the mode to `Ingress`. +7. Type the VPCE ID provided by dbt Cloud Support into the identifier box and press **Enter**. +8. Click **Create Network Rule**. + + + +9. In the **Network Policy** tab, edit the policy you want to add the rule to. This could be your account-level policy or a policy specific to the users connecting from dbt Cloud. + + +10. Add the new rule to the allowed list and click **Update Network Policy**. + + + +### Using SQL +For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt Cloud. These SQL examples demonstrate how to add a network rule and update your network policy accordingly. + +1. Create a new network rule with the following SQL: +```sql +CREATE NETWORK RULE allow_dbt_cloud_access + MODE = INGRESS + TYPE = AWSVPCEID + VALUE_LIST = (''); -- Replace '' with the actual ID provided +``` + +2. Add the rule to a network policy with the following SQL: +```sql +ALTER NETWORK POLICY + ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access'); +``` + diff --git a/website/docs/docs/core/about-core-setup.md b/website/docs/docs/core/about-core-setup.md index 16bfe18fc37..0ad6b004cac 100644 --- a/website/docs/docs/core/about-core-setup.md +++ b/website/docs/docs/core/about-core-setup.md @@ -17,6 +17,3 @@ dbt Core is an [open-source](https://github.com/dbt-labs/dbt-core) tool that ena If you're interested in using a command line interface to [develop dbt projects in dbt Cloud](/docs/cloud/about-develop-dbt), the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) lets you run dbt commands locally. The dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features). If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](https://docs.getdbt.com/guides). - -## Related docs -- [Move from dbt Core to dbt Cloud](/guides/core-to-cloud-1?step=1) diff --git a/website/docs/docs/core/installation-overview.md b/website/docs/docs/core/installation-overview.md index 2e112066048..ace217cece8 100644 --- a/website/docs/docs/core/installation-overview.md +++ b/website/docs/docs/core/installation-overview.md @@ -51,5 +51,4 @@ Most command-line tools, including dbt, have a `--help` flag that you can use to ::: -## Related docs -- [Move from dbt Core to dbt Cloud](/guides/core-to-cloud-1?step=1) + diff --git a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md index e7004e893a0..d65f7a593a0 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md @@ -63,7 +63,7 @@ Unit tests are defined in YML files in your `models/` directory and are currentl #### New `data_tests:` syntax -The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](/docs/build/data-tests#new-test-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future. +The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](/docs/build/data-tests#new-data_tests-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future. ```yml diff --git a/website/docs/docs/deploy/job-notifications.md b/website/docs/docs/deploy/job-notifications.md index b4673eb4ad2..229d2b16f7f 100644 --- a/website/docs/docs/deploy/job-notifications.md +++ b/website/docs/docs/deploy/job-notifications.md @@ -43,7 +43,7 @@ You can receive email alerts about jobs by configuring the dbt Cloud email notif ## Slack notifications -You can receive Slack alerts about jobs by setting up the Slack integration, then configuring the dbt Cloud Slack notification settings. +You can receive Slack alerts about jobs by setting up the Slack integration, then configuring the dbt Cloud Slack notification settings. dbt Cloud integrates with Slack via OAuth to ensure secure authentication. :::note If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:support@getdbt.com) for assistance. @@ -52,16 +52,28 @@ If there has been a change in user roles or Slack permissions where you no longe ### Prerequisites - You must be an administrator of the Slack workspace. - You must be an account admin to configure Slack notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users). -- Make sure the notification channel (where you want to receive alerts) is a public channel. The integration only supports public channels in the Slack workspace. +- The integration only supports _public_ channels in the Slack workspace. ### Set up the Slack integration -1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section. -1. In the **Linked accounts** section, find the Slack application and click **Link**. +1. From the gear menu, select **Account settings** and then select **Integrations** from the left sidebar. +1. Locate the **OAuth** section with the Slack application and click **Link**. -1. Allow dbt Labs to access the Slack workspace. If you are a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. + +### Logged in to Slack +If you're already logged in to Slack, the handshake only requires allowing the app access. If you're a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. +## Logged out + +If you're logged out or the Slack app/website is closed, you must authenticate before completing the integration. + +1. Complete the field defining the Slack workspace you want to integrate with dbt Cloud. + +2. Sign in with an existing identity or use email address and password. +3. Once you have authenticated successfully, accept the permissions. + + ### Configure Slack notifications 1. From the gear menu, choose **Notification settings**. @@ -80,5 +92,5 @@ If there has been a change in user roles or Slack permissions where you no longe ### Disable the Slack integration -1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section. -1. Find the Slack application in the **Linked accounts** section, click the trash can icon, and click **Unlink**. Channels that you configured will no longer receive Slack notifications. Channels that are configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit the configuration. \ No newline at end of file +1. From the gear menu, select **Account settings**. On the **Integrations** page, scroll to the **OAuth** section. +1. Click the trash can icon (on the far right of the Slack integration) and click **Unlink**. Channels that you configured will no longer receive Slack notifications. _This is not an account-wide action._ Channels configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit their configuration. diff --git a/website/docs/guides/core-cloud-2.md b/website/docs/guides/core-cloud-2.md new file mode 100644 index 00000000000..fe2d084af03 --- /dev/null +++ b/website/docs/guides/core-cloud-2.md @@ -0,0 +1,198 @@ +--- +title: 'Move from dbt Core to dbt Cloud: What you need to know' +id: core-cloud-2 +description: "Use this guide to understand the considerations and methods you need to move from dbt Core to dbt Cloud." +hoverSnippet: "Use this guide to understand the considerations and methods you need to move from dbt Core to dbt Cloud." +icon: 'guides' +hide_table_of_contents: true +tags: ['Migration','dbt Core','dbt Cloud'] +keywords: ['dbt Core','dbt Cloud','Migration', 'Move dbt', 'Migrate dbt'] +level: 'Intermediate' +recently_updated: true +--- + +## Introduction +Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. + +Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics: + +import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md'; + + + +## What you'll learn +Today thousands of companies, with data teams ranging in size from 2 to 2,000, rely on dbt Cloud to accelerate data work, increase collaboration, and win the trust of the business. Understanding what you'll need to do in order to move between dbt Cloud and your current Core deployment will help you strategize and plan for your move. + +The guide outlines the following steps: + +- [Considerations](https://docs.getdbt.com/guides/core-cloud-2?step=3): Learn about the most important things you need to think about when moving from Core to Cloud. +- [Plan your move](https://docs.getdbt.com/guides/core-cloud-2?step=4): Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more. +- [Move to dbt Cloud](https://docs.getdbt.com/guides/core-cloud-2?step=5): Review the steps to move your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository. +- [Test and validate](https://docs.getdbt.com/guides/core-cloud-2?step=6): Discover how to ensure model accuracy and performance post-move. +- [Transition and training](https://docs.getdbt.com/guides/core-cloud-2?step=7): Learn how to fully transition to dbt Cloud and what training and support you may need. +- [Summary](https://docs.getdbt.com//guides/core-cloud-2?step=8): Summarizes key takeaways and what you've learned in this guide. +- [What's next?](https://docs.getdbt.com/guides/core-cloud-2?step=9): Introduces what to expect in the following guides. + +## Considerations + +If your team has is using dbt Core today, you could be reading this guide because: +- You’ve realized the burden of maintaining that deployment. +- The person who set it up has since left. +- You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. + +This guide shares the technical adjustments and team collaboration strategies you’ll need to know to move your project from dbt Core to dbt Cloud. Each "build your own" deployment of dbt Core will look a little different, but after seeing hundreds of teams make the migration, there are many things in common. + +The most important things you need to think about when moving from dbt Core to dbt Cloud: + +- How is your team structured? Are there natural divisions of domain? +- Should you have one project or multiple? Which dbt resources do you want to standardize & keep central? +- Who should have permissions to view, develop, administer? +- How are you scheduling your dbt models to run in production? +- How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)? +- How do your data developers prefer to work? +- How do you manage different data environments and the different behaviors in those environments? + +dbt Cloud provides standard mechanisms for tackling these considerations, all of which delivers long-term benefits to your organization: +- Cross-team collaboration +- Access control +- Orchestration +- Isolated data environments + +If you have rolled out your own dbt Core deployment, you have probably come up with different answers. + +## Plan your move + +As you plan your move, consider your workflow and team layout to ensure a smooth transition. Here are some key considerations to keep in mind: + + + +You don’t need to move every team and every developer’s workflow all at once. Many customers with large dbt deployments start by moving one team and one project. + +Once the benefits of a consolidated platform are clear, move the rest of your teams and workflows. While long-term ‘hybrid’ deployments can be challenging, it may make sense as a temporary on-ramp. + + + + +Assess the users or personas involved in the pre-move, during the move, and post-move. +- **Administrators**: Plan for new [access controls](/docs/cloud/manage-access/about-user-access) in dbt Cloud, such as deciding what teams can manage themselves and what should be standardized. Determine who will be responsible for setting up and maintaining projects, data platform connections, and environments. +- **Data developers** (data analysts, data engineers, analytics engineers, business analysts): Determine onboarding order, workflow adaptation in dbt Cloud, training on [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) usage, and role changes. +- **Data consumers:** Discover data insights by using [dbt Explorer](/docs/collaborate/explore-projects) to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. + + + + + +If you have multiple teams of dbt developers, think about how to start your onboarding sequence for dbt Cloud: +- Start with downstream (like business-embedded teams) who may benefit from the dbt Cloud IDE as dev experience (less technical users) and sharing features (like auto-deferral and dbt Explorer) to share with their stakeholders, moving to more technical teams later. +- Consider setting up a [CI job](/docs/deploy/ci-jobs) in dbt Cloud (even before development or production jobs) to streamline development workflows. This is especially beneficial if there's no existing CI process. + + + + + +Discover how dbt Cloud can help simplify development, orchestration, and testing: +- **Development**: Develop dbt models, allowing you to build, test, run, and version control your dbt projects using the dbt Cloud CLI (command line interface or code editor) or dbt Cloud IDE (browser-based). +- **Orchestration**: Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval. + - Set up [a CI job](/docs/deploy/ci-jobs) to ensure developer effectiveness, and CD jobs to deploy changes as soon as they’re merged. + - Link deploy jobs together by [triggering a job](/docs/deploy/deploy-jobs#trigger-on-job-completion) when another one is completed. + - For the most flexibility, use the [dbt Cloud API](https://docs.getdbt.com/dbt-cloud/api-v2#/) to trigger jobs. This makes sense when you want to integrate dbt execution with other data workflows. +- **Continuous integration (CI)**: Use [CI jobs](/docs/deploy/ci-jobs) to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. This build-on-PR functionality is a great way to catch bugs before deploying to production. + - For many teams, dbt Cloud CI represents a major improvement compared to their previous development workflows. +- **How are you defining tests today?**: While testing production data is important, it’s not the most efficient way to catch logical errors introduced by developers You can use [unit testing](/docs/build/unit-tests) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. + + + + + +Transition to dbt Cloud's [access control](/docs/cloud/manage-access/about-user-access) mechanisms to ensure security and proper access management. dbt Cloud administrators can use dbt Cloud's permission model to control user-level access in a dbt Cloud account: +- **License-based access controls:** Users are configured with account-wide license types. These licenses control the things a user can do within the application: view project metadata, develop changes within those projects, or administer access to those projects. +- **Role-based Access Control (RBAC):** Users are assigned to *groups* with specific permissions on specific projects or all projects in the account. A user may be a member of multiple groups, and those groups may have permissions on multiple projects. + + + + + +If you require isolation between production and pre-production data environments due to sensitive data, dbt Cloud can support Development, Staging, and Production data [environments](/docs/dbt-cloud-environments). + +This provides developers with the benefits of an enhanced workflow while ensuring isolation between Staging and Production data, and locking down permissions on Prod. + + + +## Move to dbt Cloud + +This guide is your roadmap to help you think about migration strategies and what moving from dbt Core to dbt Cloud could look like. + +After reviewing the considerations and planning your move, you may want to start moving your dbt Core project to dbt Cloud: +- Check out the detailed [Move to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) guide for useful tasks and insights for a smooth transition from dbt Core to dbt Cloud. + +For a more detailed comparison of dbt Core and dbt Cloud, check out [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud). + +## Test and validate + +After [setting the foundations of dbt Cloud](https://docs.getdbt.com/guides/core-to-cloud-1?step=1), it's important to validate your migration to ensure seamless functionality and data integrity: + +- **Review your dbt project:** Ensure your project compiles correctly and that you can run commands. Make sure your models are accurate and monitor performance post-move. +- **Start cutover:** You can start the cutover to dbt Cloud by creating a dbt Cloud job with commands that only run a small subset of the DAG. Validate the tables are being populated in the proper database/schemas as expected. Then continue to expand the scope of the job to include more sections of the DAG as you gain confidence in the results. +- **Precision testing:** Use [unit testing](/docs/build/unit-tests) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. +- **Access and permissions**: Review and adjust [access controls and permissions](/docs/cloud/manage-access/about-user-access) within dbt Cloud to maintain security protocols and safeguard your data. + +## Transition and training + +Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out `dbt-core` execution for dbt Cloud execution (through the [API](/docs/dbt-cloud-apis/overview)). + +Familiarize your team with dbt Cloud's [features](/docs/cloud/about-cloud/dbt-cloud-features) and optimize development and deployment processes. Some key features to consider include: +- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest fixes and early access to new functionality for your dbt project. +- **Development tools**: Use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to build, test, run, and version control your dbt projects. +- **Documentation and Source freshness:** Automate storage of [documentation](/docs/collaborate/documentation) and track [source freshness](/docs/deploy/source-freshness) in dbt Cloud, which streamlines project maintenance. +- **Notifications and logs:** Receive immediate [notifications](/docs/deploy/monitor-jobs) for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting. +- **CI/CD:** Use dbt Cloud's [CI/CD](/docs/deploy/ci-jobs) feature to run your dbt projects in a temporary schema whenever new commits are pushed to open pull requests. This helps with catching bugs before deploying to production. + +### Beyond your move + +Now that you’ve chosen dbt Cloud as your platform, you’ve unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful [features](/docs/cloud/about-cloud/dbt-cloud-features) for analytics engineering teams. Here are some additional features you can use to unlock the full potential of dbt Cloud: + +- **Audit logs:** Use [audit logs](/docs/cloud/manage-access/audit-log) to review actions performed by people in your organization. Audit logs contain audited user and system events in real time. You can even [export](/docs/cloud/manage-access/audit-log#exporting-logs) *all* the activity (beyond the 90 days you can view in dbt Cloud). +- **dbt Cloud APIs:** Use dbt Cloud's robust [APIs](/docs/dbt-cloud-apis/overview) to create, read, update, and delete (CRUD) projects/jobs/environments project. The [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage. While the [Discovery API](/docs/dbt-cloud-apis/discovery-api) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health. +- **dbt Explorer**: Use [dbt Explorer](/docs/collaborate/explore-projects) to view your project's [resources](/docs/build/projects) (such as models, tests, and metrics) and their [lineage](https://docs.getdbt.com/terms/data-lineage) to gain a better understanding of its latest production state. (Once you have a successful job in a Production environment). +- **dbt Semantic Layer:** The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) allows you to define universal metrics on top of your models that can then be queried in your [business intelligence (BI) tool](/docs/use-dbt-semantic-layer/avail-sl-integrations). This means no more inconsistent metrics — there’s now a centralized way to define these metrics and create visibility in every component of the data flow. +- **dbt Mesh:** Use [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) to share data models across organizations, enabling data teams to collaborate on shared data models and leverage the work of other teams. + +### Additional help + +- **dbt Learn courses**: Access our free [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand training. +- **dbt Community:** Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. +- **dbt Support team:** Our [dbt Support team](/docs/dbt-support) is always available to help you troubleshoot your dbt Cloud issues. Create a support ticket in dbt Cloud and we’ll be happy to help! +- **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. + +## Summary + +This guide should now have given you some insight and equipped you with a framework for moving from dbt Core to dbt Cloud. This guide has covered the following key areas: + +- **Considerations:** Understanding the foundational steps required for a successful migration, including evaluating your current setup and identifying key considerations unique to your team's structure and workflow needs. + +- **Plan you move**: Highlighting the importance of workflow redesign, role-specific responsibilities, and the adoption of new processes to harness dbt Cloud's collaborative and efficient environment. + +- **Move to dbt Cloud**: Linking to [the guide](https://docs.getdbt.com/guides/core-to-cloud-1?step=1) that outlines technical steps required to transition your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository. + +- **Test and validate**: Emphasizing technical transitions, including testing and validating your dbt projects within the dbt Cloud ecosystem to ensure data integrity and performance. + +- **Transition and training**: Share useful transition, training, and onboarding information for your team. Fully leverage dbt Cloud's capabilities, from development tools (dbt Cloud CLI and dbt Cloud IDE) to advanced features such as dbt Explorer, the Semantic Layer, and dbt Mesh. + +## What’s next? + + + +Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to dbt Cloud. + +For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud: + + + +### Related content +- [Learn dbt Cloud](https://courses.getdbt.com/collections) courses +- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights +- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. +- [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud. +- Subscribe to the [dbt Cloud RSS alerts](https://status.getdbt.com/) + + diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md index f2ee7b016e2..2c2561b125d 100644 --- a/website/docs/guides/core-to-cloud-1.md +++ b/website/docs/guides/core-to-cloud-1.md @@ -7,6 +7,7 @@ icon: 'guides' time_to_complete: 'Total estimated time: 3-4 hours' hide_table_of_contents: true tags: ['Migration','dbt Core','dbt Cloud'] +keywords: ['dbt Core','dbt Cloud','Migration', 'Move dbt', 'Migrate dbt'] level: 'Intermediate' recently_updated: true --- @@ -17,42 +18,48 @@ recently_updated: true Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. +Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. This series is ideal for users aiming for streamlined workflows and enhanced analytics: + +import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md'; + + + + dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports: -- The [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or command line with [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) for development -- [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) for consistent metrics +- Development experiences tailored to multiple personas ([dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [dbt Cloud CLI](/docs/cloud/cloud-cli-installation)) +- Out-of-the-box [CI/CD workflows](/docs/deploy/ci-jobs) +- The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) for consistent metrics - Domain ownership of data with multi-project [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) setups - [dbt Explorer](/docs/collaborate/explore-projects) for easier data discovery and understanding Learn more about [dbt Cloud features](/docs/cloud/about-cloud/dbt-cloud-features). -dbt Core is an open-source tool that enables data teams to transform data following analytics engineering best practices using a command line interface. It must be self-hosted and maintained. +dbt Core is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for ‘single players’ and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale. -### What you'll learn +## What you'll learn This guide outlines the steps you need to take to move from dbt Core to dbt Cloud and highlights the necessary technical changes: -- [Account setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=3#account-setup): Learn how to create a dbt Cloud account, invite team members, and configure it for your team. -- [Data platform setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=4#data-platform-setup): Find out about connecting your data platform to dbt Cloud. -- [Git setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=5#git-setup): Learn to link your dbt project's Git repository with dbt Cloud. -- [Developer setup:](https://docs.getdbt.com/guides/core-to-cloud-1?step=6#developer-setup) Understand the setup needed for developing in dbt Cloud. -- [Environment variables](https://docs.getdbt.com/guides/core-to-cloud-1?step=7#environment-variables): Discover how to manage environment variables in dbt Cloud, including their priority. -- [Orchestration setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=8#orchestration-setup): Learn how to prepare your dbt Cloud environment and jobs for orchestration. -- [Models configuration](https://docs.getdbt.com/guides/core-to-cloud-1?step=9#models-configuration): Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI. -- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-1?step=10#whats-next): Summarizes key takeaways and introduces what to expect in the following guides. +- [Account setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=4): Learn how to create a dbt Cloud account, invite team members, and configure it for your team. +- [Data platform setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=5): Find out about connecting your data platform to dbt Cloud. +- [Git setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=6): Learn to link your dbt project's Git repository with dbt Cloud. +- [Developer setup:](https://docs.getdbt.com/guides/core-to-cloud-1?step=7) Understand the setup needed for developing in dbt Cloud. +- [Environment variables](https://docs.getdbt.com/guides/core-to-cloud-1?step=8): Discover how to manage environment variables in dbt Cloud, including their priority. +- [Orchestration setup](https://docs.getdbt.com/guides/core-to-cloud-1?step=9): Learn how to prepare your dbt Cloud environment and jobs for orchestration. +- [Models configuration](https://docs.getdbt.com/guides/core-to-cloud-1?step=10): Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI. +- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-1?step=11): Summarizes key takeaways and introduces what to expect in the following guides. + +### Related docs +- [Learn dbt Cloud](https://courses.getdbt.com/collections) on-demand video learning. +- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights +- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. ## Prerequisites - You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections). -- You are using dbt version 1.0 or later. +- A [supported version](/docs/dbt-versions/core) of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt. - You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**! -### Related docs -- [Learn dbt Cloud](https://courses.getdbt.com/collections) -- [Develop with dbt Cloud](/docs/cloud/about-develop-dbt) -- [Deploy jobs](/docs/deploy/deployments) -- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights -- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. - ## Account setup This section outlines the steps to set up your dbt Cloud account and configure it for your team. @@ -64,9 +71,12 @@ This section outlines the steps to set up your dbt Cloud account and configure i 3. Configure [Single Sign-On (SSO)](/docs/cloud/manage-access/sso-overview) or [Role-based access control (RBAC)](/docs/cloud/manage-access/about-user-access#role-based-access-control) for easy and secure access. - This removes the need to save passwords and secret environment variables locally. -4. In **Account settings**, enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) to only reparse changed files, saving time. +### Additional configuration +Explore these additional configurations for performance and reliability improvements: + +1. In **Account settings**, enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) to only reparse changed files, saving time. -5. In **Account settings**, enable [Git repo caching](/docs/deploy/deploy-environments#git-repository-caching) for job reliability & third-party outage protection. +2. In **Account settings**, enable [Git repo caching](/docs/deploy/deploy-environments#git-repository-caching) for job reliability & third-party outage protection. ## Data platform setup @@ -97,8 +107,10 @@ Your existing dbt project source code should live in a Git repository. In this s 1. Ensure your dbt project is in a Git repository. -2. In **Account settings**, select **Integrations**, and [connect and configure Git](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud to connect your Git repository: - - Connect with one of the [native integrations](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud (such as GitHub, GitLab, and Azure DevOps). +2. In **Account settings**, select **Integrations** to [connect your Git repository](/docs/cloud/git/git-configuration-in-dbt-cloud) to dbt Cloud: + - (**Recommended**) Connect with one of the [native integrations](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud (such as GitHub, GitLab, and Azure DevOps). + + This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use. - [Import a Git repository](/docs/cloud/git/import-a-project-by-git-url) from any valid Git URL that points to a dbt project. ### Additional configuration @@ -119,17 +131,20 @@ This section highlights the development configurations you’ll need for your db ### dbt Cloud environments -The concept of an [environment](/docs/environments-in-dbt) in dbt Cloud is the same as a `target` in dbt Core. - -The primary difference between a dbt Cloud environment and a `target` in dbt Core is that you can make these configurations through the dbt Cloud UI instead of within the `profiles.yml` file. +The most common data environments are production, staging, and development. The way dbt Core manages [environments](/docs/environments-in-dbt) is through `target`, which are different sets of connection details. -This difference streamlines the process of switching between development, staging, and production contexts, removing the need to manually edit the `profiles.yml` file. dbt Cloud environments also integrate with additional features such as job scheduling, version control, and more — making it easier to manage the full lifecycle of your dbt projects within a single platform. You can [set up](/reference/dbt-jinja-functions/target) or [customize](/docs/build/custom-target-names) target names in dbt Cloud. +[dbt Cloud environments](/docs/dbt-cloud-environments) go further by: +- Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform. +- Streamlining the process of switching between development, staging, and production contexts. +- Making it easy to configure environments through the dbt Cloud UI instead of manually editing the `profiles.yml` file. You can also [set up](/reference/dbt-jinja-functions/target) or [customize](/docs/build/custom-target-names) target names in dbt Cloud. +- Adding `profiles.yml` attributes to dbt Cloud environment settings with [Extended Attributes](/docs/dbt-cloud-environments#extended-attributes). +- Using [Git repo caching](/docs/dbt-cloud-environments#git-repository-caching) to protect you from third-party outages, Git auth failures, and more. ### Initial setup steps 1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing. 2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using. - - Once your full migration is complete, consider upgrading your environments to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) to always get the latest features and more. + - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) so you don’t have to. 3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI. - Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project. @@ -137,10 +152,11 @@ This difference streamlines the process of switching between development, stagin ### Additional configuration Explore these additional configurations to optimize your developer setup further: -1. **Custom target names** — If you’re using a [`custom target.name`](/reference/dbt-jinja-functions/target) in your project, we recommend you set them using [environment variables](/docs/build/environment-variables). Alternatively, you can update it at the developer credentials level. + +1. **Custom target names** — Using [`custom target.names`](/docs/build/custom-target-names) in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the `custom target.name` values in your developer credentials or orchestration setup, we recommend using [environment variables](/docs/build/environment-variables) as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using [`{{ target }}` logic](/reference/dbt-jinja-functions/target) which is meant for defining the data warehouse connection. ### dbt Cloud commands -1. Review the [dbt commands](/reference/dbt-commands) supported for dbt Cloud development. For example, `dbt debug` isn’t needed in dbt Cloud since the UI displays logs for your viewing. +1. Review the [dbt commands](/reference/dbt-commands) supported for dbt Cloud development. For example, `dbt init` isn’t needed in dbt Cloud as you can create a new project directly in dbt Cloud. ## Environment variables This section will help you understand how to set up and manage dbt Cloud environment variables for your project. The following categories are covered: @@ -148,13 +164,14 @@ This section will help you understand how to set up and manage dbt Cloud environ - [dbt Cloud environment variables order of precedence](/guides/core-to-cloud-1?step=7#dbt-cloud-environment-variables-order-of-precedence) - [Set environment variables in dbt Cloud](/guides/core-to-cloud-1?step=7#set-environment-variables-in-dbt-cloud) -In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined in the `profiles.yml` file. In dbt Cloud, you can set [environment variables](/docs/build/environment-variables) in the dbt Cloud user interface (UI). Read [Set up environment variables](#set-environment-variables-in-dbt-cloud) for more info. +In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined manually by the developer or within the external application running dbt. + ### Environment variables in dbt Cloud - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_`). - If your dbt Core environment variables don’t follow this naming convention, perform a [“find and replace”](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#dbt-cloud-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. -- dbt Cloud secures environment variables, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_` to obscure them in logs and the UI. +- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_` to obscure them in logs and the UI. @@ -189,7 +206,7 @@ To use the [dbt Cloud's job scheduler](/docs/deploy/job-scheduler), set up one e 1. **dbt Core version** — In your environment settings, configure dbt Cloud with the same dbt Core version. - Once your full migration is complete, we recommend upgrading your environments to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) to always get the latest features and more. You only need to do this once. -2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for automated or event-driven dbt jobs. You can use cron execution, manual, pull requests, or API triggers. +2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job. - Note that alongside [jobs in dbt Cloud](/docs/deploy/jobs), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](/docs/deploy/deployment-tools) for more information. ### Additional configuration @@ -209,21 +226,21 @@ Explore these additional configurations to optimize your dbt Cloud orchestration ### CI/CD setup -Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable [continuous integration / continuous deployment (CI/CD)](/docs/deploy/continuous-integration) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. +Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable [continuous integration / continuous deployment (CI/CD)](/docs/deploy/continuous-integration) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners. 1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt Cloud. -2. Create [a CI/CD job](/docs/deploy/ci-jobs) to optimize workflows. +2. Create [a CI/CD job](/docs/deploy/ci-jobs) to automate quality checks before code is deployed to production. 3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against. -## Models configuration +## Model development and discovery In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [dbt Cloud CLI](/docs/cloud/cloud-cli-installation). You’ll want to make sure you set up your [development environment and credentials](/docs/dbt-cloud-environments#set-developer-credentials). -1. In your [development tool](/docs/cloud/about-develop-dbt) of choice, you can review your dbt project, ensure it's set up correctly, and that you can run dbt commands. For example: +1. In your [development tool](/docs/cloud/about-develop-dbt) of choice, you can review your dbt project, ensure it's set up correctly, and run some [dbt commands](/reference/dbt-commands): - Run `dbt compile` to make sure your project compiles correctly. - Run a few models in the dbt Cloud IDE or dbt Cloud CLI to ensure you’re experiencing accurate results in development. @@ -239,22 +256,21 @@ You have learned: - How to set up your dbt Cloud account - How to connect your data platform and Git repository - How to configure your development, orchestration, and CI/CD environments -- How to set up environment variables and validate your models. +- How to set up environment variables and validate your models + -For next steps, we'll soon share other guides on how to manage your move and tips/faqs. Stay tuned! +For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud: - + + ### Related docs -- [Learn dbt Cloud](https://courses.getdbt.com/collections) -- [Develop with dbt Cloud](/docs/cloud/about-develop-dbt) -- [Deploy jobs](/docs/deploy/deployments) -- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights +- [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand learning. +- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights. - Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. +- [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud. +- Subscribe to the [dbt Cloud RSS alerts](https://status.getdbt.com/) - \ No newline at end of file + diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md new file mode 100644 index 00000000000..d343ef6b64d --- /dev/null +++ b/website/docs/guides/core-to-cloud-3.md @@ -0,0 +1,202 @@ +--- +title: 'Move from dbt Core to dbt Cloud: Optimization tips' +id: core-to-cloud-3 +description: "Use this guide to learn how to optimize your dbt Cloud experience and get answers to common questions." +hoverSnippet: "Use this guide to learn how to optimize your dbt Cloud experience and get answers to common questions." +icon: 'guides' +hide_table_of_contents: true +tags: ['Migration','dbt Core','dbt Cloud'] +keywords: ['dbt Core','dbt Cloud','Migration', 'Move dbt', 'Migrate dbt'] +level: 'Intermediate' +recently_updated: true +--- + +## Introduction + +Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. + +Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics: + +import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md'; + + + +## What you'll learn +You may have already started your move to dbt Cloud and are looking for tips to help you optimize your dbt Cloud experience. This guide includes tips and caveats for the following areas: + +- [Adapters and connections](https://docs.getdbt.com/guides/core-to-cloud-3?step=3) +- [Development tools](https://docs.getdbt.com/guides/core-to-cloud-3?step=4) +- [Orchestration](https://docs.getdbt.com/guides/core-to-cloud-3?step=5) +- [dbt Mesh](https://docs.getdbt.com/guides/core-to-cloud-3?step=6) +- [dbt Semantic Layer](https://docs.getdbt.com/guides/core-to-cloud-3?step=7) +- [dbt Explorer](https://docs.getdbt.com/guides/core-to-cloud-3?step=8) + +## Adapters and connections + +In dbt Cloud, you can natively connect to your data platform and test its [connection](/docs/connect-adapters) with a click of a button. This is especially useful for users who are new to dbt Cloud or are looking to streamline their connection setup. Here are some tips and caveats to consider: + +### Tips +- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, eliminating the need for manual updates and version discrepancies. You can **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest fixes and early access to new functionality for your dbt project. +- dbt Cloud supports a whole host of [cloud providers](/docs/cloud/connect-data-platform/about-connections), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). +- Use [Extended Attributes](/docs/deploy/deploy-environments#extended-attributes) to set a flexible [profiles.yml](/docs/core/connect-data-platform/profiles.yml) snippet in your dbt Cloud environment settings. It gives you more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment. + - For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt Cloud adapter user interface, you can use Extended Attributes to set it. + +### Caveats +- Not all parameters are available for adapters. +- A project can only use one warehouse type. + +## Development tools + +dbt Cloud empowers data practitioners to develop in the tool of their choice. It ships with a [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) (local) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) (browser-based) to build, test, run, and version control your dbt projects. + +Both development tools are tailored to suit different audiences and preferences within your team. To streamline your team’s workflow, it's important to know who will prefer the dbt Cloud IDE and who might lean towards the dbt Cloud CLI. This section aims to clarify these preferences. + +### dbt Cloud IDE +A web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. The dbt Cloud IDE makes developing fast and easy for new and seasoned data practitioners to build and test changes. + +**Who might prefer the dbt Cloud IDE?** + +- New dbt users or those transitioning from other tools who appreciate a more guided experience through a browser-based interface. +- Team members focused on speed and convenience for getting started with a new or existing project. +- Individuals who prioritize direct feedback from the IDE, such as seeing unsaved changes. + +**Key features** + +- The dbt Cloud IDE has simplified Git functionality: + - Create feature branches from the branch configured in the development environment. + - View saved but not-committed code changes directly in the IDE. +- [Format or lint](/docs/cloud/dbt-cloud-ide/lint-format) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules. +- Allows users to natively [defer to production](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows, reducing the number of objects. +- Support running multiple dbt commands at the same time through [safe parallel execution](/reference/dbt-commands#parallel-execution), a [feature](/docs/cloud/about-cloud/dbt-cloud-features) available in dbt Cloud's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process. + +The dbt Cloud IDE provides a simplified interface that's accessible to all users, regardless of their technical background. However, there are some capabilities that are intentionally not available in the dbt Cloud IDE due to its focus on simplicity and ease of use: + +- Pre-commit for automated checks before *committing* code is not available (yet). +- Mass-generating files / interacting with the file system are not available. +- Combining/piping commands, such as `dbt run -s (bash command)`, is not available. + +### dbt Cloud CLI +The dbt Cloud CLI allows you to run dbt [commands](/reference/dbt-commands#available-commands) against your dbt Cloud development environment from your local command line. For users who seek full control over their development environment and ideal for those comfortable with the command line. + +When moving from dbt Core to dbt Cloud, make sure you check the `.gitignore` file contains the [necessary folders](/docs/collaborate/git/version-control-basics#the-gitignore-file). dbt Core doesn't interact with git so dbt Cloud doesn't automatically add or verify entries in the `.gitignore` file. Additionally, if the repository already contains dbt code and doesn't require initialization, dbt Cloud won't add any missing entries to the `.gitignore file`. + +**Who might prefer the dbt Cloud CLI?** + +- Data practitioners accustomed to working with a specific set of development tooling. +- Users looking for granular control over their Git workflows (such as pre-commits for automated checks before committing code). +- Data practitioners who need to perform complex operations, like mass file generation or specific command combinations. + +**Key features** + +- Allows users to run dbt commands against their dbt Cloud development environment from their local command line with minimal configuration. +- Allows users to natively [defer to production](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows, reducing the number of objects. +- Support running multiple dbt commands at the same time through [safe parallel execution](/reference/dbt-commands#parallel-execution), a [feature](/docs/cloud/about-cloud/dbt-cloud-features) available in dbt Cloud's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process. +- Able to use Visual Studio (VS) Code extensions (such as [dbt-power-user](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user) to enhance the development experience by adding extra functionalities. + +## Orchestration + +dbt Cloud provides robust orchestration that enables you to schedule, run, and monitor dbt jobs with ease. Here are some tips and caveats to consider when using dbt Cloud's orchestration features: + +### Tips + +- Enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) between jobs in dbt Cloud to significantly speed up project parsing by only processing changed files, optimizing performance for large projects. +- [Run multiple CI/CD](/docs/deploy/continuous-integration) jobs at the same time which will not block production runs. The Job scheduler automatically cancels stale runs when a newer commit is pushed. This is because each PR will run in its own schema. +- dbt Cloud automatically [cancels](/docs/deploy/job-scheduler#run-cancellation-for-over-scheduled-jobs) a scheduled run if the existing run is still executing. This prevents unnecessary, duplicative executions. +- Protect you and your data freshness from third-party outages by enabling dbt Cloud’s [Git repository caching](/docs/deploy/deploy-environments#git-repository-caching), which keeps a cache of the project's Git repository. +- [Link deploy jobs](/docs/deploy/deploy-jobs#trigger-on-job-completion--) across dbt Cloud projects by configuring your job or using the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this. +- [Rerun your jobs](/docs/deploy/retry-jobs) from the start or the point of failure if your dbt job run completed with a status of **`Error.`** + +### Caveats +- To automate the setup and configuration of your dbt Cloud platform, you can store your job configurations as code within a repository: + - Check out our [Terraform provider.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) + - Alternatively, check out our [jobs-as-code](https://github.com/dbt-labs/dbt-jobs-as-code) repository, which is a tool built to handle dbt Cloud jobs as a well-defined YAML file. +- dbt Cloud users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](/guides/zapier-slack) and post to Slack. + +## dbt Mesh + +[dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. It allows you to make use of multiple interconnected dbt projects instead of a single large, monolithic project. + +It enables you to interface and navigate between different projects and models with [cross-project dependencies](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), enhancing collaboration and data governance. + +Here are some tips and caveats to consider when using dbt Mesh: + +### Tips +- To dynamically resolve [cross-project references](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), all developers need to develop with dbt Cloud (either with the dbt Cloud CLI or dbt Cloud IDE). Cross-project references aren't natively supported in dbt Core, except by installing the source code from upstream projects [as packages](/docs/build/packages#how-do-i-add-a-package-to-my-project) +- Link models across projects for a modular and scalable approach for your project and teams. +- Manage access to your dbt models both within and across projects using: + - **[Groups](/docs/collaborate/govern/model-access#groups)** — Organize nodes in your dbt DAG that share a logical connection and assign an owner to the entire group. + - **[Model access](/docs/collaborate/govern/model-access#access-modifiers)** — Control which other models or projects can reference this model. + - **[Model versions](/docs/collaborate/govern/model-versions)** — Enable adoption and deprecation of models as they evolve. + - **[Model contracts](/docs/collaborate/govern/model-contracts)** — Set clear expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. +- Use [dbt-meshify](https://github.com/dbt-labs/dbt-meshify) to accelerate splitting apart your monolith into multiple projects. + +### Caveats +- To use cross-project references in dbt, each dbt project must correspond to just one dbt Cloud project. We strongly discourage defining multiple projects for the same codebase, even if you're trying to manage access permissions, connect to different data warehouses, or separate production and non-production data. While this was required historically, features like [Staging environments](/docs/dbt-cloud-environments#types-of-environments), Environment-level RBAC (_coming soon_), and [Extended attributes](/docs/dbt-cloud-environments#extended-attributes) will make it unnecessary. +- Project dependencies are uni-directional, meaning they go in one direction. This means dbt checks for cycles across projects (circular dependencies) and raise errors if any are detected. However, we are considering support to allow projects to depend on each other in both directions in the future, with dbt still checking for node-level cycles while allowing cycles at the project level. +- Everyone in the account can view public model metadata, which helps users find data products more easily. This is separate from who can access the actual data, which is controlled by permissions in the data warehouse. For use cases where even metadata about a reusable data asset is sensitive, we are [considering](https://github.com/dbt-labs/dbt-core/issues/9340) an optional extension of protected models. + +Refer to the [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-4-faqs) for more questions. + +## dbt Semantic Layer + +Leverage the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by MetricFlow, to create a unified view of your business metrics, ensuring consistency across all analytics tools. Here are some tips and caveats to consider when using dbt Semantic Layer: + +### Tips +- Define semantic models and metrics once in dbt Cloud with the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) (powered by MetricFlow). Reuse them across various analytics platforms, reducing redundancy and errors. +- Use the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) to query metrics in downstream tools for consistent, reliable data metrics. +- Connect to several data applications, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more, to query your metrics. [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) include Tableau, Google Sheets, Hex, and more. +- Use [exports](/docs/use-dbt-semantic-layer/exports) to write commonly used queries directly within your data platform, on a schedule. + +### Caveats +- dbt Semantic Layer currently supports the Deployment environment for querying. Development querying experience coming soon. +- Run queries/semantic layer commands in the dbt Cloud CLI, however running queries/semantic layer commands in the dbt Cloud IDE isn’t supported *yet.* +- dbt Semantic Layer doesn't yet support SSH tunneling for [Postgres or Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections. It also doesn't support using [Single sign-on (SSO)](/docs/cloud/manage-access/sso-overview) for Semantic Layer [production credentials](/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens), however, SSO is supported for development user accounts. + +Refer to the [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs) for more information. + +## dbt Explorer + +[dbt Explorer](/docs/collaborate/explore-projects) enhances your ability to discover, understand, and troubleshoot your data assets through rich metadata and lineage visualization. Here are some tips and caveats to consider when using dbt Explorer: + +### Tips +- Use the search and filter capabilities in dbt Explorer to quickly locate models, sources, and tests, streamlining your workflow. +- View all the [different projects](/docs/collaborate/explore-multiple-projects) and public models in the account, where the public models are defined, and how they are used to gain a better understanding of your cross-project resources. +- Use the [Lenses](/docs/collaborate/explore-projects#lenses) feature, which are map-like layers for your DAG, available from your project's lineage graph. Lenses help you further understand your project’s contextual metadata at scale, especially to distinguish a particular model or a subset of models. +- Access column-level lineage (CLL) for the resources in your dbt project. + +### Caveats +- There must be at least one successful job run in the production deployment environment for Explorer to populate information. + +Familiarize yourself with dbt Explorer’s features to fully leverage its capabilities to avoid missed opportunities for efficiency gains. + +Refer to the [dbt Explorer FAQs](/docs/collaborate/dbt-explorer-faqs) for more information. + +## What's next? + + + +Congratulations on making it through the guide 🎉! + +We hope you’re equipped with useful insights and tips to help you with your move. Something to note is that moving from dbt Core to dbt Cloud isn’t just about evolving your data projects, it's about exploring new levels of collaboration, governance, efficiency, and innovation within your team. + +For the next steps, continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud: + + + + + +### Resources + +If you need any additional help or have some questions, use the following resources: + +- [dbt Cloud learn courses](https://courses.getdbt.com/collections) for on-demand video learning. +- Our [Support team](https://docs.getdbt.com/docs/dbt-support) is always available to help you troubleshoot your dbt Cloud issues. +- Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. +- Subscribe to the [dbt Cloud RSS alerts](https://status.getdbt.com/) +- Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. +- [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud. + +For tailored assistance, you can use the following resources: + +- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights +- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and move. diff --git a/website/docs/guides/manual-install-qs.md b/website/docs/guides/manual-install-qs.md index 513a402e3ac..b68c43db42d 100644 --- a/website/docs/guides/manual-install-qs.md +++ b/website/docs/guides/manual-install-qs.md @@ -28,9 +28,6 @@ If you want to edit files and run projects using the web-based dbt Integrated De After setting up BigQuery to work with dbt, you are ready to create a starter project with example models, before building your own models. -### Related docs -- [Move from dbt Core to dbt Cloud](/guides/core-to-cloud-1?step=1) - ## Create a repository The following steps use [GitHub](https://github.com/) as the Git provider for this guide, but you can use any Git provider. You should have already [created a GitHub account](https://github.com/join). @@ -475,4 +472,4 @@ For more info on how to get started, refer to [create and schedule jobs](/docs/d For more information about using dbt Core to schedule a job, refer [dbt airflow](/blog/dbt-airflow-spiritual-alignment) blog post. - \ No newline at end of file + diff --git a/website/docs/reference/dbt-jinja-functions/cross-database-macros.md b/website/docs/reference/dbt-jinja-functions/cross-database-macros.md index 334bcfe5760..e13dba0bdec 100644 --- a/website/docs/reference/dbt-jinja-functions/cross-database-macros.md +++ b/website/docs/reference/dbt-jinja-functions/cross-database-macros.md @@ -54,6 +54,7 @@ Please make sure to take a look at the [SQL expressions section](#sql-expression - [bool\_or](#bool_or) - [listagg](#listagg) - [Cast functions](#cast-functions) + - [cast](#cast) - [cast\_bool\_to\_text](#cast_bool_to_text) - [safe\_cast](#safe_cast) - [Date and time functions](#date-and-time-functions) @@ -101,6 +102,7 @@ Please make sure to take a look at the [SQL expressions section](#sql-expression - [bool\_or](#bool_or) - [listagg](#listagg) - [Cast functions](#cast-functions) + - [cast](#cast) - [cast\_bool\_to\_text](#cast_bool_to_text) - [safe\_cast](#safe_cast) - [Date and time functions](#date-and-time-functions) @@ -169,6 +171,7 @@ Please make sure to take a look at the [SQL expressions section](#sql-expression - [listagg](#listagg) [**Cast functions**](#cast-functions) +- [cast](#cast) - [cast_bool_to_text](#cast_bool_to_text) - [safe_cast](#safe_cast) @@ -777,6 +780,38 @@ array_to_string( ## Cast functions +### cast + +**Availability**: +dbt v1.8 or higher. For more information, select the version from the documentation navigation menu. + + + +__Args__: + + * `field`: [attribute name or expression](#sql-expressions). + * `type`: data type to convert to + +This macro casts a value to the specified data type. Unlike [safe\_cast](#safe_cast), this macro will raise an error when the cast fails. + +**Usage**: + +```sql +{{ dbt.cast("column_1", api.Column.translate_type("string")) }} +{{ dbt.cast("column_2", api.Column.translate_type("integer")) }} +{{ dbt.cast("'2016-03-09'", api.Column.translate_type("date")) }} +``` + +**Sample Output (PostgreSQL)**: + +```sql + cast(column_1 as TEXT) + cast(column_2 as INT) + cast('2016-03-09' as date) +``` + + + ### cast_bool_to_text __Args__: diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md index ed0b563731d..3711ea79314 100644 --- a/website/docs/reference/node-selection/syntax.md +++ b/website/docs/reference/node-selection/syntax.md @@ -111,7 +111,7 @@ Constructing and debugging your selection syntax can be challenging. To get a " dbt ls --select "path/to/my/models" # Lists all models in a specific directory. dbt ls --select "source_status:fresher+" # Shows sources updated since the last dbt source freshness run. dbt ls --select state:modified+ # Displays nodes modified in comparison to a previous state. -dbt ls --select "result:+ state:modified+ --defer --state ./" # Lists nodes that match certain [result statuses](/reference/node-selection/syntax#the-result-status) and are modified. +dbt ls --select "result:+" state:modified+ --state ./ # Lists nodes that match certain [result statuses](/reference/node-selection/syntax#the-result-status) and are modified. ``` @@ -186,7 +186,7 @@ After issuing one of the above commands, you can reference the results by adding ```bash # You can also set the DBT_STATE environment variable instead of the --state flag. -dbt run --select "result: --defer --state path/to/prod/artifacts" +dbt run --select "result:" --defer --state path/to/prod/artifacts ``` The available options depend on the resource (node) type: @@ -205,7 +205,7 @@ The available options depend on the resource (node) type: The state and result selectors can also be combined in a single invocation of dbt to capture errors from a previous run OR any new or modified models. ```bash -dbt run --select "result:+ state:modified+ --defer --state ./" +dbt run --select "result:+" state:modified+ --defer --state ./ ``` ### Fresh rebuilds diff --git a/website/snippets/_cloud-environments-info.md b/website/snippets/_cloud-environments-info.md index 4822c8a5eae..1d744aa75d7 100644 --- a/website/snippets/_cloud-environments-info.md +++ b/website/snippets/_cloud-environments-info.md @@ -113,4 +113,4 @@ Partial parsing in dbt Cloud requires dbt version 1.4 or newer. The feature does To enable, select **Account settings** from the gear menu and enable the **Partial parsing** option. - \ No newline at end of file + diff --git a/website/snippets/_core-to-cloud-guide-table.md b/website/snippets/_core-to-cloud-guide-table.md new file mode 100644 index 00000000000..415476dace0 --- /dev/null +++ b/website/snippets/_core-to-cloud-guide-table.md @@ -0,0 +1,5 @@ +|
Guide
|
Information
|
Audience
| +|------------|-------------|----------| +| [Move from dbt Core to dbt Cloud: What you need to know](/guides/core-cloud-2) | Understand the considerations and methods needed in your move from dbt Core to dbt Cloud. | Team leads
Admins | +| [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) | Learn the steps needed to move from dbt Core to dbt Cloud. | Developers
Data engineers
Data analysts | +| [Move from dbt Core to dbt Cloud: Optimization tips](/guides/core-to-cloud-3) | Learn how to optimize your dbt Cloud experience with common scenarios and useful tips. | Everyone | diff --git a/website/snippets/_packages_or_dependencies.md b/website/snippets/_packages_or_dependencies.md index 3cfcf7fbf11..696abdd8c7b 100644 --- a/website/snippets/_packages_or_dependencies.md +++ b/website/snippets/_packages_or_dependencies.md @@ -1,14 +1,13 @@ ## Use cases -Starting from dbt v1.6, `dependencies.yml` has replaced `packages.yml`. The `dependencies.yml` file can now contain both types of dependencies: "package" and "project" dependencies. +Starting from dbt v1.6, we added a new configuration file called `dependencies.yml`. The file can contain both types of dependencies: "package" and "project" dependencies. - ["Package" dependencies](/docs/build/packages#how-do-i-add-a-package-to-my-project) lets you add source code from someone else's dbt project into your own, like a library. +- ["Project" dependencies](/docs/collaborate/govern/project-dependencies) provide a different way to build on top of someone else's work in dbt. -- ["Project" dependencies](/docs/collaborate/govern/project-dependencies) provide a different way to build on top of someone else's work in dbt. +If your dbt project doesn't require the use of Jinja within the package specifications, you can simply rename your existing `packages.yml` to `dependencies.yml`. However, something to note is if your project's package specifications use Jinja, particularly for scenarios like adding an environment variable or a [Git token method](/docs/build/packages#git-token-method) in a private Git package specification, you should continue using the `packages.yml` file name. -If your dbt project doesn't require the use of Jinja within the package specifications, you can simply rename your existing `packages.yml` to `dependencies.yml`. However, something to note is if your project's package specifications use Jinja, particularly for scenarios like adding an environment variable or a [Git token method](/docs/build/packages#git-token-method) in a private Git package specification, you should continue using the `packages.yml` file name. - -There are some important differences between Package dependencies and Project dependencies: +Examine the following tabs to understand the differences and determine when should use to `dependencies.yml` or `packages.yml`. @@ -16,9 +15,9 @@ There are some important differences between Package dependencies and Project de Project dependencies are designed for the [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) and [cross-project reference](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref) workflow: - Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup. -- Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. +- Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. - Private packages are not supported in `dependencies.yml` because they intentionally don't support Jinja rendering or conditional configuration. This is to maintain static and predictable configuration and ensures compatibility with other services, like dbt Cloud. -- Use `dependencies.yml` for organization and maintainability. It can help maintain your project's organization by allowing you to specify [dbt Hub packages](https://hub.getdbt.com/) like `dbt_utils`. This reduces the need for multiple YAML files to manage dependencies. +- Use `dependencies.yml` for organization and maintainability if you're using both [cross-project refs](docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref) and [dbt Hub packages](https://hub.getdbt.com/). This reduces the need for multiple YAML files to manage dependencies. @@ -26,6 +25,7 @@ Project dependencies are designed for the [dbt Mesh](/best-practices/how-we-mesh Package dependencies allow you to add source code from someone else's dbt project into your own, like a library: +- If you only use packages like those from the [dbt Hub](https://hub.getdbt.com/), remain with `packages.yml`. - Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow. - Use `packages.yml` to include packages, including private packages, in your project's dependencies. If you have private packages that you need to reference, `packages.yml` is the way to go. - `packages.yml` supports Jinja rendering for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a [Git token method](/docs/build/packages#git-token-method) from an environment variable, into your package specifications. diff --git a/website/src/theme/DocItem/Layout/index.js b/website/src/theme/DocItem/Layout/index.js index 2ee1f8bf156..d1995bd0839 100644 --- a/website/src/theme/DocItem/Layout/index.js +++ b/website/src/theme/DocItem/Layout/index.js @@ -27,6 +27,7 @@ import {ThemeClassNames} from '@docusaurus/theme-common'; import VersionContext from '../../../stores/VersionContext' import getElements from '../../../utils/get-html-elements'; import useHashLink from '../../../utils/use-hash-link'; +import removeTrailingDashes from '../../../utils/remove-trailing-slashes'; /** * Decide if the toc should be rendered, on mobile or desktop viewports @@ -50,7 +51,7 @@ function useDocTOC() { ); // Headings to remove from TOC - const headingsToFilter = await getElements( + const headingsToFilter = document.querySelectorAll( ".tabs-container h2, .tabs-container h3, .expandable-anchor h2, .expandable-anchor h3" ); @@ -84,9 +85,13 @@ function useDocTOC() { level = null; } + // Remove trailing slashes from TOC item + // Trailing slashes come from the LifeCycle pill use on headers + const updatedHeading = removeTrailingDashes(heading) + return { - value: heading.innerHTML, - id: heading.id, + value: updatedHeading.innerHTML, + id: updatedHeading.id, level: level && level, }; }; diff --git a/website/src/utils/remove-trailing-slashes.js b/website/src/utils/remove-trailing-slashes.js new file mode 100644 index 00000000000..549e5193429 --- /dev/null +++ b/website/src/utils/remove-trailing-slashes.js @@ -0,0 +1,19 @@ +// Util function to remove trailing dashes from ID attribute on headers +export default function removeTrailingDashes(header) { + // Create copy of header element + const updatedHeader = header; + + // Get id attribute + const thisId = updatedHeader?.getAttribute("id"); + + // If header's id ends with trailing dash, remove dash + if (thisId?.endsWith("-")) { + // Remove `-` from end of ID string + updatedHeader.id = thisId?.substring(0, thisId?.length - 1); + + // Recursively run function to check for another trailing slash + removeTrailingDashes(updatedHeader); + } + + return updatedHeader; +} diff --git a/website/static/img/docs/dbt-cloud/Link-your-Slack-Profile.png b/website/static/img/docs/dbt-cloud/Link-your-Slack-Profile.png index bdc4ad2fc58..246d6a903f8 100644 Binary files a/website/static/img/docs/dbt-cloud/Link-your-Slack-Profile.png and b/website/static/img/docs/dbt-cloud/Link-your-Slack-Profile.png differ diff --git a/website/static/img/docs/dbt-cloud/accept-permissions.png b/website/static/img/docs/dbt-cloud/accept-permissions.png new file mode 100644 index 00000000000..d57a0a614c0 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/accept-permissions.png differ diff --git a/website/static/img/docs/dbt-cloud/define-workspace.png b/website/static/img/docs/dbt-cloud/define-workspace.png new file mode 100644 index 00000000000..60df32c27c9 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/define-workspace.png differ diff --git a/website/static/img/docs/dbt-cloud/snowflakeprivatelink2.png b/website/static/img/docs/dbt-cloud/snowflakeprivatelink2.png new file mode 100644 index 00000000000..bed02f9ea22 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/snowflakeprivatelink2.png differ diff --git a/website/static/img/docs/dbt-cloud/snowflakeprivatelink3.png b/website/static/img/docs/dbt-cloud/snowflakeprivatelink3.png new file mode 100644 index 00000000000..3cf828a895a Binary files /dev/null and b/website/static/img/docs/dbt-cloud/snowflakeprivatelink3.png differ