From 6a7e23aacc30bf6829f61861cb2e007a89ee0a37 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 12 Nov 2024 12:31:51 +0000 Subject: [PATCH] update sql linting to ga --- .../docs/cloud/dbt-cloud-ide/lint-format.md | 3 ++- .../docs/docs/dbt-versions/release-notes.md | 1 + website/docs/docs/deploy/ci-jobs.md | 4 ++-- .../docs/deploy/continuous-integration.md | 22 ++++++++++++------- 4 files changed, 19 insertions(+), 11 deletions(-) diff --git a/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md b/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md index d14435a97e0..abd3c86d4a8 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md @@ -81,7 +81,7 @@ To configure your own linting rules: :::tip Configure dbtonic linting rules -Refer to the [SQLFluff config file](https://github.com/dbt-labs/jaffle-shop-template/blob/main/.sqlfluff) to add the dbt code (or dbtonic) rules we use for our own projects: +Refer to the [Jaffle shop SQLFluff config file](https://github.com/dbt-labs/jaffle-shop-template/blob/main/.sqlfluff) for dbt-specific (or dbtonic) linting rules we use for our own projects:
dbtonic config code example provided by dbt Labs @@ -231,3 +231,4 @@ To avoid this, break up your model into smaller models (files) so that they are - [User interface](/docs/cloud/dbt-cloud-ide/ide-user-interface) - [Keyboard shortcuts](/docs/cloud/dbt-cloud-ide/keyboard-shortcuts) +- [SQL linting in CI jobs](/docs/deploy/continuous-integration#sql-linting) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 6759a026e02..85ec50daf64 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,6 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 +- **New**: SQL linting in CI jobs is now generally available in dbt Cloud. You can enable SQL linting in your CI jobs, using [SQLFluff](https://sqlfluff.com/), to automatically lint all SQL files in your project as a run step before your CI job builds. Refer to [SQL linting](/docs/deploy/continuous-integration#sql-linting) for more information. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index 7ab7f65796d..2b690ff6aae 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -14,7 +14,7 @@ dbt Labs recommends that you create your CI job in a dedicated dbt Cloud [deploy - You have a dbt Cloud account. - CI features: - For both the [concurrent CI checks](/docs/deploy/continuous-integration#concurrent-ci-checks) and [smart cancellation of stale builds](/docs/deploy/continuous-integration#smart-cancellation) features, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/). - - The [SQL linting](/docs/deploy/continuous-integration#sql-linting) feature is currently available in [beta](/docs/dbt-versions/product-lifecycles#dbt-cloud) to a limited group of users and is gradually being rolled out. If you're in the beta, the **Linting** option is available for use. + - For the [SQL linting](/docs/deploy/continuous-integration#sql-linting) feature, you should have [SQLFluff configured](/docs/deploy/continuous-integration#to-configure-sqlfluff-linting) in your project. - [Advanced CI](/docs/deploy/advanced-ci) features: - For the [compare changes](/docs/deploy/advanced-ci#compare-changes) feature, your dbt Cloud account must be on the [Enterprise plan](https://www.getdbt.com/pricing/) and have enabled Advanced CI features. Please ask your [dbt Cloud administrator to enable](/docs/cloud/account-settings#account-access-to-advanced-ci-features) this feature for you. After enablement, the **dbt compare** option becomes available in the CI job settings. - Set up a [connection with your Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud). This integration lets dbt Cloud run jobs on your behalf for job triggering. @@ -36,7 +36,7 @@ To make CI job creation easier, many options on the **CI job** page are set to d 4. Options in the **Execution settings** section: - **Commands** — By default, this includes the `dbt build --select state:modified+` command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when this job runs. - - **Linting** — Enable this option for dbt to [lint the SQL files](/docs/deploy/continuous-integration#sql-linting) in your project as the first step in `dbt run`. If this check runs into an error, dbt can either **Stop running on error** or **Continue running on error**. + - **Linting** — Enable this option for dbt to [lint the SQL files](/docs/deploy/continuous-integration#sql-linting) in your project as the first step in `dbt run`. If this check runs into an error, dbt can either **Stop running on error** or **Continue running on error**. - **dbt compare** — Enable this option to compare the last applied state of the production environment (if one exists) with the latest changes from the pull request, and identify what those differences are. To enable record-level comparison and primary key analysis, you must add a [primary key constraint](/reference/resource-properties/constraints) or [uniqueness test](/reference/resource-properties/data-tests#unique). Otherwise, you'll receive a "Primary key missing" error message in dbt Cloud. To review the comparison report, navigate to the [Compare tab](/docs/deploy/run-visibility#compare-tab) in the job run's details. A summary of the report is also available from the pull request in your Git provider (see the [CI report example](#example-ci-report)). diff --git a/website/docs/docs/deploy/continuous-integration.md b/website/docs/docs/deploy/continuous-integration.md index 4e152b0a97e..d50b6f44eb4 100644 --- a/website/docs/docs/deploy/continuous-integration.md +++ b/website/docs/docs/deploy/continuous-integration.md @@ -31,10 +31,10 @@ dbt Cloud deletes the temporary schema from your  w The [dbt Cloud scheduler](/docs/deploy/job-scheduler) executes CI jobs differently from other deployment jobs in these important ways: -- **Concurrent CI checks** — CI runs triggered by the same dbt Cloud CI job execute concurrently (in parallel), when appropriate. -- **Smart cancellation of stale builds** — Automatically cancels stale, in-flight CI runs when there are new commits to the PR. -- **Run slot treatment** — CI runs don't consume a run slot. -- **SQL linting** — When enabled, automatically lints all SQL files in your project as a run step before your CI job builds. +- [**Concurrent CI checks**](#concurrent-ci-checks) — CI runs triggered by the same dbt Cloud CI job execute concurrently (in parallel), when appropriate. +- [**Smart cancellation of stale builds**](#smart-cancellation-of-stale-builds) — Automatically cancels stale, in-flight CI runs when there are new commits to the PR. +- [**Run slot treatment**](#run-slot-treatment) — CI runs don't consume a run slot. +- [**SQL linting**](#sql-linting) — When enabled, automatically lints all SQL files in your project as a run step before your CI job builds. ### Concurrent CI checks @@ -56,10 +56,16 @@ When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the late CI runs don't consume run slots. This guarantees a CI check will never block a production run. -### SQL linting +### SQL linting -When enabled for your CI job, dbt invokes [SQLFluff](https://sqlfluff.com/) which is a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. By default, it lints all the changed SQL files in your project (compared to the last deferred production state). +When [enabled for your CI job](/docs/deploy/ci-jobs#set-up-ci-jobs), dbt invokes [SQLFluff](https://sqlfluff.com/) which is a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. By default, it lints all the changed SQL files in your project (compared to the last deferred production state). -If the linter runs into errors, you can specify whether dbt should stop running the job on error or continue running it on error. When failing jobs, it helps reduce compute costs by avoiding builds for pull requests that don't meet your SQL code quality CI check. +If the linter runs into errors, you can specify whether dbt should stop running the job on error or continue running it on error. When failing jobs, it helps reduce compute costs by avoiding builds for pull requests that don't meet your SQL code quality CI check. -You can use [SQLFluff Configuration Files](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#configuration-files) to override the default linting behavior in dbt. Create an `.sqlfluff` configuration file in your project, add your linting rules to it, and dbt Cloud will use them when linting. For complete details, refer to [Custom Usage](https://docs.sqlfluff.com/en/stable/gettingstarted.html#custom-usage) in the SQLFluff documentation. +#### To configure SQLFluff linting: + +- Use [SQLFluff Configuration Files](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#configuration-files) to override the default linting behavior in dbt. +- Create a `.sqlfluff` configuration file in your project, add your linting rules to it, and dbt Cloud will use them when linting. + - When configuring, you can use `dbt` as the templater (for example, `templater = dbt`) + - If you’re using the dbt Cloud IDE, dbt Cloud CLI, or any other editor, refer to [Customize linting](/docs/cloud/dbt-cloud-ide/lint-format#customize-linting) for guidance on how to add dbt-specific (or dbtonic) linting rules we use for own project. +- For complete details, refer to [Custom Usage](https://docs.sqlfluff.com/en/stable/gettingstarted.html#custom-usage) in the SQLFluff documentation.