From 2af8e00c979d1c317e32f337c57758a0e2a269b3 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 9 May 2024 11:34:17 -0400 Subject: [PATCH 1/4] Testing unlisted feature --- website/docs/vpc/test-doc.md | 112 +++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 website/docs/vpc/test-doc.md diff --git a/website/docs/vpc/test-doc.md b/website/docs/vpc/test-doc.md new file mode 100644 index 00000000000..7619211bf2b --- /dev/null +++ b/website/docs/vpc/test-doc.md @@ -0,0 +1,112 @@ +--- +title: "Upgrading to v1.8 (latest)" +id: test-doc +description: New features and changes in dbt Core v1.8 +displayed_sidebar: "docs" +unlisted: true +--- + +## Resources + +- Changelog (coming soon) +- [dbt Core CLI Installation guide](/docs/core/installation-overview) +- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) + +## What to know before upgrading + +dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). + +## Keep on latest version + +With dbt Cloud, you can get early access to many new features and functionality before they're in the Generally Available (GA) release of dbt Core v1.8 without the need to manage version upgrades. Refer to the [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) setting for more details. + +## New and changed features and functionality + +Features and functionality new in dbt v1.8. + +### New dbt Core adapter installation procedure + +Before dbt Core v1.8, whenever you would `pip install` a data warehouse adapter for dbt, `pip` would automatically install `dbt-core` alongside it. The dbt adapter directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independent of `dbt-core`. + +Beginning in v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). Going forward, your installations should explicitly include _both_ `dbt-core` _and_ the desired adapter. The new `pip` installation command should look like this: + +```shell +pip install dbt-core dbt-ADAPTER_NAME +``` + +For example, you would use the following command if you use Snowflake: +```shell +pip install dbt-core dbt-snowflake +``` + +For the time being, we have maintained install-time dependencies to avoid breaking existing scripts in surprising ways; `pip install dbt-snowflake` will continue to install the latest versions of both `dbt-core` and `dbt-snowflake`. Given that we may remove this implicit dependency in future versions, we strongly encourage you to update install scripts **now**. + +### Unit Tests + +Historically, dbt's test coverage was confined to [“data” tests](/docs/build/data-tests), assessing the quality of input data or resulting datasets' structure. + +In v1.8, we're introducing native support for [unit testing](/docs/build/unit-tests). Unit tests validate your SQL modeling logic on a small set of static inputs __before__ you materialize your full model in production. They support a test-driven development approach, improving both the efficiency of developers and the reliability of code. + +Starting from v1.8, when you execute the `dbt test` command, it will run both unit and data tests. Use the [`test_type`](/reference/node-selection/methods#the-test_type-method) method to run only unit or data tests: + +```shell + +dbt test --select "test_type:unit" # run all unit tests +dbt test --select "test_type:data" # run all data tests + +``` + +Unit tests are defined in YML files in your `models/` directory and are currently only supported on SQL models. To distinguish between the two, the `tests:` config has been renamed to `data_tests:`. Both are currently supported for backward compatibility. + +#### New `data_tests:` syntax + +The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](/docs/build/data-tests#new-data_tests-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future. + +```yml + +models: + - name: orders + columns: + - name: order_id + data_tests: + - unique + - not_null + + +``` + +#### The `--empty` flag + +The [`run`](/reference/commands/run#the-`--empty`-flag) and [`build`](/reference/commands/build#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. + +### Deprecated functionality + +The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. + +- Overriding a built-in materialization from an installed package raises a deprecation warning. +- Using a custom materialization from an installed package does not raise a deprecation warning. +- Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: + + ```sql + {% materialization view, default %} + {{ return(my_cool_package.materialization_view_default()) }} + {% endmaterialization %} + ``` + +### Managing changes to legacy behaviors + +dbt Core v1.8 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`. + +You can read more about each of these behavior changes in the following links: + +- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations) +- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces) +- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) + +## Quick hits + +- Custom defaults of [global config flags](/reference/global-configs/about-global-configs) should be set in the `flags` dictionary in [`dbt_project.yml`](/reference/dbt_project.yml), instead of in [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). Support for `profiles.yml` has been deprecated. +- New CLI flag [`--resource-type`/`--exclude-resource-type`](/reference/global-configs/resource-type) for including/excluding resources from dbt `build`, `run`, and `clone`. +- To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source. +- Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. + From 7163cf8002b146283dcdb231b145da70f4309bf5 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Fri, 9 Aug 2024 11:03:18 -0400 Subject: [PATCH 2/4] Testing unlisted --- website/docs/docs/get-started-dbt.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/get-started-dbt.md b/website/docs/docs/get-started-dbt.md index 1aba57962fd..7d3e9c3bbd8 100644 --- a/website/docs/docs/get-started-dbt.md +++ b/website/docs/docs/get-started-dbt.md @@ -1,6 +1,7 @@ --- title: "dbt Quickstarts" id: get-started-dbt +unlisted: true hide_table_of_contents: true pagination_next: null pagination_prev: null From dccf582b824debcc07f78f958b70df0c41335919 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 12 Aug 2024 12:22:16 -0400 Subject: [PATCH 3/4] Testing unlisted in next --- website/docs/docs/cloud/about-cloud/about-dbt-cloud.md | 1 + website/docs/docs/cloud/about-cloud/architecture.md | 1 + website/docs/docs/cloud/about-cloud/tenancy.md | 1 + 3 files changed, 3 insertions(+) diff --git a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md index 02f950111ea..059f9e6ff7b 100644 --- a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md +++ b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md @@ -4,6 +4,7 @@ id: "dbt-cloud-features" sidebar_label: "dbt Cloud features" description: "Explore dbt Cloud's features and learn why dbt Cloud is the fastest way to deploy dbt" hide_table_of_contents: true +unlisted: true pagination_next: "docs/cloud/about-cloud/architecture" pagination_prev: null --- diff --git a/website/docs/docs/cloud/about-cloud/architecture.md b/website/docs/docs/cloud/about-cloud/architecture.md index 52614f0cbcd..4b7a1b945b7 100644 --- a/website/docs/docs/cloud/about-cloud/architecture.md +++ b/website/docs/docs/cloud/about-cloud/architecture.md @@ -1,6 +1,7 @@ --- title: "dbt Cloud Architecture" id: "architecture" +unlisted: true description: "Information about the architecture, communication, and security of dbt Cloud" --- diff --git a/website/docs/docs/cloud/about-cloud/tenancy.md b/website/docs/docs/cloud/about-cloud/tenancy.md index efbc248f91e..26900b10cd8 100644 --- a/website/docs/docs/cloud/about-cloud/tenancy.md +++ b/website/docs/docs/cloud/about-cloud/tenancy.md @@ -2,6 +2,7 @@ title: Tenancy id: tenancy description: "Information about single tenant and multi-tenant dbt Cloud instances" +unlisted: true --- import AboutCloud from '/snippets/_test-tenancy.md'; From ab338425bb2e9f8d5e0b38960c4d4f96417e4ce9 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Fri, 8 Nov 2024 10:41:58 -0500 Subject: [PATCH 4/4] New commit to next --- website/docs/reference/global-configs/cache.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/reference/global-configs/cache.md b/website/docs/reference/global-configs/cache.md index 1a74fef8d30..349bdcd9d6f 100644 --- a/website/docs/reference/global-configs/cache.md +++ b/website/docs/reference/global-configs/cache.md @@ -13,6 +13,7 @@ There are two ways to optionally modify this behavior: - `CACHE_SELECTED_ONLY` (default `False`): Whether to limit cache population to just the resources selected in the current run. This can offer significant speed improvements when running a small subset of a large project, while still providing the benefit of caching upfront. For example, to quickly compile a model that requires no database metadata or introspective queries: + ```text dbt --no-populate-cache compile --select my_model_name