Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify privatelink can't connect across cloud providers #6371

Merged
merged 4 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions website/docs/docs/cloud/secure/about-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ sidebar_label: "About PrivateLink"

import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkHostnameWarning from '/snippets/_privatelink-hostname-restriction.md';
import CloudProviders from '/snippets/_privatelink-across-providers.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

PrivateLink enables a private connection from any dbt Cloud Multi-Tenant environment to your data platform hosted on AWS using [AWS PrivateLink](https://aws.amazon.com/privatelink/) technology. PrivateLink allows dbt Cloud customers to meet security and compliance controls as it allows connectivity between dbt Cloud and your data platform without traversing the public internet. This feature is supported in most regions across NA, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<CloudProviders type='a data platform' />

### Cross-region PrivateLink

dbt Labs has a worldwide network of regional VPCs. These VPCs are specifically used to host PrivateLink VPC endpoints, which are connected to dbt Cloud instance environments. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards. The connected services are also authenticated. Currently, we have multiple customers successfully connecting to their PrivateLink endpoints in different AWS regions within dbt Cloud.
Expand Down
3 changes: 3 additions & 0 deletions website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,14 @@ pagination_next: null

import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkSLA from '/snippets/_PrivateLink-SLA.md';
import CloudProviders from '/snippets/_privatelink-across-providers.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps will walk you through the setup of a Databricks AWS PrivateLink or Azure Private Link endpoint in the dbt Cloud multi-tenant environment.

<CloudProviders type='Databricks'/>

## Configure AWS PrivateLink

1. Locate your [Databricks instance name](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)
Expand Down
5 changes: 4 additions & 1 deletion website/docs/docs/cloud/secure/postgres-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ sidebar_label: "PrivateLink for Postgres"
import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkTroubleshooting from '/snippets/_privatelink-troubleshooting.md';
import PrivateLinkCrossZone from '/snippets/_privatelink-cross-zone-load-balancing.md';
import CloudProviders from '/snippets/_privatelink-across-providers.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed through a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as explained in the following steps.

<CloudProviders type='Postgres' />

## Configuring Postgres interface-type PrivateLink

### 1. Provision AWS resources
Expand Down Expand Up @@ -96,4 +99,4 @@ Once dbt Cloud support completes the configuration, you can start creating new c
4. Configure the remaining data platform details.
5. Test your connection and save it.

<PrivateLinkTroubleshooting features={'/snippets/_privatelink-troubleshooting.md'}/>
<PrivateLinkTroubleshooting features={'/snippets/_privatelink-troubleshooting.md'}/>
5 changes: 4 additions & 1 deletion website/docs/docs/cloud/secure/redshift-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ sidebar_label: "PrivateLink for Redshift"
import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkTroubleshooting from '/snippets/_privatelink-troubleshooting.md';
import PrivateLinkCrossZone from '/snippets/_privatelink-cross-zone-load-balancing.md';
import CloudProviders from '/snippets/_privatelink-across-providers.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

Expand All @@ -17,6 +18,8 @@ AWS provides two different ways to create a PrivateLink VPC endpoint for a Redsh

dbt Cloud supports both types of endpoints, but there are a number of [considerations](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html#managing-cluster-cross-vpc-considerations) to take into account when deciding which endpoint type to use. Redshift-managed provides a far simpler setup with no additional cost, which might make it the preferred option for many, but may not be an option in all environments. Based on these criteria, you will need to determine which is the right type for your system. Follow the instructions from the section below that corresponds to your chosen endpoint type.

<CloudProviders type='Redshift' />

:::note Redshift Serverless
While Redshift Serverless does support Redshift-managed type VPC endpoints, this functionality is not currently available across AWS accounts. Due to this limitation, an Interface-type VPC endpoint service must be used for Redshift Serverless cluster PrivateLink connectivity from dbt Cloud.
:::
Expand Down Expand Up @@ -125,4 +128,4 @@ Once dbt Cloud support completes the configuration, you can start creating new c
4. Configure the remaining data platform details.
5. Test your connection and save it.

<PrivateLinkTroubleshooting features={'/snippets/_privatelink-troubleshooting.md'}/>
<PrivateLinkTroubleshooting features={'/snippets/_privatelink-troubleshooting.md'}/>
3 changes: 3 additions & 0 deletions website/docs/docs/cloud/secure/snowflake-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,14 @@ sidebar_label: "PrivateLink for Snowflake"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import CloudProviders from '/snippets/_privatelink-across-providers.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps walk you through the setup of a Snowflake AWS PrivateLink or Azure Private Link endpoint in a dbt Cloud multi-tenant environment.

<CloudProviders type='Snowflake' />

:::note Snowflake SSO with PrivateLink
Users connecting to Snowflake using SSO over a PrivateLink connection from dbt Cloud will also require access to a PrivateLink endpoint from their local workstation.

Expand Down
1 change: 1 addition & 0 deletions website/snippets/_privatelink-across-providers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
PrivateLink endpoints can't connect across cloud providers. For a PrivateLink connection to work, both dbt Cloud and the server (like {props.type}) must be hosted on the same cloud provider. For example, AWS PrivateLink can’t connect to resources hosted on Azure, and Azure PrivateLink can’t connect to resources hosted on AWS.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to mention what alternatives to consider if users are in this situation? e.g. do they need to consider alternatives like a VPN or any other solution?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding additional options would just confuse people more. I think this is pretty good. I might tweak the last sentence a bit to something like:

"For example, dbt Cloud hosted on AWS cannot connect via PrivateLink to services hosted on Azure, and dbt Cloud hosted on Azure can’t connect via Private Link to services hosted on AWS."

Small note that PrivateLink on AWS doesn't have a space, but on Azure it does (Private Link).

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
Loading