From 611ed0cfa2487106f7f70673e760f23cebc2ad48 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sun, 12 May 2024 19:49:15 +0700 Subject: [PATCH 1/4] docs(self-hosted): external storage configurations --- src/components/sidebar.tsx | 3 +- src/docs/self-hosted/external-storage.mdx | 89 +++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) create mode 100644 src/docs/self-hosted/external-storage.mdx diff --git a/src/components/sidebar.tsx b/src/components/sidebar.tsx index 48465d8d46..9aea600b72 100644 --- a/src/components/sidebar.tsx +++ b/src/components/sidebar.tsx @@ -116,7 +116,8 @@ export default () => { Geolocation Single Sign-On (SSO) Content Security Policy (CSP) - Reverse Proxy + Reverse Proxy + External Storage Troubleshooting Support diff --git a/src/docs/self-hosted/external-storage.mdx b/src/docs/self-hosted/external-storage.mdx new file mode 100644 index 0000000000..fa8603d1f0 --- /dev/null +++ b/src/docs/self-hosted/external-storage.mdx @@ -0,0 +1,89 @@ +--- +title: External Storage +--- + + + + + After changing configuration files, re-run the ./install.sh script, to rebuild and restart the containers. See the configuration section for more information. + + + + +## Filestore + +Filestore handles storing attachment, sourcemap, and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. + +### S3 backend + +The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. + +```yaml +filestore.backend: 's3' +filestore.options: + bucket_acl: 'private' + default_acl: 'private' + access_key: '' + secret_key: '' + bucket_name: 'my-bucket' + region_name: 'auto' + endpoint_url: 'https://' + addressing_style: 'path' # For regular AWS S3, use "auto" or "virtual". For other S3-compatible API like MinIO or Ceph, use "path". + signature_version: 's3v4' +``` + +Refer to [botocore configuration](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) for valid configuration values. + + + +## Vroom + +Vroom is the service that handles profiling. By default the data for profiling is saved on local filesystem. On self-hosted deployment, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. + +### S3 backend + +```bash +# For regular AWS S3 +s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com + +# For other S3-compatible API +s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL=false +``` + +Additional environment variables should be provided: +- `AWS_ACCESS_KEY=foobar` +- `AWS_SECRET_KEY=foobar` +- `AWS_SESSION_TOKEN=foobar` (optional) + +Further explanation on the query string options: +- `region`: The AWS region for requests. +- `endpoint`: The endpoint URL (hostname only or fully qualified URI). +- `disableSSL`: A value of "true" disables SSL when sending requests. +- `s3ForcePathStyle`: A value of "true" forces the request to use path-style addressing. + +### Azure Blob Storage backend + +```bash +azblob://my-container?protocol=https&domain=yourcompany.blob.core.windows.net&localemu=false&cdn=false +``` + +Additional environment variables that should be provided (pick what's compatible with your configuration): +- `AZURE_STORAGE_ACCOUNT=foobar` - The service account name. Required if used along with `AZURE_STORAGE_KEY`, because it defines authentication mechanism to be [azblob.NewSharedKeyCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob#NewSharedKeyCredential), which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used. +- `AZURE_STORAGE_KEY=foobar` - To use a shared key credential alongside with `AZURE_STORAGE_ACCOUNT`. +- `AZURE_STORAGE_SAS_TOKEN=foobar` - To use a SAS token + +Other authentication options and details can be found on the [gocloud.dev/blob/azblob's documentation](https://pkg.go.dev/gocloud.dev@v0.37.0/blob/azureblob#hdr-URLs) + +Further explanation on the query string options: +- `domain`: Your storage domain. +- `protocol`: Network protocol (`http` or `https`). +- `cdn`: A value of "true" specifies that the blob server is a CDN. +- `localemu`: A value of "true" specifies that the blob server is a local emulator. \ No newline at end of file From bc99370fa3cb234ed08a73a74720ad876832f227 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 18 May 2024 09:40:42 +0700 Subject: [PATCH 2/4] Update src/docs/self-hosted/external-storage.mdx Co-authored-by: Alex Zaslavsky --- src/docs/self-hosted/external-storage.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/docs/self-hosted/external-storage.mdx b/src/docs/self-hosted/external-storage.mdx index fa8603d1f0..a011f7e2e9 100644 --- a/src/docs/self-hosted/external-storage.mdx +++ b/src/docs/self-hosted/external-storage.mdx @@ -54,7 +54,7 @@ Vroom is the service that handles profiling. By default the data for profiling i # For regular AWS S3 s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com -# For other S3-compatible API +# For other S3-compatible APIs s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL=false ``` From d6f0bf653291eb1aff3b2af1c1d4644b837292de Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 18 May 2024 09:40:57 +0700 Subject: [PATCH 3/4] Update src/docs/self-hosted/external-storage.mdx Co-authored-by: Alex Zaslavsky --- src/docs/self-hosted/external-storage.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/docs/self-hosted/external-storage.mdx b/src/docs/self-hosted/external-storage.mdx index a011f7e2e9..0cee5b6864 100644 --- a/src/docs/self-hosted/external-storage.mdx +++ b/src/docs/self-hosted/external-storage.mdx @@ -76,7 +76,8 @@ azblob://my-container?protocol=https&domain=yourcompany.blob.core.windows.net&lo ``` Additional environment variables that should be provided (pick what's compatible with your configuration): -- `AZURE_STORAGE_ACCOUNT=foobar` - The service account name. Required if used along with `AZURE_STORAGE_KEY`, because it defines authentication mechanism to be [azblob.NewSharedKeyCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob#NewSharedKeyCredential), which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used. +- `AZURE_STORAGE_ACCOUNT=foobar`: The service account name. Required if used along with `AZURE_STORAGE_KEY`, because it defines authentication mechanism to be [azblob.NewSharedKeyCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob#NewSharedKeyCredential), which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used. + - `AZURE_STORAGE_KEY=foobar` - To use a shared key credential alongside with `AZURE_STORAGE_ACCOUNT`. - `AZURE_STORAGE_SAS_TOKEN=foobar` - To use a SAS token From 50347c62f2d487d12a4b5648e68458c5d1f6df1f Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sun, 21 Jul 2024 08:19:46 +0700 Subject: [PATCH 4/4] feat(self-hosted): remove azure; clearly state the purpose of external storage --- src/docs/self-hosted/external-storage.mdx | 66 +++++++++++------------ 1 file changed, 30 insertions(+), 36 deletions(-) diff --git a/src/docs/self-hosted/external-storage.mdx b/src/docs/self-hosted/external-storage.mdx index 0cee5b6864..12e8d76616 100644 --- a/src/docs/self-hosted/external-storage.mdx +++ b/src/docs/self-hosted/external-storage.mdx @@ -2,24 +2,32 @@ title: External Storage --- - +In some cases, storing Sentry data on-disk is not really something people can do. Sometimes, it's better if they can offload it into some bucket storage (like AWS S3 or Google Cloud Storage). After changing configuration files, re-run the ./install.sh script, to rebuild and restart the containers. See the configuration section for more information. - +## Sentry + +The Sentry service has a abstraction called "filestore" that handles storing attachment, sourcemap (release artifacts), and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. -## Filestore +### Google Cloud Storage backend -Filestore handles storing attachment, sourcemap, and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. +The configuration for GCS backend is pointed to `sentry.filestore.gcs.GoogleCloudStorage`. You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). + +```yaml +filestore.backend: "gcs" +filestore.options: + bucket_name: "..." +``` ### S3 backend + + Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + + The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. ```yaml @@ -31,31 +39,37 @@ filestore.options: secret_key: '' bucket_name: 'my-bucket' region_name: 'auto' - endpoint_url: 'https://' + endpoint_url: 'https://' # If you're not using AWS. addressing_style: 'path' # For regular AWS S3, use "auto" or "virtual". For other S3-compatible API like MinIO or Ceph, use "path". signature_version: 's3v4' ``` Refer to [botocore configuration](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) for valid configuration values. - - ## Vroom Vroom is the service that handles profiling. By default the data for profiling is saved on local filesystem. On self-hosted deployment, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. +### Google Cloud Storage backend + +You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). + +```bash +gs://my-bucket +``` + ### S3 backend + + Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + + ```bash # For regular AWS S3 s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com # For other S3-compatible APIs -s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL=false +s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL ``` Additional environment variables should be provided: @@ -68,23 +82,3 @@ Further explanation on the query string options: - `endpoint`: The endpoint URL (hostname only or fully qualified URI). - `disableSSL`: A value of "true" disables SSL when sending requests. - `s3ForcePathStyle`: A value of "true" forces the request to use path-style addressing. - -### Azure Blob Storage backend - -```bash -azblob://my-container?protocol=https&domain=yourcompany.blob.core.windows.net&localemu=false&cdn=false -``` - -Additional environment variables that should be provided (pick what's compatible with your configuration): -- `AZURE_STORAGE_ACCOUNT=foobar`: The service account name. Required if used along with `AZURE_STORAGE_KEY`, because it defines authentication mechanism to be [azblob.NewSharedKeyCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob#NewSharedKeyCredential), which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used. - -- `AZURE_STORAGE_KEY=foobar` - To use a shared key credential alongside with `AZURE_STORAGE_ACCOUNT`. -- `AZURE_STORAGE_SAS_TOKEN=foobar` - To use a SAS token - -Other authentication options and details can be found on the [gocloud.dev/blob/azblob's documentation](https://pkg.go.dev/gocloud.dev@v0.37.0/blob/azureblob#hdr-URLs) - -Further explanation on the query string options: -- `domain`: Your storage domain. -- `protocol`: Network protocol (`http` or `https`). -- `cdn`: A value of "true" specifies that the blob server is a CDN. -- `localemu`: A value of "true" specifies that the blob server is a local emulator. \ No newline at end of file