Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud: add TiDB Serverless Export #16969

Merged
merged 19 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC-tidb-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@
- [GitHub Integration](/tidb-cloud/branch-github-integration.md)
- [Manage Spending Limit](/tidb-cloud/manage-serverless-spend-limit.md)
- [Back Up and Restore TiDB Serverless Data](/tidb-cloud/backup-and-restore-serverless.md)
- [Export Data from TiDB Serverless](/tidb-cloud/serverless-export.md)
zhangyangyu marked this conversation as resolved.
Show resolved Hide resolved
- Manage TiDB Dedicated Clusters
- [Create a TiDB Dedicated Cluster](/tidb-cloud/create-tidb-cluster.md)
- Connect to Your TiDB Dedicated Cluster
Expand Down
128 changes: 128 additions & 0 deletions tidb-cloud/serverless-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
---
title: Export Data from TiDB Serverless
summary: Learn how to export data from TiDB Serverless clusters.
---

# Export Data from TiDB Serverless

TiDB Serverless Export (Beta) is a service that enables you to export data from a TiDB Serverless cluster to local storage or an external storage service. You can use the exported data for backup, migration, data analysis, or other purposes.

While you can also export data using tools such as [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) and TiDB [Dumpling](https://docs.pingcap.com/tidb/dev/dumpling-overview), TiDB Serverless Export offers a more convenient and efficient way to export data from a TiDB Serverless cluster. It brings the following benefits:

- Convenience: the export service provides a simple and easy-to-use way to export data from a TiDB Serverless cluster, eliminating the need for additional tools or resources.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more advantage is you don't have to take care of memory allcoation. For mysqldump and dumpling, it's easy to exceed memory limitation, especially for free tier the limitation is only 256MiB.

- Isolation: the export service uses separate computing resources, ensuring isolation from the resources used by your online services.
- Consistency: the export service ensures the consistency of the exported data without causing locks, which does not affect your online services.

## Features

This section describes the features of TiDB Serverless Export.

### Export location

You can export data to local storage or [Amazon S3](https://aws.amazon.com/s3/).

> **Note:**
>
> If the size of the data to be exported is large (more than 100 GiB), it is recommended that you export it to Amazon S3.

**Local storage**

Exporting data to local storage has the following limitations:

- Exporting multiple databases to local storage at the same time is not supported.
- Exported data is saved in the stashing area and will expire after two days. You need to download the exported data in time.
- If the storage space of stashing area is full, you will not be able to export data to local storage.

**Amazon S3**

To export data to Amazon S3, you need to provide an [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for your S3 bucket. Make sure the access key has read and write access for your S3 bucket, including at least these permissions: `s3:PutObject` and `s3:ListBucket`.

### Data filtering

You can filter data by specifying the database and table you want to export. If you specify a database without specifying a table, all tables in that specified database will be exported. If you do not specify a database when you export data to Amazon S3, all databases in the cluster will be exported.

> **Note:**
>
> You must specify the database when you export data to local storage.

### Data formats

You can export data in the following formats:
shiyuhang0 marked this conversation as resolved.
Show resolved Hide resolved

- `SQL` (default): export data in SQL format.
- `CSV`: export data in CSV format.

The schema and data are exported according to the following naming conventions:

| Item | Not compressed | Compressed |
|-----------------|------------------------------------------|-----------------------------------------------------|
| Database schema | {database}-schema-create.sql | {database}-schema-create.sql.{compression-type} |
| Table schema | {database}.{table}-schema.sql | {database}.{table}-schema.sql.{compression-type} |
| Data | {database}.{table}.{0001}.{sql|csv} | {database}.{table}.{0001}.{sql|csv}.{compression-type} |

### Data compression

You can compress the exported data using the following algorithms:

- `gzip` (default): compress the exported data with gzip.
- `snappy`: compress the exported data with snappy.
- `zstd`: compress the exported data with zstd.
- `none`: do not compress the exported data.

### Cancel export

You can cancel an export task that is in the running state.

## Examples

Currently, you can manage export tasks using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md).

### Export data to local storage

1. Create an export task that specifies the database and table you want to export:

```shell
ticloud serverless export create -c <cluster-id> --database <database> --table <table>
```
qiancai marked this conversation as resolved.
Show resolved Hide resolved

You will get an export ID from the output.

2. After the export is successful, download the exported data to your local storage:

```shell
ticloud serverless export download -c <cluster-id> -e <export-id>
```
qiancai marked this conversation as resolved.
Show resolved Hide resolved

### Export data to Amazon S3

```shell
ticloud serverless export create -c <cluster-id> --bucket-uri <bucket-uri> --access-key-id <access-key-id> --secret-access-key <secret-access-key>
```

### Export with the CSV format

```shell
ticloud serverless export create -c <cluster-id> --file-type CSV
```

### Export the whole database

```shell
ticloud serverless export create -c <cluster-id> --database <database>
```

### Export with snappy compression

```shell
ticloud serverless export create -c <cluster-id> --compress snappy
```

### Cancel an export task

```shell
ticloud serverless export cancel -c <cluster-id> -e <export-id>
```

## Pricing

The export service is free during the beta period. You only need to pay for the [Request Units (RUs)](/tidb-cloud/tidb-cloud-glossary.md#request-unit) generated during the export process of successful or canceled tasks. For failed export tasks, you will not be charged.
Loading