Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release ticloud cli v1.0.0-beta.2 #18938

Merged
merged 30 commits into from
Sep 25, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4c0d724
add export cli in export doc
shiyuhang0 Sep 18, 2024
4eebcdf
update cli doc
shiyuhang0 Sep 18, 2024
a8b1729
fix lint
shiyuhang0 Sep 19, 2024
d12e68f
cluster
shiyuhang0 Sep 20, 2024
0f9d079
branch
shiyuhang0 Sep 20, 2024
477b234
add whoami
shiyuhang0 Sep 20, 2024
19a99bf
fix lint
shiyuhang0 Sep 20, 2024
adcd616
update cli toc
shiyuhang0 Sep 20, 2024
c36ce97
add sql user
shiyuhang0 Sep 20, 2024
f3d9f89
add import doc
xuanyu66 Sep 20, 2024
2c532e7
opt lint
shiyuhang0 Sep 20, 2024
1233e77
opt lint
shiyuhang0 Sep 20, 2024
8b73d7d
opt lint
shiyuhang0 Sep 20, 2024
8d1b7fc
polish
xuanyu66 Sep 20, 2024
c7baeee
Update tidb-cloud/serverless-export.md
shiyuhang0 Sep 20, 2024
e6a3da9
Update tidb-cloud/cli-reference.md
shiyuhang0 Sep 20, 2024
832df06
Update tidb-cloud/cli-reference.md
shiyuhang0 Sep 20, 2024
5f3b367
some opts
shiyuhang0 Sep 20, 2024
ca7c81d
change file names ticloud upgrade
hfxsd Sep 23, 2024
25a29a2
fix name error
hfxsd Sep 23, 2024
b931ec1
Apply suggestions from code review
shiyuhang0 Sep 24, 2024
9e4bac2
Update tidb-cloud/ticloud-upgrade.md
shiyuhang0 Sep 24, 2024
a1329b7
fix dead link
shiyuhang0 Sep 24, 2024
5983248
fix verify
shiyuhang0 Sep 24, 2024
47a5c08
fix lint
shiyuhang0 Sep 24, 2024
76eb38d
Apply suggestions from code review
shiyuhang0 Sep 25, 2024
c0671e3
Apply suggestions from code review
shiyuhang0 Sep 25, 2024
10e4cb9
Update tidb-cloud/serverless-external-storage.md
shiyuhang0 Sep 25, 2024
4fc7851
Update tidb-cloud/ticloud-import-start.md
shiyuhang0 Sep 25, 2024
de6916c
Update tidb-cloud/ticloud-serverless-export-create.md
shiyuhang0 Sep 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion TOC-tidb-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,7 @@
- auth
- [login](/tidb-cloud/ticloud-auth-login.md)
- [logout](/tidb-cloud/ticloud-auth-logout.md)
- [whoami](/tidb-cloud/ticloud-auth-whoami.md)
- serverless
- [create](/tidb-cloud/ticloud-cluster-create.md)
- [delete](/tidb-cloud/ticloud-cluster-delete.md)
Expand All @@ -701,6 +702,11 @@
- [list](/tidb-cloud/ticloud-serverless-export-list.md)
- [cancel](/tidb-cloud/ticloud-serverless-export-cancel.md)
- [download](/tidb-cloud/ticloud-serverless-export-download.md)
- sql-user
- [create](/tidb-cloud/ticloud-serverless-sql-user-create.md)
- [delete](/tidb-cloud/ticloud-serverless-sql-user-delete.md)
- [list](/tidb-cloud/ticloud-serverless-sql-user-list.md)
- [update](/tidb-cloud/ticloud-serverless-sql-user-update.md)
- [ai](/tidb-cloud/ticloud-ai.md)
- [completion](/tidb-cloud/ticloud-completion.md)
- config
Expand All @@ -713,7 +719,7 @@
- [use](/tidb-cloud/ticloud-config-use.md)
- project
- [list](/tidb-cloud/ticloud-project-list.md)
- [update](/tidb-cloud/ticloud-update.md)
- [upgrade](/tidb-cloud/ticloud-update.md)
- [help](/tidb-cloud/ticloud-help.md)
- [Table Filter](/table-filter.md)
- [Resource Control](/tidb-resource-control.md)
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
27 changes: 14 additions & 13 deletions tidb-cloud/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,20 @@ The following table lists the commands available for the TiDB Cloud CLI.

To use the `ticloud` CLI in your terminal, run `ticloud [command] [subcommand]`. If you are using [TiUP](https://docs.pingcap.com/tidb/stable/tiup-overview), use `tiup cloud [command] [subcommand]` instead.

| Command | Subcommand | Description |
|-------------------|--------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| auth | login, logout | Login and logout |
| serverless (alias: s) | create, delete, describe, list, update, spending-limit, region, shell | Manage TiDB Cloud Serverless clusters |
| serverless branch | create, delete, describe, list, shell | Manage TiDB Cloud Serverless branches |
| serverless import | cancel, describe, list, start | Manage TiDB Cloud Serverless import tasks |
| serverless export | create, describe, list, cancel, download | Manage TiDB Cloud Serverless export tasks |
| ai | - | Chat with TiDB Bot |
| completion | bash, fish, powershell, zsh | Generate completion script for specified shell |
| config | create, delete, describe, edit, list, set, use | Configure user profiles |
| project | list | Manage projects |
| update | - | Update the CLI to the latest version |
| help | cluster, completion, config, help, import, project, update | View help for any command |
| Command | Subcommand | Description |
|-----------------------|-----------------------------------------------------------------------|------------------------------------------------|
| auth | login, logout, whoami | Login and logout |
| serverless (alias: s) | create, delete, describe, list, update, spending-limit, region, shell | Manage TiDB Cloud Serverless clusters |
| serverless branch | create, delete, describe, list, shell | Manage TiDB Cloud Serverless branches |
| serverless import | cancel, describe, list, start | Manage TiDB Cloud Serverless import tasks |
| serverless export | create, describe, list, cancel, download | Manage TiDB Cloud Serverless export tasks |
| serverless sql-user | create, list, delete, update | Manage TiDB Cloud Serverless SQL users |
| ai | - | Chat with TiDB Bot |
| completion | bash, fish, powershell, zsh | Generate completion script for specified shell |
| config | create, delete, describe, edit, list, set, use | Configure user profiles |
| project | list | Manage projects |
| upgrade | - | Update the CLI to the latest version |
| help | auth, config, serverless, ai, project, upgrade, help, completion | View help for any command |

## Command modes

Expand Down
125 changes: 116 additions & 9 deletions tidb-cloud/serverless-export.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,13 @@ While you can also export data using tools such as [mysqldump](https://dev.mysql

## Export locations

You can export data to a local file or [Amazon S3](https://aws.amazon.com/s3/).
You can export data to

- A local file
- An external storage, including:
- [Amazon S3](https://aws.amazon.com/s3/)
- [Google Cloud Storage](https://cloud.google.com/storage)
- [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)

> **Note:**
>
Expand All @@ -42,11 +48,38 @@ To export data to Amazon S3, you need to provide the following information:

For more information, see [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access).

### Google Cloud Storage

To export data to Google Cloud Storage, you need to provide the following information:

- URI: `gs://<bucket-name>/<file-path>`
- Access credential: a **base64 encoded** [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your bucket. Make sure the service account key has the `storage.objects.create` permission.

For more information, see [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-gcs-access).

> **Note:**
>
> Currently, you can only export to Google Cloud Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md).

### Azure Blob Storage

To export data to Azure Blob Storage, you need to provide the following information:

- URI: `azure://<account-name>.blob.core.windows.net/<container-name>/<file-path>`
- Access credential: a [shared access signature (SAS) token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) for your Azure Blob Storage container. Make sure the SAS token has the `Read` and `Write` permissions on the `Container` and `Object` resources.

For more information, see [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access).

> **Note:**
>
> Currently, you can only export to Azure Blob Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md).

## Export options

### Data filtering

- TiDB Cloud console supports exporting data with the selected databases and tables.
- TiDB Cloud CLI supports exporting data with SQL statements and [table filters](/table-filter.md).

### Data formats

Expand All @@ -58,14 +91,15 @@ You can export data in the following formats:
- `separator`: specify the character used to separate fields in the exported data. The default separator is `,`.
- `header`: specify whether to include a header row in the exported data. The default value is `true`.
- `null-value`: specify the string that represents a NULL value in the exported data. The default value is `\N`.
- `Parquet`: export data in Parquet format. Currently, it is only supported in TiDB Cloud CLI.

The schema and data are exported according to the following naming conventions:

| Item | Not compressed | Compressed |
|-----------------|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Database schema | {database}-schema-create.sql | {database}-schema-create.sql.{compression-type} |
| Table schema | {database}.{table}-schema.sql | {database}.{table}-schema.sql.{compression-type} |
| Data | {database}.{table}.{0001}.{csv&#124;sql} | {database}.{table}.{0001}.{csv&#124;sql}.{compression-type} |
| Item | Not compressed | Compressed |
|-----------------|-------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Database schema | {database}-schema-create.sql | {database}-schema-create.sql.{compression-type} |
| Table schema | {database}.{table}-schema.sql | {database}.{table}-schema.sql.{compression-type} |
| Data | {database}.{table}.{0001}.{csv&#124;parquet&#124;sql} | {database}.{table}.{0001}.{csv&#124;sql}.{compression-type}<br/>{database}.{table}.{0001}.{compression-type}.parquet |

### Data compression

Expand All @@ -76,7 +110,58 @@ You can compress the exported CSV and SQL data using the following algorithms:
- `zstd`: compress the exported data with `zstd`.
- `none`: do not compress the exported `data`.

## Steps
You can compress the exported Parquet data using the following algorithms:

- `zstd` (default): compress the Parquet file with `zstd`.
- `gzip`: compress the Parquet file with `gzip`.
- `snappy`: compress the Parquet file with `snappy`.
- `none`: do not compress the Parquet file.

### Data conversion

When exporting data to Parquet format, The data conversion between TiDB Cloud Serverless and Parquet are as follows:
shiyuhang0 marked this conversation as resolved.
Show resolved Hide resolved

| TiDB Cloud Serverless Type | Parquest primitive type | Parquet logical type |
|----------------------------|-------------------------|----------------------------------------------|
| VARCHAR | BYTE_ARRAY | String(UTF8) |
| TIME | BYTE_ARRAY | String(UTF8) |
| TINYTEXT | BYTE_ARRAY | String(UTF8) |
| MEDIUMTEXT | BYTE_ARRAY | String(UTF8) |
| TEXT | BYTE_ARRAY | String(UTF8) |
| LONGTEXT | BYTE_ARRAY | String(UTF8) |
| SET | BYTE_ARRAY | String(UTF8) |
| JSON | BYTE_ARRAY | String(UTF8) |
| DATE | BYTE_ARRAY | String(UTF8) |
| CHAR | BYTE_ARRAY | String(UTF8) |
| VECTOR | BYTE_ARRAY | String(UTF8) |
| DECIMAL(1<=p<=9) | INT32 | DECIMAL(p,s) |
| DECIMAL(10<=p<=18) | INT64 | DECIMAL(p,s) |
| DECIMAL(p>=19) | BYTE_ARRAY | String(UTF8) |
| ENUM | BYTE_ARRAY | String(UTF8) |
| TIMESTAMP | INT64 | TIMESTAMP(unit=MICROS,isAdjustedToUTC=false) |
| DATETIME | INT64 | TIMESTAMP(unit=MICROS,isAdjustedToUTC=false) |
| YEAR | INT32 | / |
| TINYINT | INT32 | / |
| UNSIGNED TINYINT | INT32 | / |
| SMALLINT | INT32 | / |
| UNSIGNED SMALLINT | INT32 | / |
| MEDIUMINT | INT32 | / |
| UNSIGNED MEDIUMINT | INT32 | / |
| INT | INT32 | / |
| UNSIGNED INT | FIXED_LEN_BYTE_ARRAY(9) | DECIMAL(20,0) |
| BIGINT | FIXED_LEN_BYTE_ARRAY(9) | DECIMAL(20,0) |
| UNSIGNED BIGINT | BYTE_ARRAY | String(UTF8) |
| FLOAT | FLOAT | / |
| DOUBLE | DOUBLE | / |
| BLOB | BYTE_ARRAY | / |
| TINYBLOB | BYTE_ARRAY | / |
| MEDIUMBLOB | BYTE_ARRAY | / |
| LONGBLOB | BYTE_ARRAY | / |
| BINARY | BYTE_ARRAY | / |
| VARBINARY | BYTE_ARRAY | / |
| BIT | BYTE_ARRAY | / |

## Examples

### Export data to a local file

Expand Down Expand Up @@ -160,16 +245,38 @@ You can compress the exported CSV and SQL data using the following algorithms:
<div label="CLI">

```shell
ticloud serverless export create -c <cluster-id> --s3.bucket-uri <uri> --s3.access-key-id <access-key-id> --s3.secret-access-key <secret-access-key>
ticloud serverless export create -c <cluster-id> --s3.uri <uri> --s3.access-key-id <access-key-id> --s3.secret-access-key <secret-access-key> --filter "database.table"
```

- `s3.bucket-uri`: the Amazon S3 URI with the `s3://<bucket-name>/<file-path>` format.
- `s3.uri`: the Amazon S3 URI with the `s3://<bucket-name>/<file-path>` format.
- `s3.access-key-id`: the access key ID of the user who has the permission to access the bucket.
- `s3.secret-access-key`: the access key secret of the user who has the permission to access the bucket.

</div>
</SimpleTab>

### Export data to Google Cloud Storage

Currently, you can only export data to Google Cloud Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md).

```shell
ticloud serverless export create -c <cluster-id> --gcs.uri <uri> --gcs.service-account-key <service-account-key> --filter "database.table"
```

- `gcs.uri`: the URI of the Google Cloud Storage bucket with the `gs://<bucket-name>/<file-path>` format.
shiyuhang0 marked this conversation as resolved.
Show resolved Hide resolved
- `gcs.service-account-key`: the base64 encoded service account key.

### Export data to Azure Blob Storage

Currently, you can only export data to Azure Blob Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md).

```shell
ticloud serverless export create -c <cluster-id> --azblob.uri <uri> --azblob.sas-token <sas-token> --filter "database.table"
```

- `azblob.uri`: the URI of the Azure Blob Storage with the `azure://<account-name>.blob.core.windows.net/<container-name>/<file-path>` format.
shiyuhang0 marked this conversation as resolved.
Show resolved Hide resolved
- `azblob.sas-token`: the account SAS token of the Azure Blob Storage.

### Cancel an export task

To cancel an ongoing export task, take the following steps:
Expand Down
Loading
Loading