storage: The S3-compatible API of Google Cloud Storage does not support DeleteObjects

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Create a GCS bucket.
2. Create a TiCDC v7.6 changefeed with redo-log enabled (setting consistent.level = 'eventual'), and set the storage to s3://bucket/?endpoint=http://storage.googleapis.com (thus accessing GCS through their S3-compatible XML API).
3. Wait for a while.

2. What did you expect to see? (Required)

No error.

3. What did you see instead (Required)

Changefeed stopped because of incompatible API:

[2024/02/01 00:00:00.000 +00:00] [WARN] [meta_manager.go:159] ["redo: init redo meta fail"] [namespace=default] [changefeed=«redacted»] [error="[CDC:ErrS3StorageAPI]external storage api: NotImplemented: A header or query you provided requested a function that is not implemented.\n\tstatus code: 400, request id: , host id: "]

4. What is your TiDB version? (Required)

v7.6.0

[kennytm]

• No this is not an issue about TiCDC. TiCDC uses the external storage API in the BR package, thus the labels.
• The error happened because TiCDC since sink(ticdc): cleanup expired files by day for storage sink tiflow#10097 used the DeleteFiles() function introduced from ddl: clean up cloud storage files after global sort #47084.
• On S3 (and KS3), this maps to the DeleteObjects (POST /?delete) API for batch deletion in a single call. (Meanwhile other storage implementations just loop and delete each file one-by-one, including GCS).
• The DeleteObjects API is not supported in the GCS XML API. This caused the "NotImplemented" error shown in the issue description.

The GCS JSON API (gcs://) uses an entirely different approach for batch delete (https://cloud.google.com/storage/docs/batch), so I believe there is no plan for Google to support DeleteObjects in the XML API.

For this issue we either

• declare using s3:// for GCS is bad, users should just use gcs://, and close this issue, or
• do support s3:// for GCS, make DeleteFiles() detect the HTTP 400, and fallback to use a loop

[BornChanger]

@kennytm can you please open the issue to tiflow repo instead? cc @flowbehappy

[BornChanger]

/remove-component br

[kennytm]

@BornChanger I have explained why the issue is filed here in the first line in #50918 (comment). I'm keeping the labels unless there is a dedicated label for the external-storage component.