Bug Report: Backup: runtime error on v19.0.0

[L3o-pold]

Overview of the Issue

Updating from 19.0.0-rc.1 to 19.0.0 backups are no longer working.

vtctldclient --server=localhost:15999 --alsologtostderr BackupShard --concurrency=1 "$keyspace/$shard"

E0307 07:50:07.804724 62492 main.go:56] rpc error: code = Unknown desc = caught panic during Backup: runtime error: invalid memory address or nil pointer dereference

Reproduction Steps

1. Running vttablet and vtctld with following flag:

--backup_storage_implementation=s3 \
--backup_engine_implementation=xtrabackup \
--s3_backup_aws_region=us-east-1 \
--s3_backup_storage_bucket=default \
--s3_backup_force_path_style=true \
--s3_backup_aws_endpoint=https://* \
--s3_backup_storage_root=*

2. Run backup

vtctldclient --server=localhost:15999 --alsologtostderr BackupShard --concurrency=1 "$keyspace/$shard"

3. see the error

E0307 07:50:07.804724 62492 main.go:56] rpc error: code = Unknown desc = caught panic during Backup: runtime error: invalid memory address or nil pointer dereference

Binary Version

vtctld version Version: 19.0.0 (Git revision 97fdf78e0f916df2a1404f78208307c9a8de78e1 branch 'HEAD') built on Wed Mar  6 15:31:14 UTC 2024 by vitess@buildkitsandbox using go1.22.1 linux/amd64

Operating System and Environment details

docker vitess/lite images

Log Fragments

vttablet I0307 07:22:48.006242       1 s3.go:359] StartBackup: [s3] dir: commerce/-80, name: 2024-03-07.072248.d2-0000100010, bucket: default
vttablet E0307 07:22:48.007079       1 rpc_server.go:53] TabletManager.Backup(concurrency:1) on d2-0000100010 panic: runtime error: invalid memory address or nil pointer dereference
vttablet runtime/panic.go:261 (0x456e17)
vttablet runtime/signal_unix.go:881 (0x456de5)
vttablet net/http/transport.go:518 (0x71788d)
vttablet net/http/roundtrip.go:17 (0x6fbe52)
vttablet net/http/client.go:259 (0x6b97a3)
vttablet net/http/client.go:180 (0x6b9057)
vttablet net/http/client.go:724 (0x6bb37b)
vttablet net/http/client.go:590 (0x1c990a4)
vttablet github.com/aws/[email protected]/aws/corehandlers/handlers.go:120 (0x1c9908e)
vttablet github.com/aws/[email protected]/aws/corehandlers/handlers.go:112 (0x1c98710)
vttablet github.com/aws/[email protected]/aws/request/handlers.go:267 (0x1c3f375)
vttablet github.com/aws/[email protected]/aws/request/request.go:602 (0x1c42b8c)
vttablet github.com/aws/[email protected]/aws/request/request.go:555 (0x1c423e4)
vttablet github.com/aws/[email protected]/service/s3/api.go:6081 (0x1cd0a3c)
vttablet vitess.io/vitess/go/vt/mysqlctl/s3backupstorage/s3.go:488 (0x1d91b53)
vttablet vitess.io/vitess/go/vt/mysqlctl/s3backupstorage/s3.go:360 (0x1d90d46)
vttablet vitess.io/vitess/go/vt/mysqlctl/backup.go:148 (0x1331b08)
vttablet vitess.io/vitess/go/vt/vttablet/tabletmanager/rpc_backup.go:168 (0x1669247)
vttablet vitess.io/vitess/go/vt/vttablet/grpctmserver/server.go:526 (0x1aec44f)
vttablet vitess.io/vitess/go/vt/proto/tabletmanagerservice/tabletmanagerservice_grpc.pb.go:1793 (0x1acf206)
vttablet google.golang.org/[email protected]/server.go:1680 (0x977587)
vttablet google.golang.org/[email protected]/server.go:1801 (0x979175)
vttablet google.golang.org/[email protected]/server.go:1027 (0x9722aa)
vttablet runtime/asm_amd64.s:1695 (0x479460)

[mattlord]

@L3o-pold since we don't have a test case, is it possible for you to try out the PR branch? #15421 We're pretty confident that fixes the issue, but any confirmation would be great if possible. In any event, thank you for reporting it!

[L3o-pold]

@mattlord I'll look for building the PR branch and try out.

I was asking myself if we are not missing something in our Vitess release process. I find it weird that a release version was not similar as the pre-release (RC) one.

It could be frustrating to find a bug that was not present during the RC testing.

Should we not improve our release process to release RC version with new commit before releasing a final version? It may delay the final release but it could improve the stability of the project :)

[mattlord]

This can happen with any bug fix in any release. The real problem IMO is that we don't have a good mock test framework that is then used to cover all of the functionality for the s3 backup storage plugin.

[deepthi]

Should we not improve our release process to release RC version with new commit before releasing a final version? It may delay the final release but it could improve the stability of the project :)

You make some good points. We should consider this. Making sure that the release commit is the same as a previously released candidate does give us better confidence.

[frouioui]

You make some good points. We should consider this. Making sure that the release commit is the same as a previously released candidate does give us better confidence.

That would imply doing several RC releases though.

[L3o-pold]

but any confirmation would be great if possible. In any event, thank you for reporting it!

@mattlord I can confirm it fixed the issue

[mattlord]

but any confirmation would be great if possible. In any event, thank you for reporting it!

@mattlord I can confirm it fixed the issue

Thank you, @L3o-pold ! I'm sorry that you ran into this, but appreciate you reporting it as always.