tso client receive GOAWAY when idle with keepalive

[yongman]

Bug Report

What did you do?

Create tso client with grpc keepalive enabled.

            let cb = ChannelBuilder::new(self.env.clone())
                .max_send_message_len(-1)
                .max_receive_message_len(-1)
                .keepalive_time(Duration::from_secs(10))
                .keepalive_timeout(Duration::from_secs(3))
                .max_reconnect_backoff(Duration::from_secs(5))
                .initial_reconnect_backoff(Duration::from_secs(1));

The parameter is same as the argument with legacy pd tso client in TiKV. The tso stream will receive GOAWAY after the stream is idle for more than 30 seconds.

What did you expect to see?

The connection should not be disconnected.

What did you see instead?

[2024/04/23 14:26:09.673 +08:00] [INFO] [<unknown>] ["ipv4:127.0.0.1:2479: Got goaway [11] err={\"created\":\"@1713853569.672943563\",\"description\":\"GOAWAY received\",\"file\":\"/data/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.10.3+1.44.0-patched/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc\",\"file_line\":1057,\"grpc_status\":14,\"http2_error\":11,\"raw_bytes\":\"too_many_pings\"}"]
[2024/04/23 14:26:09.673 +08:00] [ERROR] [<unknown>] ["Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to \"too_many_pings\""]

What version of PD are you using (pd-server -V)?