Quic Client Creates Excessive Number of Quinn Endpoints

[ryleung-solana]

Problem

The Quic client creates a new Quinn Endpoint for each connection pool. This can result in excessive port usage in large networks (as each Endpoint uses a separate UDP port).

Proposed Solution

The above behavior was based on performance problems in Quinn observed by @lijunwangs with a single Quinn Endpoint. We should root-cause this perf problem and propose a fix to Quinn, so that we may use a single Quinn Endpoint for all the connection pools.

[ryleung-solana]

Benched on GCE with bench-tps, and did not see any significant difference. However, results may be biased by #32179 which causes inconsistent runs of low TPS in general. Would like to see this fixed and get some consistent runs before concluding that this is no longer a problem.

[apfitzge]

The issues as described in #32179 began with changes to connection cache which affects quic.

[behzadnouri]

From my observations at least, this still seems to be an issue.
i.e. same quinn Endpoint creates contention across connections or send/receive operations on that Endpoint..

[ryleung-solana]

@behzadnouri interesting... what stats are you looking at that show this as a bottleneck?

[behzadnouri]

@behzadnouri interesting... what stats are you looking at that show this as a bottleneck?

Testing turbine on quic, I see evidence that hints to contention on quic endpoint across connections or send/receive operations.

[github-actions]

This repository is no longer in use. Please re-open this issue in the agave repo: https://github.com/anza-xyz/agave