Use shared pool of CUDA streams instead of thread-local pools

[msimberg]

The idea is that the thread-local streams (three by default) may limit concurrency of kernels within a worker thread. Using a single but larger shared pool of streams should lead to more possible concurrency, especially if many independent kernels are submitted quickly after each other from the same worker thread. At the same time, having e.g. three thread-local streams per thread on a Grace CPU is fake concurrency: CUDA will not actually allow allow a concurrency of 3×72=216. Using a shared pool limits the number of created streams to something a bit more manageable, and does not scale unnecessarily with the number of worker threads. Finally, using the same streams concurrently from different threads should not create thread-safety issues.

This change does not noticeably change the performance on P100, MI250, or H100 GPUs.

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ ∅ (target: 90.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (a6a945b)	18222	13762	75.52%
Head commit (b0bc0cf)	18222 (+0)	13762 (+0)	75.52% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1294)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[msimberg]

On old Piz Daint (with P100s) this change does not seem to have a big impact on performance, which is good. I still want to try on H100s before I mark this as ready for review.

[msimberg]

On LUMI (MI250) and Alps (GH200) using this with 32 normal and high priority streams in the pool performs very close to the old thread-local streams (three per thread).