build: use profile-guided optimization

[nvanbenschoten]

Go 1.20 (https://tip.golang.org/doc/go1.20) added support for profile-guided optimization (PGO). Go 1.21 (https://tip.golang.org/doc/go1.21) further improves PGO and marks it as ready for general use.

PGO promises to improve the performance of Go programs:

Performance improvements vary depending on application behavior, with most programs from a representative set of Go programs seeing between 2 and 7% improvement from enabling PGO.

PGO currently achieves this by inline hot function calls, but the Go team expects it to become more capable in future releases: golang/go#62463.

Once #109773 lands, we should test out PGO and explore integrating it into our CI pipeline.

Jira issue: CRDB-31351

Epic CRDB-41952

[nvanbenschoten]

Go 1.22 is expanding PGO's capabilities:

Profile-guided Optimization (PGO) builds can now devirtualize a higher proportion of calls than previously possible. Most programs from a representative set of Go programs now see between 2 and 14% improvement from enabling PGO.

[lyang24]

I read more about pgo its very interesting - the hard part is it needs a pprof to guide the optimization, do you have guidances on how get a proper profile?

bazel already support pgo, once i get profile i can drop into the folder that contains the main.go, and add pgoprofile = ":<profile_name>.pprof", to
https://github.com/lyang24/cockroach/blob/d0bb7e47c6649dce2d8efa0c26b999ed5d161fe6/pkg/cmd/cockroach/BUILD.bazel#L25-L30

[rickystewart]

Hi @lyang24, in release-24.1 you can manually collect a profile on the Advanced Debug page or if you wish there is a method to set up automatic profiling. Here is a draft commit showing how to add profiles to the build (the profiles here are one-offs for an internal experiment that we did). You could also directly use the profiles from this commit if you like. If you do please be aware that we do not intend to merge this commit as-is and the provided profiles may not be representative and the PGO build may not be "production-ready"; your mileage may vary and this is not supported by Cockroach Labs.

Note that building with PGO is substantially slower than building without it. It also has implications for caching with respect to the build especially if you are doing development, running tests, etc.

[tbg]

When this gets tackled we'll also want to include a sysbench profile. (I'm happy to provide one, just DM me).