wal: consider lowering failover threshold dynamically

[jbowens]

Failing over the WAL incurs some additional cost (buffering within the WAL writer, synchronization overhead and CPU from double writing). Depending on the distribution of write latencies of the underlying disk and the workload, these costs are minor relative to the write latency of some value X. Ideally we would find X and use it as a failover threshold in order to insulate the workload from high tail latencies.

See internal slack thread https://cockroachlabs.slack.com/archives/C02HW5362HX/p1729872497185199?thread_ts=1729781880.487219&cid=C02HW5362HX

Jira issue: PEBBLE-287