pageserver: implement buffer bypass for aligned parts of the write

[yliang412]

After implementing #9693, our buffered writer always copies writes into internal aligned buffer and write them out once a buffer is full due to O_DIRECT alignment requirement.

For larger writes (e.g. download_object, image layer values), it might make sense to implement buffer bypass for aligned parts of the write and avoid copying majority of the data into the internal buffer. Doing so allows us to save cpu cycles and potentially improve throughput.

Requirements

• Create a wrapper type that wraps around bytes::Bytes. Make sure the view into the buffer is aligned at construction, impl aligned marker trait.

The actual workflow is something like

• Memcpy unaligned head part of the buffer into the internal buffer and then flush.
  • We might need to pad zeros so that a single blob is contiguous on disk, and contiguous in vectored read. However this introduces internal fragmentation.
• flush aligned part of the buffer.
• Memcpy unaligned tail into the internal buffer.

For instance:

# `X` represents the bytes originally in the internal aligned buffer.
# `E` represents the bytes originally in the external maybe not-aligned buffer.

IoBufferMut                        Bytes
+-------+-------+-------+-------   ----+-------+----
XXXXXXXXXXXXXXXXXXXXXXXXXX         EEEEEEEEEEEEEEEEE
+-------+-------+-------+-------   ----+-------+----
A       A       A       A              A       A
       
                          ||
                          \/

IoBufferMut (flush)                    IoBufOwnedSlice<Bytes> (flush)
+-------+-------+-------+-------       +-------
XXXXXXXXXXXXXXXXXXXXXXXXXX00EEEE       EEEEEEEE
+-------+-------+-------+-------       +-------
A       A       A       A              A       

IoBufferMut                     
+-------+-------+-------+-------
EEEEE
+-------+-------+-------+-------
A       A       A       A     

Related

Using a mpsc channel to send request with oneshot channel callbacks to recycle the buffer might work better than current two mpsc channel based utils::sync::Duplex.