db: consider adding IterContext

[jbowens]

The iterator stack has a host of configuration and global state. Today a lot of this configuration exists outside the base.InternalIterator interface. This forces iterator construction to propagate this configuration, every internal iterator in the stack to save a copy, and every internal iterator to zero this state on Close. Iterator.Close has an unfortunately high cost on workloads that use an iterator for a one-time seek (#4049) in part due to zeroing these structures.

We might reduce CPU and reduce the memory used by an iterator by propagating some of this information down the iterator stack during an operation by pointer. For example, we could define a base.IterContext struct:

type IterContext struct {
	Compare    Compare
	Equal      Equal
	Split      Split
	LowerBound []byte
	UpperBound []byte
	Prefix     []byte
	Stats      InternalIteratorStats
	Context    context.Context
        Logger     Logger
        Comparer   Comparer
}

Every iteration method of the InternalIterator interface could accept a *IterContext as the first parameter, which would point into a field of the pebble.Iterator struct when iterating using a top-level pebble.Iterator. With a quick survey, it looks like we would be able to remove:

• mergingIter:
  • logger (16 bytes)
  • split (8 bytes)
  • prefix (24 bytes)
  • lower (24 bytes)
  • upper (24 bytes)
  • stats (8 bytes)
• levelIter:
  • ctx (16 bytes)
  • logger (16 bytes)
  • comparer (8 bytes)
  • cmp (8 bytes)
  • split (8 bytes)
  • lower (24 bytes)
  • upper (24 bytes)
  • prefix (24 bytes)
• singleLevelIterator:
  • ctx (16 bytes)
  • cmp (8 bytes)
  • lower (24 bytes)
  • upper (24 bytes)
• colblk.DataBlockIter
  • cmp (8 bytes)
  • split (8 bytes)
• colblk.IndexIter
  • cmp (8 bytes)
  • split (8 bytes)
• arenaskl.Iterator
  • lower (24 bytes)
  • upper (24 bytes)

There are some tricky bits where global state diverges from an individual iterator's state. For example the levelIter avoids propagating bounds if a sstable falls wholly within the bounds. I think we'd still see benefit because, for example, some iterators maintain both global-level bounds and bounds to use within a more constrained iteration context.

Additionally, we might see some benefit to keeping frequently accessed fields (cmp, split) shared by improving cache locality.

Also propagating the iteration prefix through Next will facilitate #3794, because today leaf iterators don't retain the iteration prefix.

Jira issue: PEBBLE-289

[RaduBerinde]

Maybe we can somehow tie the valueBlockReader allocation to the IterContext too - you would only be able to read a lazy value if the IterContext is still valid.