kafka-log-Cleaner.adoc

Cleaner

Cleaner is created exclusively for a CleanerThread.

Cleaner requires that the number of slots in the given OffsetMap multiplied by the given dupBufferLoadFactor is greater than 1 or the following exception is thrown:

offset map is too small to fit in even a single message, so log cleaning will never make progress. You can increase log.cleaner.dedupe.buffer.size or decrease log.cleaner.threads

Cleaner uses Cleaner [id]: prefix for logging (with the given ID).

Tip	Cleaner uses kafka.log.LogCleaner logger.

Creating Cleaner Instance

Cleaner takes the following to be created:

• ID

• OffsetMap

• ioBufferSize

• maxIoBufferSize

• dupBufferLoadFactor (of the CleanerConfig of the parent LogCleaner) (default: 0.9d)

• Throttler

• Time

• "Check Done" function (TopicPartition ⇒ Unit)

Cleaner initializes the internal properties.

Cleaning Log — clean Method

clean(cleanable: LogToClean): (Long, CleanerStats)

clean…​FIXME

Note	clean is used exclusively when CleanerThread is requested to clean a log.

Cleaning Log — doClean Method

doClean(
  cleanable: LogToClean,
  deleteHorizonMs: Long): (Long, CleanerStats)

doClean…​FIXME

Note	doClean is used when Cleaner is requested to clean a log.

cleanSegments Internal Method

cleanSegments(
  log: Log,
  segments: Seq[LogSegment],
  map: OffsetMap,
  deleteHorizonMs: Long,
  stats: CleanerStats,
  transactionMetadata: CleanedTransactionMetadata): Unit

cleanSegments…​FIXME

Note	cleanSegments is used exclusively when Cleaner is requested to clean a log.

cleanInto Internal Method

cleanInto(
  topicPartition: TopicPartition,
  sourceRecords: FileRecords,
  dest: LogSegment,
  map: OffsetMap,
  retainDeletesAndTxnMarkers: Boolean,
  maxLogMessageSize: Int,
  transactionMetadata: CleanedTransactionMetadata,
  lastRecordsOfActiveProducers: Map[Long, LastRecord],
  stats: CleanerStats): Unit

cleanInto…​FIXME

Note	cleanInto is used exclusively when Cleaner is requested to cleanSegments.

buildOffsetMap Internal Method

buildOffsetMap(
  log: Log,
  start: Long,
  end: Long,
  map: OffsetMap,
  stats: CleanerStats): Unit

buildOffsetMap…​FIXME

Note	buildOffsetMap is used exclusively when Cleaner is requested to doClean.

buildOffsetMapForSegment Internal Method

buildOffsetMapForSegment(
  topicPartition: TopicPartition,
  segment: LogSegment,
  map: OffsetMap,
  startOffset: Long,
  maxLogMessageSize: Int,
  transactionMetadata: CleanedTransactionMetadata,
  stats: CleanerStats): Boolean

buildOffsetMapForSegment…​FIXME

Note	buildOffsetMapForSegment is used exclusively when Cleaner is requested to buildOffsetMap.

Internal Properties

Name	Description
decompressionBufferSupplier	FIXME Used when…​FIXME
readBuffer	FIXME Used when…​FIXME
writeBuffer	FIXME Used when…​FIXME