Producer and consumer diagnostics

[soujiro32167]

I'd like to create middleware for producers and consumers, to do things like distributed tracing, logging, metrics and so on.

Here is an example of instrumenting a producer and consumer with a variation of trace4cats

As you can see, its pretty clunky: I have to backfill all the aliases. It would be nice to be able to do something like

val tracedProducer = producer
  .tapChunk(c => onChunkProduced(c))
  .tapAck(a => onAckReceived(a))

val tracedCosumer = consumer
  .tapChunk(c => onChunkConsumed(c))
  .tapCommit(c => onCommit(c))

[svroonland]

See #1048

[erikvanoosten]

@svroonland We do not actually have diagnostics for producers yet and as far as I know there is no issue requesting those. Should we keep this one open?

[svroonland]

@soujiro32167 This could be of interest for you https://github.com/pierangeloc/zio-theseus/tree/main/modules/tracing-kafka/src/main/scala/io/tuliplogic/ziotoolbox/tracing/kafka

[svroonland]

Also we have consumer metrics since v2.7.3

[erikvanoosten]

@erikvanoosten I'm not sure what to do with this issue. Tracing, logging and metrics are probably covered somewhat by your work on open telemetry / open tracing. Producer already exposes some metrics via the underlying apache producer. What do you think?

The consumer already has a diagnostics API which can you use for most of these use cases.

Indeed, tracing is covered by #1345 and #1356.

In #1345 we create a mechanism to wrap producers with the @@ annotation syntax. You still need to duplicate all the methods of Producer but it is made a lot easier by providing the DefaultProducer trait.

If we complete and merge #1356, I am not sure if we still want to merge the annotation part #1345 as well. Its a lot of code for a small use case.

Indeed, more metrics can be had from the Java consumer / producer. We don't expose those via zio-metrics, but a bridge could be added (I'd say: PRs welcome 😉).

IMHO for logging you are better of with putting this directly in the consumer/producer and not as a wrapper. Again: too much code for a small use case.

The API proposed in this issue is nice, but it looks it will result in a lot of non-composable code in the library. Perhaps we are better off with 'producer diagnostics', something similar to the consumer diagnostics.

In other words: lots of options and feedback is greatly appreciated.

[svroonland]

@erikvanoosten I'm not sure what to do with this issue. Tracing, logging and metrics are probably covered somewhat by your work on open telemetry / open tracing. Producer already exposes some metrics via the underlying apache producer. What do you think?

[soujiro32167]

I would love something like consumer diagnostics for the producer