Refactor `SummerBuilder` to make it possible to use `Counters` on `SummerBuilder` creation #738

ttim · 2017-06-27T06:48:10Z

Currently the way to customize Summer instances is not perfect - we have an old 'legacy' way based on a lot of different options and 'new' way with setting SummerBuilder directly.
At the same time with new way it's impossible to fully utilize Counters because jobId is needed to create them. Also new way is quite verbose.

This PR adds a way to use Counters in SummerBuilder and couple of SummerBuilders: Null, sync and async.

…mmer` creation

ttim · 2017-06-27T06:48:55Z

There is an issue for that: #711

codecov-io · 2017-06-27T07:11:25Z

Codecov Report

Merging #738 into develop will decrease coverage by 0.13%.
The diff coverage is 39.28%.

@@             Coverage Diff             @@
##           develop     #738      +/-   ##
===========================================
- Coverage    71.99%   71.86%   -0.14%     
===========================================
  Files          152      153       +1     
  Lines         3721     3735      +14     
  Branches       208      202       -6     
===========================================
+ Hits          2679     2684       +5     
- Misses        1042     1051       +9

Impacted Files	Coverage Δ
...om/twitter/summingbird/online/option/AllOpts.scala	`62.5% <0%> (-20.84%)`	⬇️
...om/twitter/summingbird/online/option/Summers.scala	`32.35% <32.35%> (ø)`
...la/com/twitter/summingbird/storm/BuildSummer.scala	`50% <55%> (+6.52%)`	⬆️
.../twitter/summingbird/memory/ConcurrentMemory.scala	`97.14% <0%> (+1.42%)`	⬆️
.../main/scala/com/twitter/summingbird/Producer.scala	`77.27% <0%> (+1.51%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f8dd8f...22532c0. Read the comment docs.

ttim · 2017-06-27T07:34:55Z

@johnynek I know here is quite a lot of code without test coverage, if you think it worth it to add tests on that - I can do this. But personally I think it's ok to leave this kind of code without tests.

pankajroark · 2017-06-27T16:23:45Z

summingbird-online/src/main/scala/com/twitter/summingbird/online/option/AllOpts.scala

-case class SummerConstructor(get: SummerBuilder)
+case class SummerConstructor(get: SummerConstructorSpec)
+
+trait SummerConstructorSpec {


Why is it called SummerConstructorSpec, there's no SummerConstructor in definition.

After offline discussion renamed it to SummerWithCountersBuilder.

pankajroark · 2017-06-27T16:27:21Z

summingbird-online/src/main/scala/com/twitter/summingbird/online/option/AllOpts.scala

+case class SummerConstructor(get: SummerConstructorSpec)
+
+trait SummerConstructorSpec {
+  def builder(counter: Name => Counter with Incrementor): SummerBuilder


tbh, I think we should just change the definition of SummerBuilder to take the counterBuilder as argument and default to a no-op counterBuilder. SummerBuilder is used very rarely and I think it's ok to break it. There are only a couple places that we'll need to fix at Twitter. Seems better than adding a level of indirection here.
@johnynek wdyt?

I did this change in backward compatible manner - it's still possible to create SummerWithCountersBuilder out of plain SummerBuilder.

pankajroark · 2017-06-27T19:32:01Z

summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala

+        val valueCombinerCrushSize = option(DEFAULT_VALUE_COMBINER_CACHE_SIZE)
+        val doCompact = option(CompactValues.default)
+        Summers.async(
+          cacheSize, flushFrequency, softMemoryFlush, asyncPoolSize, doCompact, valueCombinerCrushSize


too long, move each param to separate line.

johnynek

👍

some minor comments, but it looks like a nice refactor.

johnynek · 2017-06-28T18:24:23Z

summingbird-online/src/main/scala/com/twitter/summingbird/online/option/AllOpts.scala

+case class SummerConstructor(get: SummerWithCountersBuilder)
+
+trait SummerWithCountersBuilder {
+  def create(counter: Name => Counter with Incrementor): SummerBuilder


I am not crazy about with what about Name => (Counter, Incrementor) do they have to be bound together?

I prefer to keep it this way (or maybe change to just Incrementor). It should be the same object, not both of them.

Changed to just Incrementor.

johnynek · 2017-06-28T18:24:53Z

summingbird-online/src/main/scala/com/twitter/summingbird/online/option/AllOpts.scala

-case class SummerConstructor(get: SummerBuilder)
+case class SummerConstructor(get: SummerWithCountersBuilder)
+
+trait SummerWithCountersBuilder {


you probably want this to extend Serializable or we may have some corner case issues.

Actually I prefer to keep it non serializable, this code (creating actual SummerBuilder with Counters) should happen only on submitter node, and SummerBuilder is an object which should be serializable (and it is).

Commented in code

johnynek · 2017-06-28T18:27:46Z

summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala

+  def apply(builder: StormTopologyBuilder, node: StormNode): SummerBuilder = {
+    val summerBuilder = builder.get[SummerConstructor](node)
+      .map { case (_, constructor) => constructor.get }.getOrElse {
+      logger.info(s"[${builder.getNodeName(node)}] use legacy way of getting summer builder")


can we indent here?

ttim · 2017-06-30T03:54:51Z

@johnynek @pankajroark can you look again did I point all your comments? Thanks!

johnynek · 2017-06-30T19:51:13Z

👍

Refactor SummerBuilder to make it possible to use Counters on `Su…

1b1f86f

…mmer` creation

ttim requested review from pankajroark and johnynek June 27, 2017 06:49

Introduce summerConstructorSpec to make logging output nicer

c6bb6ac

pankajroark reviewed Jun 27, 2017

View reviewed changes

SummerConstructorSpec -> SummerWithCountersBuilder

1f8a744

pankajroark approved these changes Jun 28, 2017

View reviewed changes

johnynek reviewed Jun 29, 2017

View reviewed changes

Pankaj's and Oscar's comments

22532c0

ttim merged commit c272b2a into twitter:develop Jun 30, 2017

ttim deleted the summer_build_params branch July 1, 2017 00:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor `SummerBuilder` to make it possible to use `Counters` on `SummerBuilder` creation #738

Refactor `SummerBuilder` to make it possible to use `Counters` on `SummerBuilder` creation #738

ttim commented Jun 27, 2017

ttim commented Jun 27, 2017

codecov-io commented Jun 27, 2017 •

edited

Loading

ttim commented Jun 27, 2017

pankajroark Jun 27, 2017

ttim Jun 27, 2017

pankajroark Jun 27, 2017

ttim Jun 27, 2017

pankajroark Jun 27, 2017

johnynek left a comment

johnynek Jun 28, 2017

ttim Jun 29, 2017

ttim Jun 29, 2017

johnynek Jun 28, 2017

ttim Jun 29, 2017

ttim Jun 29, 2017

johnynek Jun 28, 2017

ttim Jun 29, 2017

ttim commented Jun 30, 2017

johnynek commented Jun 30, 2017

Refactor SummerBuilder to make it possible to use Counters on SummerBuilder creation #738

Refactor SummerBuilder to make it possible to use Counters on SummerBuilder creation #738

Conversation

ttim commented Jun 27, 2017

ttim commented Jun 27, 2017

codecov-io commented Jun 27, 2017 • edited Loading

Codecov Report

ttim commented Jun 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnynek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttim commented Jun 30, 2017

johnynek commented Jun 30, 2017

Refactor `SummerBuilder` to make it possible to use `Counters` on `SummerBuilder` creation #738

Refactor `SummerBuilder` to make it possible to use `Counters` on `SummerBuilder` creation #738

codecov-io commented Jun 27, 2017 •

edited

Loading