Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

partitioner sometimes violates the n + 1 rule #1804

Open
johnynek opened this issue Feb 16, 2018 · 6 comments
Open

partitioner sometimes violates the n + 1 rule #1804

johnynek opened this issue Feb 16, 2018 · 6 comments

Comments

@johnynek
Copy link
Collaborator 

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 5 was not less than or equal to 3
[info]     Location: (WritePartitionerTest.scala:101)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(Mapped(CoGroupedPipe(Pair(IdentityReduce(scala.math.Ordering$Int$@1457f692,WithDescriptionTypedPipe(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(FlatMapped(WithDescriptionTypedPipe(ForceToDisk(WithDescriptionTypedPipe(Fork(WithDescriptionTypedPipe(Fork(EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),org.scalacheck.GenArities$$Lambda$441/1591575691@c8f3e1d),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),org.scalacheck.GenArities$$Lambda$441/1591575691@261ad18c),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),None,List(),ReflexiveEquality()),IdentityReduce(scala.math.Ordering$Int$@1457f692,WithDescriptionTypedPipe(WithDescriptionTypedPipe(MapValues(WithDescriptionTypedPipe(FlatMapped(SourcePipe(com.twitter.scalding.source.FixedTypedText(uMvxarCxpAubxith6fe8axecny)),org.scalacheck.GenArities$$Lambda$441/1591575691@18de4736),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),org.scalacheck.GenArities$$Lambda$441/1591575691@7a77d7b0),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),None,List(),ReflexiveEquality()),<function3>)),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))
[info]     )

It would be nice to fix this, and maybe even get it take exactly the same number of steps as cascading.
@johnynek
Copy link
Collaborator Author

another example:

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 4 was not less than or equal to 3
[info]     Location: (WritePartitionerTest.scala:101)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(FlatMapped(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),List((ctse,false))),WithDescriptionTypedPipe(Fork(EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),org.scalacheck.GenArities$$Lambda$441/1312778964@9247412),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))
[info]     )

@johnynek
Copy link
Collaborator Author

another:

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 4 was not less than or equal to 3
[info]     Location: (WritePartitionerTest.scala:101)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(IterablePipe(List(0, 1358996629, -1, 1176242286, -799433509, 1695053710, -1, 2147483647, 2147483647, 562222783, 2147483647, -1, 1215429577, 1037781522, 2147483647, -1, 1167458391, 0, 2147483647, 2147483647, 2147483647, 1, 1, -2147483648, 2147483647, 1962348912)),WithDescriptionTypedPipe(MergedTypedPipe(IterablePipe(List(-738719769, 1, 1, 318544573, 729584317)),WithDescriptionTypedPipe(Fork(WithDescriptionTypedPipe(Fork(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(SourcePipe(com.twitter.scalding.source.FixedTypedText(wkprxxmjzdqdcsvcqzB)),EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))
[info]     )

@johnynek johnynek mentioned this issue Feb 16, 2018
Merged
@johnynek
Copy link
Collaborator Author

Looks like this still rarely fails:

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 4 was not less than or equal to 2
[info]     Location: (WritePartitionerTest.scala:102)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(Filter(IterablePipe(List(-2147483648, 283231238, -2147483648, -1945810342, 0, 0, 915272271, 2147483647, -576715933, 287388012, -1, -1749491001, -1, -1046239738, 0, -272081409, 57139275, -434808636, -911187964, -126527959, 2147483647)),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),WithDescriptionTypedPipe(Fork(EmptyTypedPipe),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),WithDescriptionTypedPipe(WithDescriptionTypedPipe(WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),List((hssiml2nqfch5jbI2fpctMdiunkgqoidshq,false))),List((fxg6nrzx,false)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))
[info]     )

@johnynek johnynek reopened this Feb 18, 2018
@johnynek
Copy link
Collaborator Author 

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 3 was not less than or equal to 1
[info]     Location: (WritePartitionerTest.scala:102)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(FlatMapped(WithDescriptionTypedPipe(Fork(WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),SourcePipe(com.twitter.scalding.source.FixedTypedText(fubkmllkDpjlekh6lnwjqcdwaQrwSr8J3Rq0))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),WithDescriptionTypedPipe(TrappedPipe(WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(Mapped(EmptyTypedPipe,<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),com.twitter.scalding.source.FixedTypedText(azyydPsd3fykcqnh),Single(com.twitter.scalding.TupleGetter$IntGetter$@3db992f4)),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))
[info]     )

@johnynek johnynek mentioned this issue Feb 20, 2018
Merged
@johnynek
Copy link
Collaborator Author

Here is another example of this:

[info] - the total number of steps is not more than cascading *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 3 was not less than or equal to 1
[info]     Location: (WritePartitionerTest.scala:101)
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(Filter(WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),org.scalacheck.GenArities$$Lambda$441/1121672652@3ed6ad6a),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),WithDescriptionTypedPipe(FlatMapped(WithDescriptionTypedPipe(ForceToDisk(EmptyTypedPipe),List((org.scalacheck.Gen$R.map(Gen.scala:237),true))),scala.Function1$$Lambda$10/1663166483@19568812),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R.map(Gen.scala:237),true)))
[info]     )

@johnynek
Copy link
Collaborator Author

the past two had ForceToDisk(EmptyTypedPipe). I wonder if that causes an issue for the partitioner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@johnynek