Snappy for compression codec using velox as backend #6697
Unanswered
himanshu-zetta
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I need to run gluten on spark with snappy as shuffle compression codec.
As gluten velox backends api support only lz4 and zstd as shown in the figure below:
so, I added snappy in list using this Set("lz4", "zstd", "snappy").
This compiled and I was able to run the spark queries with spark.io.compression.codec=snappy.
But the issue I'm facing which seems like there is no compression happening during snappy with gluten as when I compared the shuffle write with base I observed ~2x write in case of gluten.
@PHILO-HE , @FelixYBW, @weiting-chen could you help me, what else I need to do to use snappy codec for shuffle compression
Beta Was this translation helpful? Give feedback.
All reactions