Document using with YARN #35

devinsba · 2017-03-08T19:30:09Z

I'll preface this by saying I have zero operational knowledge about spark so...

In order to use this in AWS with EMR we will need to be able to connect to a YARN controlled Spark cluster. I'm not sure if currently the project even supports YARN but it would be good to know and have some documentation around if/when it does.

codefromthecrypt · 2017-03-09T13:24:10Z

looks like normally yarn is a different distribution, which special cases the magic value "yarn" as the cluster name. It might be possible to hack our main class to do the same, or we document using spark-submit, treating this app as a normal jar.

here are some related links

https://spark.apache.org/docs/1.6.3/running-on-yarn.html
http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/

devinsba · 2017-03-09T13:27:31Z

EDIT: jinx again.. 😄

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document using with YARN #35

Document using with YARN #35

devinsba commented Mar 8, 2017

codefromthecrypt commented Mar 9, 2017 •

edited

Loading

devinsba commented Mar 9, 2017 •

edited

Loading

Document using with YARN #35

Document using with YARN #35

Comments

devinsba commented Mar 8, 2017

codefromthecrypt commented Mar 9, 2017 • edited Loading

devinsba commented Mar 9, 2017 • edited Loading

codefromthecrypt commented Mar 9, 2017 •

edited

Loading

devinsba commented Mar 9, 2017 •

edited

Loading