* Update the docs related Livy build and Spark 2 support

Change-Id: Id59c628e73da46a124f718b192fc92336ac8eace

* Updating the doc

Change-Id: Ibdeb06fde53d359e77aace780a6093b2cc6a70b7

Also added documentation for pyspark.

%table returns wrong header if input is a list of rows. Workaround the issue by converting row to dict.

- SparkYarnAppSpec.can kill spark-submit while it's running.

Added sparkSession to expose Spark 2.0 SparkSession in JobContext. If SparkSession is not supported, it will throw an exception.

Example:
JobHandle<String> handler = client.submit(new Job<String>() {
  @OverRide
  public String call(JobContext jc) throws Exception {
	SparkSession session = jc.sparkSession();
	return session.version();
  }
});

- When RSC connection times out, it should kill the application thru cluster manager to make sure nothing is leaked.

…domly. (#251)

- Redirect stderr to stdout to avoid synchronization between stdout and stderr.

…ter (#235)

Users can list both livy-repl_2.10 and livy-repl_2.11 jars and their dependencies in livy.repl.jars. Livy automatically picks the right jars according to Spark's Scala version and Scala jar versioning convention.

…ding (#252)

* LIVY-271. traceback could be json list or json object which is misleading

- InteractiveSessionSpec.should error out the session if the interpreter dies.

…me specific error. (#243)

- Instead of using regex, parse for internal frames manually.
- Repl in Scala 2.11 cleans internal frames internally. Skip internal frames cleaning.

InteractiveSessionSpec.should report an error if accessing an unknown variable

Change-Id: I1ff274c189212ad7c92dbaa597af68e947364b22

)

Travis will upload logs for diagnosis.

…e. (#247)

Notebook applications like Jupyter might crash while livy is running. No one will clean up its corresponding Livy session and will be leaked.
Heartbeat is added to address this. To keep a session alive, the notebook application must continously make GET requests to the interactive session. If no GET request is made within the heartbeat interval, livy-server will delete the session regardless to its state (busy, idle).

Heartbeat is per session and is controlled by session property "heartbeatTimeoutInSecond". Its default is 0 and it means heartbeat is disabled. To enable heartbeat, please create the session with non-zero "heartbeatTimeoutInSecond" in the create request.

* Add livy REST API to get Livy build info

Change-Id: I56c5116f6f74357d06ceabcad38ac5988003ccf5

* Change some typos

Change-Id: If47570766e19b98b88de521aa57dfae1ae5b48d9

* Address the comments

Change-Id: I961979534281e7bb5030a8c471e5245c7dedca82

* Make Livy server request log configurable

Change-Id: If880908aa1afb02522a9278ddda185715a1f98f0

* Address the comments

Change-Id: I04bb29bf873d7f6341afb42ad4aad34cab35900c

…started and expose the state of RSCClient (#268)

Change-Id: Iae8b04393c4c52d87e87bd451916d6191007db08

…#264)

Currently livy.rsc.jars is a RSC configuration, which means user should specify this configuration each time when creating a session, this is semantically incorrect and inconvenient, also equivalent to livy.repl.jars, we should change this to Livy configuration.

Besides current doc uses livy.jars as a configuration name, no code honors this configuration name, this should also be updated.

…park. (#270)

HiveContext is always created no matter whether we enable hiveContext through spark.repl.enableHiveContext. The root cause is that we depends on shell.py of spark. and unfortunately HiveContext would not initialize itself when created, but defer its initialization until any methods is called. This change would call sqlContext.tables() to check whether hiveContext can work properly.

…#265)

* Add Spnego support for Java Job API client

Change-Id: I8cce80d75a549e573da70be9ef1aeb94457829a4

* Add Python Job API kerberos support

Change-Id: I022b3c14a439321deda3ae5e0cacbc2b18ef9830

* Fix python style

Change-Id: I9e9f00dde93c471c6fa1fd5f71842945f527c73b

* Address the comments

Change-Id: I4994122e8e28276efe4f00fc7ff8e02e0a7a6c60

* LIVY-280. Livy session leakage issue when submitting app timeout

* address comments

* address comments

Change-Id: I014b79102c0759b96c89a5c9177abd286903402a

…e. (#269)

In `Rpc.call()`, RPC function call messages are enqueued to the send buffer of the Channel on `Channel.write()` call because it's called outside of the netty Channel IO event loop.
RPC replies are handled by `RpcDispatcher.channelRead0`. It calls `RpcDispatcher.writeMessage()` which calls `Channel.write()` to send RPC reply messages to the Channel. `RpcDispatcher.channelRead0` is executed in the event loop which means `Channel.write()` writes messages directly to the socket and bypassing the send buffer.
When one side of the RPC channel is making a RPC call but at the same time received another RPC call from the other side. Then this race condition might happen:
- Send call header
- Send reply header
- Send reply payload
- Send call payload
This means RPC reply messages might preempt the on going RPC call and messages will be sent out of order.

To fix this, we have to prevent`RpcDispatcher.channelRead0` from being called while `Rpc.call()` is sending messages. One of the solutions is changing Rpc.call()` to write messages in the event loop.

…PSHOT. (#278)

…eter. (#280)

…uster (#286)

Change-Id: I32dd970bfa1150e3b47ecbfd7c1a7c348bb7e259

Fixed broken integration test on Spark 2.1.

#291)

Emit warnings instead of throwing fatal errors.

#293)

Change-Id: Ibf549db71b0f6dfc2004c3ff602e92860a38a093

Currently a session timeout can't be configured below 1h. This is due to a hard-coded value that was missed in the clean up of LIVY-114 and LIVY-127.

Removed the limit and the conf value (default of 1h) will be used.

Specified fields for client test. Travis CI looks to be defaulted to python 2.7, so it never came across this issue.

Current session gc mechanism has some issues:
- Stopped session still needs to wait to timeout to gc-ed.
-Batch session will be gc-ed unexpectedly in run-time when timing out, which makes long running application impossible.
- Sometimes user doesn't want to stop idle sessions.

Changes of this commit:
- Never check the activity of batch session, which means batch session will only be gc-ed after stop.
- Add a configuration to turn off activity check for interactive session, which meets some usage scenarios.
- Add a configuration to control how long a finished session state will be kept in memory before cleaned out.

To avoid OOM for long running sessions, introduce statement retention mechanism to remove old statements.

Also refactor the statement state code to make it more clear.

…teractive sessions. (#297)

Passing driverProcess to SparkApp for interactive sessions so session/log will return spark-submit log.

- Added instructions to livy-client template.
- Filled out livy-client.conf.template.
- Updated conf naming to all camelCase, switched conf files to templates and updated .gitignore.

Updated pom files that used 4 space indent to use 2 space indent to match the remaining pom files.

* Change Hadoop dependencies to Apache Hadoop

Change-Id: I5500cc0061b03af1587700b7b0a7a147bbf0a333

* Change to use Apache Hadoop 2.7.3

Change-Id: Ied6a69e28fb4a9e4d1dcd97595c7ab253f1eaf82

- Livy server url is exposed as http URL even https is enabled, so we should handle this.
- Livy server SSL keystore password and key password currently set to same configurations, which should be separated.

…sion timeout check is off'. (#310)

Change-Id: I9daec6ac7fdb2c9f727cbab6db042ec144c4edb8

…urn 1 on error. (#311)

Currently if livy-server fails to start it will still return 0 (success), this has been fixed. I also added a livy-server status command that will return a status output similar to that of livy-server start when it's already running or livy-server stop when it's already stopped. Lastly I updated all output text to use livy-server instead of livy_server since that's the actual name of the script.

Tested manually.

* LIVY-313. Fixed SparkRInterpreter always returning success.

- Stopped redirecting stderr to stdout.
- Continue to read ErrorStream (it was only being read once).
- Checking for any errors returned by stderr before returning success.

* Fixing scalastyle check error

* Changing the way errors are handled in SparkRInterpreter

* Fixing scalastyle check error

* Updating SparkRSessionSpec

…300)

Updates to Livy configurations

- Added config deprecation with alternatives to ClientConf, HTTPConf, RSCConf, and LivyConf.
- Added framework for deprecation without alternatives when the need arises.
- Updated naming conventions in code and templates to use - instead of _ or camelCase and deprecated previous configs.
- Updated TestClientConf and added a new test.

…#312)

Some Spark packages are depending on scala-reflect 2.11.0 and it conflicts with Spark's scala version 2.11.8.
This's not Livy's fault actually but doesn't hurt to make Livy more fault tolerant.
Since the scala-reflect jar with the correct version must already be in CLASSPATH, fixed livy-repl to not load user supplied scala-reflect jars.

#315)

Change-Id: Ia8b635dbb9e8ef8e55bbe566967c5abaa5e07020

Change-Id: Id301749a5e678666df771aa8f02cce149b56ff93

…e issue (#323)

* Refactor statement progress tracker to fix binary compatible issue

Change-Id: Ie91fd77472aeebe138bd6711a0baa82269a6b247

* refactor again to simplify the code

Change-Id: I9380bcb8dd2b594250783633a3c68e290ac7ea28

* isolate statementId to job group logic

Change-Id: If554aee2c0b3d96b54804f94cbb8df9af7843ab4

…#319)

* squash of original ui poc code

* Initial All Sessions Page code

* finished all-sessions page and cleaned up code

* Moved metrics and added ui redirect, added no sessions message

* added conf to template, cleaned up missed code from last commit

* UI enable/disable no long configureable, always on

* nit

* Address review and fixed some html/css errors

* Fixed ITs and added a redirect to metrics when ui is disabled

* Add session information log when session is created

Change-Id: I8702ebe1d893cf328b4490e5b5d09f3afd02b7ce

* add more logs

Change-Id: I969edf4e680e59e3ecfbaa50431ae61bd03d4795

* simplify the code

Change-Id: If2c7876ddb6020ace3872c9ae639d69d58b02f48

* Address the comments

Change-Id: Iba5f10ad73ff67b97af8292769b254346b8ef7c8

…igurable (#329)

* Provide Jetty Http Request/Response Header Size Configuration

Kerberos over http requires larger header sizes than the default,
or the HTTP Error 413 Request entity too large will be returned.
This patch increases the default for Livy to 128K and also allows
this to be configurable.

* Change the conf key name

Change-Id: Id274c8cc60d30e5d778f9c447502b7e7a789a8f0

…ons (#331)

* Add unit test to verify large header size configuration

Change-Id: I6c231f9fc9773d1ea40313661b7c49ccfaa44796

* Style fix

Change-Id: I24e617f95fd3e45a674a6b5a691428f0fdabcd89

* Style fix

Change-Id: I91e71979499da8ebba41223b6fe41862de168d03

* Revert the changes

Change-Id: Id7bbd1b2378867c8534b900bc7ba9b1234a9b985

* Add configurations to livy.conf.template

Change-Id: I84d428869bc5cc22aa7f00c6c603ea4a6b052964

…ule six not found issue (#341)

Change-Id: I7ba64be56354be5a03e6f6b349990af6b662e457

…334)

* Code changes in RPCserver for user provided port

* Indentation Changes

* Indentation Changes

* Indentation Changes

* Indentation Changes

* Configuring Port Range

* Documentation Changed

* launcher.port.range will take care of launching RPC

* Checkstyle changes

* Checkstyle changes

* Dummy push

* Code changes

* Changed BindException Handling to SocketException Handling

* Changed Import Order

* Code changes to increase port range

* Set Port isConntect to true

* Indentation Changes & port range in livy-client.conf.template

* Indentation changes

* Changed visibilty of method private

* Indentation Changes

* Indenetation Changes

* Unit test case to test port range

* Checkstyle changes

* Unit test case for port range

* Added comment for Port Range Configuration and increase port range for unit test case