-
Notifications
You must be signed in to change notification settings - Fork 15
User Guide 1.1.0
In order to use DDS in your Spark shell, you need to add it to your classpath and import the DDS core functions. Depending on the Scala version your Spark is compiled in, select the correct jar. Assuming your Spark is built with 2.11, start your Spark shell with the following parameter and import the core functionality:
spark-shell --jars spawncamping-dds-1.1.0_2.11.2.jarimport de.frosner.dds.core.DDS._DDS comes with a lightweight web server that serves the results and charts to your browser. It pushes JSON objects to the JavaScript front-end that will then display them using HTML, CSS and SVG. The server needs to be started once after the Spark shell has loaded. It can be used for the entire session. However, you can stop and restart it as often as you like.
Note that a visualization is served only once. So currently only one browser window is able to consume the result (the one that asks quickest).
The server can be started by calling the start() function in the Spark shell. You can also specify the interface and port the server should listen to. To start the server listening to 192.168.0.5:8081, execute the following command:
start("192.168.0.5", 8081)The server can be stopped by calling the stop() function in the Spark shell.
To get a list of all available functions use the help() function. To get help for particular functions, you can pass the function name to help(name: String).
Besides the functions based on RDDs, DDS offers a bunch of generic plotting functions not related to Spark. They can be used to plot charts or display tables in the browser from any data source you have.
line[N](values: Seq[N])(implicit num: Numeric[N])Prints a line chart with the given values and a default label.
lines[N](labels: Seq[String], values: Seq[Seq[N]])(implicit num: Numeric[N])Prints a line chart with multiple lines. The label labels(x) corresponds to the value sequence values(x).
bar[N](values: Seq[N])(implicit num: Numeric[N])Prints a bar chart with the given values and a default label.
bars[N](labels: Seq[String], values: Seq[Seq[N]])(implicit num: Numeric[N])Prints a bar chart with multiple bars. The label labels(x) corresponds to the value sequence values(x).
pie[K, V](keyValuePairs: Iterable[(K, V)])(implicit num: Numeric[V])Prints a pie chart with one segment for each key-value pair in the input collection. The keys are assumed to be unique (e.g. values already reduced).
show[V](sequence: Seq[V])(implicit tag: TypeTag[V])Shows the given value sequence in a table. DDS can show sequences of simple values (e.g. strings, numbers) or composite ones (collections, case classes). The values of that sequence need to have a type tag, i.e. if they are custom classes they need to be defined top level.
table(head: Seq[String], rows: Seq[Seq[Any]])Prints a table with the given row-wise data. The head(x) corresponds to each column, e.g. rows(0)(x).
def show[V](rdd: RDD[V], sampleSize: Int = 20)(implicit tag: TypeTag[V])Prints the first lines of the given RDD in a table. The second (optional) argument determines the number of rows to show. DDS can show RDDs of simple values (e.g. strings, numbers) or composite ones (collections, case classes). The values of that RDD need to have a type tag, i.e. if they are custom classes they need to be defined top level.
summarize[N](values: RDD[N])(implicit num: Numeric[N] = null)Shows some basic summary statistics for the given RDD. If the RDD has numeric values, count, sum, min, max, mean, stdev, and variance are calculated. If the RDD has nominal values, DDS will calculate mode and cardinality for you.
groupAndSummarize(readyToGroup: RDD[(K, N)])(implicit num: Numeric[N])Shows some basic summary statistics for each of the groups defined by the given key. It is assumed that there are key-value pairs in each input row, where the key can be used for grouping.
summarizeGroups(grouped: RDD[(K, Iterable[N])])(implicit num: Numeric[N])Shows some basic summary statistics for each of the given groups. It is assumed that there is one input row per group and is usually a result of a group-by operation on an RDD.
groupAndPie(readyToGroup: RDD[(K, N)])(reduceFunction: (N, N) => N)(implicit num: Numeric[N])Computes a pie chart visualizing the numeric values per group. It is assumed that there are key-value pairs in each input row, where the key can be used for grouping. DDS will apply the given reduce function to the values of each group before plotting the reduced value in a segment.
pieGroups(grouped: RDD[(K, Iterable[N])])(reduceFunction: (N, N) => N)(implicit num: Numeric[N])Computes a pie chart visualizing the numeric values per group. It is assumed that there is one input row per group and is usually a result of a group-by operation on an RDD. DDS will apply the given reduce function to the values of each group before plotting the reduced value in a segment.
DDS comes with a set of interactive visualizations at the front-end side. The following sections explains only the visualizations that are truly interactive, i.e. the user can modify it by clicking and pressing buttons. Simple visualizations like tables or line charts are not covered here.
The parallel coordinates plot is a visualization technique to explore high dimensional data. Instead of drawing the axes orthogonally like in a 2D scatter plot, they are drawn in a parallel fashion. One row / data point corresponds to a polyline through all the axes. DDS currently supports two interactions with the parallel coordinates plot: Axis reordering and filtering. DDS offers parallel coordinates for all tabular data.
To reorder the axes, drag one axis on its label to a different position. This can be helpful to detect patterns like correlations or clusters in the data.
You can filter the values by selecting a range in one of the dimensions. Just click and span a filter region on an axis. DDS supports filters on multiple dimensions simultaneously. To change the filter just move it around or change its size. To delete a filter, just click on a different part of the axis that is not in range of the filter.