Developer Guide

Developer Guide

Development Environment

The following list contains recommendations for a convenient DDS development environment:

• POSIX oriented operating system (Mac OS X, Linux, ...)
• git
• Scala
• SBT
• Intellij + Scala plugin
• Chrome (especially the developer tools)
• Python + jsbeautifier (in case you want to automatically format your JavaScript code)

You do not have to use the stuff mentioned there but then things might not be as smooth as they could be .

Development Cycle

1. Grab an issue (talk to @FRosner for suggestions)
2. Create an issue branch (git checkout -b issue/<issueNo>)
3. Develop your feature there (put # in commit message)
4. Rebase against master often (you can push a new branch for each rebase, e.g. issue/1337-2, issue/1337-3 etc.)
5. Once you are done, drop a message in the issue mentioning @FRosner and requesting a review (maybe create a pull request as well)
6. React to the reviewer comments and go back to 3. if needed
7. Wait for the branch to be merged

Adding DDS Visualizations

In order to add new visualizations to DDS, you have to do three major steps:

1. Provide a custom implementation of a Servable that stores the data to visualize [Scala]
2. Write one or multiple DDS core functions that send your Servable to the web server [Scala]
3. Tell draw.js how to draw that servable [JavaScript]
4. Extend Visualization prototype and implement the draw and clean-up methods
5. Populate your custom visualization object inside draw.js with the transmitted data from the servable

Extending the Servable Trait

A Servable is basically any object that can be serialized into a JavaScript object. This will be done automatically for you, so you just have to provide three things: (1) a servableType, (2) a spray.json.JsValue which represents the data you want to transport to the front-end and (3) a title.

The servableType will be used in the if statement in the main.js to pick the correct JavaScript prototype for your data. So you need to add another if branch there looking for your type. Make sure that the type string is not already used by another servable.

The JsValue can be anything (an object, an array, a primitive). It will be serialized to a JSON object and put into the content field of the servable object.

The title is a String value and will be displayed in the left of the corresponding header div.

So a simple servable storing a single number representing a row count of an RDD could look like this:

case class RowCount(count: Long) extends Servable {
  override val servableType: String = "rowCount"
  override def contentAsJson: JsValue = JsNumber(count)
  override val title: String = "Count"
}

Then, RowCount(2l) will be serialized to:

{
  "type" : "rowCount",
  "title" : "Count",
  "content" : 2
}

Writing a DDS Core Function

All DDS core functions should be member of the de.frosner.dds.core.DDS object. Although a core function can (and should if possible) call other core functions, eventually there needs to be a call to the private serve(servable: Servable) method, which will transmit the servable to the web UI.

So consider you created your own RowCount extends Servable that is supposed to transmit the row count of an RDD. The corresponding function could look like this:

def rowCount[V](rdd: RDD[V]): Unit = {
  serve(RowCount(rdd.count))
}

Creating a Visualization

The next step is simply to add another if statement to the main.js that is picking up your servable implementation:

// ...
} else if (servable.type == "rowCount") {
  // draw the visualization
} 
// ...

After you did this, you should write the code that actually displays something. To make the visualization reusable, it is advised to implement the Visualization.js prototype (defined as a require.js module). All extending prototypes need to define a _draw(servableContent) and a _clear() method. Documentation about these methods is available in the Visualization.js.

If we just wanted to print the row count as simple HTML, the visualization could look the following:

define(function(require) {

  var Visualization = require("Visualization");

  function RowCount() {}

  RowCount.prototype = new Visualization();
  RowCount.prototype.constructor = Visualization;
  RowCount.prototype.parent = Visualization.prototype;
  
  RowCount.prototype._draw = function(count) {
    this._content.innerHTML = "Row count: " + count;
  }
  
  RowCount.prototype._clear = function() {
    // we don't have any header buttons to clear
  }

  return RowCount;

}

Last but not least, you need to create and add your RowCount to the draw.js:

// ...
} else if (servable.type == "rowCount") {
  toDraw = new RowCount()
    .data(servable.content);
} 
// ...