Home

Welcome to the Orchestrator wiki!

Advanced Configuration Examples

Upload a file using POST

When having an endpoint where to upload a file, for example for audio or video processing, that can be done with a configuration file like the following:

ip = "ip.to.upload.com"
port = 9000
method = "POST"
requestUrl = "uploadThisFile"
response.json.path = "uuid"
outputField = "upload_uuid"
uploadFile {
  name = "file",
  mime = "video/mp4",
  filePath = "${video_path}"
}

That configuration file will look into each document for a field video_path and upload that file to the endpoint defined, in this case that would be a POST to htpp://ip.to.upload.com/uploadThisFile using the defined mime type. The bitcode for the video will be posted within a field called file.

Polling

With polling the pipeline will try again and again a certain request until a defined field contains certain string.

The parameter requestDelayMs (which is optional for every module definition) can be used to define the waiting time between calls. The execution will stop when the timeout defined in the pipeline configuration, in the parameter executionTimeoutSeconds is reached.

For example, having this configuration for a module:

ip = "some.service.com"
port = 9000
method = "GET"
requestUrl = "status/${upload_uuid}"
response.json.path = "data.status"
polling.condition = "DONE"
outputField = "processing_status"
requestTimeoutSeconds=3000
requestDelayMs=60000

That would make calls to hhtp://some.service.com:9000/status/asdasdasdasd if asdasdasdasd is upload_uuid in the input document until the value of data.status is set to "DONE". The time between calls will be 60 seconds as defined in requestDelayMs. That will continue until the condition is met or the executionTimeoutSeconds is met.

Using pivots (similar to a flatmap)

This should be used when the output of a module returns an array of results and each of those results should be processed separately in subsequent steps of the Orchestrator pipeline.

For example, in a pipeline for processing audio files, first there are some steps that extract emotions from audio. Then, the next step makes a transcription of such audio and will split that into sentences. We finally want to have of those sentences processed separately.

Using "Pivots" a new document will be created for each sentence with that particular sentence in the output field and with a copy of the rest of the fields.

So, taking this module configuration as an example:

ip = "some.example.ip"
port = 1234
method = "GET"
requestUrl = "extractTranscription?uuid=${uuid}"
response.json.map {
  arousal = "entries.emotions.arousal"
  valence = "entries.emotions.valence"
  text = "Text"
}
response.json.pivotPath = "response"
pivotId="${video_id}"
pivotName="videoId"
outputField = "audioAnalysis"

And this as the input

{ "uuid":"asdasd234233452",
  "videoId":"video 1"
  "sentiment": 1,
  "author": "someguy",
}

Being the following json the response of the module:

{"results": 
    [ {"entries":
         {
            "arousal":3,
            "valence": 5,
            "text": "first sentence"
         }
      }
    ,
    {"entries":
        {
           "arousal":1,
            "valence": 3,
            "text": "second sentence"
        }
    ]
}

Then the output would be:

[
{ "uuid":"asdasd234233452",
  "videoId":"video 1"
  "sentiment": 1,
  "author": "someguy",
  "audioAnalysis":  {
            "arousal": 1,
            "valence": 3,
            "text": "first sentence"
         }
},
{ "uuid":"asdasd234233452",
  "videoId":"video 1"
  "sentiment": 1,
  "author": "someguy",
  "audioAnalysis":  {
            "arousal":3,
            "valence": 5,
            "text": "second sentence"
         }
}
]