Skip to content

3. flow averager

Jump to bottom
Alexandre Marcireau edited this page Jun 27, 2018 · 27 revisions

goal: develop an application to calculate the flow average every 100 ms from an Event Stream file

duration: 30 minutes

prerequisites:

result files: basics/3_flow_averager

steps

  1. role of the Tarsier library
  2. functions generating functions
  3. type-agnostic algorithms
  4. the code

role of the Tarsier library

Complex computer vision algorithms can often be described as sequential pipelines of simpler algorithms. The Tarsier library provides a collection of such algorithms, meant to be assembled into pipelines to solve specific problems. The provided event handlers are classes with overloaded call operators (the first solution presented in the DVS event stream counter tutorial). They are observables as well, associated with make functions. The next section details these concepts.

functions generating functions

Let f be an event handler with the signature:

f(event) -> void

If f is an event-triggered observable (besides being an event handler), it may trigger an event while handling one. Let g be the event handler for the observable f. We call f_g the compile-time specialized version of f such that g handles its events. f_g is equivalent to:

f_g(event) -> void {
    // first, do the usual f calculations
    // under certain conditions, a new event is constructed
    if (condition) {
        output_type new_event(...);
        g(new_event); // g handles new_event
    }
}

In order to generate f_g, the generic event handler f is associated with a function make_f, with the following behavior:

make_f(g) {
    return f_g;
}

This syntax has several benefits. First, complex event handlers can be chained using nested make functions. Assuming three event handlers f, g and h, it is possible to build the specialized handler f_g_h by calling:

auto f_g_h = make_f(make_g(h));

Second, it can be generalized to handlers with several output types. If f may generate events with either the type output_type_1 or output_type_2, the function make_f will have the behavior:

make_f(g, h) {
    return f_gh;
}

where g (respectively h) handles events with the type output_type_1 (respectively output_type_2).

type-agnostic algorithms

The Tarsier library does not provide event types. Rather, its algorithms use templates to be compatible with types provided by the user or other libraries. When the observable type (or output type) is different from the handled type (or input type), a conversion function is required. As an example, the tarsier::compute_flow event handler's constructor expects two functions:

  • event_to_flow_event(event event, float vx, float vy) -> flow_event must convert an handled event and computed speeds along the x and y axes into a flow event
  • handle_flow_event(flow_event) -> void must handle a generated flow event

event and flow_event are user-provided types. Thanks to this syntax, a generic flow algorithm can be implemented and used with any event type with at least the properties t, x and y. It is left to the user to decide which properties are passed on to the generated flow event.

A complete example is given in the next section.

the code

Duplicate the project created in the previous tutorial, and name the copied directory flow_averager. Rename flow_averager/source/dvs_event_stream_counter.cpp to flow_averager/source/flow_averager.cpp, and change the solution and project names in flow_averager/premake4.lua. Then, install the Tarsier library by running from the flow_averager directory:

cd third_party
git submodule add https://github.com/neuromorphic-paris/tarsier.git
git submodule update --init --recursive

The application will use the following event handlers:

  • sepia::make_split<sepia::type::dvs> handles a DVS event, and expects two handlers: one for increase (ON) events, and one for decrease (OFF) events. The output events are sepia::simple_event objects, which are DVS events without the is_increase property.
  • tarsier::make_compute_flow<sepia::simple_event, flow_event> handles sepia::simple_event objects, and produces flow_event objects. The latter will be provided by the application.

Write the following content to flow_averager/source/flow_averager.cpp:

#include "../third_party/sepia/source/sepia.hpp"
#include "../third_party/tarsier/source/compute_flow.hpp"
#include <iostream>

/// filename points to the Event Stream file to read.
const auto filename = sepia::join(
    {sepia::dirname(SEPIA_DIRNAME), "third_party", "sepia", "third_party", "event_stream", "examples", "dvs.es"});

/// flow_event contains the parameters of a flow update.
struct flow_event {
    uint64_t t;
    float vx;
    float vy;
} __attribute__((packed));

int main() {
    // read the header
    const auto header = sepia::read_header(sepia::filename_to_ifstream(filename));

    return 0;
}

The custom type flow_event will be used with the flow algorithm. There are no constraints on this type's properties. An application willing to display the flow as a vector field would use a flow_event type with x and y properties. Since the application aims to average the flow regardless its spatial position, these properties are not required. __attribute__((packed)) is a non-standard feature (but supported by all modern compilers) reducing the amount of memory required to store the object.

The function sepia::read_header returns a sepia::header object with properties type, width and height. These parameters will be used to initialize the flow computation algorithm. Write the following content to flow_averager/source/flow_averager.cpp:

#include "../third_party/sepia/source/sepia.hpp"
#include "../third_party/tarsier/source/compute_flow.hpp"
#include <iostream>

/// filename points to the Event Stream file to read.
const auto filename = sepia::join(
    {sepia::dirname(SEPIA_DIRNAME), "third_party", "sepia", "third_party", "event_stream", "examples", "dvs.es"});

/// flow_event contains the parameters of a flow update.
struct flow_event {
    uint64_t t;
    float vx;
    float vy;
} __attribute__((packed));

int main() {
    const auto header = sepia::read_header(sepia::filename_to_ifstream(filename));

    sepia::join_observable<sepia::type::dvs>(
        sepia::filename_to_ifstream(filename),
        sepia::make_split<sepia::type::dvs>(
            tarsier::make_compute_flow<sepia::simple_event, flow_event>(
                header.width,
                header.height,
                2,    // spatial window's radius
                1e6,  // temporal window
                10,   // minimum number of events in the spatio-temporal window required to trigger a flow event
                [](sepia::simple_event simple_event, float vx, float vy) -> flow_event {
                    return {simple_event.t, vx, vy};
                },
                [](flow_event flow_event) {}),
            [](sepia::simple_event) {}));
    return 0;
}

To use the tarsier::compute_flow handler, the associated header "../third_party/tarsier/source/compute_flow.hpp" is included. No linking is required. Unlike the Sepia library, which provides a single header, each Tarsier handler has its own header.

The function tarsier::make_compute_flow<sepia::simple_event, flow_event> is documented here.

The lambda function [](sepia::simple_event simple_event, float vx, float vy) -> flow_event converts an input event into a flow event. Since the flow coordinates are not useful to calculate the average, simple_event.x and simple_event.y are not used.

[](flow_event flow_event) {} is the event handler for flow events. It does nothing for now.

[](sepia::simple_event) {} is the event handler for light decrease events. It does nothing. sepia::make_split<sepia::type::dvs> uses a class-based event handler (provided by Tarsier) as first parameter, and a lambda function-based event handler as second parameter.

To calculate the average flow, you need several state variables captured by reference. Write the following content to flow_averager/source/flow_averager.cpp:

#include "../third_party/sepia/source/sepia.hpp"
#include "../third_party/tarsier/source/compute_flow.hpp"
#include <iostream>

/// filename points to the Event Stream file to read.
const auto filename = sepia::join(
    {sepia::dirname(SEPIA_DIRNAME), "third_party", "sepia", "third_party", "event_stream", "examples", "dvs.es"});

/// flow_event contains the parameters of a flow update.
struct flow_event {
    uint64_t t;
    float vx;
    float vy;
} __attribute__((packed));

int main() {
    const auto header = sepia::read_header(sepia::filename_to_ifstream(filename));

    float vx_sum = 0;
    float vy_sum = 0;
    uint64_t previous_t = 0;
    std::size_t count = 0;

    sepia::join_observable<sepia::type::dvs>(
        sepia::filename_to_ifstream(filename),
        sepia::make_split<sepia::type::dvs>(
            tarsier::make_compute_flow<sepia::simple_event, flow_event>(
                header.width,
                header.height,
                2,    // spatial window's radius
                1e6,  // temporal window
                10,   // minimum number of events in the spatio-temporal window required to trigger a flow event
                [](sepia::simple_event simple_event, float vx, float vy) -> flow_event {
                    return {simple_event.t, vx, vy};
                },
                [&](flow_event flow_event) {
                    ++count;
                    vx_sum += flow_event.vx;
                    vy_sum += flow_event.vy;
                    if (flow_event.t >= previous_t + 100000) {
                        std::cout << "average flow: (" << (flow_event.vx * 1e6) << ", " << (flow_event.vy * 1e6)
                                  << ") pixels / second" << std::endl;
                        previous_t = flow_event.t;
                        count = 0;
                    }
                }),
            [](sepia::simple_event) {}));
    return 0;
}

You can now compile and execute the application by running from the flow_averager directory:

premake4 gmake
cd build
make
release/flow_averager

Windows users must run premake4 vs2010 instead, and open the generated solution with Visual Studio.

You are ready for the ATIS event stream viewer tutorial.

Clone this wiki locally