4. ATIS event stream viewer
goal: develop an application to display events emitted by an ATIS Event Stream file
duration: 1 hour
prerequisites:
- Having completed the flow averager tutorial
result files: basics/4_atis_event_stream_viewer
Open a terminal and run:
sudo apt install qtbase5-dev qtdeclarative5-dev qml-module-qtquick-controls qml-module-qtquick-controls2 # GUI toolkitOpen a terminal and run:
brew install qt # GUI toolkitDownload and install Qt. Select the latest 32-bits version when asked. You may want to restrict the installation to your platform, as the default setup will install pre-compiled versions for other platforms as well, and take up a lot of space. After the installation, open a command prompt as administrator and run:
mklink /D c:\Qt\opt c:\Qt\5.11.1\msvc2015You may need to change 5.11.1 and msvc2015 to match your Qt version and platform.
Finally, add c:\Qt\opt\bin to your path and reboot your computer.
The ATIS camera is an extension to DVS cameras. Each pixel has two light sensitive areas, and two logic circuits. The first one behaves like a DVS's, and produces light increase (ON) and light decrease (OFF) events. The second circuit is triggered by the first one, and measures absolute luminance. Rather than transmitting a value on a fixed number of bits, the exposure measurement circuit sends a pair of events called threshold crossings. The first threshold crossing is sent when light integration starts. The second threshold crossing is sent when the integrated luminance reaches a tunable threshold. The luminance is proportional to the inverse of the time difference (or ∆t) between the two threshold crossings. The sepia::atis_event type provided by the Sepia library is defined as:
namespace sepia {
struct atis_event {
/// t represents the event's timestamp.
uint64_t t;
/// x represents the coordinate of the event on the sensor grid alongside the horizontal axis.
/// x is 0 on the left, and increases from left to right.
uint16_t x;
/// y represents the coordinate of the event on the sensor grid alongside the vertical axis.
/// y is 0 on the bottom, and increases bottom to top.
uint16_t y;
/// is_threshold_crossing is false if the event is a change detection, and true if it is a threshold crossing.
bool is_threshold_crossing;
/// change detection: polarity is false if the light is decreasing.
/// exposure measurement: polarity is false for a first threshold crossing.
bool polarity;
};
}Duplicate the project created in the previous tutorial, and name the copied directory atis_event_stream_viewer. Rename atis_event_stream_viewer/source/flow_averager.cpp to atis_event_stream_viewer/source/atis_event_stream_viewer.cpp, and change the solution and project names in atis_event_stream_viewer/premake4.lua. Then, install the Chameleon library by running from the atis_event_stream_viewer directory:
cd third_party
git submodule add https://github.com/neuromorphic-paris/chameleon.gitChameleon components use the Qt framework. The latter requires an extra compilation step, called moc, which must happen before the actual C++ compilation. Chameleon provides the script qt.lua to use Qt in a premake project. Write the following content to atis_event_stream_viewer/premake4.lua:
local qt = require 'third_party/chameleon/qt'
solution 'atis_event_stream_viewer'
configurations {'release', 'debug'}
location 'build'
project 'atis_event_stream_viewer'
kind 'ConsoleApp'
language 'C++'
location 'build'
files {'source/*.qml', 'source/*.cpp'}
buildoptions {'-std=c++11'}
linkoptions {'-std=c++11'}
files(qt.moc({
'third_party/chameleon/source/background_cleaner.hpp',
'third_party/chameleon/source/dvs_display.hpp',
'third_party/chameleon/source/delta_t_display.hpp'},
'build/moc'))
includedirs(qt.includedirs())
libdirs(qt.libdirs())
links(qt.links())
buildoptions(qt.buildoptions())
linkoptions(qt.linkoptions())
defines {'SEPIA_COMPILER_WORKING_DIRECTORY="' .. project().location .. '"'}
configuration 'release'
targetdir 'build/release'
defines {'NDEBUG'}
flags {'OptimizeSpeed'}
configuration 'debug'
targetdir 'build/debug'
defines {'DEBUG'}
flags {'Symbols'}
configuration 'linux'
links {'pthread'}Each Chameleon component used in your application must appear in qt.moc's first argument. If you change this list, you must run premake4 gmake before recompiling.
The ATIS event stream viewer uses two display: chameleon::dvs_display shows ON and OFF events, and chameleon::delta_t_display shows luminance encoded in ∆ts between threshold crossings. chameleon::background_cleaner is used to reset the GPU's buffers before drawing the other displays.
To use Chameleon components, an application must create and execute a QGuiApplication object. Write the following content to atis_event_stream_viewer/source/atis_event_stream_viewer.cpp:
#include "../third_party/chameleon/source/background_cleaner.hpp"
#include "../third_party/chameleon/source/dvs_display.hpp"
#include "../third_party/chameleon/source/delta_t_display.hpp"
#include "../third_party/sepia/source/sepia.hpp"
#include <QtGui/QGuiApplication>
#include <QtQml/QQmlApplicationEngine>
#include <QtQml/QQmlContext>
/// filename points to the Event Stream file to read.
const auto filename = sepia::join(
{sepia::dirname(SEPIA_DIRNAME), "third_party", "sepia", "third_party", "event_stream", "examples", "atis.es"});
int main(int argc, char* argv[]) {
// read the header
const auto header = sepia::read_header(sepia::filename_to_ifstream(filename));
// create the Qt Application
QGuiApplication app(argc, argv);
// register Chameleon types
qmlRegisterType<chameleon::background_cleaner>("Chameleon", 1, 0, "BackgroundCleaner");
qmlRegisterType<chameleon::dvs_display>("Chameleon", 1, 0, "DvsDisplay");
qmlRegisterType<chameleon::delta_t_display>("Chameleon", 1, 0, "DeltaTDisplay");
// pass the header properties to qml
QQmlApplicationEngine application_engine;
application_engine.rootContext()->setContextProperty("header_width", header.width);
application_engine.rootContext()->setContextProperty("header_height", header.height);
// load the view and setup the window properties for OpenGL rendering
application_engine.loadData(
#include "atis_event_stream_viewer.qml"
);
auto window = qobject_cast<QQuickWindow*>(application_engine.rootObjects().first());
{
QSurfaceFormat format;
format.setDepthBufferSize(24);
format.setStencilBufferSize(8);
format.setVersion(3, 3);
format.setProfile(QSurfaceFormat::CoreProfile);
window->setFormat(format);
}
// retrieve pointers to the displays generated by qml
auto dvs_display = window->findChild<chameleon::dvs_display*>("dvs_display");
auto delta_t_display = window->findChild<chameleon::delta_t_display*>("delta_t_display");
// run the Qt Application
return app.exec();
}Modern Qt application use QML. The interface is described in a second file (here, atis_event_stream_viewer.qml), loaded during compilation and written in the QML language. The C++ code may retrieve pointers to the objects instantiated by the underlying engine, in order to interact with the user interface. Chameleon displays are updated with their push functions. These functions are templated, therefore compatible with any event type with the expected properties. The latter vary from display to display, and are documented here. Create the file atis_event_stream_viewer/source/atis_event_stream_viewer.qml with the following content:
R""(
import Chameleon 1.0
import QtQuick 2.7
import QtQuick.Layouts 1.1
import QtQuick.Window 2.2
Window {
id: window
visible: true
width: header_width * 2
height: header_height
Timer {
interval: 20
running: true
repeat: true
onTriggered: {
dvs_display.trigger_draw();
delta_t_display.trigger_draw();
}
}
BackgroundCleaner {
width: window.width
height: window.height
}
RowLayout {
width: window.width
height: window.height
spacing: 0
DvsDisplay {
objectName: "dvs_display"
id: dvs_display
Layout.fillWidth: true
Layout.fillHeight: true
Layout.alignment: Qt.AlignHCenter | Qt.AlignVCenter
canvas_size: Qt.size(header_width, header_height)
}
DeltaTDisplay {
objectName: "delta_t_display"
id: delta_t_display
Layout.fillWidth: true
Layout.fillHeight: true
Layout.alignment: Qt.AlignHCenter | Qt.AlignVCenter
canvas_size: Qt.size(header_width, header_height)
}
}
}
)""R""( and )"" delimit a string literal. This syntax makes compile-time loading of the QML file possible.
Qt requires the instruction app.exec() to be executed by the main thread. In order to handle the events at the same time, a thread must be created. The Sepia library provides the function sepia::make_observable<sepia::type::atis>, which has a syntax similar to sepia::join_observable<sepia::type::atis>. However, instead of reading and handling all the events before returning, sepia::make_observable<sepia::type::atis> creates a thread to handle the events and returns immediately. Compared to sepia::join_observable<sepia::type::atis>, it has two extra parameters:
-
handle_exceptionis an observable forstd::exception_ptr. It is called when the observable is interrupted, either because something unexpected happened or because the end of file was reached. -
must_restartis called when the end of file is reached. It must returntrueif the observable is to loop (restart reading from the beginning) andfalseotherwise. Iftrueis returned,handle_exceptionis not called.
Write the following content to atis_event_stream_viewer/source/atis_event_stream_viewer.cpp:
#include "../third_party/chameleon/source/background_cleaner.hpp"
#include "../third_party/chameleon/source/dvs_display.hpp"
#include "../third_party/chameleon/source/delta_t_display.hpp"
#include "../third_party/sepia/source/sepia.hpp"
#include "../third_party/tarsier/source/stitch.hpp"
#include <QtGui/QGuiApplication>
#include <QtQml/QQmlApplicationEngine>
#include <QtQml/QQmlContext>
/// filename points to the Event Stream file to read.
const auto filename = sepia::join(
{sepia::dirname(SEPIA_DIRNAME), "third_party", "sepia", "third_party", "event_stream", "examples", "atis.es"});
/// exposure_measurement holds the parameters of an exposure measurement.
struct exposure_measurement {
uint64_t delta_t;
uint16_t x;
uint16_t y;
} __attribute__((packed));
int main(int argc, char* argv[]) {
// read the header
const auto header = sepia::read_header(sepia::filename_to_ifstream(filename));
// create the Qt Application
QGuiApplication app(argc, argv);
// register Chameleon types
qmlRegisterType<chameleon::background_cleaner>("Chameleon", 1, 0, "BackgroundCleaner");
qmlRegisterType<chameleon::dvs_display>("Chameleon", 1, 0, "DvsDisplay");
qmlRegisterType<chameleon::delta_t_display>("Chameleon", 1, 0, "DeltaTDisplay");
// pass the header properties to qml
QQmlApplicationEngine application_engine;
application_engine.rootContext()->setContextProperty("header_width", header.width);
application_engine.rootContext()->setContextProperty("header_height", header.height);
// load the view and setup the window properties for OpenGL rendering
application_engine.loadData(
#include "atis_event_stream_viewer.qml"
);
auto window = qobject_cast<QQuickWindow*>(application_engine.rootObjects().first());
{
QSurfaceFormat format;
format.setDepthBufferSize(24);
format.setStencilBufferSize(8);
format.setVersion(3, 3);
format.setProfile(QSurfaceFormat::CoreProfile);
window->setFormat(format);
}
// retrieve pointers to the displays generated by qml
auto dvs_display = window->findChild<chameleon::dvs_display*>("dvs_display");
auto delta_t_display = window->findChild<chameleon::delta_t_display*>("delta_t_display");
// create the event handling pipeline
auto observable = sepia::make_observable<sepia::type::atis>(
sepia::filename_to_ifstream(filename),
sepia::make_split<sepia::type::atis>(
[&](sepia::dvs_event dvs_event) { dvs_display->push(dvs_event); },
tarsier::make_stitch<sepia::threshold_crossing, exposure_measurement>(
header.width,
header.height,
[](sepia::threshold_crossing threshold_crossing, uint64_t delta_t) -> exposure_measurement {
return {delta_t, threshold_crossing.x, threshold_crossing.y};
},
[&](exposure_measurement exposure_measurement) { delta_t_display->push(exposure_measurement); })),
[](std::exception_ptr) {},
[]() { return true; });
// run the Qt Application
return app.exec();
}The exception handler does nothing: exceptions are ignored by the application. The event handling pipeline uses tarsier::stitch (documented here) to calculate ∆ts between threshold crossings. chameleon::t_delta_display takes care of mapping the ∆ts to the screen's grey levels.
You can now compile and execute the application by running from the atis_event_stream_viewer directory:
premake4 gmake
cd build
make
release/atis_event_stream_viewerWindows users must run premake4 vs2010 instead, and open the generated solution with Visual Studio.
You completed the tutorials basics! For complementary information, see: