2 Different types of visualization
Here we will look at how you can interpret the different visualizations coming out of VideoAnalysis.
The aim of creating such alternate displays from video recordings is to uncover features, structures and similarities within the material itself, and in relation to, for example, score material.
From a musical point of view, one core challenge here has been to create visualisation techniques that can capture different types of temporal levels. For example, we may observe the instantaneous unfolding of sound and motion, while at the same time keeping an internal memory of the trajectories of a sequence, and an overall image of longer patterns.
VideoAnalysis can generate both dynamic and static visualizations, as well as some quantitative data:
- dynamic visualisations (video files)
- motion video
- motion history video
- static visualisations (images)
- motion average image
- motiongrams
- videograms
- motion data (csv files)
- quantity of motion
- centroid of motion
- area of motion
We will examine each of these in a little more detail.
The analysis method used in VideoAnalysis is based on a technique called "frame differencing". Here we create a motion image by calculating the absolute pixel difference between subsequent frames in the video file:
The end result is an image where only the pixels that have changed between the frames are displayed. This can be interesting in itself, but motion images are also the starting point for many other video visualization and analysis techniques.
A motion video is a series of motion images, each showing only the motion happening between the two last frames in the original video file.
The quality of the raw motion image depends on the quality of the original video stream. Small changes in the lighting, camera motion, compression artefacts, etc. may all influence the final image. Such visual "noise" can be removed by applying a simple low-pass filter to remove pixels below a certain threshold. This can be found in the Filter settings in VideoAnalysis.
The number goes between 0 and 1, where 0 will let through all pixels, while 1 will only let through completely white pixels. The default setting is 0.1, and this usually works quite well for most videos. If you want more detail you can try setting it down to 0.05, or if you have too much noise you can raise it to 0.2 or 0.3. That way it is possible to remove some of the unwanted noise in the image.
The filtered motion image is used for further processing.
While a motion history image may reveal information about the spatial aspects of a motion sequence over a fairly short period of time, it is possible to use a motiongram to display longer sequences. This display is created by plotting the normalised mean values of the rows of a series of motion images. The motiongram makes it possible to see both the location and quantity of motion of a video sequence over time, and is thus an efficient way of visualising longer motion sequences.
A motiongram is only a reduced display of a series of motion images, with no analysis being done. It might help to think of the motiongram as a display of a collapsed series of pictures, or “stripes,” where each “stripe” summarises the content of a whole motion image.
Dependent on the frame rate of the video file, motiongrams can be created from recordings as short as a few seconds to several hours. For short recordings it is possible to follow detailed parts of a body, particularly if there are relevant colours in the image, while motiongrams of longer recordings will mainly reveal larger sections of motion. Motiongrams work well together with audio spectrograms, and other types of temporal displays such as graphs of motion or sound features.
While motiongrams are good for getting a general sense of the spatiotemporal distribution of a video file, a motion average image is good for getting a general sense of the space covered. This may be thought of as similar to an "open shutter" on a camera, since the function just averages (and normalizes) over all the frames.
If the motion is evenly distributed in the image, the motion average image may not contain much information. But if a person has been standing still in one spot, it will show up very clearly. Such an average image may also be useful in combination with the motion average image.
One dynamic visualization technique is that of creating a motion history video, which gives you an idea of how the motion has been unfolding in time.