Video playback API overhaul

[ivan-mogilko]

Problem

Video playback API in AGS is rudimentary, it only allows to play a single video, blocking whole game, and nothing else may be updated nor drawn on screen while this video is playing. While this is not the most popular kind of request, there are still users who would like to be able to make proper FMV-style games with AGS, or at least have more control over cinematic cutscenes.
At the moment this may only be achieved using a plugin, but using plugins is always a last resort, as they don't always have full sync with what engine does (think of audio), and one has to ensure that the plugin is built for all wanted platforms.

Long story short, there's a certain demand to have a better video playback API. The basic requirements are:

• be able to run video while the rest of the game is running too;
• be able to render video on an arbitrary game object (gui, overlay, room object);
• be able to run multiple videos at once;
• and all the above should support playing video's audio track too.

We had a discussion about this on forums a while back:
https://www.adventuregamestudio.co.uk/forums/engine-development/feature-request-videos-non-blocking-playback-with-guis/

Following is based on my comment there, but this is a draft idea, so may be discussed and rethought further.

Proposal

In regards to the Script API:

1. There should be a kind of VideoPlayback struct in script. VideoPlayback object is returned from PlayVideo (or similar) function, and provides means for controlling video state, including but not necessarily limited to: Pause, Resume, Seek functions, Length and Position properties.
2. VideoPlayback does not draw anything on screen itself. Instead it has a Graphic property, that lets access a dynamic sprite, created and "owned" by the Playback object, where the video frames are decoded to. This property may then be used to draw or assign this sprite anywhere, in a same way as any other sprite is used in the engine.
3. For Audio support, VideoPlayback should have a AudioChannel* property, which lets to access a channel where video's audio track is being played. NOTE: if we modify audio API prior to this, then this should be changed accordingly.
   EDIT: later I had strong doubts about using AudioChannel for this purpose, as that gives audio playback controls, which will overcomplicate things. Please refer to my comment below: Video playback API overhaul #2155 (comment)

In regards to the underlying implementation:

1. The decoding must happen on a separate thread to not unnecessarily load up or block the game's thread. It is important to remember that the video playback may work with its own fps setting, not necessarily matching the game's fps. So a frame may theoretically be decoded once per a number of game updates, or multiple times per a single game update.
2. Of course this raises a question: how do video and game sync? what will happen if the decoder tries to decode next frame while the game is trying to render previous one on screen? The solution I could come up with at this point is to implement a back-buffer mechanism for this sprite. That means that instead of having 1 bitmap, the sprite has 2 associated bitmaps. One of them is considered to be sync with the game, and another is the one that video decoder writes to. Upon entering a game frame update, these bitmaps switch: a previous back-buffer becomes an active sprite's surface, and the previous active surface becomes back-buffer, that a video decoder will write to when necessary.
3. Thus, video sync with the game happens once per game frame, perhaps when all the other game animations are updated.
4. The video's audio track should be played through standard audio system, and playback "placed" on one of the "logical channels", accessible in script.

There may be a number of secondary questions. Like do we need video to have its own "audio type", for easier managing default volume and reserve channels? do we need to have videos in the project explorer in the Editor (like some users suggest)? and so forth. But these all may be decided later, after the main bulk of work is done.

[ericoporto]

I am leaving here the link for Godot docs on case it's useful (the Godot engine also only supports Theora playback).

https://docs.godotengine.org/en/stable/tutorials/animation/playing_videos.html

The difference that I think matters most when going from video running exclusively on the screen vs video running just on an in-game element, is the video FPS.

When the video runs exclusively on screen like today, it can in theory have "any" fps (like 25fps) but when video is run in-game elements, it really is best if the encoded fps is an integer multiple or fraction of the game (say 30fps video in a 60fps game). I don't think there's much to do here other than telling this somewhere in the manual.

[ivan-mogilko]

Right, above docs reminded me, the VideoPlayback should have "autoplay" property, and a command to go frame by frame, for a custom pacing control.

[ericoporto]

Just checking around other engines

• LÖVE approach on Video
  • newVideo function
  • Video Object
  • Video Steam
• GameMaker Video Playback
  • GameMaker Manual Reference on Video
  • video how-to

So löve has also strictly Theora support (I guess this is how it goes in FOSS projects) and GameMaker doesn't specify support leaving for the platforms. About the API itself, it looks like the general idea that we want contains the features available in those - GameMaker is a bit weird like if you draw each frame using a DrawingSurface.

Löve has an interesting thing on the filter used for scaling when rendering the video - either linear or nearest neighbor and if the filter is applied either when sizing up or down, this is interesting because I can see linear when sizing down and nearest when sizing up in a mixed size pixelart game.

Ah about script names, I wonder if one day we will have VIEWS as enum, because then they would be named like eViewNameOfView, and this leaves the single v letter available for videos in script name like vIntro in Vincent's example in the forum post.

[ivan-mogilko]

Löve has an interesting thing on the filter used for scaling when rendering the video - either linear or nearest neighbor and if the filter is applied either when sizing up or down

We could have separate filters for the whole game too.

Ah about script names, I wonder if one day we will have VIEWS as enum, because then they would be named like eViewNameOfView,

Um, that naming does actually sound like a dubious idea, as people might have a habit of calling views starting with 'v' already. Just because something is enum for strictness does not mean that we are obliged to enforce 'e' letter. Besides, views are 99% much more common than videos in AGS games.

Another thing, View may eventually become a managed struct for consistency with other objects in game, and be addressed with a pointer (but this is just to mention a possibility).

[ericoporto]

We could use vc as in Video Clip - like vcIntro.Play() - or other alternative is to call videos as movies and then we use the letter m that is just not allocated to anything.

[ivan-mogilko]

There's something that I realized that trying to add an interface for "audio player" into plugin api (#2256).

If you have a Video playing, then the audio control should likely be restricted to what it may do.
For instance, the AudioChannel can Seek, change playback Speed, Pause and Stop. I do not think it's a good idea to let users do that for the audio track coming from a video, as not only that would be difficult to implement, but even if working will desync video & audio.

Which means that we should not put video's audio onto a AudioChannel, at least not with the controls it has now.

Likely there has to be two separate script structs, one that defines a audio player (playback control), and another that defines audio manipulator, or output (settings control). "Audio player" may as well inherit "manipulator". But Video player struct then should only expose a "manipulator" for audio.

[ericoporto]

It could be AudioChannelBase or AudioChannelSimple and then AudioChannel just extends that, so that the video could give a pointer for the more restrictive interface.

[ivan-mogilko]

Oh, but the names in api should reflect what the object does. I think, if anything, the "base" is actually something that controls playback. On the other hand, remembering the discussions about better audio api, the future of "channel" is not clear, so I would not like to bind any more names to that.

This is something that adjusts running audio data.
I would propose "AudioFilter", but that name sounds rather like a base class for a range of custom implementations. Maybe we'll support a list of user-made "AudioFilters" attached to this object in some future.
There may be a derivative from that name though.

Other terms that come to mind are: AudioSettings, AudioControl (?), Modifier, Panel, Tumbler, Tuner ... AudioLevels? AudioProperties? ...

[AlanDrake]

Playback ?

[ivan-mogilko]

Playback ?

But "playback" is something that controls playback, this would rather be a name for an object that has pause/resume/stop/seek etc, everything that AudioChannel has now.

[AlanDrake]

Right. AudioMixer maybe, for this specific case of exposing limited stuff like Pan/Volume?

[ivan-mogilko]

Right. AudioMixer maybe, for this specific case of exposing limited stuff like Pan/Volume?

I thought about this, but is not "mixer" something that mixes 2+ audio streams into one?

[AlanDrake]

I thought about this, but is not "mixer" something that mixes 2+ audio streams into one?

It can be. At it's core is generally a device or digital panel where you set volume, panning, bass, treble, etc, for one or more streams.
Technically, isn't that's what is gonna happen with a video with stereo or multi-channel anyway?

If you want an alternative term just for these audio properties "mixer" should be an intuitive choice.

[ivan-mogilko]

If you want an alternative term just for these audio properties "mixer" should be an intuitive choice.

Supposing so, but what if in the future we'd want an actual mixer in script api, that's lets mix sound from 2+ channels (or whatever audio source is called then) and this name will be already taken by an object that works as a audio modifier?

EDIT: this makes me think further, that since the video's sound is also a sound stream, then one might like to pass it ("plug in") into the mixer as well... so it has to be a kind of thing that you may pass somewhere as a "audio stream"...

Perhaps it may make sense to theorize a new audio API first, prior to video; or along with it.

EDIT2: since love2d video was posted as an example above, it's curious that they did not care about this issue, and simply return a regular "audio source" from Video:
https://love2d.org/wiki/Video:getSource
https://love2d.org/wiki/Source
I wonder how that works, what will happen if you seek, rewind or pause video's audio source. Either it ignores the command, or affects the video, as if video's controls were used.

[AlanDrake]

If the VideoPlayback object is going to exclusively control an audio voice by itself, you might also simply add Volume/Panning directly into VideoPlayback and call it a day without intermeditate objects or other layers.

That way you can postpone the design of a new Audio API and still have something usable.
Designing script API is difficult, we should reserve the right to change things until we're satisfied, it's hard to get it right the first time...

[ivan-mogilko]

Resolved by #2338