AGS 4: non-blocking video playback

[ivan-mogilko]

For #2155.

This is not a complete functionality, but a draft, meant to test out the idea. Random mistakes are to be expected.

1. Rewrote VideoPlayer class, so that it buffers video frames and prepares a "ready frame" when the time is right, but does not draw anything itself. The plan for the VideoPlayer is that it can work in 2 modes: autoplay and frame-by-frame mode. In the first mode it calculates timing when the next frame should be presented, and switches the "ready frame". In the frame-by-frame mode it will be in paused state but advance to next frame by a call to NextFrame(). Audio is not played in the frame-by-frame mode.
2. Picked out BlockingVideoPlayer class which is basically a "game state" singleton that runs VideoPlayer, blocking the rest of the game. This is used for the old PlayVideo command.
3. Implemented "video core", similar to existing "audio core", with a running thread, which polls video players.
4. Wrote a video object control, that starts and stops videos, and acquires their "ready frames", placing them into the spriteset, as dynamic sprites (so that they don't get deleted). Besides being video frames, these act entirely like any sprite, so may be assigned anywhere, e.g. to Overlay. When frame is changed, game objects are notified of a sprite's change as usual, and redraw themselves as necessary.
5. Implemented new VideoPlayer API as follows:

enum PlaybackState {
  ePlaybackOn = 2,
  ePlaybackPaused = 3,
  ePlaybackStopped = 4
};

builtin managed struct VideoPlayer {
  import static VideoPlayer* Open(const string filename, bool autoPlay=true, RepeatStyle=eOnce);
  /// Starts or resumes the playback.
  import void Play();
  /// Pauses the playback.
  import void Pause();
  /// Advances video by 1 frame, will pause video when called.
  import void NextFrame();
  /// Changes playback to continue from the specified frame.
  import void SeekFrame(int frame);
  /// Changes playback to continue from the specified position in milliseconds.
  import void SeekMs(int position);
  /// Stops the video completely.
  import void Stop();

  /// Gets current frame index.
  import readonly attribute int Frame;
  /// Gets total number of frames in this video.
  import readonly attribute int FrameCount;
  /// Gets this video's framerate (number of frames per second).
  import readonly attribute float FrameRate;
  /// Gets the number of sprite this video renders to.
  import readonly attribute int Graphic;
  /// The length of the currently playing video, in milliseconds.
  import readonly attribute int LengthMs;
  /// Gets/sets whether the video should loop.
  import attribute bool Looping;
  /// The current playback position, in milliseconds.
  import readonly attribute int PositionMs;
  /// The speed of playing (1.0 is default).
  import attribute float Speed;
  /// Gets the current playback state (playing, paused, etc).
  import readonly attribute PlaybackState State;
  /// The volume of this video's sound, from 0 to 100.
  import attribute int Volume;
};

A key point is a Graphic property that returns an index of a sprite where the video renders to. This sprite may then be assigned to anything that may normally have a sprite, such as Overlay, GUI button, etc.

Basic script example may be:

VideoPlayer *video = VideoPlayer.Open("video.ogv", true, eRepeat);
if (video)
{
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    over = null;
    video = null;
}

---

TODO:

• Proper script API
• Fix timing after resuming
• Autofix audio/video desync
• Support speed setting
• Support seek, APEG does not provide this!
• Stubs for AGS_NO_VIDEO_PLAYER build (?)

POTENTIAL OPTIMIZATIONS

• Buffer frame textures along with the bitmaps
• Store "opaque" flag per sprite, so that updating sprite's texture would be a bit faster. This may be still done separately, as a general improvement, if "buffer texture" path is taken.
• Might have separate locks for buffering and processing/control, as buffering only touches the queue and frame pool, and might as well keep running while the ready frame is being retrieved (or rather, we should let retrieve the ready frame while the decoding and buffering is run). This may be done in "audio core" too, but with video this is more important, as video frames take more time to process.

OTHER THINGS:

• Consider finding/writing a new library code for working with ogg theora, because APEG is very old, and does not have all required functionality prepared.

[ericoporto]

I know this is ugly but it was the best place I found at the time to throw the polling for audio

ags/Engine/ac/timer.cpp

Lines 72 to 74 in 84ab2b5

	#if defined(AGS_DISABLE_THREADS)
	audio_core_entry_poll();
	#endif

The video thread will require polling there too as we can't use threads in the Emscripten port currently.

[ericoporto]

The CI failure in Android just looks like it failed to download the SDL sources it uses when building it, maybe just rerunning that task should make it complete now.

[ivan-mogilko]

Pushed a proper Video API (still a draft though), which now looks like this:

enum PlaybackState {
  ePlaybackOn = 2,
  ePlaybackPaused = 3,
  ePlaybackStopped = 4
};

builtin managed struct VideoPlayer {
  import static VideoPlayer* Open(const string filename, bool autoPlay=true, RepeatStyle=eOnce);
  /// Starts or resumes the playback.
  import void Play();
  /// Pauses the playback.
  import void Pause();
  /// Advances video by 1 frame, will pause video when called.
  import void NextFrame();
  /// Changes playback to continue from the specified frame.
  import void SeekFrame(int frame);
  /// Changes playback to continue from the specified position in milliseconds.
  import void SeekMs(int position);
  /// Stops the video completely.
  import void Stop();

  /// Gets current frame index.
  import readonly attribute int Frame;
  /// Gets total number of frames in this video.
  import readonly attribute int FrameCount;
  /// Gets this video's framerate (number of frames per second).
  import readonly attribute float FrameRate;
  /// Gets the number of sprite this video renders to.
  import readonly attribute int Graphic;
  /// The length of the currently playing video, in milliseconds.
  import readonly attribute int LengthMs;
  /// Gets/sets whether the video should loop.
  import attribute bool Looping;
  /// The current playback position, in milliseconds.
  import readonly attribute int PositionMs;
  /// The speed of playing (1.0 is default).
  import attribute float Speed;
  /// Gets the current playback state (playing, paused, etc).
  import readonly attribute PlaybackState State;
  /// The volume of this video's sound, from 0 to 100.
  import attribute int Volume;
};
#endif

Most of the functionality works, except Seek and few less important getters.
Unfortunately APEG library does not support Seek, so either this has to be written, or a completely new library found/written for working with ogg theora decoder. Maybe looking for a better library (or even writing one ourselves) should be the next task after implementing the formally working video API.

[ericoporto]

Uhm, maybe add a CanSeek in the video API for now and return false? Is there a chance we sometime use a new stream class type that can't seek?

[ivan-mogilko]

Rebased, squashed commits.

Made Seek return new position or error value. Seek(0) now works as a rewind so long as stream may be reused.

[ivan-mogilko]

Rebased and redone PR after #2348. Now this PR contains much less commits.
Refactored "video core" interface in the similar way to "audo core" in #2344.
Another difference with the previous variant is that I made BlockingVideoPlayer also use "video core" instead of containing its own VideoPlayer with a thread.

Old branch is still available here, temporarily:
https://github.com/ivan-mogilko/ags-refactoring/tree/ags4--newvideoapi2--oldvariant

[ericoporto]

This is a typo, it should be player->Poll() here.

[ivan-mogilko]

Hmm, actually I removed the video polling inside BlockingVideoPlayer, because it's polled in WaitForNextFrame in case of "no threads" build.