Seeking can produce different video frames in Firefox and Chrome

[mzur]

Seeking to the time of a video annotation can produce different vieo frames in Firefox and Chrome (e.g. video 6018). This can result in erroneously displayed annotation locations in the video.

This seems to be a problem caused by the video encoding (presumably through conversion with VLC). Judging by the Largo patches generated by ffmpeg with the same annotation timestamp, Firefox seems to show the wrong video frame.

[mzur]

I produced a minimal working example. This uses an original video not transcoded with VLC. Based on this, Firefox would be correct.

<video id="video" src="http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4"></video>

<script type="text/javascript">
   let video = document.getElementById('video');
   video.currentTime = 501.95;
</script>

This renders as follows (compare the ears and body):

(Chrome)

(Firefox)

(ffmpeg -ss 00:08:21.95 -i bbb_sunflower_1080p_60fps_normal.mp4 -vframes 1 -q:v 2 output.jpg)

[mzur]

This time Firefox doesn't produce the same result as ffmpeg.

Example video with reduced size that can be uploaded here:

bbb-clip.mp4

Updated code:

<video id="video" src="bbb-clip.mp4"></video>


<script type="text/javascript">
   let video = document.getElementById('video');
   video.currentTime = 53.05;
</script>

(chrome)

(firefox)

(ffmpeg -ss 00:00:53.05 -i bbb-clip.mp4 -vframes 1 -q:v 2 output.jpg)

[mzur]

Filed a bug for Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1739125

[mzur]

This issue was confirmed to be a Firefox bug. Maybe we should show a warning for users who use Firefox in a version that is affected by this.

[mzur]

I've now found an example for an annotation that might be shown incorrectly in Chrome. It's this annotation. The Largo thumbnail is below, which should be the correct time and position from Ffmpeg. Firefox seems to show the same frame than the thumbnail but Chrome does not.

= Biigle\VideoAnnotation {#4829
    id: 930610,
    frames: "[397.005997]",
    points: "[[241.76,460.01]]",
    video_id: 12683,
    shape_id: 1,
    created_at: "2023-01-18 10:07:29",
    updated_at: "2023-01-18 10:07:29",
  }

(thumbnail)

(Chrome)

(Firefox)

We should do an experiment with the video and extract the frame at the time of the annotation with Ffmpeg and also build a minimal HTML document that seeks the video to the same time. Then check the document with Firefox and Chrome. If Chrome is wrong, look for a filed bug or else file a bug similar to the one for Firefox (see above).

[mzur]

I now confirmed that indeed Chrome shows a different frame than Firefox and FFmpeg (ffmpeg -ss 397.005997 -i original.mp4 -frames:v 1 frame.jpg). It becomes obvious with this minimal HTML:

<video id="video" src="original.mp4"></video>

<script type="text/javascript">
   let video = document.getElementById('video');
   video.currentTime = 397.0060;
</script>

It turned out that this time an incorrect position of the "moov atom" was the problem. The atom should be placed at the start of the file but here it apparently was at the end:

$ ffmpeg -v trace -i original.mp4 2>&1 | grep -e type:\'mdat\' -e type:\'moov\'
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55dbdacb8d40] type:'mdat' parent:'root' sz: 205354407 48 205452833
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55dbdacb8d40] type:'moov' parent:'root' sz: 98386 205354455 205452833

The atom van be moved to the beginning with: ffmpeg -i original.mp4 -vcodec copy -acodec copy -movflags faststart output.mp4 (ref).

$ ffmpeg -v trace -i output.mp4 2>&1 | grep -e type:\'mdat\' -e type:\'moov\'
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5585b801dd40] type:'moov' parent:'root' sz: 98394 40 205452841
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5585b801dd40] type:'mdat' parent:'root' sz: 205354407 98442 205452841

Now Chrome displays the correct frame at the time.

All MP4 videos should have their moov atom at the start. Should we display a warning message if this is not the case?

[mzur]

This will be resolved with: https://bugzilla.mozilla.org/show_bug.cgi?id=1735300

[davidsean]

I've been investigating a similar issue and ran across an interesting case that seems unrelated to the moov atom.
I've confirmed that moov comes before mdat and even re-encoded with -movflags faststart but I still see a discrepancy between HTML and FFMPEG.

The timestamp in question (extracted from the query-param) is t=1931415 (or 193.1415s)

I've noticed that "next/prev" frame navigation increments the ts query param in steps of 0.0100 s.
The video has a framerate of fps=29.97002997002997
which means t=1931415 falls at the frame 5787.2577422--- basically it sits between two frames.

So what frame should be extracted? I did a bit of poking and I think FFMPG always rounds up to the next frame, whereas HTML (at least on Chrome) and OpenCV rounds down to the previous frame.
See my tests below (I've added some annotations to highlight the differences).

The first column has t=1930929 (or 193.0929s), which falls exactly on a frame. All frame extractions are identical.
The second column is the problem case from biigle at t=1931415 and we see that there are different rounding behaviors.
The third colum attempts to jump the the next legitimate frame, but 1931262 lacks the needed precision, and we are stuck at frame 5787.998 which doesn't quite cross over to 5788 territory.
The fourth column uses 1931263 (the next available timestamp using the query-param model) which pushes the frame number over to the next one (which is 5788).

I think this could be addressed by using a frame number as a query-parameter.
Here is a gist to reproduce the test https://gist.github.com/davidsean/0e4619e715fc45b02c75f241216cb934

EDITE: I am starting to think that supporting framerates of 29.97 is going to be super annoying in the context of adding temporal annotations. I think it needs to be discouraged. perhaps in a best-practices guide. Showing a fps warning similar to the moov atom warning could be the best route.

[mzur]

Hi, thanks for making the effort to investigate this! This is really one of the hardest issues in BIIGLE right now and I still don't see a viable solution. Using a frame number as query parameter would be correct. However, the HTML video API does not expose this. Also, it does not expose exact timestamps for security reasons. To work around this, BIIGLE seeks the video to the current (inexact) timestamp whenever the video is paused in the hope that this will reliably produce the same video frame the next time the inexact timestamp is used again. We also can't get information on the framerate or the moov atom in the frontend (although this could be done once in the backend).

The only (for us, currently) non-viable solution I see would be to let BIIGLE re-encode all uploaded videos to a very specific and well-defined format. With this, we could implement video playback with the WebCodecs API, which gives us access to individual video frames etc. Other than that, I don't see a way to improve the status quo if what you say is true and annotations/thumbnails are still off with a correct moov atom...

[davidsean]

I created a series of reference videos that shows the frame number, time, frame rate and status of the -faststart flag.
They are available in this projetct: https://biigle.de/projects/3379
use this invite link https://biigle.de/project-invitations/dfed272d-ab80-4719-b6ef-b7bc5b908f51

as a test, I chose the 5fps video and placed a laser point when the moving circle touches a gridline

I did the same thing with both video variants moov at end and at start of video, in order to compare the largo thumbnails.

First thing I noticed was that the experimental next/prev frame navigation works well, but the ts query param is off (does not represent the time of the frame).

The largo thumbnails shows a different frame both both variants:

I think the issue I am seeing more accurately falls under #433

Feel free to reproduce and validate the moov flag in the reference videos.

[mzur]

Thanks again for investigating! Did you update the annotations in the meantime? I see these Largo thumbnails:

[davidsean]

I did play around with updating annotation by snapping/quantizing annotation timestamps to frames. It's possible that I updated these without realizing.

[mzur]

We had another user report this issue just now. I see two different cases:

• A video with the moov atom at the start: Chrome displays a completely wrong frame. Firefox is only slightly different (maybe 1 frame off) than Ffmpeg. The user annotated with Chrome and the object is not even visible in the Largo thumbnail. This is something new, as I assumed that videos with moov correctly positioned should be displayed fine in Chrome. This doesn't seem to be the case at least for some videos. I'm not sure what we can do about this.

• A video with the moov atom at the end: Both Firefox and Chrome display the same frame for the annotation. Ffmpeg selects a different frame, though. I assume both browsers display the wrong frame here. I raised the priority of Implement MP4 moov atom validation #983 so users will get a warning if the moov atom is placed incorrectly.