The decoder returned a frame that is past the expected one

[runitao]

Version

1.41.0

Describe the bug.

I use fn.readers.video to decode videos, the minimum reproducible example will report exception（see relevant log output).

I have test 4 cases, just the case 1 reports exception, others are well.

python dali_issue.py filelist_1.txt   # case 1
python dali_issue.py filelist_2.txt   # case 2, just reorder the video files
python dali_issue.py filelist_3.txt   # case 3, only 349047839_5.mp4
python dali_issue.py filelist_4.txt   # case 4, only 349064170_5.mp4

The video files in all cases are 349064170_5.mp4 and/or 349047839_5.mp4

PS. I test with weekly build too, the problem is still here.

NVIDIA-SMI 535.154.05
Driver Version: 535.154.05
CUDA Version: 12.2

Minimum reproducible example

git clone https://github.com/runitao/dali_example.git

Relevant log output

[/opt/dali/dali/operators/reader/loader/video_loader.h:179] file_list_include_preceding_frame uses the default value False. In future releases, the default value will be changed to True.
iter 0 label [0 0 0 0]
140492320241408 Exception in thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: ./349064170_5.mp4
Traceback (most recent call last):
File "dali_issue.py", line 56, in
video, label = pipe.run()
File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1327, in run
return self.outputs()
File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1165, in outputs
return self._outputs()
File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1250, in _outputs
return self._pipe.Outputs()
RuntimeError: Critical error in pipeline:
Error in GPU operator nvidia.dali.fn.readers.video,
which was used in the pipeline definition with the following traceback:

File "dali_issue.py", line 27, in video_pipeline
video, label, start_frame_num, timestamps = fn.readers.video(

encountered:

Error in worker thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: ./349064170_5.mp4

Other/Misc.

No response

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report

[JanuszL]

Let me check this and get back to you with more info.

[JanuszL]

It seems that the video decoder incorrectly returns frame/time skipping the frame we are expecting to get. Let me check this with a video decoder team.

[JanuszL]

Tracked internally as nv4899180.

[runitao]

Hi @JanuszL , is there any progress? With the latest repo, the minimum reproduce example still does't work.

[JanuszL]

I'm afraid there is not. It is not up to my team now.

[Pipickin]

@JanuszL, hello! Could you let me know if there’s any new information on the bug? I'm trying to use fn.readers.video to train my model, but I periodically get this error—not even on the first video pass. I recreate the annotations each epoch due to the specifics of the training, so the videos are the same, but different intervals may appear.

[JanuszL]

I'm afraid I didn't get any update from the relevant team.

[0909wlgur]

Hi, @runitao.
In my case, the bugs occurred in the video files, and it is suspected that the issue originates from the audio tracks. While the DALI documentation states that only the image track is read, it seems there are instances where the PTS from the audio track is referenced. In my case, the audio tracks in the videos had a variable frame rate instead of a constant frame rate. Since I didn’t need the audio tracks, I removed them, and after doing so, the error no longer occurred, and decoding worked correctly.

Therefore, I recommend converting all streams in the video file to a constant frame rate to avoid such issues.

[Pipickin]

Hi, guys!
I have figured out that there are some troubles with the video encoding/decoding formats. My source videos were in the hevc format (h265) and when I have tried to get clip from 1500 frame to 1548 I have got an error, if I took clip from 1499 to 1547 or from 1501 to 1549 then it worked fine. But when I had converted video to the h264 then it worked even with 1500 frame. I also tested hevc_nvenc format and it works! I don't know why but maybe it could help the other developers.