How can I modify Frame Data directly?

[kurochan]

I want to manipulate frame images directly from go.
For example, to overlay other images, add effects, insert generated frames etc....

I know ffmpeg has a filter feature, but this time I want to manipulate the frame directly.

How can I do this?

It seems to me that a function like Frame.SetData([]byte) would be a good idea, but if it doesn't exist, is there another way?

[asticode]

For now you can "export" the frame's data to a Go image and manipulate it, but you can't "save" your manipulations to a Frame. I remember trying to implement exactly that, one or two years ago but couldn't find a clean way while ensuring proper memory management and planes processing. I'll think about it and see if I come up with a new idea 👍

[asticode]

@kurochan I've created the branch frame-data-write with the following:

• FrameData.SetBytes(b []byte, align int) error which allows you to copy a buffer inside a frame's data
• FrameData.FromImage(src image.Image) error which allows you to copy a Go image.Image inside a frame's data

Bare in mind that the developer is responsible for handling the writability of the frame. For that, I've added

• Frame.MakeWritable()
• Frame.IsWritable()

I'm not sure I've implemented the most efficient way to do this but I couldn't think of a better way. Also, memory management shouldn't be a problem, but for those features only time can tell 😬

Could you try this new branch and let me know whether this work as expected? If so, I'll merge it in master 👍

[asticode]

No worries, do my changes work properly?

[kurochan]

@asticode The compilation succeeds and works, but the picture appears to be partially corrupted.
I wrote code to draw a blue rectangle based on the sample and go run main.go -i input.ts -o out.ts .

Could you review that my implementation is correct?
https://gist.github.com/kurochan/083b1cb4fb1439e90a633afa6a7bee5f
Here are the main changes: https://gist.github.com/kurochan/083b1cb4fb1439e90a633afa6a7bee5f#file-draw_avframe-go-L522-L539

[asticode]

As I've stated in my previous comment:

Bare in mind that the developer is responsible for handling the writability of the frame

FYI I've also added this fact to the documentation in the code 👍

Your frame's buffer is referenced in multiple places therefore you can't write data manually into it without causing the problems you're seeing. What you need to do is call f.MakeWritable() error here. After doing so, the content I generated using the code you've shared was correct 👍

Therefore I'm merging my branch into both dev and master.

Let me know whether you need a tag 👍

[kurochan]

@asticode I understood the points you made. And it worked as intended 🎉

Let me know whether you need a tag 👍

Yes, I would like you to release a new version 👍

Thank you for your quick implementation!

[asticode]

I've created a v0.22.0 tag 👍

[kurochan]

Implemented at v0.22.0 🎉