YUV422P to RGB and back to YUV422P format

[Kalit31]

Hello again,
I have been tinkering around with the library. I have the following usecase and need some suggestions/approaches to efficiently implement this.

I have a UDP video stream, from which I read packets, which are then decoded into frames in YUV422P format. I want to convert these decoded frames to RGB format and send to a python script (adds some AR filters on top of this images) which returns back filtered images in RGB format. Then, I would convert these RGB frames back to YUV422P format, encode them into packets and push into a WebRTC track.

Note: I have been using MediaPipe for AR-related processing. Apparently, they do not have support in Go.

Here are some code snippets:

func writeH264ToTrackAR(track *webrtc.TrackLocalStaticSample) {
	astiav.RegisterAllDevices()
	initTestSrc()
	defer freeVideoCoding()

	ticker := time.NewTicker(h264FrameDuration)
	for ; true; <-ticker.C {
		if err = inputFormatContext.ReadFrame(decodePacket); err != nil {
			if errors.Is(err, astiav.ErrEof) {
				break
			}
			panic(err)
		}
		decodePacket.RescaleTs(videoStream.TimeBase(), decodeCodecContext.TimeBase())

		if err = decodeCodecContext.SendPacket(decodePacket); err != nil {
			panic(err)
		}
	
		for {
			// Read Decoded Frame
			if err = decodeCodecContext.ReceiveFrame(decodeFrame); err != nil {
				if errors.Is(err, astiav.ErrEof) || errors.Is(err, astiav.ErrEagain) {
					fmt.Println("In error block")
					break
				}
				panic(err)
			}

			initVideoEncoding()

			// Send the frame to Python for AR filtering
			filteredFrame, err := sendFrameToPython(decodeFrame)
			if err != nil {
				panic(err)
			}
			
			pts++
			filteredFrame.SetPts(pts)

			// Encode the filtered frame
			if err = encodeCodecContext.SendFrame(filteredFrame); err != nil {
				panic(err)
			}

			for {
				// Read encoded packets
				encodePacket = astiav.AllocPacket()
				if err = encodeCodecContext.ReceivePacket(encodePacket); err != nil {
					if errors.Is(err, astiav.ErrEof) || errors.Is(err, astiav.ErrEagain{
						break
					}
					panic(err)
				}
				// Write H264 to track
				if err = track.WriteSample(media.Sample{Data: encodePacket.Data(), Duration: h264FrameDuration}); err != nil {
					panic(err)
				}
			}
		}
	}
}

func sendFrameToPython(frame *astiav.Frame) (*astiav.Frame, error) {
	img, _ := astiavFrameToImage(frame)
	if img == nil {
		return nil, errors.New("failed to convert ASTIAV frame to image")
	}

        // Send and receive frame to python script for processing

	// Convert image back to ASTIAV frame
	filteredFrame := imageToASTIAVFrame(receivedImg)

	return filteredFrame, nil
}

Ref: https://github.com/pion/example-webrtc-applications/tree/master/ffmpeg-send

Questions:

1. Any suggestions to handle this usecase?
2. Is there any library support to convert YUV422P frames to RGB and vice versa?

Any help would be appreciated. Thanks in advance!

[asticode]

There are 2 ways to convert pixel formats using go-astiav directly:

• use a software scale context
• use filters

Since the pion example you've linked uses software scale context and since it's the simplest one to setup, I think that'd be better for you to try this first.

You can basically create a software scale context to convert from YUV422P into RGB and another one to convert from RGB to YUV422P. For the first one you'd have to make sure source and destination width and height are the same but source pixel format is YUV422P and destination pixel format is RGB here then, as in the example, use ScaleFrame() to convert pixel format. You can then use the same logic with your second software scale context to convert back to YUV422P.

Does that answer your question?

[Kalit31]

Hi @asticode,
Thanks for your response!
This was helpful. A quick followup - the ScaleFrame() function call would give me a frame in RGBA format. I want to convert this frame to an object of type Image.image. I implemented this logic using the below code block:

	width := frame.Width()
	height := frame.Height()
	img := image.NewRGBA(image.Rect(0, 0, int(width), int(height)))
	frame.Data().ToImage(img)

However, can you give some example of how to convert an object of type Image.image to an astiav.Frame?

[asticode]

You can check out the frame data manipulation example. It shows the different methods you'd need to use (some methods were added very recently).

Let me know whether you're having problems with those methods 👍

[Kalit31]

Hi @asticode,
I did go through the example you shared above. But, the libavutil/imgutils.c file is throwing an assertion error.

Here is the code snippet which I am testing:

reader := bytes.NewReader(processedFrameData)

// Decode the JPEG image from the reader
processed_img, err := jpeg.Decode(reader)
if err != nil {
	return nil, fmt.Errorf("failed to decode image: %w", err)
}

fmt.Println("HEIGHT: ", processed_img.Bounds().Dy())
fmt.Println("WIDTH: ", processed_img.Bounds().Dx())
processedFrame = astiav.AllocFrame()
processedFrame.SetHeight(processed_img.Bounds().Dy())
processedFrame.SetWidth(processed_img.Bounds().Dx())
processedFrame.SetPixelFormat(astiav.PixelFormatRgba)

align := 0
if err := processedFrame.AllocBuffer(align); err != nil {
	return nil, fmt.Errorf("main: allocating buffer failed: %w", err)
}

if err := processedFrame.AllocImage(align); err != nil {
	return nil, fmt.Errorf("main: allocating image failed: %w", err)
}

if err := processedFrame.MakeWritable(); err != nil {
	return nil, fmt.Errorf("main: making frame writable failed: %w", err)
}

if err := processedFrame.Data().FromImage(processed_img);  err != nil {
	return frame, fmt.Errorf("converting processed image to frame failed: %w", err)
}

The FromImage function throws the below error:

HEIGHT:  720
WIDTH:  1280

Assertion ((src_linesize) >= 0 ? (src_linesize) : (-(src_linesize))) >= bytewidth failed at libavutil/imgutils.c:350
SIGABRT: abort
PC=0x7f776a7de00b m=12 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 33 gp=0xc00009ae00 m=12 mp=0xc000580808 [syscall]:
runtime.cgocall(0x91b7b0, 0xc000477950)
	/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000477928 sp=0xc0004778f0 pc=0x51082b
github.com/asticode/go-astiav._Cfunc_av_image_copy(0x7f76ec000b80, 0x7f76ec000bc0, 0xc0000e9280, 0xc00002c4c0, 0x1a, 0x500, 0x2d0)
	_cgo_gotypes.go:3457 +0x45 fp=0xc000477950 sp=0xc000477928 pc=0x908dc5
github.com/asticode/go-astiav.(*frameDataFrame).copyPlanes.func4(0xc0000363f0?, 0xc0000e9280, 0xc00002c4c0)

The processed image data which I am receiving from the other python script (which I mentioned in previous comments) is correct - I tested by dumping the image.Image object into an image file.

I believe I am not configuring the processedFrame variable in the correct manner. Can you assist here?

I do not have much understanding about the align variable as well. Can you share some insights on this as well?

Thanks a lot for your prompt responses!

[asticode]

Can you first try with align := 1 and let me know whether that works better?

The internet will have a better technical answer than mine regarding aligment but as a rule of thumb I usually set align := 1 for video and align := 0 for audio.

If that still doesn't work, could you send me a jpeg that is causing the error?

[Kalit31]

Hi @asticode,
I think I figured out the issue. I was feeding in an image of type image.Image instead of image.RGBA. The pixel format of the frame into which I was trying to load the image was set to astiav.PixelFormatRgba. After converting the image.Image object to image.RGBA and feeding it to the FromImage() function, things work fine!

Thanks for your timely assistance🙏!
You may close the discussion.

[asticode]

Great! Hope you enjoy the lib! ❤️