FFmpeg iOS Get UIImage from AVFrame - ios

Question

I am having trouble getting a UIImage out of the frames I am reading into my iOS FFmpeg project. I need to be able to read a frame in, and then convert this to a UIImage in order to display the frame in a UIImageView. My code appears to be reading in the frames, but I am lost as to how to convert them as there is little documentation on how to do this. Can anyone help?
while (!finished) {
if (av_read_frame(_formatContext, &packet) >= 0) {
if (packet.stream_index == _videoStream) {
int ret = avcodec_send_packet(_codecContext, &packet);
if (ret < 0 || ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
printf("av_codec_send_packet error ");
}
while (ret >= 0) {
ret = avcodec_receive_frame(_codecContext, _frame);
if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
printf("avcodec_receive_frame error ");
}
finished = true;
}
}
av_packet_unref(&packet);
}
}

Answer 1

You should know about pixel formats like rgb and yuv. Videos almost always uses yuv formats like yuv420p. Then study AVFrame structure, here some info:
AVFormat.format : Current frame's pixel format i.e. AV_PIX_FMT_YUV420P
AVFormat.width : Horizontal length of current frame (hence width) unit: pixels
AVFormat.height : Vertical length of current frame (hence height) unit: pixels
Now where is the actual frame buffer you might ask, it is in AVFormat.data[n]
n can be 0-3. Depending on the format, just first one may contain whole frame or all 4 of them. I.e. yuv420p uses 0, 1, and 2. Their linesizes (aka strides) can be obtained reading corresponding AVFormat.linesize[n] value.
As for yuv420p:
data[0] is Y plane
data[1] is U plane
data[2] is V plane
If you multiply linesize[0] with AVFrame.height, you'll get size of that plane (Y) as number of bytes.
I don't know about UIImage structure (or whatever it is), if it requeris a specific format like RGB, you need to convert your AVFrame to that format using swscale.
Here some examples: https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/scaling_video.c
In libav (ffmpeg) scaling (resizing) and pixel format conversion are done via same function.
Hope these helps.