FFMPEG making requests for each frame when decoding a stream, slow performance - ios

Question

I am having an issue playing MOV camera captured files from an iPhone. My FFMPEG implementation has no problem playing most file formats, this issue is exclusive only for camera captured MOV.
When trying to open the file, I can see in the logs that many requests are made, each requests decoding only one frame, before making a new request which results the video being buffered extremely slowly.
It takes roughly a minute to buffer about a few seconds of the video.
Another thing to mention is that the very same problematic file is played without an issue locally. The problem is when trying to decode while streaming.
I compiled my code on Xcode 11, iOS SDK 13, with cocoapods mobile-ffmpeg-https 4.2.
Here is a rough representation of my code, its pretty standard:
Here is how I open AVFormatContext:
AVFormatContext *context = avformat_alloc_context();
context->interrupt_callback.callback = FFMPEGFormatContextIOHandler_IO_CALLBACK;
context->interrupt_callback.opaque = (__bridge void *)(handler);
av_log_set_level(AV_LOG_TRACE);
int result = avformat_open_input(&context, [request.urlAsString UTF8String], NULL, NULL);
if (result != 0) {
if (context != NULL) {
avformat_free_context(context);
}
return nil;
}
result = avformat_find_stream_info(context, NULL);
if (result < 0) {
avformat_close_input(&context);
return nil;
}
Video decoder is opened like so, audio decoder is nearly identical
AVCodecParameters *params = context->streams[streamIndex]->codecpar;
AVCodec *codec = avcodec_find_decoder(params->codec_id);
if (codec == NULL) {
return NULL;
}
AVCodecContext *codecContext = avcodec_alloc_context3(codec);
if (codecContext == NULL) {
return NULL;
}
codecContext->thread_count = 6;
int result = avcodec_parameters_to_context(codecContext, params);
if (result < 0) {
avcodec_free_context(&codecContext);
return NULL;
}
result = avcodec_open2(codecContext, codec, NULL);
if (result < 0) {
avcodec_free_context(&codecContext);
return NULL;
}
I read the data from the server like so:
AVPacket packet;
int result = av_read_frame(formatContext, &avPacket);
if (result == 0) {
avcodec_send_packet(codecContext, &avPacket);
// .... decode ....
}
Logs after opening the decoders:
// [tls] Request is made here
// [tls] Request response headers are here
Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:2048
Probing mp3 score:1 size:2048
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] type:'ftyp' parent:'root' sz: 20 8 23077123
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] ISO: File Type Major Brand: qt
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] type:'wide' parent:'root' sz: 8 28 23077123
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] type:'mdat' parent:'root' sz: 23066642 36 23077123
// [tls] Request is made here
// [tls] Request response headers are here
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 0, sample 4, dts 133333
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 1, sample 48, dts 1114558
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 2, sample 1, dts 2666667
[h264 # 0x116080200] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
// [tls] Request is made here
// [tls] Request response headers are here
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 0, sample 4, dts 133333
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 1, sample 48, dts 1114558
[mov,mp4,m4a,3gp,3g2,mj2 # 0x115918e00] stream 2, sample 1, dts 2666667
[h264 # 0x116080200] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
// [tls] Request is made here
// [tls] Request response headers are here
// ...
These are some warnings I found in the log
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] interrupted
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] stream 0: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] stream 1: start_time: 0.000 duration: 11.832
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] stream 2: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] stream 3: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] format: start_time: 0.000 duration: 11.833 bitrate=15601 kb/s
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] Could not find codec parameters for stream 0 (Video: h264, 1 reference frame (avc1 / 0x31637661), none(bt709, left), 1920x1080, 1/1200, 15495 kb/s): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[mov,mp4,m4a,3gp,3g2,mj2 # 0x11c030800] After avformat_find_stream_info() pos: 23077123 bytes read:16293 seeks:1 frames:0
Also when calling avformat_open_input(...), 2 GET requests are made, before the returning.
Notice the "Probing mp3 score:1", that is not shown for other MOV files or any other files.
I have tried different versions of ffmpeg, I have tried messing around with the delays of the stream, I tried removing my custom interrupt callback, nothing have worked.
Code works fine with any other videos I have tested (mp4, mkv, avi).
Metadata of the test file:
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
creation_time : 2019-04-14T08:17:03.000000Z
com.apple.quicktime.make: Apple
com.apple.quicktime.model: iPhone 7
com.apple.quicktime.software: 12.2
com.apple.quicktime.creationdate: 2019-04-14T11:17:03+0300
Duration: 00:00:16.83, bitrate: N/A
Stream #0:0(und), 0, 1/600: Video: h264, 1 reference frame (avc1 / 0x31637661), none(bt709), 1920x1080 (0x0), 0/1, 15301 kb/s, 30 fps, 30 tbr, 600 tbn (default)
Metadata:
creation_time : 2019-04-14T08:17:03.000000Z
handler_name : Core Media Video
encoder : H.264
Stream #0:1(und), 0, 1/44100: Audio: aac (mp4a / 0x6134706D), 44100 Hz, mono, 100 kb/s (default)
Metadata:
creation_time : 2019-04-14T08:17:03.000000Z
handler_name : Core Media Audio
Stream #0:2(und), 0, 1/600: Data: none (mebx / 0x7862656D), 0/1, 0 kb/s (default)
Metadata:
creation_time : 2019-04-14T08:17:03.000000Z
handler_name : Core Media Metadata
Stream #0:3(und), 0, 1/600: Data: none (mebx / 0x7862656D), 0/1, 0 kb/s (default)
Metadata:
creation_time : 2019-04-14T08:17:03.000000Z
handler_name : Core Media Metadata

Answer 1

I found a fix for this (sorta of):
Set the io_open callback of the AVFormatContext to your own function, and then when that is called, after you call the default io_open you change the buffer size.
static int (*IO_OPEN_DEFAULT)(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options);
int IO_OPEN_OVERRIDE(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options) {
int result = IO_OPEN_DEFAULT(s, pb, url, flags, options);
pb[0]->buffer_size = 41239179;
return result;
}
This fixes the issue. The value you set it to is usually very large (20 to 40MB). You can get that value from the second network request bytes range when you are opening the format context (first network request is made with bytes range 0-* and then the second network request is made with bytes range XXXX-* where XXXX should be the buffer size).
The reason why this fixes the issue, is because by buffering all of that data, aviocontext no longer needs to make a new network request to get the audio data. The audio data is already buffered (or at least the first position is).
There might be a better way to fix this issue, it seems like apple MOV files separate their video and audio data with these massive chunks for some reason and that causes ffmpeg to make a million network requests, for each frame.