FFMPEG eats all the memory - memory

Question

Probably one of the most cliche question but here is the problem: So I have a Ubuntu Server running as an isolated machine only to handle FFMPEG jobs. It has 4 vCPU, 4GB RAM, 80GB storage. I am currently using this script to convert a video into HLS playlist: https://gist.github.com/maitrungduc1410/9c640c61a7871390843af00ae1d8758e This works fine for all video including 4K recorded from iPhone. However, I am trying to add watermark so I changed the line 106 of this script
from:
cmd+=" ${static_params} -vf scale=w=${widthParam}:h=${heightParam}"
to:
cmd+=" ${static_params} -filter_complex [1]colorchannelmixer=aa=0.5,scale=iw*0.1:-1[wm];[0][wm]overlay=W-w-5:H-h-5[out];[out]scale=w=${widthParam}:h=${heightParam}[final] -map [final]"
Now this works flawlessly in videos from Youtube or other sources but as soon as I am trying to use 4K videos from iPhone, the RAM usage grows from 250MB to 3.8GB in less than minute and crashes the entire process. So I looked out for some similar question:
FFmpeg Concat Filter High Memory Usage
https://github.com/jitsi/jibri/issues/269
https://superuser.com/questions/1509906/reduce-ffmpeg-memory-usage-when-joining-videos
ffmpeg amerge Error while filtering: Cannot allocate memory
I understand that FFMPEG requires high amount of memory consumption but I am unsure what's the exact way to process video without holding the stream in the memory but instead release any memory allocation in real-time. Even if we decide to work without watermark, It still hangs around 1.8GB RAM for processing 5 seconds 4K video and this create a risk of what if our user upload rather longer video than it will eventually crash down the server. I have thought about ulimit but this does seem like restricting FFMPEG instead of writing an improved command. Let me know how I can tackle this problem. Thanks

Answer 1

Okay, I found a solution. The problem is that the 4K video has extremely higher bitrate and it will load on your RAM to process the filter_complex which will eventually kill your process. To tackle this problem first thing I did was to transcode the input video to H264 format (you can put custom bitrate if you want to but I left that one out).
So I added this new command after line 58 of this script https://gist.github.com/maitrungduc1410/9c640c61a7871390843af00ae1d8758e
ffmpeg -i SOURCE.MOV -c:a aac -ar 48000 -c:v libx264 -profile:v main -crf 19 -preset ultrafast /home/myusername/myfolder/out.mp4
Now that we have a new processed out.mp4. We will go down the script line 121 and remove it. The reason for doing this is to stop FFMPEG from overloading all the command at once. Now we will remove line 107 to 109 and do this:
filters=[1]colorchannelmixer=aa=0.5,scale=iw*0.1:-1[wm];[0][wm]overlay=W-w-5:H-h-5[out];[out]scale=w=${widthParam}:h=${heightParam}[final]
cmd=""
cmd+=" ${static_params} -filter_complex ${filters} -map [final]"
cmd+=" -b:v ${bitrate} -maxrate ${maxrate%.*}k -bufsize ${bufsize%.*}k -b:a ${audiorate}"
cmd+=" -hls_segment_filename ${target}/${name}_%03d.ts ${target}/${name}.m3u8"
ffmpeg ${misc_params} -i /home/myusername/myfolder/out.mp4 -i mylogo.png ${cmd}
So now we are running FFMPEG inside a loop to handle per resolution basis output. This will eliminate the overloading of all filters in memory at once. You might even wanna remove line 53 depending on your use case.
Test
4K HEVC iPhone video of 1.2 minute long (453MB)
transcoding to H264 - Memory Usage stayed at 750MB
HLS + watermark - Memory Usage stayed between 430MB to 1.1GB
4K HEVC LG HDR video of 1.13 minute long (448MB)
transcoding to H264 - Memory Usage stayed at 800MB
HLS + watermark - Memory Usage stayed between 380MB to 850MB
My final thoughts
FFMPEG is a power eater. The total number of core/memory requirement will depend majorly on how much video you want to process. In my case, We only wanted to support upto 500MB videos so our test for 4K video processing is fitting the need but if you have larger video requirement then you have to test with more RAM/CPU core at hand
It's never a good idea to run parallel FFMPEG. Processing videos in a batch-wise will ensure optimum use of available resources and lesser chance for breaking your system in the middle of the night
Always run FFMPEG in an isolated machine away from your webserver, database, mail server, etc
Increasing resource is not always the answer. While we tend to first conclude that more resource === more stability is not always right. I've read enough thread about how even 64GB Ram with 32 core fails to keep up with FFMPEG so your best bet is to first improve your command or segregate commands into smaller command to handle the resources as effectively as possible
I'm not an expert in FFMPEG but I think this information will help someone who might have similar question.