We are using Web Audio API to play and manipulate audio in a web app.
When trying to decode large mp3 files (around 5MB) the memory usage spikes upwards in Safari on iPad, and if we load another similar size file it will simply crash.
It seems like Web Audio API is not really usable when running on the iPad unless we use small files.
Note that the same code works well on Chrome Desktop version - Safari version does complain on high memory usage.
Does anybody knows how to get around this issue? or what's the memory limit for playing audio files using Web Audio on an iPad?
Thanks!
Decoded audio files weight a lot more in RAM than on disk. A single sample uses 4 bytes (32-bit float). This translates to 230 MB of RAM for 10 minutes of audio at 48 000 Hz sample rate and in stereo. One hour of audio at the same sample rate and with stereo will take ~1,3 GB of RAM!
So, if you decode a lot of files, you can consume big amounts of RAM. My suggestion is to "undecode" files that you don't need (just "forget" unneeded audio buffers, so garbage collector can free memory).
You can also use mono audio files instead of stereo, that should reduce memory usage by half.
Note, that decoded audio files are always resampled to device's sample rate. This means that using audio with low sample rates won't help with memory usage.
Related
We are hosting mp3 files on AWS s3. We have built a web app (in React) that will play back the mp3s. However, it sometimes becomes distorted when played in Safari on iOS. The strange thing is that this does not happen all the time.
Here is the original file (sometimes distorted): https://sayyit-prod-static-assets.s3.amazonaws.com/static/audio/Darrin+M.+McMahon.original.mp3
Here is the file sounds when distorted: https://sayyit-prod-static-assets.s3.amazonaws.com/static/audio/WhatsApp+Video+2019-09-26+at+11.06.49+AM.mp4
Now, this distortion only happens when playing it through our app. When we provide a direct link to s3 (like I did above), it works. The distortion also happens when linking directly to s3 in our app.
Here are some ideas:
The mp3 file is broken
When going directly to the S3 link, it downloads entirely, which seems to allow the mp3 file to play perfectly
Any help would be greatly appreciated.
The sample rate on this MP3 file is 16 kHz. That's very low (not abnormal for voice), but also uncharacteristically low for a 128k MP3. I suspect that there's a bug with the resampler (as the iPhone hardware is locked to 48 kHz anyway), or that you're hitting an edge case bug with the decoder.
I'd recommend that you stop using MP3 and solve a few things at once. While MP3 is of acceptable quality, it's quality for a given bitrate isn't as good as alternatives. These days, you should consider using Opus. It's supported on iOS if muxed into a CAF file, and is extremely efficient. You could drop the bitrate down to 48k for voice and still have excellent quality. And, you'll bypass whatever resampling or decoding issue you're having now all in one go.
I'm using AVAssetDownloadTask to download some FairPlay-encrypted audio. As per guidelines, the audio is split up into small chunks to allow switching between bitrates during streaming. Our chunks are about 6 seconds each, which means less than 100 kb in size.
The download speed of this process is pretty bad. I've seen speeds between 85 KB/s and 250 KB/s. This is on a connection where when I download a new Xcode beta, I get several megabytes per second.
I'm guessing that the slow speed is due to having to make a separate request for each segment, which is a lot of overhead. I've tried inspecting the download traffic using Charles, and even though it shows one HTTPS connection per download task, the request body size continually ticks upward over the lifetime of the download. I tried downloading a 100MB test file from the same server where the audio files live and it came down at a few megabytes per second.
My question: what are best practices for getting good download performance using AVAssetDownloadTask? Should the segments be larger? Should there be a separate file that is one large chunk for downloading? Or is this behavior weird, suggesting I've got something configured wrong?
I'm using a Raspberry Pi 2 to route wifi-eth connections. So from the eth side I have a computer that will connect to internet using the Pi wifi connection. On the Raspberry I started htop to monitor the CPUs load, then on the computer I started chrome and played a 20-minute 1080 video. The load on the CPU didn't seem to go beyond 5% anyhow. After that I closed youtube tab and started a download of a binary file of 5GB from the first row here (https://testdebit.info/). Well, I noticed that CPU load was much more higher, around 10%!
Any explanation of such a difference?
It has to do with compression and how video is encoded. A normal file can be compressed, but nothing like that of a video stream.
A video stream can achieve very high compressions due to the predictable characteristics of video, e.g. video from one frame to another doesn't change much. As such, video will send a whole frame (I-frame) and then update it with just the changes (P-frame). It's even possible to do backward prediction (B-frame). Here's a wikipedia reference.
Yes, I hear your next unspoken question: Doesn't more compression mean more CPU time to uncompress? That's true for a lot of types of compression, such as that used by zip files. But since raw video is not very information dense over time, you have compression techniques that in essence reduce the amount of data you send with very little CPU usage.
I hope this helps.
I have used the vlc plugin(vlc web plugin 2.1.3.0) in Firefox to display the receiving live stream from my server into my browser. and i need to display 16 channels into one web page, but when i play more than 10 channels in the same time, i show that the processor is 100% and some breaking in the video appear. i have checked the plugin-memory in the running task, i have showed that around 45 MB from memory is dedicated for each video (so 10 channels : 10 * 45 = 450 MB).
kindly, do you have any method to reduce the consumption of the VLC plugin to allow the display of 16 channels in the same time ?
best regards,
There is no way to do that correctly. You could probably save a few megabytes by disabling audio decoding if there are audio tracks in one of your 16 streams in case you don't need them. Except for that, 45MB per stream is quite reasonable in terms of VLC playback and won't be able to go much below that, unless you reduce the video dimensions.
Additionally, your problem is probably not the use of half a giga byte of memory (Chrome and Firefox easily manage to use that much memory by themselves if you open a few tabs), but that VLC exceeds your CPU capacity. Make sure not to use windowless playback since this is less efficient that the normal windowed mode.
VLC 2.2 will improve the performance of the webplugins on windows by adding hardware acceleration known from the standalone application.
I'm just researching at the moment the possibility of writing an app to record an hours worth of video/audio for a specific use case.
As the video will be an hour long I would want to encode on-the-fly and not after the recording has finished to keep disk usage to a minimum.
Do the video capture APIs write a large uncompressed file to disk that has to be encoded after or can they encode on-the-fly resulting in a optimised file written to disk?
It's important that the video is recorded at a lower resolution than the iPhone's advertised 720/1080p as I need to keep the file sizes down due to length of video (which will need to be uploaded).
Any information you have would be appreciated or even just a pointer in the right direction.
No they do not record uncompressed to disk (unless this is what you want). You can specify to record to a MOV/MP4 and have the video encoded in H264. Additionally you can control the average bit rate of the encoding. You can also specify the capture size, and output encoding size along with scaling options if needed. For demo code check out AVCamDemo in the WWDC 2010 sample code. This demo code may now be available in the docs.