I've been looking at using AVAudioPlayer to play sounds. The problem here is the format- I have a bunch of WAV samples in a buffer but no WAV header at all. AVAudioPlayer doesn't seem to allow you to just set the format, it has to try to interpret it from the NSData buffer. Having to copy over the entire sound before it can start playing is not going to be a good experience for my users.
If I have a buffer of WAV samples (not a WAV file in memory), how can I play it back with AVAudioPlayer without crippling my performance with a gigantic copy?
Instead of copying the entire (gigantic) sound, just copy a WAV file header in front the first sample of the existing buffer. When initially obtaining your wave data into memory (file load, download, or synthesis, etc.), make sure there is at least 44 bytes of padding in front, so there's room to copy the header in. If it's a buffer snippet, save a bit of the data, copy the WAV header in, play the sound, and then restore the original 44 bytes.
Related
I have an iOS app that downloads videos from Firebase (Cloud Firestore) with a feed similar to Instagram/TikTok. However, I can't get the videos to be readily available before a user scrolls to the video. Any tips would be super helpful. How does TikTok do it? Do they save a whole bunch of videos to file in the background on load?
My current VideoDownloadManager:
Checks if video is already loaded in temp cache or local file,
and if not:
Downloads video url (view Firebase's downloadURL)
Returns the video url for immediate use (before playing it still
has some delay for buffering)
Stores the video url in a temp
cache (in case user scrolls away and scrolls back)
Begins to write video to file and removes video from temp cache on completion
With the current set up, videos play efficiently after they are downloaded. But if a video is not downloaded yet and a user scrolls to that video, it takes too long for (#1/2) above to complete and buffer enough to play. I am already using OperationQueues and prioritizing the current video over any other background videos - but this isn't fast enough still.
TikTok videos are almost always readily available as a user scrolls. What the secret?
Thanks for your help!
I have a few tips for you:
1- when you are loading your stream you should start preheating video URLs in a background thread.
2- try not to download complete files, and just cache or buffer a small amount of file like 1MB.
3- with .mp4 files you can play videos even if they are not downloaded completely.
4- start full download when the video starts playing based on the buffer rate or video length.
5- try to use videos with the smallest file size and bitrates. when you are creating them, try to convert them to a convenient format. my suggestion will be:
Video:
video bit rate -> 12.5
video size -> 960x540
conversion format -> h264
Sound:
rate -> 44100
encoding bit rate -> 96000
6- check if the video has a buffered range of more than 25% when you are going to start the playback.
7- don't forget to do the downloads in a temp folder and clean that folder regularly. This helps avoid huge app size, the consequence of not doing that will may lead users to delete your app!!.
For iOS developers: this is my videoConverter.
also, you can use this cache video player GSPlayer
I have an app that plays a sound file every time the screen is touched. For some reason, the app will crash every once in a while with the following error:
reason: 'Resource tick.mp3 can not be loaded'
In case you need it, here is how I play the file each time the screen is tapped:
runAction(SKAction.playSoundFileNamed("tick.mp3", waitForCompletion: false))
This does not happen very often, maybe 1 in 10 runs of the app. Most of the time everything works as expected. I wish I knew what I am doing to cause the crash but I have no clue! I am just tapping away seemingly no different than the times when it doesn't crash. Then all of a sudden I get this issue...
If you play the sound via a playSound function, it will work
var soundFile = SKAction.playSoundFileNamed("bark.wav", waitForCompletion: false)
playSound(soundFile)
playSound:
func playSound(soundVariable : SKAction)
{
runAction(soundVariable)
}
First of all, it looks like that you are using mp3 file to play (short) sound effects. When using mp3 the audio is compressed. In memory, it will have different, bigger size. Also there is a decoding performance penalty (decoding takes CPU time). The most important thing, and the reason why I am talking about mp3 files can be found in docs:
When using hardware-assisted decoding, the device can play only a
single instance of one of the supported formats at a time. For
example, if you are playing a stereo MP3 sound using the hardware
codec, a second simultaneous MP3 sound will use software decoding.
Similarly, you cannot simultaneously play an AAC and an ALAC sound
using hardware. If the iPod application is playing an AAC or MP3 sound
in the background, it has claimed the hardware codec; your application
then plays AAC, ALAC, and MP3 audio using software decoding.
As you can see,the problem is that only one mp3 file at a time can be played using hardware. If you play more than one mp3 at a time, they will be decoded with software and that is slow.
So, I would recommend you to use .wav or .caf files to play sound effects. mp3 would be probably good for background music.
About crashing issue:
try to use .wav or .caf files instead of .mp3
try to hold a strong reference to the SKAction and reuse it as suggested by Reece Kenney.
Im currently researching the mp3 format in order to build an mp3 decoder.
After some thinking I figured out that the simplest way to calculate the length of the song would be to divide the size by the bitrate (taking in account the size of the ID3 tag etc.), and transform the result to minutes. Using this method on a few songs I got accurate times.
I always assumed the time of the song is the length of the pure audio data, but in this method, frames are also "considered" part of the song (when calculating the time).
Also, the I understood that the audio data in the mp3 file is compressed, so when its decompressed the size of it will be larger of course, and then the time calculation seems un accurate.
Am I missing something here? because it just doesnt make any sense to me that the songs length is calculated with the compressed data and not the uncompressed ones, and frames which are a DWORD each are not ignored.
I always assumed the time of the song is the length of the pure audio data, but in this method, frames are also "considered" part of the song (when calculating the time). Also, the I understood that the audio data in the mp3 file is compressed, so when its decompressed the size of it will be larger of course, and then the time calculation seems un accurate.
When a media stream, such as an MP3 file, is compressed with a constant bitrate, that bitrate reflects the compressed size of the data, not the uncompressed size. So your math is fine.
What will throw you off with this approach is metadata tags (e.g, ID3) -- those are part of the file size, but are not counted in the bitrate, since they aren't audio data. Luckily, those tend to be relatively small, so they won't affect your results much.
I need to append video frames taken from stream to existing mp4 file.
I found that I need to use mdat and stco atom to update chunks table.
Is it posible to append new frames/chunks at all? I need to do this in ios app.
Such appending is possible, but it may not be practical -- especially if you are wanting to add one frame at a time.
MP4 is very much a "baked" format that is difficult to modify after it has been generated. This is because the structural data in the "moov" box contains file offsets to important media data in the "mdat" box. If the moov box is positioned before the mdat box (common to allow progressive download), then any data added to the moov box (i.e. references to new keyframes in your appended video) will push the mdat box further away. Not only would you have to rewrite the file, but you'd need to update all the file offsets accordingly. (Perhaps there are some clever tricks for keeping the moov box size constant...) If the mdat box is positioned first, the operation would still be awkward because you'd need to copy the moov box into memory, append your new video to the end of the mdat, update moov fields accordingly, and append the new moov box.
If you can gather all new video frames, and at the end of the recording add them during a rewrite operation, this may be workable. You could also look into Fragmented MP4 (using "moof" boxes), but I'm not sure how widespread the support for reading such files is.
I'm trying to display the contents of a video file (let's just say without the audio for now) onto a UV mapped 3D object in OpenGL. I've done a fair bit in OpenGL but have no idea where to begin in video file handling, and most of the examples out there seems to be for getting video frames from cameras, which is not what I'm after.
At the moment I feel if I can get individual frames of the video as CGImageRef I'd be set, so I'm wondering how to do this? Perhaps there are even be better ways to do this? Where should I start and what's the most straight forward file format for video playback on iOS? .mov?
Apologies; typing on an iPhone so I'll be a little brief.
Create an AVURLAsset with the URL of your video - which can be a local file URL if you like. Anything QuickTime can do is fine, so MOV or M4V in H.264 is probably the best source.
Query the asset for tracks of type AVMediaTypeVideo. You should get just one unless your source video has multiple camera angles of something like that, so just taking objectAtIndex:0 should give you the AVAssetTrack you want.
Use that to create an AVAssetReaderTrackOutput. Probably you want to specify kCVPixelFormatType_32BGRA.
Create an AVAssetReader using the asset; attach the asset reader track output as an output. And call startReading.
Henceforth you can call copyNextSampleBuffer on the track output to get new CMSampleBuffers, putting you in the same position as if you were taking input from the camera. So you can lock that to get at pixel contents and push those to OpenGL via Apple's BGRA extension.
You're probably going to have to use a player layer and flatten its contents into a bitmap context. See the documentation for AVPlayerLayer. The performance might be very poor though.