I am interested in recording media using an AVCaptureSession in iOS while playing media back using an AVPlayer (specifically, I am playing back audio and recording video, but I'm not sure it matters).
The problem is, when I play the resulting media back together later, they are out of sync. Is it possible to synchronize them, either by ensuring that playback and recording start simultaneously, or by discovering what the offset is between them? I probably need the sync to be on the order of 10 ms. It is unreasonable to assume that I can always capture audio (since the user may use headphones), so syncing via analysis of original and recorded audio is not an option.
This question suggests that it's possible to end playback and recording simultaneously and determine the initial offset from the resulting lengths that way, but I'm unclear how to get them to end simultaneously. I have two cases: 1) the audio playback runs out, and 2), the user hits the "stop recording" button.
This question suggests priming and then applying a fixed, but possibly device-dependent delay, which is obviously a hack, but if it's good enough for audio it's obviously worth considering for video.
Is there another media layer I can use to perform the required synchronization?
Related: this question is unanswered.
If you are specifically using AVPlayer to playback Audio and i would suggest you to use AudioQueueServices for the same. Its seamless and fast as it reads buffer by buffer and play pause is faster than AVPLlayer
There can also be the possibility that you are missing the initial statement of [avPlayer prepareToPlay] which might be causing much overhead for it to sync before playing the Audio.
Hope it helps you.
Related
I need to have a series of sound samples (audio files) being played back at the touch of a button. The audio samples need to be played back simultaneously and precisely (think 4 voices in a piece of music).
I managed to do this with several instances of AVAudioPlayer but it will go out of sync.
Reading about it, due to lack of precision it seems to not be the right choice for I'm trying to do.
Audio queue (is this part of Core Audio?) seems to be able to do what I want but I can hardly find any code bits in Swift to setup what I’m trying to do, which is:
Load the audio file, prepare it to be played, then play it (I would trigger it with an NSTimer).
Is this straightforward to implement with audio queue or should I look elsewhere?
If you could point me into the right direction I would be very grateful.
Thanks a lot!
As is demonstrated in this answer, I have recently learned how to play audio files using both AVAudioPlayer and AudioToolbox. I have successfully played a single audio test file using both methods. However, I want to ask about which one I should actually use in my app (or if it even matters).
Here are the relevant characteristics of my app:
There are about 800 audio clips.
Most of the clips last less than one second.
Any could be chosen to be played at random by the user, but only a small subset will be used on any particular run.
No special volume control or playback options are needed.
These are my questions:
Which method for playing a sound would be better? Why?
Should I preload the sounds or just load them when they are needed? I'm guessing that preloading 800 sounds every time is a bad idea. But if I wait to load them until they are needed, I am worried about performance (ie, a noticeable pause before the clip is played)
Do I need to play sounds on a background thread?
So my concerns in choosing which audio player to go with are memory and performance. I couldn't tell from any of the documentation that I saw which is better in this case.
There is a talking cat app well known for iOS devices, in which you speak your voice and he repeats. Analyzing this app, you'll see that it stops talking when you stop talking, that is, it stops to capture the audio when not receive another voice.
I was giving a analyzing the methods of AVAudioRecorder class, and not found any method in which to capture when the User stop to talking or recorder stops to receive external audio.
How can I capture when the audio recorder stops to receiving audio.
Process the audio stream as it is coming through. You can look at the frequency and volume of the stream. From there you can determine if the user has stopped talking.
I suggest frequency and volume as the recorder still picks up background audio. If the volume drops dramatically then the sounds the recorder is picking up must be further away from the device than before. The frequency can also lend itself to:
A.) Filter out the background audio in the audio used to replay the audio with a pitch change or any other changes. etc.
B.) I do not know the limits of frequency for the average human. But this covers the use case where the user has stopped talking, but have moved the device in such a way that the recorder still picks up load shuffling from moving fingers near the mic.
I have an AVMutableComposition with 2 audio tracks and one video track. I'm using the composition to string about 40 different video clips from .mov files, putting the video content of each clip in the video track of my composition and the audio in the audio track. The second audio track I use for music.
I also have a synchronized layer for titles graphics.
When I play this composition using an AVPlayer, the audio slowly gets out of sync. It takes about 4 minutes to start becoming noticeable. It seems like if I only string together a handfull of longer clips the problem is not as apparent, it is when there are many clips shorter clips (~40 in my test) that it gets really bad.
Pausing and Playing doesn't re-sync the audio, however seeking does. In other words, if I let the video play to the end, towards the end the lip sync gets noticeably off even if I pause and play throughout, however, if I seek to a time towards the end the audio gets back in sync.
My hacky solution for now is to seek to the currentTime + 1 frame every minute or so. This creates an unpleasant jump in the video caused by a lag in the seek operation, so not a good solution.
Exporting with an ExportSession doesn't present this problem, audio remains in sync in the output movie.
I'm wondering if the new masterClock property in the AVPlayer is the answer to this, and if it is, how is it used?
I had the same issue and fixed it, among many other audio and video things, by specifying times timescales in the following manner:
CMTime(seconds: my_seconds, preferredTimescale: CMTimeScale(600))
Before, my time scale was CMTimeScale(NSEC_PER_SEC). That caused me jittery when composing clips at a different frame rate, plus this audio out-of-sync that Eddy mentions here.
In spite of looking like a magic number, 600 is a common multiple of 24, 30, 60 and 120. These are usual frame rates for different purposes. The common multiple avoids dragging around rounding problems when composing multiple clips.
I have been using MPMovieplayer and the playableDuration to check the available duration of a movie.
The duration always seems to be ~1 second further than my current duration and basically I would like to increase this.
I have tried to use the prepareToPlay but this seems to do nothing noticeable to the playable Duration.
I have tried to set as many parameters as possible to attempt to try and change the value via setting the defaults pre-emptively such as the MPMovieSourceType, MediaType and alike, but all to no avail.
Just to clear a few things up firstly: I am using both MPMoviePlayer and AVplayer which both play different streams simultaneously as the video/audio I am using is split.
EDIT
Seems like I overlooked the file size affecting the stream and should have read more in the apple resources then elsewhere, but as far as I can tell the issue is: the file size is too large and therefore a server side media segmenter has to be implemented.
Apple Resource on Media Segmenting