I'm trying to create a metronome for iOS in Swift. I'm using a GCD dispatch queue to time an AVAudioPlayer. The variable machineDelay is being used to time the player, but its running slower than the time I'm asking of it.
For example, if I ask for a delay of 1sec, it plays at 1.2sec. 0.749sec plays at about 0.92sec, and 0.5sec plays at about 0.652sec. I could try to compensate by adjusting for this discrepancy but I feel like there's something I'm missing here.
If there's a better way to do this altogether, please give suggestions. This is my first personal project so I welcome ideas.
Here are the various functions that should apply to this question:
func milliseconds(beats: Int) -> Double {
let ms = (60 / Double(beats))
return ms
}
func audioPlayerDidFinishPlaying(player: AVAudioPlayer, successfully flag: Bool) {
if self.playState == false {
return
}
playerPlay(playerTick, delay: NSTimeInterval(milliseconds(bpm)))
}
func playerPlay(player: AVAudioPlayer, delay: NSTimeInterval) {
let machineDelay: Int64 = Int64((delay - player.duration) * Double(NSEC_PER_SEC))
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, machineDelay),dispatch_get_main_queue(), { () -> Void in
player.play()
})
}
I have never really done anything with sound on iOS but I can tell you why you are getting those inconsistent timings.
What happens when you use dispatch_after() is that some timer is set somewhere in the OS and at some point soon after it expires, it puts your block on the queue. "at some point after" is going to be short, but depending on what the OS is doing, it will almost certainly not be close to zero.
The main queue is serviced by the main thread using the run loop. This means your task to play the sound is competing for use of the CPU with all the UI functionality. This means that the chance of it playing the sound immediately is quite low.
Finally, the completion handler will fire at some short time after the sound finishes playing but not necessarily straight away.
All of these little delays add up to the latency you are seeing. Unfortunately, depending on what the device is doing, that latency can vary. This is never going to work for something that needs precise timings.
There are, I think, a couple of ways to achieve what you want. However, audio programming is beyond my area of expertise. You probably want to start by looking at Core Audio. My five minutes of research suggests either Audio Queue Services or OpenAL, but those five minutes are literally everything I know about sound on iOS.
dispatch_after is not intended for sample accurate callbacks.
If you are writing audio applications there is no way to escape, you need to implement some CoreAudio code in one way or another.
It will "pull" specific counts of samples. Do the math (figuratively ;)
Related
I'm using AVAudioEngine for audio in an iOS game application. A problem I've encountered is that AVAudioPlayerNode.play() takes a long time to execute, which can be a problem in real-time applications such as games.
play() just activates the player node - you don't have to call it every time you play a sound. As such, it doesn't have to be called that often, but it does have to be called occasionally, such as to activate the player initially, or after it's been deactivated (which happens in some situations). Even if only called occasionally, the long execution times can be a problem, especially if you need to call play() on multiple players at once.
The execution time for play() seems to be proportional to the value of AVAudioSession.ioBufferDuration, which you can request to be changed using AVAudioSession.setPreferredIOBufferDuration(). Here's some code I'm using to test this:
import AVFoundation
import UIKit
class ViewController: UIViewController {
private let engine = AVAudioEngine()
private let player = AVAudioPlayerNode()
private let ioBufferSize = 1024.0 // Or 256.0
override func viewDidLoad() {
super.viewDidLoad()
let audioSession = AVAudioSession.sharedInstance()
try! audioSession.setPreferredIOBufferDuration(ioBufferSize / 44100.0)
try! audioSession.setActive(true)
engine.attach(player)
engine.connect(player, to: engine.mainMixerNode, format: nil)
try! engine.start()
print("IO buffer duration: \(audioSession.ioBufferDuration)")
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
if player.isPlaying {
player.stop()
} else {
let startTime = CACurrentMediaTime()
player.play()
let endTime = CACurrentMediaTime()
print("\(endTime - startTime)")
}
}
}
Here are some sample timings for play() that I got using a buffer size of 1024 (which I believe is the default):
0.0218
0.0147
0.0211
0.0160
0.0184
0.0194
0.0129
0.0160
Here are some sample timings using a buffer size of 256:
0.0014
0.0029
0.0033
0.0023
0.0030
0.0039
0.0031
0.0032
As you can see above, for a buffer size of 1024, execution times tend to be in the 15-20 ms range (around a full frame at 60 FPS). With a buffer size of 256, it's around 3 ms - not as bad, but still costly when you only have ~17 ms per frame to work with.
This is on an iPad Mini 2 running iOS 12.4.2. This is obviously an old device, but the results I see on the simulator seem similarly proportional, so it seems to have more to do with the buffer size and the behavior of the function itself than with the hardware being used. I don't know what's going on under the hood, but it seems possible that play() blocks until the beginning of the next audio cycle, or something like that.
Requesting a lower buffer size seems like a partial solution, but there are some potential drawbacks. According to the documentation here, lower buffer sizes can mean more disk access when streaming from a file, and irrespective of that, the request may not be honored at all. Also, here, someone reports playback problems related to low buffer sizes. Taking all this into account, I'm disinclined to pursue this as a solution.
That leaves me with execution times for play() in the 15-20 ms range, which typically means a missed frame at 60 FPS. If I arrange things so that only one call to play() is made at a time, and only infrequently, maybe it won't be noticeable, but it's not ideal.
I've searched for information and asked about this in other places, but it seems either not many people are encountering this behavior in practice, or it isn't an issue for them.
AVAudioEngine is intended for use in real-time applications, so if I'm right that AVAudioPlayerNode.play() blocks for a significant amount of time proportional to the buffer size, that seems like a design issue. I realize this probably isn't an issue many are dealing with, but I'm posting here to ask if anyone has encountered this specific issue with AVAudioEngine, and if so, if there's any insight, suggestions, or workarounds anyone can offer.
I've investigated this fairly thoroughly. Here are my findings.
Having now tested the behavior on a variety of devices and iOS versions (including the latest version at the time of this writing, 13.2), and having had others test it as well, my current conclusion is that the long execution times for AVAudioPlayerNode.play() are inherent and that there's no obvious workaround. As noted in my original post, the execution times can be reduced by requesting a lower buffer duration, but as discussed earlier, this doesn't seem like a viable solution.
I heard from a credible source that calling play() on a background thread (e.g. using Grand Central Dispatch) should be safe, and indeed this would be one way to solve the problem. However, although it may technically be safe to call play() (or other AVAudioEngine-related functions) on different threads, I'm skeptical as to whether this is a good idea (further explanation below).
The documentation doesn't state this as far as I can tell, but AVAudioEngine will throw NSException's under various circumstances, which, without special handling, will result in application termination in Swift.
One of the things that will cause an NSException to be thrown is if you call AVAudioPlayerNode.play() while the engine is not running. Obviously if you only have your own code to worry about, you can take steps to ensure this doesn't occur.
However, iOS itself will sometimes stop the engine of its own accord, for example when an audio interruption occurs. If you call play() subsequent to that and before restarting the engine, an NSException will be thrown. It's fairly easy to avoid this mistake if all your calls to play() are on the main thread, but multithreading complicates the issue and seems like it could introduce the risk of accidentally calling play() after the engine has been stopped. Although there may be ways to work around this, multithreading seems to introduce undesirable complexity and fragility, so I've opted not to pursue it.
My current strategy is as follows. For the reasons discussed earlier, I'm not using multithreading. Instead, I'm doing everything I can to reduce the number of calls to play(), both overall and per-frame. This includes, among other things, only supporting stereo audio (for various reasons, supporting both mono and stereo can lead to more calls to play(), which is undesirable).
Lastly, I also investigated alternatives to AVAudioEngine. OpenAL is still supported on iOS, but is deprecated. A custom implementation using low-level APIs such as Audio Queue Services or Audio Units would be a possibility, but would be non-trivial. I've also looked at some open-source solutions, but the options I looked at use AVAudioEngine under the hood themselves and therefore suffer from the same problems, and/or have other shortcomings or limitations of their own. Of course there are also commercial options available, which may provide a solution for some developers.
I’m manually writing a video recorder. Unfortunately it’s necessary if you want to record video and use ARKit at the same time. I’ve got most of it figured out, but now I need to optimize it a bit because my phone gets pretty hot running ARKit, Vision and this recorder all at once.
To make the recorder, you need to use an AVAssetWriter with an AVAssetWriterInput (and AVAssetWriterInputPixelBufferAdaptor). The input has a isReadyForMoreMediaData property you need to check before you can write another frame. I’m recording in real-time (or as close to as possible).
Right now, when ARKit.ARSession gives me a new session I immediately pass it to the AVAssetWriterInput. What I want to do is add it to a queue, and have loop check to see if there’s samples available to write. For the life of me I can’t figure out how to do that efficiently.
I want to just run a while loop like this, but it seems like it would be a bad idea:
func startSession() {
// …
while isRunning {
guard !pixelBuffers.isEmpty && writerInput.isReadyForMoreMediaData else {
continue
}
// process sample
}
}
Can I run this a separate thread from the ARSession.delegateQueue? I don't want to run into issues with CVPixelBuffers from the camera being retained for too long.
I am working on an iOS application that, say on a button click, launches several threads, each executing a piece of Open GL code. These threads either have a different EAGLContext set on them, or if they use same EAGLContext, then they are synchronised (i.e. 2 threads don't set same EAGLContext in parallel).
Now suppose the app goes into background. As per Apple's documentation, we should stop all the OpenGL calls in applicationWillResignActive: callback so that by the time applicationDidEnterBackground: is called, no further GL calls are made.
I am using dispatch_queues to create background threads. For e.g.:
__block Byte* renderedData; // some memory already allocated
dispatch_sync(glProcessingQueue, ^{
[EAGLContext setCurrentContext:_eaglContext];
glViewPort(...)
glBindFramebuffer(...)
glClear(...)
glDrawArrays(...)
glReadPixels(...) // read in renderedData
}
use renderedData for something else
My question is - how to handle applicationWillResignActive: so that any such background GL calls can be not just stopped, but also be able to resume on applicationDidBecomeActive:? Should I wait for currently running blocks to finish before returning from applicationWillResignActive:? Or should I just suspend glProcessingQueue and return?
I have also read that similar is the case when app is interrupted in other ways, like displaying an alert, a phone call, etc.
I can have multiple such threads at any point of time, invoked by possibly multiple ViewControllers, so I am looking for some scalable solution or design pattern.
The way I see it you need to either pause a thread or kill it.
If you kill it you need to ensure all resources are released which means again calling openGL most likely. In this case it might actually be better to simply wait for the block to finish execution. This means the block must not take too long to finish which is impossible to guarantee and since you have multiple contexts and threads this may realistically present an issue.
So pausing seems better. I am not sure if there is a direct API to pause a thread but you can make it wait. Maybe a s system similar to this one can help.
The linked example seems to handle exactly what you would want; it already checks the current thread and locks that one. I guess you could pack that into some tool as a static method or a C function and wherever you are confident you can pause the thread you would simply do something like:
dispatch_sync(glProcessingQueue, ^{
[EAGLContext setCurrentContext:_eaglContext];
[ThreadManager pauseCurrentThreadIfNeeded];
glViewPort(...)
glBindFramebuffer(...)
[ThreadManager pauseCurrentThreadIfNeeded];
glClear(...)
glDrawArrays(...)
glReadPixels(...) // read in renderedData
[ThreadManager pauseCurrentThreadIfNeeded];
}
You might still have an issue with main thread if it is used. You might want to skip pause on that one otherwise your system may simply never wake up again (not sure though, try it).
So now you are look at interface of your ThreadManager to be something like:
+ (void)pause {
__threadsPaused = YES;
}
+ (void)resume {
__threadsPaused = NO;
}
+ (void)pauseCurrentThreadIfNeeded {
if(__threadsPaused) {
// TODO: insert code for locking until __threadsPaused becomes false
}
}
Let us know what you find out.
I'm making a new stopwatch Application with Watchkit, first my application is very simple like this:
first of all i tried to make a playButtonPressed to start a timer:
#IBAction func playButtonPressed() {
println("playButton pressed")
timer = NSTimer.scheduledTimerWithTimeInterval(0.01, target: self, selector: Selector("updateTimer"), userInfo: nil, repeats: true)
startTime = NSDate()
}
with my updateTimer function like this:
func updateTimer() {
duration = NSDate().timeIntervalSinceDate(startTime)
println("updateTimer: \(dateStringFromTimeInterval(duration))")
timeLabel.setText(dateStringFromTimeInterval(duration))
}
the dateStringFromTimeInterval function can help me make a dateString with duration is TimeInterval variable.
every thing is ok with my output on Debug area, i can see the dateString at printOut. But my label is lagging for setting the timeLabel as you can see here:
I don't know why, can any one can help me fix that or may be it is a bug of apple watchkit right now? i don't know it'll be lag on real device or not.
Thanks
You have several good questions in here. Unfortunately I have nothing but bad news for you. I have been working extensively with WKInterfaceTimers over the past couple of weeks and they have severe limitations and bugs associated with them. I have a couple responses broken down here in detail.
Issue 1 - Using a WKInterfaceDate as a Timer
This is going to be really frowned upon by Apple and I wouldn't doubt this would be possible grounds for rejection. As #mattt mentions, you don't want to use an NSTimer to flip the date value. Each time you try to switch the date label, Apple lumps all those changes together and pushes them from the Extension on the iPhone to the Watch over WiFi or Bluetooth. It does this to try to optimize battery life.
Because of this, you will never be able to accurately display the time on the Watch in the way that you are currently doing. The proper way to do this is to use a WKInterfaceTimer.
Issue 2 - Using a WKInterfaceTimer
While WKInterfaceTimers are built to do exactly what you want, they have some limitations. The major one for your use case is that it only does up to second precision, not millisecond. Secondly, the timers are extremely inaccurate. I've seen them anywhere from 50ms to 950ms off. I've followed this radar to bring the issue to Apple's attention.
In summary, your current implementation is not going to be super accurate and will be frowned upon by Apple, and the WKInterfaceTimer is extremely inaccurate and can only perform second precision.
Sorry for the downer answer. :-(
Hi !
I'm building a timer using GCD for the purpose of playing a sound at a specific interval, to be more precise, it's a metronome sound. I've been trying for days to solve my issue but nothing. Everything is good but when I set my tempo to a bigger value , let's say 150 bpm or 200 bpm, when the sound starts for the first time, it fires very quickly(almost like two sounds in the same time meaning it does not have the expected interval) and after this , it calibrates. I start the sound the second time , all is good... so this happens only the first time I resume my dispatch source so I'm guessing it has something to do with loading the sound from the disk , like in this post : Slow start for AVAudioPlayer the first time a sound is played . For my sound I used at first an instance of AVAudioPlayer with prepareToPlay and play and also created it in the AppDelegate class, it hasn't work...I have even tried the SoundManager class developed by #NickLockwood,same issue. At present, I'm using a SystemSoundID. As for the timers, this is my first GCD timer , I've already tried the classical NSTimer, CADisplayLink and other timers found on git... all in vain.
Another interesting issue is that with the other timers , everything is perfect on the simulator but on the device the same glitch.
Here's the code, I hope someone will bring me to the light.
-(void)playButtonAction //
{
if (_metronomeIsAnimatingAndPLaying == NO)
{
[self startAnimatingArm]; // I start my animation and create my timer
metronomeTimer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0));
dispatch_source_set_timer(metronomeTimer,dispatch_time(DISPATCH_TIME_NOW, duration * NSEC_PER_SEC),duration * NSEC_PER_SEC,duration *NSEC_PER_SEC);
dispatch_source_set_event_handler(metronomeTimer, ^{[self playTick];});
dispatch_resume(metronomeTimer);
_metronomeIsAnimatingAndPLaying = YES;
}
}
-(void)playTick
{
AudioServicesPlaySystemSound(appDeleg.soundID); // soundID is created in appDelegate
}
In my application didFinishLaunching
NSString *path = [[NSBundle mainBundle] pathForResource:#"tick"
ofType:#"caf"];
AudioServicesCreateSystemSoundID((CFURLRef)[NSURL fileURLWithPath:path]
, &_soundID);
And BPM setter and getter :
- (NSUInteger)bpm
{
return round(60.0 / duration);
}
- (void)setBpm:(NSUInteger)bpm
{
if (bpm >= MaxBPM) {
bpm = MaxBPM;
} else if (bpm <= MinBPM) {
bpm = MinBPM;
}
duration = (60.0 / bpm);
}
This arrangement will fundamentally never work.
GCD is a thread-pool designed to facilitate task-level parallelism. It is usually asynchronous and non real-time. These are almost precisely the opposite characteristics to those required in an audio application.
Each thread servicing a GCD queue is contending with other threads in the system for an opportunity to execute. Furthermore, the queue may be busy at requested time processing something else. If that something else is long-running - and long-running tasks are precisely the kind of thing that GCD is made for - the scheduler may pre-empt the thread before the operation has completed and penalise the queue; it may wait a long time for service.
The Manpage for GCD states the following about timers on GCD queues:
A best effort attempt is made to submit the event handler block to the target queue at the specified time; however, actual invocation may occur at a later time.
NSTimer will not be any better. Its documentation states A timer is not a real-time mechanism. Since you'll probably run this on the application's main run-loop, it will also be very unpredictable.
The solution to this problem is to use lower-level audio APIs - specifically Audio Units. The advantage of doing so is that soft-syth units have an event queue which is serviced by the unit's render handler. This runs on a real-time thread, and offers extremely robust and predictable service. Since you can queue a considerable number of events with timestamps in the future, your timing requirements are now quite loose. You could safely use either GCD or a NSTimer for this.