I am working on a musical app with an arpeggio/sequencing feature that requires great timing accuracy. Currently, using `Timer' I have achieved an accuracy with an average jitter of ~5ms, but a max jitter of ~11ms, which is unacceptable for fast arpeggios of 8th, 16th notes & 32nd notes especially.
I've read the 'CADisplayLink' is more accurate than 'Timer', but since it is limited to 1/60th of a second for it's accuracy (~16-17ms), it seems like it would be a less accurate approach than what I've achieved with Timer.
Would diving into CoreAudio be the only way to achieve what I want? Is there some other way to achieve more accurate timing?
I did some testing of Timer and DispatchSourceTimer (aka GCD timer) on iPhone 7 with 1000 data points with an interval of 0.05 seconds. I was expecting GCD timer to be appreciably more accurate (given that it had a dedicated queue), but I found that they were comparable, with standard deviation of my various trials ranging from 0.2-0.8 milliseconds and maximum deviation from the mean of about 2-8 milliseconds.
When trying mach_wait_until as outlined in Technical Note TN2169: High Precision Timers in iOS / OS X, I achieved a timer that was roughly 4 times as accurate than what I achieved with either Timer or GCD timers.
Having said that, I'm not entirely confident of the mach_wait_until is the best approach, as the determination of the specific policy values for thread_policy_set seem to be poorly documented. But the code below reflects the values I used in my tests, using code adapted from How to set realtime thread in Swift? and TN2169:
var timebaseInfo = mach_timebase_info_data_t()
func configureThread() {
mach_timebase_info(&timebaseInfo)
let clock2abs = Double(timebaseInfo.denom) / Double(timebaseInfo.numer) * Double(NSEC_PER_SEC)
let period = UInt32(0.00 * clock2abs)
let computation = UInt32(0.03 * clock2abs) // 30 ms of work
let constraint = UInt32(0.05 * clock2abs)
let THREAD_TIME_CONSTRAINT_POLICY_COUNT = mach_msg_type_number_t(MemoryLayout<thread_time_constraint_policy>.size / MemoryLayout<integer_t>.size)
var policy = thread_time_constraint_policy()
var ret: Int32
let thread: thread_port_t = pthread_mach_thread_np(pthread_self())
policy.period = period
policy.computation = computation
policy.constraint = constraint
policy.preemptible = 0
ret = withUnsafeMutablePointer(to: &policy) {
$0.withMemoryRebound(to: integer_t.self, capacity: Int(THREAD_TIME_CONSTRAINT_POLICY_COUNT)) {
thread_policy_set(thread, UInt32(THREAD_TIME_CONSTRAINT_POLICY), $0, THREAD_TIME_CONSTRAINT_POLICY_COUNT)
}
}
if ret != KERN_SUCCESS {
mach_error("thread_policy_set:", ret)
exit(1)
}
}
I then could do:
private func nanosToAbs(_ nanos: UInt64) -> UInt64 {
return nanos * UInt64(timebaseInfo.denom) / UInt64(timebaseInfo.numer)
}
private func startMachTimer() {
Thread.detachNewThread {
autoreleasepool {
self.configureThread()
var when = mach_absolute_time()
for _ in 0 ..< maxCount {
when += self.nanosToAbs(UInt64(0.05 * Double(NSEC_PER_SEC)))
mach_wait_until(when)
// do something
}
}
}
}
Note, you might want to see if when hasn't already passed (you want to make sure that your timers don't get backlogged if your processing can't be completed in the allotted time), but hopefully this illustrates the idea.
Anyway, with mach_wait_until, I achieved greater fidelity than Timer or GCD timers, at the cost of CPU/power consumption as described in What are the do's and dont's of code running with high precision timers?
I appreciate your skepticism on this final point, but I suspect it would be prudent to dive into CoreAudio and see if it might offer a more robust solution.
For acceptable musically accurate rhythms, the only suitable timing source is using Core Audio or AVFoundation.
I'm working on a sequencer App myself, and I would defiantly recommend using AudioKit for those purposes.
It has a its own sequencer class.
https://audiokit.io/
Related
I apologize for not knowing the proper terminology for everything here. I'm a fairly new programmer, and entirely new to Swift. The task I'm trying to accomplish is to display a current speed in MPH. I've found that using the "CoreLocation" and storing locations in an array and using "locations.speed "to display the speed is quite slow and does not refresh as often as I want.
My thought was to get an initial speed value using the "MapKit" and "CoreLocation" method, then feed that initial speed value into a function using the accelerometer to provide a quicker responding speedometer. I would do this by integrating the accelerometer values and adding the initial velocity. This was the best solution I could come up with to get a more accurate speedometer with a better refresh rate.
I'm having a couple of issues currently:
First Issue: I don't know how to get an initial speed value from a function using location data as parameters into a function using accelerometer data as parameters.
Second Issue: Even when assuming an initial speed of 0, my current program displays a value that keeps increasing infinitely. I'm not sure what the issue is that is causing this.
I will show you the portion of my code responsible for this, and would appreciate any insight any of you may have!
For my First Issue, here is my GPS data Function:
func provideInitSpeed(_ manager: CLLocationManager, didUpdateLocations locations: [CLLocation])->Double {
let location = locations[0]
return ((location.speed)*2.23693629) //returns speed value converted to MPH
}
I'm not sure how to make a function call to retrieve this value in my Accelerometer function.
For my Second Issue, here is my Accelerometer Function with assumed starting speed of 0:
motionManager.startAccelerometerUpdates(to: OperationQueue.current!) {
(data,error) in
if let myData = data {
//getting my acceleration data and rounding the values off to the hundredths place to reduce noise
xAccel = round((myData.acceleration.x * g)*100)/100
yAccel = round((myData.acceleration.y * g)*100)/100
zAccel = round((myData.acceleration.z * g)*100)/100
// Integrating accel vals to get velocity vals *Possibly where error occurs* I multiply the accel values by the change in time, which is currently set at 0.2 seconds.
xVel += xAccel * self.motionManager.accelerometerUpdateInterval
yVel += yAccel * self.motionManager.accelerometerUpdateInterval
zVel += zAccel * self.motionManager.accelerometerUpdateInterval
// Finding total speed; Magnitude of Velocity
totalSpeed = sqrt(pow(xVel,2) + pow(yVel,2) + pow(zVel,2))
// if-else statment for further noise reduction. note: "zComp" just adjusts for the -1.0 G units z acceleration value that the phone reads by default for gravity
if (totalSpeed - zComp) > -0.1 && (totalSpeed - zComp) < 0.1 {
self.view.reloadInputViews()
self.speedWithAccelLabel.text = "\(0.0)"
} else {
// Printing totalSpeed
self.view.reloadInputViews()
self.speedWithAccelLabel.text = "\(abs(round((totalSpeed - zComp + /*where initSpeed would go*/)*10)/10))"
}
}//data end
}//motionManager end
I'm not sure why but the speed this function displays is always increasing by about 4 mph every refresh of the label.
This is my first time using Stack Overflow, so I apologize for any stupid mistakes I might have made!
Thanks a lot!
I'm trying to generate a spectrogram from an AVAudioPCMBuffer in Swift. I install a tap on an AVAudioMixerNode and receive a callback with the audio buffer. I'd like to convert the signal in the buffer to a [Float:Float] dictionary where the key represents the frequency and the value represents the magnitude of the audio on the corresponding frequency.
I tried using Apple's Accelerate framework but the results I get seem dubious. I'm sure it's just in the way I'm converting the signal.
I looked at this blog post amongst other things for a reference.
Here is what I have:
self.audioEngine.mainMixerNode.installTapOnBus(0, bufferSize: 1024, format: nil, block: { buffer, when in
let bufferSize: Int = Int(buffer.frameLength)
// Set up the transform
let log2n = UInt(round(log2(Double(bufferSize))))
let fftSetup = vDSP_create_fftsetup(log2n, Int32(kFFTRadix2))
// Create the complex split value to hold the output of the transform
var realp = [Float](count: bufferSize/2, repeatedValue: 0)
var imagp = [Float](count: bufferSize/2, repeatedValue: 0)
var output = DSPSplitComplex(realp: &realp, imagp: &imagp)
// Now I need to convert the signal from the buffer to complex value, this is what I'm struggling to grasp.
// The complexValue should be UnsafePointer<DSPComplex>. How do I generate it from the buffer's floatChannelData?
vDSP_ctoz(complexValue, 2, &output, 1, UInt(bufferSize / 2))
// Do the fast Fournier forward transform
vDSP_fft_zrip(fftSetup, &output, 1, log2n, Int32(FFT_FORWARD))
// Convert the complex output to magnitude
var fft = [Float](count:Int(bufferSize / 2), repeatedValue:0.0)
vDSP_zvmags(&output, 1, &fft, 1, vDSP_length(bufferSize / 2))
// Release the setup
vDSP_destroy_fftsetup(fftsetup)
// TODO: Convert fft to [Float:Float] dictionary of frequency vs magnitude. How?
})
My questions are
How do I convert the buffer.floatChannelData to UnsafePointer<DSPComplex> to pass to the vDSP_ctoz function? Is there a different/better way to do it maybe even bypassing vDSP_ctoz?
Is this different if the buffer contains audio from multiple channels? How is it different when the buffer audio channel data is or isn't interleaved?
How do I convert the indices in the fft array to frequencies in Hz?
Anything else I may be doing wrong?
Update
Thanks everyone for suggestions. I ended up filling the complex array as suggested in the accepted answer. When I plot the values and play a 440 Hz tone on a tuning fork it registers exactly where it should.
Here is the code to fill the array:
var channelSamples: [[DSPComplex]] = []
for var i=0; i<channelCount; ++i {
channelSamples.append([])
let firstSample = buffer.format.interleaved ? i : i*bufferSize
for var j=firstSample; j<bufferSize; j+=buffer.stride*2 {
channelSamples[i].append(DSPComplex(real: buffer.floatChannelData.memory[j], imag: buffer.floatChannelData.memory[j+buffer.stride]))
}
}
The channelSamples array then holds separate array of samples for each channel.
To calculate the magnitude I used this:
var spectrum = [Float]()
for var i=0; i<bufferSize/2; ++i {
let imag = out.imagp[i]
let real = out.realp[i]
let magnitude = sqrt(pow(real,2)+pow(imag,2))
spectrum.append(magnitude)
}
Hacky way: you can just cast a float array. Where reals and imag values are going one after another.
It depends on if audio is interleaved or not. If it's interleaved (most of the cases) left and right channels are in the array with STRIDE 2
Lowest frequency in your case is frequency of a period of 1024 samples. In case of 44100kHz it's ~23ms, lowest frequency of the spectrum will be 1/(1024/44100) (~43Hz). Next frequency will be twice of this (~86Hz) and so on.
4: You have installed a callback handler on an audio bus. This is likely run with real-time thread priority and frequently. You should not do anything that has potential for blocking (it will likely result in priority inversion and glitchy audio):
Allocate memory (realp, imagp - [Float](.....) is shorthand for Array[float] - and likely allocated on the heap`. Pre-allocate these
Call lengthy operations such as vDSP_create_fftsetup() - which also allocates memory and initialises it. Again, you can allocate this once outside of your function.
I'm doing some glsl fractals, and I'd like to make the calculations bail if they're taking too long to keep the frame rate up (without having to figure out what's good for each existing device and any future ones).
It would be nice if there were a timer I could check every 10 iterations or something....
Failing that, it seems the best approach might be to track how long it took to render the previous frame (or previous N frames) and change the "iterate to" number dynamically as a uniform...?
Or some other suggestion? :)
As it appears there's no good way to do this in the GPU, one can do a simple approach to "tune" the "bail after this number of iterations" threshold outside the loop, once per frame.
CFTimeInterval previousTimestamp = CFAbsoluteTimeGetCurrent();
// gl calls here
CFTimeInterval frameDuration = CFAbsoluteTimeGetCurrent() - previousTimestamp;
float msecs = frameDuration * 1000.0;
if (msecs < 0.2) {
_dwell = MIN(_dwell + 16., 256.);
} else if (msecs > 0.4) {
_dwell = MAX(_dwell - 4., 32.);
}
So my "dwell" is kept between 32 and 256, and more optimistically raised than decreased, and is pushed as a uniform in the "gl calls here" section.
I am creating a very simple game using Swift and SpriteKit and I am moving a ball on the screen using the accelerometer data (acceleration x,y).
I would say the code works fine but I have noticed that sometimes (often right when I open the app) the accelerometer data is not correct and delayed for few seconds.
Why is that happening?
I am using the following code to read the accelerometer data:
if motionManager.accelerometerAvailable == true {
motionManager.startAccelerometerUpdatesToQueue(NSOperationQueue.currentQueue(), withHandler:{
data, error in
self.accX = CGFloat(data.acceleration.x)
self.accY = CGFloat(data.acceleration.y)
})
}
And the function update to apply some impulse to the ball:
override func update(currentTime: CFTimeInterval) {
var impulse = CGVectorMake(accX, accY)
var obj = childNodeWithName("ball") as SKSpriteNode
obj.physicsBody?.applyImpulse(impulse)
}
Am i missing something?
Thank you
With any accelerometer data, it is a good idea to run it through a filter to smooth out any irregular spikes. Here is my favorite:
double filteredAcceleration[3];
memset(filteredAcceleration, 0, sizeof(filteredAcceleration));
CMAccelerometerData *newestAccel = motionManager.accelerometerData;
filteredAcceleration[0] = (filteredAcceleration[0]*(1.0-alpha)) + (newestAccel.acceleration.x*alpha);
filteredAcceleration[1] = (filteredAcceleration[1]*(1.0-alpha)) + (newestAccel.acceleration.y*alpha);
filteredAcceleration[2] = (filteredAcceleration[2]*(1.0-alpha)) + (newestAccel.acceleration.z*alpha);
alpha can be any value from 0 to 1. The closer to 1 the more responsive it will be, the closer to zero the more smooth it will be. My favorite value on the iPhone is 0.2 It is a good compromise for smooth yet responsive for a game like doodle jump, or possibly moving a ball around.
I don't know why the accelerometer data is incorrect/delayed on startup, my guess would be that the hardware has to wake up and calibrate itself, but regardless of the why, if you implement a filter, it will smooth out these irregularities, and they won't be nearly as noticeable.
I have given priority to both functions and the issue seems fixed.
let priority = DISPATCH_QUEUE_PRIORITY_DEFAULT
dispatch_async(dispatch_get_global_queue(priority, 0)) {
// do some task
dispatch_async(dispatch_get_main_queue()) {
// code with priority
}
}
I'm trying to detect three actions: when a user begins walking, jogging, or running. I then want to know when the stop. I've been successful in detecting when someone is walking, jogging, or running with the following code:
- (void)update:(CMAccelerometerData *)accelData {
[(id) self setAcceleration:accelData.acceleration];
NSTimeInterval secondsSinceLastUpdate = -([self.lastUpdateTime timeIntervalSinceNow]);
if (labs(_acceleration.x) >= 0.10000) {
NSLog(#"walking: %f",_acceleration.x);
}
else if (labs(_acceleration.x) > 2.0) {
NSLog(#"jogging: %f",_acceleration.x);
}
else if (labs(_acceleration.x) > 4.0) {
NSLog(#"sprinting: %f",_acceleration.x);
}
The problem I run into is two-fold:
1) update is called multiple times every time there's a motion, probably because it checks so frequently that when the user begins walking (i.e. _acceleration.x >= .1000) it is still >= .1000 when it calls update again.
Example Log:
2014-02-22 12:14:20.728 myApp[5039:60b] walking: 1.029846
2014-02-22 12:14:20.748 myApp[5039:60b] walking: 1.071777
2014-02-22 12:14:20.768 myApp[5039:60b] walking: 1.067749
2) I'm having difficulty figuring out how to detect when the user stopped. Does anybody have advice on how to implement "Stop Detection"
According to your logs, accelerometerUpdateInterval is about 0.02. Updates could be less frequent if you change mentioned property of CMMotionManager.
Checking only x-acceleration isn't very accurate. I can put a device on a table in a such way (let's say on left edge) that x-acceleration will be equal to 1, or tilt it a bit. This will cause a program to be in walking mode (x > 0.1) instead of idle.
Here's a link to ADVANCED PEDOMETER FOR SMARTPHONE-BASED ACTIVITY TRACKING publication. They track changes in the direction of the vector of acceleration. This is the cosine of the angle between two consecutive acceleration vector readings.
Obviously, without any motion, angle between two vectors is close to zero and cos(0) = 1. During other activities d < 1. To filter out noise, they use a weighted moving average of the last 10 values of d.
After implementing this, your values will look like this (red - walking, blue - running):
Now you can set a threshold for each activity to separate them. Note that average step frequency is 2-4Hz. You should expect current value to be over the threshold at least few times in a second in order to identify the action.
Another helpful publications:
ERSP: An Energy-efficient Real-time Smartphone Pedometer (analyze peaks and throughs)
A Gyroscopic Data based Pedometer Algorithm (threshold detection of gyro readings)
UPDATE
_acceleration.x, _accelaration.y, _acceleration.z are coordinates of the same acceleration vector. You use each of these coordinates in d formula. In order to calculate d you also need to store acceleration vector of previous update (with i-1 index in formula).
WMA just take into account 10 last d values with different weights. Most recent d values have more weight, therefore, more impact on resulting value. You need to store 9 previous d values in order to calculate current one. You should compare WMA value to corresponding threshold.
if you are using iOS7 and iPhone5S, I suggest you look into CMMotionActivityManager which is available in iPhone5S because of the M7 chip. It is also available in a couple of other devices:
M7 chip
Here is a code snippet I put together to test when I was learning about it.
#import <CoreMotion/CoreMotion.h>
#property (nonatomic,strong) CMMotionActivityManager *motionActivityManager;
-(void) inSomeMethod
{
self.motionActivityManager=[[CMMotionActivityManager alloc]init];
//register for Coremotion notifications
[self.motionActivityManager startActivityUpdatesToQueue:[NSOperationQueue mainQueue] withHandler:^(CMMotionActivity *activity)
{
NSLog(#"Got a core motion update");
NSLog(#"Current activity date is %f",activity.timestamp);
NSLog(#"Current activity confidence from a scale of 0 to 2 - 2 being best- is: %ld",activity.confidence);
NSLog(#"Current activity type is unknown: %i",activity.unknown);
NSLog(#"Current activity type is stationary: %i",activity.stationary);
NSLog(#"Current activity type is walking: %i",activity.walking);
NSLog(#"Current activity type is running: %i",activity.running);
NSLog(#"Current activity type is automotive: %i",activity.automotive);
}];
}
I tested it and it seems to be pretty accurate. The only drawback is that it will not give you a confirmation as soon as you start an action (walking for example). Some black box algorithm waits to ensure that you are really walking or running. But then you know you have a confirmed action.
This beats messing around with the accelerometer. Apple took care of that detail!
You can use this simple library to detect if user is walking, running, on vehicle or not moving. Works on all iOS devices and no need M7 chip.
https://github.com/SocialObjects-Software/SOMotionDetector
In repo you can find demo project
I'm following this paper(PDF via RG) in my indoor navigation project to determine user dynamics(static, slow walking, fast walking) via merely accelerometer data in order to assist location determination.
Here is the algorithm proposed in the project:
And here is my implementation in Swift 2.0:
import CoreMotion
let motionManager = CMMotionManager()
motionManager.accelerometerUpdateInterval = 0.1
motionManager.startAccelerometerUpdatesToQueue(NSOperationQueue.mainQueue()) { (accelerometerData: CMAccelerometerData?, error: NSError?) -> Void in
if((error) != nil) {
print(error)
} else {
self.estimatePedestrianStatus((accelerometerData?.acceleration)!)
}
}
After all of the classic Swifty iOS code to initiate CoreMotion, here is the method crunching the numbers and determining the state:
func estimatePedestrianStatus(acceleration: CMAcceleration) {
// Obtain the Euclidian Norm of the accelerometer data
accelerometerDataInEuclidianNorm = sqrt((acceleration.x.roundTo(roundingPrecision) * acceleration.x.roundTo(roundingPrecision)) + (acceleration.y.roundTo(roundingPrecision) * acceleration.y.roundTo(roundingPrecision)) + (acceleration.z.roundTo(roundingPrecision) * acceleration.z.roundTo(roundingPrecision)))
// Significant figure setting
accelerometerDataInEuclidianNorm = accelerometerDataInEuclidianNorm.roundTo(roundingPrecision)
// record 10 values
// meaning values in a second
// accUpdateInterval(0.1s) * 10 = 1s
while accelerometerDataCount < 1 {
accelerometerDataCount += 0.1
accelerometerDataInASecond.append(accelerometerDataInEuclidianNorm)
totalAcceleration += accelerometerDataInEuclidianNorm
break // required since we want to obtain data every acc cycle
}
// when acc values recorded
// interpret them
if accelerometerDataCount >= 1 {
accelerometerDataCount = 0 // reset for the next round
// Calculating the variance of the Euclidian Norm of the accelerometer data
let accelerationMean = (totalAcceleration / 10).roundTo(roundingPrecision)
var total: Double = 0.0
for data in accelerometerDataInASecond {
total += ((data-accelerationMean) * (data-accelerationMean)).roundTo(roundingPrecision)
}
total = total.roundTo(roundingPrecision)
let result = (total / 10).roundTo(roundingPrecision)
print("Result: \(result)")
if (result < staticThreshold) {
pedestrianStatus = "Static"
} else if ((staticThreshold < result) && (result <= slowWalkingThreshold)) {
pedestrianStatus = "Slow Walking"
} else if (slowWalkingThreshold < result) {
pedestrianStatus = "Fast Walking"
}
print("Pedestrian Status: \(pedestrianStatus)\n---\n\n")
// reset for the next round
accelerometerDataInASecond = []
totalAcceleration = 0.0
}
}
Also I've used the following extension to simplify significant figure setting:
extension Double {
func roundTo(precision: Int) -> Double {
let divisor = pow(10.0, Double(precision))
return round(self * divisor) / divisor
}
}
With raw values from CoreMotion, the algorithm was haywire.
Hope this helps someone.
EDIT (4/3/16)
I forgot to provide my roundingPrecision value. I defined it as 3. It's just plain mathematics that that much significant value is decent enough. If you like you provide more.
Also one more thing to mention is that at the moment, this algorithm requires the iPhone to be in your hand while walking. See the picture below. Sorry this was the only one I could find.
My GitHub Repo hosting Pedestrian Status
You can use Apple's latest Machine Learning framework CoreML to find out user activity. First you need to collect labeled data and train the classifier. Then you can use this model in your app to classify user activity. You may follow this series if are interested in CoreML Activity Classification.
https://medium.com/#tyler.hutcherson/activity-classification-with-create-ml-coreml3-and-skafos-part-1-8f130b5701f6