Detect when a chopping motion has been made - Swift - ios

Question

I'm starting with my first app for iOS and I am trying to get gyro data to play a whip sound when you flick your phone.
From what I can tell, I should be using CoreMotion to get the state of the gyro, then doing some math to work out when a whip-like gesture is made, and then to run my function?
This is what I have so far - this is my ContentView.swift file.
import SwiftUI
import AVFoundation
import CoreMotion
let popSound = Bundle.main.url(forResource: "whip", withExtension: "mp3")
var audioPlayer = AVAudioPlayer()
var motionManager: CMMotionManager!
func audioPlayback() {
do {
audioPlayer = try AVAudioPlayer(contentsOf: popSound!)
audioPlayer.play()
} catch {
print("couldn't load sound file")
}
}
struct ContentView: View {
var body: some View {
Text("Press button!")
Button(action: {
audioPlayback()
}, label: {
Text("Press me!")
})
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
Currently it's set to a button. Can someone link me to a resource, or walk me though this?

Answer 1

Usually when dealing with such devices you can either get a current snapshot or you can request to get a feed of snapshot changes. In your case the promising methods seem to be startAccelerometerUpdates and startDeviceMotionUpdates for CMMotionManager. I am pretty sure somewhere in there should be sufficient information to nearly detect a gesture you are describing.
If you dig into these methods you will see that you get a feed of "frames" where each frame describes a situation at certain time.
Since you are detecting a gesture you are not interested into a single frame but rather a series of frames. So probably the first thing you need is some object to which you can append frames and this object will evaluate if current set of frames corresponds to your gesture or not. It should also be able to discard data which is not interesting. For instance frames older than 3 seconds can be discarded as this gesture should never need more than 3 seconds.
So now your problem is split into 2 parts. First part is creating an object that is able to collect frames. I would give it a public method like appendFrame or appendSnapshot. Then keep collecting frames on it. The object also needs to be able to report back that it has detected a required gesture so that you play a sound at that point. Without the detection you should be able to mock for instance that after 100 frames the buffer is cleared and that notification is reported back which then triggers the sound. So no detection at this point but everything else.
The second part is the detection itself. You now have a pool of samples, frames or snapshots. You can at any time aggregate data anyway you want to. You would probably use a secondary thread to process the data so the UI is not laggy and to be able to throttle how much CPU power you put into it. As for the detection itself I would say you may try to create some samples and try figure out the "math" part. When you have some idea or can at least preset the community with some recordings you could ask another specific question about that. It does look like a textbook example to use Machine Learning for instance.
From mathematical point of view there may be some shortcuts. A very simple example would be just looking at the direction of your device as normalized direction(x, y, z). I think you can actually already get that very easily from native components. In a "chopping" motion we expect that rotation suddenly (nearly) stopped and was recently (nearly) 90 degrees offset from current direction.
Speed:
Assuming you have an array of direction such as let directions[(x: CGFloat, y: CGFloat, z: CGFloat)] then you could identify some rotation speed changes with length of cross product.
let rotationSpeed = length(cross(directions[index], directions[index+1]))
the speed should always be a value between 0 and 1 where a maximum of 1 would mean 90 degrees change. Hope it never comes to that and you are always in values between 0 and 0.3. If you DO get to values larger than 0.5 then frame-rate of your device is too low and samples are best just discarded.
Using this approach you can map your rotations from array of vectors to array of speeds rotationSpeeds: [Float] which becomes more convenient for you. You are now looking within this array if there is a part where the rotation speed suddenly drops from high value to low value. What those values are you will need to test yourself and tweak them. But a "sudden drop" may not be on only 2 sequential samples. You need to find for instance 5 high speed frames followed by 2 low speed frames. Rather even more than that.
Now that you found such a point you found a candidate for end of your chop. At this point you can now go backwards and check all frames going back in time up to somewhere between 0.5 and 1.0 seconds from candidate (again a value you will need to try out yourself). If any of this frame is nearly 90 degrees away from candidate then you have your gesture. Something like the following should do:
length(cross(directions[index], directions[candidateIndex])) > 0.5
where the 0.5 is again something you will need to test. The closer to 1.0 the more precise the gesture needs to be. I think 0.5 should be pretty good to begin with.
Perhaps you can play with the following and see if you can get satisfying results:
struct Direction {
let x: Float
let y: Float
let z: Float
static func cross(_ a: Direction, _ b: Direction) -> Direction {
Direction(x: a.y*b.z - a.z*b.y, y: a.z*b.x - a.x*b.z, z: a.x*b.y - a.y*b.z) // Needs testing
}
var length: Float { (x*x + y*y + z*z).squareRoot() }
}
class Recording<Type> {
private(set) var samples: [Type] = [Type]()
func appendSample(_ sample: Type) { samples.append(sample) }
}
class DirectionRecording: Recording<Direction> {
func convertToSpeedRecording() -> SpeedRecording {
let recording = SpeedRecording()
if samples.count > 1 { // Need at least 2 samples
for index in 0..<samples.count-1 {
recording.appendSample(Direction.cross(samples[index], samples[index+1]).length)
}
}
return recording
}
}
class SpeedRecording: Recording<Float> {
func detectSuddenDrops(minimumFastSampleCount: Int = 4, minimumSlowSampleCount: Int = 2, maximumThresholdSampleCount: Int = 2, minimumSpeedTreatedAsHigh: Float = 0.1, maximumSpeedThresholdTreatedAsLow: Float = 0.05) -> [Int] { // Returns an array of indices where sudden drop occurred
var result: [Int] = [Int]()
// Using states to identify where in the sequence we currently are.
// The state should go none -> highSpeed -> lowSpeed
// Or the state should go none -> highSpeed -> thresholdSpeed -> lowSpeed
enum State {
case none
case highSpeed(sequenceLength: Int)
case thresholdSpeed(sequenceLength: Int)
case lowSpeed(sequenceLength: Int)
}
var currentState: State = .none
samples.enumerated().forEach { index, sample in
if sample > minimumSpeedTreatedAsHigh {
// Found a high speed sample
switch currentState {
case .none: currentState = .highSpeed(sequenceLength: 1) // Found a first high speed sample
case .lowSpeed: currentState = .highSpeed(sequenceLength: 1) // From low speed to high speed resets it back to high speed step
case .thresholdSpeed: currentState = .highSpeed(sequenceLength: 1) // From threshold speed to high speed resets it back to high speed step
case .highSpeed(let sequenceLength): currentState = .highSpeed(sequenceLength: sequenceLength+1) // Append another high speed sample
}
} else if sample > maximumSpeedThresholdTreatedAsLow {
// Found a sample somewhere between fast and slow
switch currentState {
case .none: break // Needs to go to high speed first
case .lowSpeed: currentState = .none // Low speed back to threshold resets to beginning
case .thresholdSpeed(let sequenceLength):
if sequenceLength < maximumThresholdSampleCount { currentState = .thresholdSpeed(sequenceLength: sequenceLength+1) } // Can still stay inside threshold
else { currentState = .none } // In threshold for too long. Reseting back to start
case .highSpeed: currentState = .thresholdSpeed(sequenceLength: 1) // A first transition from high speed to threshold
}
} else {
// A low speed sample found
switch currentState {
case .none: break // Waiting for high speed sample sequence
case .lowSpeed(let sequenceLength):
if sequenceLength < minimumSlowSampleCount { currentState = .lowSpeed(sequenceLength: sequenceLength+1) } // Not enough low speed samples yet
else { result.append(index); currentState = .none } // Got everything we need. This is a HIT
case .thresholdSpeed: currentState = .lowSpeed(sequenceLength: 1) // Threshold can always go to low speed
case .highSpeed: currentState = .lowSpeed(sequenceLength: 1) // High speed can always go to low speed
}
}
}
return result
}
}
func recordingContainsAChoppingGesture(recording: DirectionRecording, minimumAngleOffset: Float = 0.5, maximumSampleCount: Int = 50) -> Bool {
let speedRecording = recording.convertToSpeedRecording()
return speedRecording.detectSuddenDrops().contains { index in
for offset in 1..<maximumSampleCount {
let sampleIndex = index-offset
guard sampleIndex >= 0 else { return false } // Can not go back any further than that
if Direction.cross(recording.samples[index], recording.samples[sampleIndex]).length > minimumAngleOffset {
return true // Got it
}
}
return false // Sample count drained
}
}