AudioKit - AKOperationGenerator with AKParameters - CPU Issue - ios

I need help with sending AKParameters to the AKOperationGenerator. My current solution use a lot of CPU. Is there a better way how to do it?
Here is my example code:
import AudioKit
class SynthVoice: AKNode {
override init() {
let synth = AKOperationGenerator { p in
//(1) - 30% CPU
let osc: AKOperation = AKOperation.squareWave(frequency: p[0], amplitude: p[1], pulseWidth: p[2])
//(2) - 9% CPU
//let osc: AKOperation = AKOperation.squareWave(frequency: 440, amplitude: 1, pulseWidth: 0.5)
return osc
}
synth.parameters[0] = 880
synth.parameters[1] = 1
synth.parameters[2] = 0.5
super.init()
self.avAudioNode = synth.avAudioNode
synth.start()
}
}
class ViewController: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
let mixer: AKMixer = AKMixer([SynthVoice(), SynthVoice(), SynthVoice(), SynthVoice(), SynthVoice(), SynthVoice()])
AudioKit.output = mixer
AudioKit.start()
}
}
I need 6 voice osc bank with envelope filter for each voice. I did not find any OSC-bank with envelope filter in AudioKit, so I started to write my own via AKOperationGenerator... But the CPU is too high. (About 100% in my project - 6 AKOperationGenerator with PWM square osc and envelope filter and a lot of AKParameters that can be changed via UI)
Thanks for any response.

I'd definitely do this at the DSP Kernel level. Its C/C++ but its really not too bad. Use one of the AKOscillatorBank type nodes as your model, but in addition to having an amplitude envelope, put in a filter envelope the same way. We're releasing an open source synth that does this exact thing in a few months if you can wait.

Related

Can TFLite's CoreMLDelegate use GPU and CPU simultaneously in iOS?

I've been successfully using tflite's MetalDelegate in my app. When I switch to CoreMLDelegate, it runs my (float) tflite model (MobileNet) entirely on CPU, showing 0 GPU use. I am running this on iPhone 11MaxPro, which is a compatible device. During the initialization I noticed the following line:
“CoreML delegate: 29 nodes delegated out of 31 nodes, with 2 partitions”.
Any ideas why? How do I make CoreMLDelegate use both GPU and CPU on iOS? I downloaded the mobilenet_v1_1.0_224.tflite model file from here.
import AVFoundation
import UIKit
import SpriteKit
import Metal
var device: MTLDevice!
var commandQueue: MTLCommandQueue!
private var total_latency:Double = 0
private var total_count:Double = 0
private var sstart = TimeInterval(NSDate().timeIntervalSince1970)
class ViewController: UIViewController {
...
}
// MARK: CameraFeedManagerDelegate Methods
extension ViewController: CameraFeedManagerDelegate {
func didOutput(pixelBuffer: CVPixelBuffer) {
let currentTimeMs = Date().timeIntervalSince1970 * 1
guard (currentTimeMs - previousInferenceTimeMs) >= delayBetweenInferencesMs else { return }
previousInferenceTimeMs = currentTimeMs
// 1. First create the Metal device and command queue in viewDidLoad():
device = MTLCreateSystemDefaultDevice()
commandQueue = device.makeCommandQueue()
var timestamp = NSDate().timeIntervalSince1970
let start = TimeInterval(timestamp)
// 2. Access the shared MTLCaptureManager and start capturing
let capManager = MTLCaptureManager.shared()
let myCaptureScope = capManager.makeCaptureScope(device: device)
myCaptureScope.begin()
let commandBuffer = commandQueue.makeCommandBuffer()!
// Do Metal work
// Pass the pixel buffer to TensorFlow Lite to perform inference.
result = modelDataHandler?.runModel(onFrame: pixelBuffer)
// 3.
// encode your kernel
commandBuffer.commit()
myCaptureScope.end()
timestamp = NSDate().timeIntervalSince1970
let end = TimeInterval(timestamp)
//var end = NSDate(timeIntervalSince1970: TimeInterval(myTimeInterval))
total_latency += (end - start)
total_count += 1;
let rfps = total_count/(end - sstart)
let fps = total_count/(end - start)
let stri = "Time: " + String(end - start) + " avg: " + String(total_latency/total_count)+" count: " + String(total_count)+" rfps: "+String(rfps)+" fps: "+String(fps)
print(stri)
// Display results by handing off to the InferenceViewController.
DispatchQueue.main.async {
guard let finalInferences = self.result?.inferences else {
self.resultLabel.text = ""
return
}
let resultStrings = finalInferences.map({ (inference) in
return String(format: "%# %.2f",inference.label, inference.confidence)
})
self.resultLabel.text = resultStrings.joined(separator: "\n")
}
}
2020-08-22 07:09:39.783215-0400 ImageClassification[3039:645963] coreml_version must be 2 or 3. Setting to 3.
2020-08-22 07:09:39.785103-0400 ImageClassification[3039:645963] Created TensorFlow Lite delegate for Metal.
2020-08-22 07:09:39.785505-0400 ImageClassification[3039:645963] Metal GPU Frame Capture Enabled
2020-08-22 07:09:39.786110-0400 ImageClassification[3039:645963] Metal API Validation Enabled
2020-08-22 07:09:39.927854-0400 ImageClassification[3039:645963] Initialized TensorFlow Lite runtime.
2020-08-22 07:09:39.928928-0400 ImageClassification[3039:645963] CoreML delegate: 29 nodes delegated out of 31 nodes, with 2 partitions
thanks for trying out Core ML delegate. Can you share the version of TFLite you used, and a code that you used to initialize Core ML delegate? Also, can you confirm that you're trying to run float model, not quantized one?
the latency might be different depending on what you measure, but when measuring solely the inference time my iPhone 11 Pro shows 11ms for CPU, and 5.5ms for Core ML delegate.
Neural Engine utilization is not captured by profiler, but if you're seeing that latency and high CPU utilization, it might indicate your model is running solely on CPU. You can also try the time profiler to figure out which part is consuming the most resources.

AudioKit - How to use AKAmplitudeTracker threshold callback?

AudioKit include a great tool to track signal amplitude: AKAmplitudeTracker
This tracker can be init with a thresholdCallback, I suppose that the callback should trigger when the threshold is reach.
I'm playing with the MicrophoneAnalysis example and I can't find a way to trigger my callback.
Here is my code:
var mic: AKMicrophone!
var trackerAmplitude: AKAmplitudeTracker!
var silence: AKBooster!
AKSettings.audioInputEnabled = true
mic = AKMicrophone()
trackerAmplitude = AKAmplitudeTracker(mic, halfPowerPoint: 10, threshold: 0.01, thresholdCallback: { (success) in
print("thresholdCallback: \(success)")
})
trackerAmplitude.start()
silence = AKBooster(trackerAmplitude, gain: 0)
AudioKit.output = silence
I tried to play with the halfPowerPoint and threshold values, but even with vey low values I cannot find a way to print anything :/
Whereas when I'm printing trackerAmplitude.amplitude, I've got values higher than 0.01
Is there something I'm missing ?
The following code works. Tested with AudioKit 4.9, Xcode 11.2, macOS Playground.
This might be an issue of AudioKit, but threshold must be changed via property to activate tracking, as shown below...
import AudioKitPlaygrounds
import AudioKit
let mic = AKMicrophone()
AKSettings.audioInputEnabled = true
let amplitudeTracker = AKAmplitudeTracker(mic, halfPowerPoint: 10, threshold: 1, thresholdCallback: { (success) in
print("thresholdCallback: \(success)")
})
AudioKit.output = amplitudeTracker
try AudioKit.start()
amplitudeTracker.threshold = 0.01 // !! MUST BE SET VIA PROPERTY
amplitudeTracker.start()
mic?.start()
import PlaygroundSupport
PlaygroundPage.current.needsIndefiniteExecution = true

AudioKit - How to use frequency filters with microphone?

I'm using the AudioKit library with Swift for developing a simple iOS application that should be able to listen to frequencies of the range 3.000Hz - maybe 6000Hz.
So I'd like to just track the input frequency of the microphone within this range and to achieve this I tried to apply a filter effect on the input microphone to avoid picking up the frequency of several unwanted noises.
var mic: AKMicrophone
var silence: AKBooster
var filter: AKHighPassFilter
var tracker: AKFrequencyTracker
public init() {
mic = AKMicrophone()
filter = AKHighPassFilter(mic)
filter.cutoffFrequency = 3000 // just get frequencyies above 3000Hz (highpass)
filter.resonance = 0
tracker = AKFrequencyTracker(filter)
silence = AKBooster(tracker, gain: 0)
}
func start() {
AKSettings.audioInputEnabled = true
AudioKit.output = silence
AudioKit.start()
}
func print() {
print(tracker.frequency)
}
To sum that up: I know that the filter is changing something - but I can not really apply a frequency-filter for the range 3.000Hz + because I'm also getting values below 3.000Hz (like 2.0000 / 500 / etc) for the filtered frequency.
The AudiKit website has included some examples how to use the filters - But I can find no examples how to apply filters on the input-microphone to get a filtered frequency? http://audiokit.io/playgrounds/Filters/
Am I doing something the wrong way?
Is this really the functionality of the AudioKit-filters or didn't I get the right sense of filters?
Is there another way to filter for frequency-ranges?

Can't load AKAudioFile into AKSampler

I'm trying to use AKSampler on a simple iOS project to load a file and play it when tapping the device's screen.
I did the same steps with AKSamplePlayer and it worked fine, But I rather use the AKSampler, and also I get a strong feeling of missing something.
I've tried the play() method, and also the one with the midi note.
Which one is right? Do they both work?Besides, AudioKit looks so promising.
Here is my code:
import UIKit
import AudioKit
class ViewController: UIViewController
{
var sampler = AKSampler()
var tapRecognizer = UITapGestureRecognizer()
override func viewDidLoad()
{
super.viewDidLoad()
do
{
let file = try AKAudioFile(readFileName: "AH_G2.wav")
try sampler.loadAudioFile(file)
}
catch
{
print("No Such File...")
}
view.addGestureRecognizer(tapRecognizer)
view.isUserInteractionEnabled = true
tapRecognizer.addTarget(self, action: #selector(viewTapped))
AudioKit.output = sampler
AudioKit.start()
}
#objc private func viewTapped()
{
sampler.play(noteNumber: 60, velocity: 80, channel: 0)
print("tapped...")
}
}
Edit:
My problem is actually with the loadAudioFile method, the AKAudioFile itself is good, and the AKSampler plays a default sine sound.
I tried also the AKAudioFile methods for creating player and sampler didn't.
let file = try AKAudioFile (readFileName: "AH_G2.wav")
player = file.player
sampler = file.sampler
I also tried to add the wav file using the menu, no change.
If you look at the implementation, there is just the one play() method, but it has default values for noteNumber, velocity, and channel:
#objc open func play(noteNumber: MIDINoteNumber = 60,
velocity: MIDIVelocity = 127,
channel: MIDIChannel = 0) {
samplerUnit.startNote(noteNumber, withVelocity: velocity, onChannel: channel)
}
Changing the MIDI note will change the pitch/speed of the sample playback (60 is standard, 72 is double speed, 48 would be half speed etc), and changing the velocity will change the volume.
NB: the title of your post is 'AKSampler doesn't play', but I ran your code (changing the sample, of course) and it played just fine on my iPad.
I've tried a different audio file and it worked fine.
The first file was a mono file, so my conclusion here is that the AKSampler does not support mono files. Would love to hear more on that.

Perform Audio Analysis with FFT

I've been stuck on this problem for days now and have looked through nearly every related StackOverflow page. Through this, I now have a much greater understanding of what FFT is and how it works. Despite this, I'm having extreme difficulties implementing it into my application.
In short, what I am trying to do is make a spectrum visualizer for my application (Similar to this). From what I've gathered, I'm pretty sure I need to use the magnitudes of the sound as the heights of my bars. So with all this in mind, currently I am able to analyze an entire .caf file all at once. To do this, I am using the following code:
let audioFile = try! AVAudioFile(forReading: soundURL!)
let frameCount = UInt32(audioFile.length)
let buffer = AVAudioPCMBuffer(PCMFormat: audioFile.processingFormat, frameCapacity: frameCount)
do {
try audioFile.readIntoBuffer(buffer, frameCount:frameCount)
} catch {
}
let log2n = UInt(round(log2(Double(frameCount))))
let bufferSize = Int(1 << log2n)
let fftSetup = vDSP_create_fftsetup(log2n, Int32(kFFTRadix2))
var realp = [Float](count: bufferSize/2, repeatedValue: 0)
var imagp = [Float](count: bufferSize/2, repeatedValue: 0)
var output = DSPSplitComplex(realp: &realp, imagp: &imagp)
vDSP_ctoz(UnsafePointer<DSPComplex>(buffer.floatChannelData.memory), 2, &output, 1, UInt(bufferSize / 2))
vDSP_fft_zrip(fftSetup, &output, 1, log2n, Int32(FFT_FORWARD))
var fft = [Float](count:Int(bufferSize / 2), repeatedValue:0.0)
let bufferOver2: vDSP_Length = vDSP_Length(bufferSize / 2)
vDSP_zvmags(&output, 1, &fft, 1, bufferOver2)
This works fine and outputs a long array of data. However, the problem with this code is it analyzes the entire audio file at once. What I need is to be analyzing the audio file as it is playing, very similar to this video: Spectrum visualizer.
So I guess my question is this: How do you perform FFT analysis while the audio is playing?
Also, on top of this, how do I go about converting the output of an FFT analysis to actual heights for a bar? One of the outputs I received for an audio file using the FFT analysis code from above was this: http://pastebin.com/RBLTuGx7. The only reason for the pastebin is due to how long it is. I'm assuming I average all these numbers together and use those values instead? (Just for reference, I got that array by printing out the 'fft' variable in the code above)
I've attempted reading through the EZAudio code, however I am unable to find how they are reading in samples of audio in live time. Any help is greatly appreciated.
Here's how it is done in AudioKit, using EZAudio's FFT tools:
Create a class for your FFT that will hold the data:
#objc public class AKFFT: NSObject, EZAudioFFTDelegate {
internal let bufferSize: UInt32 = 512
internal var fft: EZAudioFFT?
/// Array of FFT data
public var fftData = [Double](count: 512, repeatedValue: 0.0)
...
}
Initialize the class and setup the FFT. Also install the tap on the appropriate node.
public init(_ input: AKNode) {
super.init()
fft = EZAudioFFT.fftWithMaximumBufferSize(vDSP_Length(bufferSize), sampleRate: 44100.0, delegate: self)
input.avAudioNode.installTapOnBus(0, bufferSize: bufferSize, format: AKManager.format) { [weak self] (buffer, time) -> Void in
if let strongSelf = self {
buffer.frameLength = strongSelf.bufferSize;
let offset: Int = Int(buffer.frameCapacity - buffer.frameLength);
let tail = buffer.floatChannelData[0];
strongSelf.fft!.computeFFTWithBuffer(&tail[offset], withBufferSize: strongSelf.bufferSize)
}
}
}
Then implement the callback to load your internal fftData array:
#objc public func fft(fft: EZAudioFFT!, updatedWithFFTData fftData: UnsafeMutablePointer<Float>, bufferSize: vDSP_Length) {
dispatch_async(dispatch_get_main_queue()) { () -> Void in
for i in 0...511 {
self.fftData[i] = Double(fftData[i])
}
}
}
AudioKit's implementation may change so you should check https://github.com/audiokit/AudioKit/ to see if any improvements were made. EZAudio is at https://github.com/syedhali/EZAudio

Resources