i have setup an EZAudio in swift to calculate the fft of the realtime mic input, and then i run a special algorithm over the fft data.
My problem is i can access the fft data when i put this in the view controller, with dispatch_async.(See code the last func)
class MasterKey:NSObject,EZMicrophoneDelegate, EZAudioFFTDelegate{
var microphone: EZMicrophone!
var fft: EZAudioFFTRolling!
var tone:String = ""
var sampleRate:Float = 0.0
var fftWindowSize:vDSP_Length = 8192
var keys:MKHRangeToKey!
init(tone:String){
super.init()
self.tone = tone
/*
* setup all dependencys for the fft analysis
*/
//setup audio session
let session = AVAudioSession.sharedInstance()
do{
try session.setCategory(AVAudioSessionCategoryPlayAndRecord)
try session.setActive(true)
}catch{
print("Audio Session setup Fails")
}
//create a mic instance
microphone = EZMicrophone(delegate: self, startsImmediately: true)
self.sampleRate = Float(microphone.audioStreamBasicDescription().mSampleRate)
//create a fft instace
fft = EZAudioFFTRolling(windowSize: fftWindowSize, sampleRate: sampleRate, delegate: self)
//start the mic
microphone.startFetchingAudio()
self.keys = MKHRangeToKey(tone: tone, sampleRate: sampleRate, fftWindowSize: Int(fftWindowSize))
}
//get the mic data
func microphone(microphone: EZMicrophone!, hasAudioReceived buffer: UnsafeMutablePointer<UnsafeMutablePointer<Float>>, withBufferSize bufferSize: UInt32, withNumberOfChannels numberOfChannels: UInt32) {
//calc the fft
if fft != nil{
fft.computeFFTWithBuffer(buffer[0], withBufferSize: bufferSize)
}
}
//get the fft data from last calculstion
func fft(fft: EZAudioFFT!, updatedWithFFTData fftData: UnsafeMutablePointer<Float>, bufferSize: vDSP_Length){
dispatch_async(dispatch_get_main_queue(), {
print(fftData)
})
}
}
But how can i put this in a separate class to call it when i needed?
Pleas pleas Help
You indicated you're using Swift. Why not just create a separate AudioFunctions.swift file and move the function (and anything related) there? You can call it from anywhere in your app without worrying about an include.
Important Note: A function doesn't have to belong to a class.
Related
I'm working on adding a feature to an existing app, to take audio input from the device microphone, convert it to frequency domain via an FFT and sends it to a coreML model. I'm using a standard AVCaptureDevice:
guard let microphone = AVCaptureDevice.default(.builtInMicrophone,
for: .audio,
position: .unspecified),
let microphoneInput = try? AVCaptureDeviceInput(device: microphone) else {
fatalError("Can't create microphone.")
}
The issue is, I require a custom sample rate to be defined for the microphone. Following Apple's documentation, setPreferredSampleRate (link) should be able to do that in a range between 8000-48000 Hz. However no matter which value I choose, the sample rate won't change, and no error is thrown:
print("Microphone sample rate: ", AVAudioSession.sharedInstance().sampleRate)
do { var flag = try AVAudioSession.sharedInstance().setPreferredSampleRate(20000) }
catch { print("Unable to set microphone sampling rate!") }
print("Microphone sample rate: ", AVAudioSession.sharedInstance().sampleRate)
Output:
Microphone sample rate: 48000.0
Microphone sample rate: 48000.0
How could I define the sampling rate for iOS devices?
EDIT:
Following the suggestion of using AVAudioConverter to resample microphone input, what's the most efficient way of doing this, considering I'm using AVCaptureAudioDataOutputSampleBufferDelegate and the corresponding captureOutput method to collect raw audio input from the microphone:
extension AudioSpectrogram: AVCaptureAudioDataOutputSampleBufferDelegate {
public func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
var audioBufferList = AudioBufferList()
var blockBuffer: CMBlockBuffer?
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
sampleBuffer,
bufferListSizeNeededOut: nil,
bufferListOut: &audioBufferList,
bufferListSize: MemoryLayout.stride(ofValue: audioBufferList),
blockBufferAllocator: nil,
blockBufferMemoryAllocator: nil,
flags: kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
blockBufferOut: &blockBuffer)
guard let data = audioBufferList.mBuffers.mData else {
return
}
if self.rawAudioData.count < self.sampleCount * 2 {
let actualSampleCount = CMSampleBufferGetNumSamples(sampleBuffer)
let ptr = data.bindMemory(to: Int16.self, capacity: actualSampleCount)
let buf = UnsafeBufferPointer(start: ptr, count: actualSampleCount)
rawAudioData.append(contentsOf: Array(buf))
}
while self.rawAudioData.count >= self.sampleCount {
let dataToProcess = Array(self.rawAudioData[0 ..< self.sampleCount])
self.rawAudioData.removeFirst(self.hopCount)
self.processData(values: dataToProcess)
}
}
I have gone through the Apple Sample Code on Equalizing Audio with vDSP, where the audio file is filtered in AVAudioSourceNode and reproduced.
My objective is to do exactly the same, but instead of taking the audio from an audio file, take it in real-time from the microphone. Is it possible to do so in AVAudioEngine? A couple of ways to do so are based on installTap or AVAudioSinkNode, as described in First strategy and Second strategy sections.
So far, I got a bit closer to my objective with the following 2 strategies.
First strategy
// Added new class variables
private lazy var sinkNode = AVAudioSinkNode { (timestep, frames, audioBufferList) -> OSStatus in
let ptr = audioBufferList.pointee.mBuffers.mData?.assumingMemoryBound(to: Float.self)
var monoSamples = [Float]()
monoSamples.append(contentsOf: UnsafeBufferPointer(start: ptr, count: Int(frames)))
self.page = monoSamples.
for frame in 0..<frames {
print("sink: " + String(monoSamples[Int(frame)]))
}
return noErr
}
// AVAudioEngine connections
engine.attach(sinkNode)
// Audio input is passed to the AVAudioSinkNode and the [Float] array is pased to the AVAudioSourceNode through the _page_ variable
engine.connect(input, to: sinkNode, format: formatt)
engine.attach(srcNode)
engine.connect(srcNode,
to: engine.mainMixerNode,
format: format)
engine.connect(engine.mainMixerNode,
to: engine.outputNode,
format: format)
// The AVAudioSourceNode access the self.page array through the getSinalElement() function.
private func getSignalElement() -> Float {
return page.isEmpty ? 0 : page.removeFirst()
}
This approach made it possible to play the audio through the AVAudioSourceNode, but, the audio stops playing after a few seconds (even though, I still successfully get the self.page array in AVAudioSourceNode) and the app finally crashes.
2 strategy
In a similar approach, I used installtap
engine.attach(srcNode)
engine.connect(srcNode,
to: engine.mainMixerNode,
format: format)
engine.connect(engine.mainMixerNode,
to: engine.outputNode,
format: format)
input.installTap(onBus: 0, bufferSize:1024, format:formatt, block: { [weak self] buffer, when in
let arraySize = Int(buffer.frameLength)
let samples = Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count:arraySize))
self!.page = samples
})
// The AVAudioSourceNode access the self.page array through the getSinalElement() function.
private func getSignalElement() -> Float {
return page.isEmpty ? 0 : page.removeFirst()
}
The outcome after implementing Second strategy is the same as in First strategy. Which can be the issues making these approaches fail?
You can use AvAudioEngine().inputNode like following:
let engine = AVAudioEngine()
private lazy var srcNode = AVAudioSourceNode { _, _, frameCount, audioBufferList -> OSStatus in
return noErr
}
// Attach First
engine.attach(srcNode)
// Then connect nodes
let input = engine.inputNode
engine.connect(input, to: srcNode, format: input.inputFormat(forBus: 0))
It is important to use input.inputFormat(...) as format type.
do{
try audioSession.setCategory(.playAndRecord, mode: .default, options: [.mixWithOthers, .defaultToSpeaker,.allowBluetoothA2DP,.allowAirPlay,.allowBluetooth])
try audioSession.setActive(true)
} catch{
print(error.localizedDescription)
}
engine.attach(player)
//Add this only you want putch
let pitch = AVAudioUnitTimePitch()
// pitch.pitch = 1000 //Filtered Voice
//pitch.rate = 1 //Normal rate
// engine.attach(pitch)
engine.attach(srcNode)
engine.connect(srcNode,
to: engine.mainMixerNode,
format: engine.inputNode.inputFormat(forBus: 0))
engine.connect(engine.mainMixerNode,
to: engine.outputNode,
format: engine.inputNode.inputFormat(forBus: 0))
engine.prepare()
engine.inputNode.installTap(onBus: 0, bufferSize: 512, format: engine.inputNode.inputFormat(forBus: 0)) { (buffer, time) -> Void in
// self.player.scheduleBuffer(buffer)
let arraySize = Int(buffer.frameLength)
let samples = Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count:arraySize))
self.page = samples
print("samples",samples)
}
engine.mainMixerNode.outputVolume = 0.5
I am doing research over four days, But I am not found any solution for calling over Bluetooth between two iOS devices within a distance.
I found that audio streaming is possible between two iOS devices using multipeer connectivity framework but this is not helpful for me. I want real time voice chat between two devices over Bluetooth.
Is there any CO-DAC for voice over Bluetooth?
My code is:
var engine = AVAudioEngine()
var file: AVAudioFile?
var player = AVAudioPlayerNode()
var input:AVAudioInputNode?
var mixer:AVAudioMixerNode?
override func viewDidLoad() {
super.viewDidLoad()
mixer = engine.mainMixerNode
input = engine.inputNode
engine.connect(input!, to: mixer!, format: input!.inputFormat(forBus: 0))
}
#IBAction func btnStremeDidClicked(_ sender: UIButton) {
mixer?.installTap(onBus: 0, bufferSize: 2048, format: mixer?.outputFormat(forBus: 0), block: { (buffer: AVAudioPCMBuffer, AVAudioTime) in
let byteWritten = self.audioBufferToData(audioBuffer: buffer).withUnsafeBytes {
self.appDelegate.mcManager.outputStream?.write($0, maxLength: self.audioBufferToData(audioBuffer: buffer).count)
}
print(byteWritten ?? 0)
print("Write")
})
do {
try engine.start()
}catch {
print(error.localizedDescription)
}
}
func audioBufferToData(audioBuffer: AVAudioPCMBuffer) -> Data {
let channelCount = 1
let bufferLength = (audioBuffer.frameCapacity * audioBuffer.format.streamDescription.pointee.mBytesPerFrame)
let channels = UnsafeBufferPointer(start: audioBuffer.floatChannelData, count: channelCount)
let data = Data(bytes: channels[0], count: Int(bufferLength))
return data
}
Thanks in Advance :)
Why is MultipeerConnectivity not helpful for you? It is a great way to stream audio over bluetooth or even wifi.
When you call this:
audioEngine.installTap(onBus: 0, bufferSize: 17640, format: localInputFormat) {
(buffer, when) -> Void in
You need to use the buffer, which has type AVAudioPCMBuffer. You then need to convert that to NSData and write to the outputStream that you would've opened with the peer:
data = someConverstionMethod(buffer)
_ = stream!.write(data.bytes.assumingMemoryBound(to: UInt8.self), maxLength: data.length)
Then on the other device you need to read from the stream and convert from NSData back to an AVAudioPCMBuffer, and then you can use an AVAudioPlayer to playback the buffer.
I have done this before with a very minimal delay.
I'm trying to make my iphone play a tune without using prerecorded files. What are my options here? AVAudioEngine, AudioKit? I've looked at them, but the learning curve is relatively steep for something I'm hoping is easy. They also seem like tools for creating sound effect given a PCM buffer window.
I'd like to be able to do something like
pitchCreator.play(["C4", "E4", "G4"], durations: [1, 1, 1])
Preferrably sounding like an instrument or at least not like a pure sine wave.
EDIT: The below code has been replaced by AudioKit
To anyone wondering this; I did make it work (kind of) using code similar to the one below.
class PitchCreator {
var engine: AVAudioEngine
var player: AVAudioPlayerNode
var mixer: AVAudioMixerNode
var buffer: AVAudioPCMBuffer
init() {
engine = AVAudioEngine()
player = AVAudioPlayerNode()
mixer = engine.mainMixerNode;
buffer = AVAudioPCMBuffer(PCMFormat: player.outputFormatForBus(0), frameCapacity: 100)
buffer.frameLength = 4096
engine.attachNode(player)
engine.connect(player, to: mixer, format: player.outputFormatForBus(0))
}
func play(frequency: Float) {
let signal = self.createSignal(frequency, amplitudes: [1.0, 0.5, 0.3, 0.1], bufferSize: Int(buffer.frameLength), sampleRate: Float(mixer.outputFormatForBus(0).sampleRate))
for i in 0 ..< signal.count {
buffer.floatChannelData.memory[i] = 0.5 * signal[i]
}
do {
try engine.start()
player.play()
player.scheduleBuffer(buffer, atTime: nil, options: .Loops, completionHandler: nil)
} catch {}
}
func stop() {
engine.stop()
player.stop()
}
func createSignal(frequency: Float, amplitudes: [Float], bufferSize: Int, sampleRate: Float) -> [Float] {
let π = Float(M_PI)
let T = sampleRate / frequency
var x = [Float](count: bufferSize, repeatedValue: 0.0)
for k in 0 ..< x.count {
for h in 0 ..< amplitudes.count {
x[k] += amplitudes[h] * sin(2.0 * π * Float(h + 1) * Float(k) / T)
}
}
return x
}
}
But it doesn't sound good enough so I've gone with sampling the notes I need and just use AVAudioPlayer instead to play them.
Background: I found one of Apple WWDC sessions called "AVAudioEngine in Practice" and am trying to make something similar to the last demo shown at 43:35 (https://youtu.be/FlMaxen2eyw?t=2614). I'm using SpriteKit instead of SceneKit but the principle is the same: I want to generate spheres, throw them around and when they collide the engine plays a sound, unique to each sphere.
Problems:
I want a unique AudioPlayerNode attached to each SpriteKitNode so that I can play a different sound for each sphere. i.e Right now, if I create two spheres and set a different pitch for each of their AudioPlayerNode, only the most recently created AudioPlayerNode seems to be playing, even when the original sphere collides. During the demo, he mentions "I'm tying a player, a dedicated player to each ball". How would I go about doing that?
There are audio clicks/artefacts every time a new collision happens. I'm assuming this has to do with the AVAudioPlayerNodeBufferOptions and/or the fact that I'm trying to create, schedule and consume buffers very quickly each time contact occurs, which is not the most efficient method. What would be a good work around for this?
Code: As mentioned in the video, "...for every ball that's born into this world, a new player node is also created". I have a separate class for the spheres, with a method that returns a SpriteKitNode and also creates an AudioPlayerNode every time it is called :
class Sphere {
var sphere: SKSpriteNode = SKSpriteNode(color: UIColor(), size: CGSize())
var sphereScale: CGFloat = CGFloat(0.01)
var spherePlayer = AVAudioPlayerNode()
let audio = Audio()
let sphereCollision: UInt32 = 0x1 << 0
func createSphere(position: CGPoint, pitch: Float) -> SKSpriteNode {
let texture = SKTexture(imageNamed: "Slice")
let collisionTexture = SKTexture(imageNamed: "Collision")
// Define the node
sphere = SKSpriteNode(texture: texture, size: texture.size())
sphere.position = position
sphere.name = "sphere"
sphere.physicsBody = SKPhysicsBody(texture: collisionTexture, size: sphere.size)
sphere.physicsBody?.dynamic = true
sphere.physicsBody?.mass = 0
sphere.physicsBody?.restitution = 0.5
sphere.physicsBody?.usesPreciseCollisionDetection = true
sphere.physicsBody?.categoryBitMask = sphereCollision
sphere.physicsBody?.contactTestBitMask = sphereCollision
sphere.zPosition = 1
// Create AudioPlayerNode
spherePlayer = audio.createPlayer(pitch)
return sphere
}
Here's my Audio Class with which I create AudioPCMBuffers and AudioPlayerNodes
class Audio {
let engine: AVAudioEngine = AVAudioEngine()
func createBuffer(name: String, type: String) -> AVAudioPCMBuffer {
let audioFilePath = NSBundle.mainBundle().URLForResource(name as String, withExtension: type as String)!
let audioFile = try! AVAudioFile(forReading: audioFilePath)
let buffer = AVAudioPCMBuffer(PCMFormat: audioFile.processingFormat, frameCapacity: UInt32(audioFile.length))
try! audioFile.readIntoBuffer(buffer)
return buffer
}
func createPlayer(pitch: Float) -> AVAudioPlayerNode {
let player = AVAudioPlayerNode()
let buffer = self.createBuffer("PianoC1", type: "wav")
let pitcher = AVAudioUnitTimePitch()
let delay = AVAudioUnitDelay()
pitcher.pitch = pitch
delay.delayTime = 0.2
delay.feedback = 90
delay.wetDryMix = 0
engine.attachNode(pitcher)
engine.attachNode(player)
engine.attachNode(delay)
engine.connect(player, to: pitcher, format: buffer.format)
engine.connect(pitcher, to: delay, format: buffer.format)
engine.connect(delay, to: engine.mainMixerNode, format: buffer.format)
engine.prepare()
try! engine.start()
return player
}
}
In my GameScene class I then test for collision, schedule a buffer and play the AudioPlayerNode if contact has occurred
func didBeginContact(contact: SKPhysicsContact) {
let firstBody: SKPhysicsBody = contact.bodyA
if (firstBody.categoryBitMask & sphere.sphereCollision != 0) {
let buffer1 = audio.createBuffer("PianoC1", type: "wav")
sphere.spherePlayer.scheduleBuffer(buffer1, atTime: nil, options: AVAudioPlayerNodeBufferOptions.Interrupts, completionHandler: nil)
sphere.spherePlayer.play()
}
}
I'm new to Swift and only have basic knowledge of programming so any suggestion/criticism is welcome.
I've been working on AVAudioEngine in scenekit and trying to do something else, but this will be what you are looking for:
https://developer.apple.com/library/mac/samplecode/AVAEGamingExample/Listings/AVAEGamingExample_AudioEngine_m.html
It explains the process of:
1-Instantiating your own AVAudioEngine sub-class
2-Methods to load PCMBuffers for each AVAudioPlayer
3-Changing your Environment node's parameters to accomodate the reverb for the large number of pinball objects
Edit: Converted, tested and added a few features:
1-You create a subclass of AVAudioEngine, name it AudioLayerEngine for example. This is to access the AVAudioUnit effects such as distortion, delay, pitch and many of the other effects available as AudioUnits.
2-Initialise by setting up some configurations for the audio engine, such as rendering algorithm, exposing the AVAudioEnvironmentNode to play with 3D positions of your SCNNode objects or SKNode objects if you are in 2D but want 3D effects
3-Create some helper methods to load presets for each AudioUnit effect you want
4-Create a helper method to create an audio player then add it to whatever node you want, as many times as you want since that SCNNode accepts a .audioPlayers methods which returns [AVAudioPlayer] or [SCNAudioPlayer]
5-Start playing.
I've pasted the entire class for reference so that you can then structure it as you wish, but keep in mind that if you are coupling this with SceneKit or SpriteKit, you use this audioEngine to manage all your sounds instead of SceneKit's internal AVAudioEngine. This means that you instantiate this in your gameView during the AwakeFromNib method
import Foundation
import SceneKit
import AVFoundation
class AudioLayerEngine:AVAudioEngine{
var engine:AVAudioEngine!
var environment:AVAudioEnvironmentNode!
var outputBuffer:AVAudioPCMBuffer!
var voicePlayer:AVAudioPlayerNode!
var multiChannelEnabled:Bool!
//audio effects
let delay = AVAudioUnitDelay()
let distortion = AVAudioUnitDistortion()
let reverb = AVAudioUnitReverb()
override init(){
super.init()
engine = AVAudioEngine()
environment = AVAudioEnvironmentNode()
engine.attachNode(self.environment)
voicePlayer = AVAudioPlayerNode()
engine.attachNode(voicePlayer)
voicePlayer.volume = 1.0
outputBuffer = loadVoice()
wireEngine()
startEngine()
voicePlayer.scheduleBuffer(self.outputBuffer, completionHandler: nil)
voicePlayer.play()
}
func startEngine(){
do{
try engine.start()
}catch{
print("error loading engine")
}
}
func loadVoice()->AVAudioPCMBuffer{
let URL = NSURL(fileURLWithPath: NSBundle.mainBundle().pathForResource("art.scnassets/sounds/interface/test", ofType: "aiff")!)
do{
let soundFile = try AVAudioFile(forReading: URL, commonFormat: AVAudioCommonFormat.PCMFormatFloat32, interleaved: false)
outputBuffer = AVAudioPCMBuffer(PCMFormat: soundFile.processingFormat, frameCapacity: AVAudioFrameCount(soundFile.length))
do{
try soundFile.readIntoBuffer(outputBuffer)
}catch{
print("somethign went wrong with loading the buffer into the sound fiel")
}
print("returning buffer")
return outputBuffer
}catch{
}
return outputBuffer
}
func wireEngine(){
loadDistortionPreset(AVAudioUnitDistortionPreset.MultiCellphoneConcert)
engine.attachNode(distortion)
engine.attachNode(delay)
engine.connect(voicePlayer, to: distortion, format: self.outputBuffer.format)
engine.connect(distortion, to: delay, format: self.outputBuffer.format)
engine.connect(delay, to: environment, format: self.outputBuffer.format)
engine.connect(environment, to: engine.outputNode, format: constructOutputFormatForEnvironment())
}
func constructOutputFormatForEnvironment()->AVAudioFormat{
let outputChannelCount = self.engine.outputNode.outputFormatForBus(1).channelCount
let hardwareSampleRate = self.engine.outputNode.outputFormatForBus(1).sampleRate
let environmentOutputConnectionFormat = AVAudioFormat(standardFormatWithSampleRate: hardwareSampleRate, channels: outputChannelCount)
multiChannelEnabled = false
return environmentOutputConnectionFormat
}
func loadDistortionPreset(preset: AVAudioUnitDistortionPreset){
distortion.loadFactoryPreset(preset)
}
func createPlayer(node: SCNNode){
let player = AVAudioPlayerNode()
distortion.loadFactoryPreset(AVAudioUnitDistortionPreset.SpeechCosmicInterference)
engine.attachNode(player)
engine.attachNode(distortion)
engine.connect(player, to: distortion, format: outputBuffer.format)
engine.connect(distortion, to: environment, format: constructOutputFormatForEnvironment())
let algo = AVAudio3DMixingRenderingAlgorithm.HRTF
player.renderingAlgorithm = algo
player.reverbBlend = 0.3
player.renderingAlgorithm = AVAudio3DMixingRenderingAlgorithm.HRTF
}
}