Playing audio from microphone - ios

Question

Goal: To stream audio/video from one device to another.
Problem: I managed to get both audio and video but the audio won't play on the other side.
Details:
I have created an app that will transmit A/V data from one device to another over the network. To not go into too much detail I will show you where I am stuck. I managed to listen to the output delegate, where I extract the audio information, convert it into Data and pass it on to a delegate that I've created.
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
// VIDEO | code excluded for simplicity of this question as this part works
// AUDIO | only deliver the frames if you are allowed to
if self.produceAudioFrames == true {
// process the audio buffer
let _audioFrame = self.audioFromSampleBuffer(sampleBuffer)
// process in async
DispatchQueue.main.async {
// pass the audio frame to the delegate
self.delegate?.audioFrame(data: _audioFrame)
}
}
}
The helper func that converts the SampleBuffer (not my code, can't find source. I know found it here on SO) :
func audioFromSampleBuffer(_ sampleBuffer: CMSampleBuffer) -> Data {
var audioBufferList = AudioBufferList()
var data = Data()
var blockBuffer : CMBlockBuffer?
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer,
nil,
&audioBufferList,
MemoryLayout<AudioBufferList>.size,
nil,
nil,
0,
&blockBuffer)
let buffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers,
count: Int(audioBufferList.mNumberBuffers))
for audioBuffer in buffers {
let frame = audioBuffer.mData?.assumingMemoryBound(to: UInt8.self)
data.append(frame!, count: Int(audioBuffer.mDataByteSize))
}
// dev
//print("audio buffer count: \(buffers.count)") | this returns 2048
// give the raw data back to the caller
return data
}
Note: Before sending over the network, I convert the data returned from the helper func like so: let payload = Array(data)
That is the host's side.
On the client side I am receiving the payload as [UInt8] and this where I am stuck. I tried multiple things but none worked.
func processIncomingAudioPayloadFromFrame(_ ID: String, _ _Data: [UInt8]) {
let readableData = Data(bytes: _Data) // back from array to the data before we sent it over the network.
print(readableData.count) // still 2048 even after recieving from network, So I am guessing data is still intact
let x = self.bytesToAudioBuffer(_Data) // option two convert into a AVAudioPCMBuffer
print(x) // prints | <AVAudioPCMBuffer#0x600000201e80: 2048/2048 bytes> | I am guessing it works
// option one | play using AVAudioPlayer
do {
let player = try AVAudioPlayer(data: readableData)
try AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryPlayback)
try AVAudioSession.sharedInstance().setActive(true)
player.prepareToPlay()
player.play()
print(player.volume) // doing this to see if this is reached
}catch{
print(error) // gets error | Error Domain=NSOSStatusErrorDomain Code=1954115647 "(null)"
}
}
Here is the helper func that converts [UInt8] into AVAudioPCMBuffer:
func bytesToAudioBuffer(_ buf: [UInt8]) -> AVAudioPCMBuffer {
// format assumption! make this part of your protocol?
let fmt = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100,
channels: 1, interleaved: true)
let frameLength = UInt32(buf.count) / fmt.streamDescription.pointee.mBytesPerFrame
let audioBuffer = AVAudioPCMBuffer(pcmFormat: fmt, frameCapacity: frameLength)
audioBuffer.frameLength = frameLength
let dstLeft = audioBuffer.floatChannelData![0]
// for stereo
// let dstRight = audioBuffer.floatChannelData![1]
buf.withUnsafeBufferPointer {
let src = UnsafeRawPointer($0.baseAddress!).
bindMemory(to: Float.self, capacity: Int(frameLength))
dstLeft.initialize(from: src, count: Int(frameLength))
}
return audioBuffer
}
Questions:
Is it possible to even play directly from [UInt8]?
How can I play the AVAudioPCMBuffer payload using the AudioEngine?
Is it possible?
How can I play the audio on the client side.
Footnote: The comments in the code should give you some hint for the output I hope. Also I don't want to save to a file or anything file related as I just want to amplify the mic for real-time listening, I have no interest in saving the data.

Answer 1

I have used the same code, for playing audio file on Carrier call.
Please try and let me know the results :
Objective Code :
NSString *soundFilePath = [[NSBundle mainBundle]
pathForResource:self.bgAudioFileName ofType: #"mp3"];
NSURL *fileURL = [[NSURL alloc] initFileURLWithPath:soundFilePath ];
myAudioPlayer = [[AVAudioPlayer alloc] initWithContentsOfURL:fileURL
error:nil];
myAudioPlayer.numberOfLoops = -1;
NSError *sessionError = nil;
// Change the default output audio route
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
// get your audio session somehow
[audioSession setCategory:AVAudioSessionCategoryMultiRoute
error:&sessionError];
BOOL success= [audioSession
overrideOutputAudioPort:AVAudioSessionPortOverrideNone
error:&sessionError];
[audioSession setActive:YES error:&sessionError];
if(!success)
{
NSLog(#"error doing outputaudioportoverride - %#", [sessionError
localizedDescription]);
}
[myAudioPlayer setVolume:1.0f];
[myAudioPlayer play];
Swift version :
var soundFilePath: String? = Bundle.main.path(forResource:
bgAudioFileName, ofType: "mp3")
var fileURL = URL(fileURLWithPath: soundFilePath ?? "")
myAudioPlayer = try? AVAudioPlayer(contentsOf: fileURL)
myAudioPlayer.numberOfLoops = -1
var sessionError: Error? = nil
// Change the default output audio route
var audioSession = AVAudioSession.sharedInstance()
// get your audio session somehow
try? audioSession.setCategory(AVAudioSessionCategoryMultiRoute)
var success: Bool? = try?
audioSession.overrideOutputAudioPort(AVAudioSessionPortOverrideNone
as? AVAudioSessionPortOverride ?? AVAudioSessionPortOverride())
try? audioSession.setActive(true)
if !(success ?? false) {
print("error doing outputaudioportoverride - \
(sessionError?.localizedDescription)")
}
myAudioPlayer.volume = 1.0
myAudioPlayer.play()