Mute Audio AVAssetWriterInput while recording - ios

I am recording video and audio using an AVAssetWriter to append CMSampleBuffer from AVCaptureVideoDataOutput and AVCaptureAudioDataOutput respectively. What I want to do is at the user discretion mute the audio during the recording.
I assuming the best way is to some how create an empty CMSampleBuffer like
CMSampleBufferRef sb;
CMSampleBufferCreate(kCFAllocatorDefault, NULL, YES, NULL, NULL, NULL, 0, 1, &sti, 0, NULL, &sb);
[_audioInputWriter appendSampleBuffer:sb];
CFRelease(sb);
but that doesn't work, so I am assuming that I need to create a silent audio buffer. How do I do this and is there a better way?

I have done this before by calling a function that processes the data in the SampleBuffer and zeros all of it. Might need to modify this if your audio format is not using an SInt16 sample size.
You can also use this same technique to process the audio in other ways.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
if(isMute){
[self muteAudioInBuffer:sampleBuffer];
}
}
- (void) muteAudioInBuffer:(CMSampleBufferRef)sampleBuffer
{
CMItemCount numSamples = CMSampleBufferGetNumSamples(sampleBuffer);
NSUInteger channelIndex = 0;
CMBlockBufferRef audioBlockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t audioBlockBufferOffset = (channelIndex * numSamples * sizeof(SInt16));
size_t lengthAtOffset = 0;
size_t totalLength = 0;
SInt16 *samples = NULL;
CMBlockBufferGetDataPointer(audioBlockBuffer, audioBlockBufferOffset, &lengthAtOffset, &totalLength, (char **)(&samples));
for (NSInteger i=0; i<numSamples; i++) {
samples[i] = (SInt16)0;
}
}

Related

Video Capture Compressed with VTCompressionSession plays too fast

I am receiving raw RGBA data from a AVCaptureVideoDataOutput and using VTCompressionSession to compress it to a raw H264 stream.
The problem I have is that the resulting stream plays too fast (playing in VLC), about 3x the real speed.
I am using the presentation times and durations from the captured data. Using AVFileMovieOutput works correctly, but I want more control over the compression.
I have tried setting kVTCompressionPropertyKey_ExpectedFrameRate but that makes no difference.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
CMTime presentationTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
CMTime duration = CMSampleBufferGetDuration(sampleBuffer);
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
OSStatus encodeStatus = VTCompressionSessionEncodeFrame(compressionSession, pixelBuffer, presentationTime, duration, NULL, NULL, NULL);
if (encodeStatus != noErr) {
NSLog(#"Encode error.");
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
}
I'm:
CFAbsoluteTime currentTime = CFAbsoluteTimeGetCurrent();
CMTime presentationTimeStamp = CMTimeMake(currentTime*27000000, 27000000);
VTCompressionSessionEncodeFrame(_enc_session, imageBuffer, presentationTimeStamp, kCMTimeInvalid, NULL, NULL, NULL);
Also. How do you init your compression session? What 'k' parameters do you set to what?

Recording audio and passing the data to a UIWebView (JavascriptCore) on iOS 8/9

We have an app that is mostly a UIWebView for a heavily javascript based web app. The requirement we have come up against is being able to play audio to the user and then record the user, play back that recording for confirmation and then send the audio to a server. This works in Chrome, Android and other platforms because that ability is built into the browser. No native code required.
Sadly, the iOS (iOS 8/9) web view lacks the ability to record audio.
The first workaround we tried was recording the audio with an AudioQueue and passing the data (LinearPCM 16bit) to a JS AudioNode so the web app could process the iOS audio exactly the same way as other platforms. This got to a point where we could pass the audio to JS, but the app would eventually crash with a bad memory access error or the javascript side just could not keep up with the data being sent.
The next idea was to save the audio recording to a file and send partial audio data to JS for visual feedback, a basic audio visualizer displayed during recording only.
The audio records and plays back fine to a WAVE file as Linear PCM signed 16bit. The JS visualizer is where we are stuck. It is expecting Linear PCM unsigned 8bit so I added a conversion step that may be wrong. I've tried several different ways, mostly found online, and have not found one that works which makes me think something else is wrong or missing before we even get to the conversion step.
Since I don't know what or where exactly the problem is I'll dump the code below for the audio recording and playback classes. Any suggestions would be welcome to resolve, or bypass somehow, this issue.
One idea I had was to record in a different format (CAF) using different format flags. Looking at the values that are produced, non of the signed 16bit ints come even close to the max value. I rarely see anything above +/-1000. Is that because of the kLinearPCMFormatFlagIsPacked flag in the AudioStreamPacketDescription? Removing that flag cuases the audio file to not be created because of an invalid format. Maybe switching to CAF would work but we need to convert to WAVE before sending the audio back to our server.
Or maybe my conversion from signed 16bit to unsigned 8bit is wrong? I have also tried bitshifting and casting. The only difference is, with this conversion all the audio values get compressed to between 125 and 130. Bit shifting and casting change that to 0-5 and 250-255. That doesn't really solve any problems on the JS side.
The next step would be, instead of passing the data to JS run it through a FFT function and produce values to be used directly by JS for the audio visualizer. I'd rather figure out if I have done something obviously wrong before going that direction.
AQRecorder.h - EDIT: updated audio format to LinearPCM 32bit Float.
#ifndef AQRecorder_h
#define AQRecorder_h
#import <AudioToolbox/AudioToolbox.h>
#define NUM_BUFFERS 3
#define AUDIO_DATA_TYPE_FORMAT float
#define JS_AUDIO_DATA_SIZE 32
#interface AQRecorder : NSObject {
AudioStreamBasicDescription mDataFormat;
AudioQueueRef mQueue;
AudioQueueBufferRef mBuffers[ NUM_BUFFERS ];
AudioFileID mAudioFile;
UInt32 bufferByteSize;
SInt64 mCurrentPacket;
bool mIsRunning;
}
- (void)setupAudioFormat;
- (void)startRecording;
- (void)stopRecording;
- (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData;
- (Boolean)isRunning;
#end
#endif
AQRecorder.m - EDIT: updated audio format to LinearPCM 32bit Float. Added FFT step in processSamplesForJS instead of sending audio data directly.
#import <AVFoundation/AVFoundation.h>
#import "AQRecorder.h"
#import "JSMonitor.h"
#implementation AQRecorder
void AudioQueueCallback(void * inUserData,
AudioQueueRef inAQ,
AudioQueueBufferRef inBuffer,
const AudioTimeStamp * inStartTime,
UInt32 inNumberPacketDescriptions,
const AudioStreamPacketDescription* inPacketDescs)
{
AQRecorder *aqr = (__bridge AQRecorder *)inUserData;
if ( [aqr isRunning] )
{
if ( inNumberPacketDescriptions > 0 )
{
AudioFileWritePackets(aqr->mAudioFile, FALSE, inBuffer->mAudioDataByteSize, inPacketDescs, aqr->mCurrentPacket, &inNumberPacketDescriptions, inBuffer->mAudioData);
aqr->mCurrentPacket += inNumberPacketDescriptions;
[aqr processSamplesForJS:inBuffer->mAudioDataBytesCapacity audioData:inBuffer->mAudioData];
}
AudioQueueEnqueueBuffer(inAQ, inBuffer, 0, NULL);
}
}
- (void)debugDataFormat
{
NSLog(#"format=%i, sampleRate=%f, channels=%i, flags=%i, BPC=%i, BPF=%i", mDataFormat.mFormatID, mDataFormat.mSampleRate, (unsigned int)mDataFormat.mChannelsPerFrame, mDataFormat.mFormatFlags, mDataFormat.mBitsPerChannel, mDataFormat.mBytesPerFrame);
}
- (void)setupAudioFormat
{
memset(&mDataFormat, 0, sizeof(mDataFormat));
mDataFormat.mSampleRate = 44100.;
mDataFormat.mChannelsPerFrame = 1;
mDataFormat.mFormatID = kAudioFormatLinearPCM;
mDataFormat.mFormatFlags = kLinearPCMFormatFlagIsFloat | kLinearPCMFormatFlagIsPacked;
int sampleSize = sizeof(AUDIO_DATA_TYPE_FORMAT);
mDataFormat.mBitsPerChannel = 32;
mDataFormat.mBytesPerPacket = mDataFormat.mBytesPerFrame = (mDataFormat.mBitsPerChannel / 8) * mDataFormat.mChannelsPerFrame;
mDataFormat.mFramesPerPacket = 1;
mDataFormat.mReserved = 0;
[self debugDataFormat];
}
- (void)startRecording/
{
[self setupAudioFormat];
mCurrentPacket = 0;
NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: #"AudioFile.wav"];
CFURLRef url = CFURLCreateWithString(kCFAllocatorDefault, (CFStringRef)recordFile, NULL);;
OSStatus *stat =
AudioFileCreateWithURL(url, kAudioFileWAVEType, &mDataFormat, kAudioFileFlags_EraseFile, &mAudioFile);
NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:stat userInfo:nil];
NSLog(#"AudioFileCreateWithURL OSStatus :: %#", error);
CFRelease(url);
bufferByteSize = 896 * mDataFormat.mBytesPerFrame;
AudioQueueNewInput(&mDataFormat, AudioQueueCallback, (__bridge void *)(self), NULL, NULL, 0, &mQueue);
for ( int i = 0; i < NUM_BUFFERS; i++ )
{
AudioQueueAllocateBuffer(mQueue, bufferByteSize, &mBuffers[i]);
AudioQueueEnqueueBuffer(mQueue, mBuffers[i], 0, NULL);
}
mIsRunning = true;
AudioQueueStart(mQueue, NULL);
}
- (void)stopRecording
{
mIsRunning = false;
AudioQueueStop(mQueue, false);
AudioQueueDispose(mQueue, false);
AudioFileClose(mAudioFile);
}
- (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData
{
int sampleCount = audioDataBytesCapacity / sizeof(AUDIO_DATA_TYPE_FORMAT);
AUDIO_DATA_TYPE_FORMAT *samples = (AUDIO_DATA_TYPE_FORMAT*)audioData;
NSMutableArray *audioDataBuffer = [[NSMutableArray alloc] initWithCapacity:JS_AUDIO_DATA_SIZE];
// FFT stuff taken mostly from Apples aurioTouch example
const Float32 kAdjust0DB = 1.5849e-13;
int bufferFrames = sampleCount;
int bufferlog2 = round(log2(bufferFrames));
float fftNormFactor = (1.0/(2*bufferFrames));
FFTSetup fftSetup = vDSP_create_fftsetup(bufferlog2, kFFTRadix2);
Float32 *outReal = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));
Float32 *outImaginary = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));
COMPLEX_SPLIT mDspSplitComplex = { .realp = outReal, .imagp = outImaginary };
Float32 *outFFTData = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));
//Generate a split complex vector from the real data
vDSP_ctoz((COMPLEX *)samples, 2, &mDspSplitComplex, 1, bufferFrames / 2);
//Take the fft and scale appropriately
vDSP_fft_zrip(fftSetup, &mDspSplitComplex, 1, bufferlog2, kFFTDirection_Forward);
vDSP_vsmul(mDspSplitComplex.realp, 1, &fftNormFactor, mDspSplitComplex.realp, 1, bufferFrames / 2);
vDSP_vsmul(mDspSplitComplex.imagp, 1, &fftNormFactor, mDspSplitComplex.imagp, 1, bufferFrames / 2);
//Zero out the nyquist value
mDspSplitComplex.imagp[0] = 0.0;
//Convert the fft data to dB
vDSP_zvmags(&mDspSplitComplex, 1, outFFTData, 1, bufferFrames / 2);
//In order to avoid taking log10 of zero, an adjusting factor is added in to make the minimum value equal -128dB
vDSP_vsadd(outFFTData, 1, &kAdjust0DB, outFFTData, 1, bufferFrames / 2);
Float32 one = 1;
vDSP_vdbcon(outFFTData, 1, &one, outFFTData, 1, bufferFrames / 2, 0);
// Average out FFT dB values
int grpSize = (bufferFrames / 2) / 32;
int c = 1;
Float32 avg = 0;
int d = 1;
for ( int i = 1; i < bufferFrames / 2; i++ )
{
if ( outFFTData[ i ] != outFFTData[ i ] || outFFTData[ i ] == INFINITY )
{ // NAN / INFINITE check
c++;
}
else
{
avg += outFFTData[ i ];
d++;
//NSLog(#"db = %f, avg = %f", outFFTData[ i ], avg);
if ( ++c >= grpSize )
{
uint8_t u = (uint8_t)((avg / d) + 128); //dB values seem to range from -128 to 0.
NSLog(#"%i = %i (%f)", i, u, avg);
[audioDataBuffer addObject:[NSNumber numberWithUnsignedInt:u]];
avg = 0;
c = 0;
d = 1;
}
}
}
[[JSMonitor shared] passAudioDataToJavascriptBridge:audioDataBuffer];
}
- (Boolean)isRunning
{
return mIsRunning;
}
#end
Audio playback and recording contrller classes
Audio.h
#ifndef Audio_h
#define Audio_h
#import <AVFoundation/AVFoundation.h>
#import "AQRecorder.h"
#interface Audio : NSObject <AVAudioPlayerDelegate> {
AQRecorder* recorder;
AVAudioPlayer* player;
bool mIsSetup;
bool mIsRecording;
bool mIsPlaying;
}
- (void)setupAudio;
- (void)startRecording;
- (void)stopRecording;
- (void)startPlaying;
- (void)stopPlaying;
- (Boolean)isRecording;
- (Boolean)isPlaying;
- (NSString *) getAudioDataBase64String;
#end
#endif
Audio.m
#import "Audio.h"
#import <AudioToolbox/AudioToolbox.h>
#import "JSMonitor.h"
#implementation Audio
- (void)setupAudio
{
NSLog(#"Audio->setupAudio");
AVAudioSession *session = [AVAudioSession sharedInstance];
NSError * error;
[session setCategory:AVAudioSessionCategoryPlayAndRecord error:&error];
[session setActive:YES error:nil];
recorder = [[AQRecorder alloc] init];
mIsSetup = YES;
}
- (void)startRecording
{
NSLog(#"Audio->startRecording");
if ( !mIsSetup )
{
[self setupAudio];
}
if ( mIsRecording ) {
return;
}
if ( [recorder isRunning] == NO )
{
[recorder startRecording];
}
mIsRecording = [recorder isRunning];
}
- (void)stopRecording
{
NSLog(#"Audio->stopRecording");
[recorder stopRecording];
mIsRecording = [recorder isRunning];
[[JSMonitor shared] sendAudioInputStoppedEvent];
}
- (void)startPlaying
{
if ( mIsPlaying )
{
return;
}
mIsPlaying = YES;
NSLog(#"Audio->startPlaying");
NSError* error = nil;
NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: #"AudioFile.wav"];
player = [[AVAudioPlayer alloc] initWithContentsOfURL:[NSURL fileURLWithPath:recordFile] error:&error];
if ( error )
{
NSLog(#"AVAudioPlayer failed :: %#", error);
}
player.delegate = self;
[player play];
}
- (void)stopPlaying
{
NSLog(#"Audio->stopPlaying");
[player stop];
mIsPlaying = NO;
[[JSMonitor shared] sendAudioPlaybackCompleteEvent];
}
- (NSString *) getAudioDataBase64String
{
NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: #"AudioFile.wav"];
NSError* error = nil;
NSData *fileData = [NSData dataWithContentsOfFile:recordFile options: 0 error: &error];
if ( fileData == nil )
{
NSLog(#"Failed to read file, error %#", error);
return #"DATAENCODINGFAILED";
}
else
{
return [fileData base64EncodedStringWithOptions:0];
}
}
- (Boolean)isRecording { return mIsRecording; }
- (Boolean)isPlaying { return mIsPlaying; }
- (void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag
{
NSLog(#"Audio->audioPlayerDidFinishPlaying: %i", flag);
mIsPlaying = NO;
[[JSMonitor shared] sendAudioPlaybackCompleteEvent];
}
- (void)audioPlayerDecodeErrorDidOccur:(AVAudioPlayer *)player error:(NSError *)error
{
NSLog(#"Audio->audioPlayerDecodeErrorDidOccur: %#", error.localizedFailureReason);
mIsPlaying = NO;
[[JSMonitor shared] sendAudioPlaybackCompleteEvent];
}
#end
The JSMonitor class is a bridge between the UIWebView javascriptcore and the native code. I'm not including it because it doesn't do anything for audio other than pass data / calls between these classes and JSCore.
EDIT
The format of the audio has changed to LinearPCM Float 32bit. Instead of sending the audio data it is sent through an FFT function and the dB values are averaged and sent instead.
Core Audio is a pain to work with. Fortunately, AVFoundation provides AVAudioRecorder to record video and also gives you access to the average and peak audio power that you can send to back to your JavaScript to update your UI visualizer. From the docs:
An instance of the AVAudioRecorder class, called an audio recorder,
provides audio recording capability in your application. Using an
audio recorder you can:
Record until the user stops the recording
Record for a specified duration
Pause and resume a recording
Obtain input audio-level data that you can use to provide level
metering
This Stack Overflow question has an example of how to use AVAudioRecorder.

Can not play audio in didReceiveData method of Multipeer Connectivity

My aim is to stream voice data to multiple devices using multipeer connnectivity.
I am using AVCaptureSession to access voice data from microphone using AVCaptureDevice type AVMediaTypeAudio.
In a custom AVCaptureAudioDataOutput class i am getting those audio data and want to stream that to all connected peers.
//sending data using Multipeer Connectivity
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
// NSLog(#"---A U D I O :%#",sampleBuffer);
AudioBufferList audioBufferList;
NSMutableData *data= [NSMutableData data];
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
for( int y=0; y< audioBufferList.mNumberBuffers; y++ ){
AudioBuffer audioBuffer = audioBufferList.mBuffers[y];
Float32 *frame = (Float32*)audioBuffer.mData;
[data appendBytes:frame length:audioBuffer.mDataByteSize];
}
CFRelease(blockBuffer);
[_session sendData:data toPeers:_session.connectedPeers withMode:MCSessionSendDataReliable error:nil];
}
In the Receiver application I am getting all nsdata in didRecievedData delegate method in MCSessionDelegate.
But I am not getting any way to play that raw NSData in Reciever application.

Playing Audio on iOS from Socket connection

Hope you can help me with this issue, I have seen a lot of questions related to this, but none of them really helps me to figure out what I am doing wrong here.
So on Android I have an AudioRecord which is recording audio and sending the audio as byte array over a socket connection to clients. This part was super easy on Android and is working perfectly.
When I started working with iOS I found out there is no easy way to go about this, so after 2 days of research and plugging and playing this is what I have got. Which still does not play any audio. It makes a noise when it starts but none of the audio being transferred over the socket is being played. I confirmed that the socket is receiving data by logging each element in the buffer array.
Here is all the code I am using, a lot is reused from a bunch of sites, can't remember all the links. (BTW using AudioUnits)
First up, audio processor:
Play Callback
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
/**
This is the reference to the object who owns the callback.
*/
AudioProcessor *audioProcessor = (__bridge AudioProcessor*) inRefCon;
// iterate over incoming stream an copy to output stream
for (int i=0; i < ioData->mNumberBuffers; i++) {
AudioBuffer buffer = ioData->mBuffers[i];
// find minimum size
UInt32 size = min(buffer.mDataByteSize, [audioProcessor audioBuffer].mDataByteSize);
// copy buffer to audio buffer which gets played after function return
memcpy(buffer.mData, [audioProcessor audioBuffer].mData, size);
// set data size
buffer.mDataByteSize = size;
}
return noErr;
}
Audio processor initialize
-(void)initializeAudio
{
OSStatus status;
// We define the audio component
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output; // we want to ouput
desc.componentSubType = kAudioUnitSubType_RemoteIO; // we want in and ouput
desc.componentFlags = 0; // must be zero
desc.componentFlagsMask = 0; // must be zero
desc.componentManufacturer = kAudioUnitManufacturer_Apple; // select provider
// find the AU component by description
AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);
// create audio unit by component
status = AudioComponentInstanceNew(inputComponent, &audioUnit);
[self hasError:status:__FILE__:__LINE__];
// define that we want record io on the input bus
UInt32 flag = 1;
// define that we want play on io on the output bus
status = AudioUnitSetProperty(audioUnit,
kAudioOutputUnitProperty_EnableIO, // use io
kAudioUnitScope_Output, // scope to output
kOutputBus, // select output bus (0)
&flag, // set flag
sizeof(flag));
[self hasError:status:__FILE__:__LINE__];
/*
We need to specifie our format on which we want to work.
We use Linear PCM cause its uncompressed and we work on raw data.
for more informations check.
We want 16 bits, 2 bytes per packet/frames at 44khz
*/
AudioStreamBasicDescription audioFormat;
audioFormat.mSampleRate = SAMPLE_RATE;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 1;
audioFormat.mBitsPerChannel = 16;
audioFormat.mBytesPerPacket = 2;
audioFormat.mBytesPerFrame = 2;
// set the format on the output stream
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
kInputBus,
&audioFormat,
sizeof(audioFormat));
[self hasError:status:__FILE__:__LINE__];
/**
We need to define a callback structure which holds
a pointer to the recordingCallback and a reference to
the audio processor object
*/
AURenderCallbackStruct callbackStruct;
/*
We do the same on the output stream to hear what is coming
from the input stream
*/
callbackStruct.inputProc = playbackCallback;
callbackStruct.inputProcRefCon = (__bridge void *)(self);
// set playbackCallback as callback on our renderer for the output bus
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Global,
kOutputBus,
&callbackStruct,
sizeof(callbackStruct));
[self hasError:status:__FILE__:__LINE__];
// reset flag to 0
flag = 0;
/*
we need to tell the audio unit to allocate the render buffer,
that we can directly write into it.
*/
status = AudioUnitSetProperty(audioUnit,
kAudioUnitProperty_ShouldAllocateBuffer,
kAudioUnitScope_Output,
kInputBus,
&flag,
sizeof(flag));
/*
we set the number of channels to mono and allocate our block size to
1024 bytes.
*/
audioBuffer.mNumberChannels = 1;
audioBuffer.mDataByteSize = 512 * 2;
audioBuffer.mData = malloc( 512 * 2 );
// Initialize the Audio Unit and cross fingers =)
status = AudioUnitInitialize(audioUnit);
[self hasError:status:__FILE__:__LINE__];
NSLog(#"Started");
}
Start Playing
-(void)start;
{
// start the audio unit. You should hear something, hopefully <img src="http://www.stefanpopp.de/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley">
OSStatus status = AudioOutputUnitStart(audioUnit);
[self hasError:status:__FILE__:__LINE__];
}
Adding data to the buffer
-(void)processBuffer: (AudioBufferList*) audioBufferList
{
AudioBuffer sourceBuffer = audioBufferList->mBuffers[0];
// we check here if the input data byte size has changed
if (audioBuffer.mDataByteSize != sourceBuffer.mDataByteSize) {
// clear old buffer
free(audioBuffer.mData);
// assing new byte size and allocate them on mData
audioBuffer.mDataByteSize = sourceBuffer.mDataByteSize;
audioBuffer.mData = malloc(sourceBuffer.mDataByteSize);
}
// loop over every packet
// copy incoming audio data to the audio buffer
memcpy(audioBuffer.mData, audioBufferList->mBuffers[0].mData, audioBufferList->mBuffers[0].mDataByteSize);
}
Stream connection callback (Socket)
-(void)stream:(NSStream *)aStream handleEvent:(NSStreamEvent)eventCode
{
if(eventCode == NSStreamEventHasBytesAvailable)
{
if(aStream == inputStream) {
uint8_t buffer[1024];
UInt32 len;
while ([inputStream hasBytesAvailable]) {
len = (UInt32)[inputStream read:buffer maxLength:sizeof(buffer)];
if(len > 0)
{
AudioBuffer abuffer;
abuffer.mDataByteSize = len; // sample size
abuffer.mNumberChannels = 1; // one channel
abuffer.mData = buffer;
int16_t audioBuffer[len];
for(int i = 0; i <= len; i++)
{
audioBuffer[i] = MuLaw_Decode(buffer[i]);
}
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = abuffer;
NSLog(#"%", bufferList.mBuffers[0]);
[audioProcessor processBuffer:&bufferList];
}
}
}
}
}
The MuLaw_Decode
#define MULAW_BIAS 33
int16_t MuLaw_Decode(uint8_t number)
{
uint8_t sign = 0, position = 0;
int16_t decoded = 0;
number =~ number;
if(number&0x80)
{
number&=~(1<<7);
sign = -1;
}
position= ((number & 0xF0) >> 4) + 5;
decoded = ((1<<position) | ((number&0x0F) << (position - 4)) |(1<<(position-5))) - MULAW_BIAS;
return (sign == 0) ? decoded : (-(decoded));
}
And the code that opens the connection and initialises the audio processor
CFReadStreamRef readStream;
CFWriteStreamRef writeStream;
CFStreamCreatePairWithSocketToHost(NULL, (CFStringRef)#"10.0.0.14", 6000, &readStream, &writeStream);
inputStream = (__bridge_transfer NSInputStream *)readStream;
outputStream = (__bridge_transfer NSOutputStream *)writeStream;
[inputStream setDelegate:self];
[outputStream setDelegate:self];
[inputStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
[outputStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
[inputStream open];
[outputStream open];
audioProcessor = [[AudioProcessor alloc] init];
[audioProcessor start];
[audioProcessor setGain:1];
I believe the issue in my code is with the socket connection callback, that I am not doing the right thing with the data.
I solved this in the end, see my answer here
I intended putting the code here, but it would be a lot of copy pasting

Encoding raw YUV420P to h264 with AVCodec on iOS

I am trying to encode a single YUV420P image gathered from a CMSampleBuffer to an AVPacket so that I can send h264 video over the network with RTMP.
The posted code example seems to work as avcodec_encode_video2 returns 0 (Success) however got_output is also 0 (AVPacket is empty).
Does anyone have any experience with encoding video on iOS devices that might know what I am doing wrong?
- (void) captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
// sampleBuffer now contains an individual frame of raw video frames
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
// access the data
int width = CVPixelBufferGetWidth(pixelBuffer);
int height = CVPixelBufferGetHeight(pixelBuffer);
int bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0);
unsigned char *rawPixelBase = (unsigned char *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
// Convert the raw pixel base to h.264 format
AVCodec *codec = 0;
AVCodecContext *context = 0;
AVFrame *frame = 0;
AVPacket packet;
//avcodec_init();
avcodec_register_all();
codec = avcodec_find_encoder(AV_CODEC_ID_H264);
if (codec == 0) {
NSLog(#"Codec not found!!");
return;
}
context = avcodec_alloc_context3(codec);
if (!context) {
NSLog(#"Context no bueno.");
return;
}
// Bit rate
context->bit_rate = 400000; // HARD CODE
context->bit_rate_tolerance = 10;
// Resolution
context->width = width;
context->height = height;
// Frames Per Second
context->time_base = (AVRational) {1,25};
context->gop_size = 1;
//context->max_b_frames = 1;
context->pix_fmt = PIX_FMT_YUV420P;
// Open the codec
if (avcodec_open2(context, codec, 0) < 0) {
NSLog(#"Unable to open codec");
return;
}
// Create the frame
frame = avcodec_alloc_frame();
if (!frame) {
NSLog(#"Unable to alloc frame");
return;
}
frame->format = context->pix_fmt;
frame->width = context->width;
frame->height = context->height;
avpicture_fill((AVPicture *) frame, rawPixelBase, context->pix_fmt, frame->width, frame->height);
int got_output = 0;
av_init_packet(&packet);
avcodec_encode_video2(context, &packet, frame, &got_output)
// Unlock the pixel data
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
// Send the data over the network
[self uploadData:[NSData dataWithBytes:packet.data length:packet.size] toRTMP:self.rtmp_OutVideoStream];
}
Note: It is known that this code has memory leaks because I am not freeing the memory that is dynamically allocated.
UPDATE
I updated my code to use #pogorskiy method. I only try to upload the frame if got output returns 1 and clear the buffer once I am done encoding video frames.

Resources