Is there a way to cancel pre-processing like echo cancellation and noise suppression in audio recorder in iOS?
I'm using AVAudioRecorder with meteringEnabled=true, and I get the average decibel level using averagePowerForChannel (docs).
I am trying to measure ambient noise near the phone, and iPhone 8 seems to amplify low noises or cancel them out if I start to speak. For example, if background music has an absolute decibel level of 30 - iOS seems to amplify it. When I start to speak even quietly - the dB level drops significantly.
But since I want to measure ambient noise - I don't want this pre-processing.
I tried setInputGain (docs) but isInputGainSettable is always false - therefore, I can't take this approach.
Is there a way to cancel any amplification or pre-processing like echo cancellation and noise suppression?
You can enable and disable AEC,AGC - using the AudioUnitSetProperty
https://developer.apple.com/documentation/audiotoolbox/1440371-audiounitsetproperty
Here is some code snippet for the same.
lResult = AudioUnitSetProperty(lAUAudioUnit,
kAUVoiceIOProperty_BypassVoiceProcessing,
kAudioUnitScope_Global,
lInputBus,
&lFalse,
sizeof(lFalse));
lResult = AudioUnitSetProperty(lAUAudioUnit,
kAUVoiceIOProperty_VoiceProcessingEnableAGC,
kAudioUnitScope_Global,
lInputBus,
&lFalse,
sizeof(lFalse));
What the app needs is access to unprocessed audio after disabling the AGC (Auto Gain Control) filters on the audio channel. To get access to raw and unprocessed audio, turn on Measurement mode in iOS.
As appears in the iOS documentation here, "Measurement" mode is a mode that indicates that your app is performing measurement of audio input or output.
This mode is intended for apps that need to minimize the amount of system-supplied signal processing to input and output signals. If recording on devices with more than one built-in microphone, the primary microphone is used.
The javascript code I used to modify this (using nativescript), before recording, is this:
// Disable AGC
const avSession = AVAudioSession.sharedInstance();
avSession.setCategoryModeOptionsError(AVAudioSessionCategoryRecord, AVAudioSessionModeMeasurement, null);
i was try above solution , but it didn't work for me.
below is my code :
var componentDesc: AudioComponentDescription
= AudioComponentDescription(
componentType: OSType(kAudioUnitType_Output),
componentSubType: OSType(kAudioUnitSubType_RemoteIO),
componentManufacturer: OSType(kAudioUnitManufacturer_Apple),
componentFlags: UInt32(0),
componentFlagsMask: UInt32(0) )
var osErr: OSStatus = noErr
let component: AudioComponent! = AudioComponentFindNext(nil, &componentDesc)
var tempAudioUnit: AudioUnit?
osErr = AudioComponentInstanceNew(component, &tempAudioUnit)
var outUnit = tempAudioUnit!
var lFalse = UInt32(1)
let lInputBus = AudioUnitElement(1)
let outputBus = AudioUnitElement(0)
let lResult = AudioUnitSetProperty(outUnit,
kAUVoiceIOProperty_BypassVoiceProcessing,
kAudioUnitScope_Global,
lInputBus,
&lFalse,
0);
var flag = Int32(1)
let Result = AudioUnitSetProperty(outUnit,
kAUVoiceIOProperty_VoiceProcessingEnableAGC,
kAudioUnitScope_Global,
outputBus,
&flag,
0);
Related
I am using AudioUnit class for recording and playback. During recording i can listen sound. Problem is that when i use sample rate 44100 then it working fine but if i use sample rate 8000 then it generating noise. After recording with 8000 sample rate when i play then there is no noise, there is actual sound.
Means only the time of recording it generate noise with actual sound.
My AudioStreamBasicDescription setting is-
audioStreamDescription.Format = AudioFormatType.LinearPCM;
audioStreamDescription.FormatFlags = AudioFormatFlags.LinearPCMIsSignedInteger |
AudioFormatFlags.LinearPCMIsPacked;
audioStreamDescription.SampleRate = 8000; // 44100;
audioStreamDescription.BitsPerChannel = 16;
audioStreamDescription.ChannelsPerFrame = 1;
audioStreamDescription.BytesPerFrame = (16 / 8);
audioStreamDescription.FramesPerPacket = 1;
audioStreamDescription.BytesPerPacket = audioStreamDescription.BytesPerFrame * audioStreamDescription.FramesPerPacket;
audioStreamDescription.Reserved = 0;
AudioUnit setting is-
public void prepareAudioUnit()
{
// Getting AudioComponent Remote output
_audioComponent = AudioComponent.FindComponent(AudioTypeOutput.Remote);
// creating an audio unit instance
audioUnit = new AudioUnit.AudioUnit(_audioComponent);
// turning on microphone
audioUnit.SetEnableIO(true, AudioUnitScopeType.Input, 1 );
audioUnit.SetEnableIO(true, AudioUnitScopeType.Output, 0 );
// setting audio format
var austat = audioUnit.SetFormat(audioStreamDescription, AudioUnitScopeType.Output, 1);
var austatInput = audioUnit.SetFormat(audioStreamDescription, AudioUnitScopeType.Input, 0);
//audioUnit.SetSampleRate(8000.0f, AudioUnitScopeType.Output, 0);
//audioUnit.SetSampleRate(8000.0f, AudioUnitScopeType.Input, 1);
// setting callback method
audioUnit.SetRenderCallback(render_CallBack, AudioUnitScopeType.Input, 0);
audioUnit.Initialize();
}
Now, my main question is how i can remove that noise which is comming with actual sound?
If i am not able to explain properly then please let me know.
This is a problem that's come up in my app after the introduction of the iPhone 6s and 6s+, and I'm almost positive that it is because the new model's built-in mic is stuck recording at 48kHz (you can read more about this here). To clarify, this was never a problem with previous phone models that I've tested. I'll walk through my Audio Engine implementation and the varying results at different points depending on the phone model further below.
So here's what's happening - when my code runs on previous devices I get a consistent number of audio samples in each CMSampleBuffer returned by the AVCaptureDevice, usually 1024 samples. The render callback for my audio unit graph provides an appropriate buffer with space for 1024 frames. Everything works great and sounds great.
Then Apple had to go make this damn iPhone 6s (just kidding, it's great, this bug is just getting to my head) and now I get some very inconsistent and confusing results. The AVCaptureDevice now varies between capturing 940 or 941 samples and the render callback now starts making a buffer with space for 940 or 941 sample frames on the first call, but then immediately starts increasing the space it reserves on subsequent calls up to 1010, 1012, or 1024 sample frames, then stays there. The space it ends up reserving varies by session. To be honest, I have no idea how this render callback is determining how many frames it prepares for the render, but I'm guessing it has to do with the sample rate of the Audio Unit that the render callback is on.
The format of the CMSampleBuffer comes in at 44.1kHz sample rate no matter what the device is, so I'm guessing theres some sort of implicit sample rate conversion that happens before I'm even receiving the CMSampleBuffer from the AVCaptureDevice on the 6s. The only difference is that the preferred hardware sample rate of the 6s is 48kHz opposed to earlier versions at 44.1kHz.
I've read that with the 6s you do have to be ready to make space for a varying number of samples being returned, but is the kind of behavior I described above normal? If it is, how can my render cycle be tailored to handle this?
Below is the code that is processing the audio buffers if you care to look further into this:
The audio samples buffers, which are CMSampleBufferRefs, come in through the mic AVCaptureDevice and are sent to my audio processing function that does the following to the captured CMSampleBufferRef named audioBuffer
CMBlockBufferRef buffer = CMSampleBufferGetDataBuffer(audioBuffer);
CMItemCount numSamplesInBuffer = CMSampleBufferGetNumSamples(audioBuffer);
AudioBufferList audioBufferList;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(audioBuffer,
NULL,
&audioBufferList,
sizeof(audioBufferList),
NULL,
NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
&buffer
);
self.audioProcessingCallback(&audioBufferList, numSamplesInBuffer, audioBuffer);
CFRelease(buffer);
This is putting the the audio samples into an AudioBufferList and sending it, along with the number of samples and the retained CMSampleBuffer, to the below function that I use for audio processing. TL;DR the following code sets up some Audio Units that are in an Audio Graph, using the CMSampleBuffer's format to set the ASBD for input, runs the audio samples through a converter unit, a newTimePitch unit, and then another converter unit. I then start a render call on the output converter unit with the number of samples that I received from the CMSampleBufferRef and put the rendered samples back into the AudioBufferList to subsequently be written out to the movie file, more on the Audio Unit Render Callback below.
movieWriter.audioProcessingCallback = {(audioBufferList, numSamplesInBuffer, CMSampleBuffer) -> () in
var ASBDSize = UInt32(sizeof(AudioStreamBasicDescription))
self.currentInputAudioBufferList = audioBufferList.memory
let formatDescription = CMSampleBufferGetFormatDescription(CMSampleBuffer)
let sampleBufferASBD = CMAudioFormatDescriptionGetStreamBasicDescription(formatDescription!)
if (sampleBufferASBD.memory.mFormatID != kAudioFormatLinearPCM) {
print("Bad ASBD")
}
if(sampleBufferASBD.memory.mChannelsPerFrame != self.currentInputASBD.mChannelsPerFrame || sampleBufferASBD.memory.mSampleRate != self.currentInputASBD.mSampleRate){
// Set currentInputASBD to format of data coming IN from camera
self.currentInputASBD = sampleBufferASBD.memory
print("New IN ASBD: \(self.currentInputASBD)")
// set the ASBD for converter in's input to currentInputASBD
var err = AudioUnitSetProperty(self.converterInAudioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0,
&self.currentInputASBD,
UInt32(sizeof(AudioStreamBasicDescription)))
self.checkErr(err, "Set converter in's input stream format")
// Set currentOutputASBD to the in/out format for newTimePitch unit
err = AudioUnitGetProperty(self.newTimePitchAudioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0,
&self.currentOutputASBD,
&ASBDSize)
self.checkErr(err, "Get NewTimePitch ASBD stream format")
print("New OUT ASBD: \(self.currentOutputASBD)")
//Set the ASBD for the convert out's input to currentOutputASBD
err = AudioUnitSetProperty(self.converterOutAudioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
0,
&self.currentOutputASBD,
ASBDSize)
self.checkErr(err, "Set converter out's input stream format")
//Set the ASBD for the converter out's output to currentInputASBD
err = AudioUnitSetProperty(self.converterOutAudioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
0,
&self.currentInputASBD,
ASBDSize)
self.checkErr(err, "Set converter out's output stream format")
//Initialize the graph
err = AUGraphInitialize(self.auGraph)
self.checkErr(err, "Initialize audio graph")
self.checkAllASBD()
}
self.currentSampleTime += Double(numSamplesInBuffer)
var timeStamp = AudioTimeStamp()
memset(&timeStamp, 0, sizeof(AudioTimeStamp))
timeStamp.mSampleTime = self.currentSampleTime
timeStamp.mFlags = AudioTimeStampFlags.SampleTimeValid
var flags = AudioUnitRenderActionFlags(rawValue: 0)
err = AudioUnitRender(self.converterOutAudioUnit,
&flags,
&timeStamp,
0,
UInt32(numSamplesInBuffer),
audioBufferList)
self.checkErr(err, "Render Call on converterOutAU")
}
The Audio Unit Render Callback that is called once the AudioUnitRender call reaches the input converter unit is below
func pushCurrentInputBufferIntoAudioUnit(inRefCon : UnsafeMutablePointer<Void>, ioActionFlags : UnsafeMutablePointer<AudioUnitRenderActionFlags>, inTimeStamp : UnsafePointer<AudioTimeStamp>, inBusNumber : UInt32, inNumberFrames : UInt32, ioData : UnsafeMutablePointer<AudioBufferList>) -> OSStatus {
let bufferRef = UnsafeMutablePointer<AudioBufferList>(inRefCon)
ioData.memory = bufferRef.memory
print(inNumberFrames);
return noErr
}
Blah, this is a huge brain dump but I really appreciate ANY help. Please let me know if there's any additional information you need.
Generally, you handle slight variations in buffer size (but a constant sample rate in and out) by putting the incoming samples in a lock-free circular fifo, and not removing any blocks of samples from that circular fifo until you have a full size block plus potentially some safety padding to cover future size jitter.
The variation in size probably has to do with the sample rate converter ratio not being a simple multiple, the resampling filter(s) needed, and any buffering needed for the resampling process.
1024 * (44100/48000) = 940.8
So that rate conversion might explain the jitter between 940 and 941 samples. If the hardware is always shipping out blocks of 1024 samples at a fixed rate of 48 kHz, and you need that block resampled to 44100 for your callback ASAP, there's a fraction of a converted sample that eventually needs to be output on only some output callbacks.
I've written some code to play multi-instrument general MIDI files on iOS. It works fine in iOS 7, but stopped working on iOS 8.
I've stripped it down to its essence here. Instead of creating 16 channels for my multi-channel mixer, I just create one sampler node, and map all the tracks to that channel. It still exhibits the same problem as the multi-sampler version. None of the Audio Toolbox calls return an error code (they all return 0) in iOS 7 or iOS 8. The sequence plays through the speakers in iOS 7, on both the simulator and on iPhone/iPad devices. Run the exact same code on the iOS 8 simulator, or an iPhone/iPad device, and no sound is produced.
If you comment out the call to [self initGraphFromMIDISequence], it plays on iOS 8 with the default sine-wave sound.
#implementation MyMusicPlayer {
MusicPlayer _musicPlayer;
MusicSequence _musicSequence;
AUGraph _processingGraph;
}
- (void)playMidi:(NSURL*)midiFileURL {
NewMusicSequence(&_musicSequence);
MusicSequenceFileLoad(_musicSequence, CFBridgingRetain(midiFileURL), 0, 0);
NewMusicPlayer(&_musicPlayer);
MusicPlayerSetSequence(_musicPlayer, _musicSequence);
[self initGraphFromMIDISequence];
MusicPlayerPreroll(_musicPlayer);
MusicPlayerStart(_musicPlayer);
}
// Sets up an AUGraph with one channel whose instrument is loaded from a sound bank.
// Maps all the tracks of the MIDI sequence onto that channel. Basically this is a
// way to replace the default sine-wave sound with another (single) instrument.
- (void)initGraphFromMIDISequence {
NewAUGraph(&_processingGraph);
// Add one sampler unit to the graph.
AUNode samplerNode;
AudioComponentDescription cd = {};
cd.componentManufacturer = kAudioUnitManufacturer_Apple;
cd.componentType = kAudioUnitType_MusicDevice;
cd.componentSubType = kAudioUnitSubType_Sampler;
AUGraphAddNode(_processingGraph, &cd, &samplerNode);
// Add a Mixer unit node to the graph
cd.componentType = kAudioUnitType_Mixer;
cd.componentSubType = kAudioUnitSubType_MultiChannelMixer;
AUNode mixerNode;
AUGraphAddNode(_processingGraph, &cd, &mixerNode);
// Add the Output unit node to the graph
cd.componentType = kAudioUnitType_Output;
cd.componentSubType = kAudioUnitSubType_RemoteIO; // Output to speakers.
AUNode ioNode;
AUGraphAddNode(_processingGraph, &cd, &ioNode);
AUGraphOpen(_processingGraph);
// Obtain the mixer unit instance from its corresponding node, and set the bus count to 1.
AudioUnit mixerUnit;
AUGraphNodeInfo(_processingGraph, mixerNode, NULL, &mixerUnit);
UInt32 const numChannels = 1;
AudioUnitSetProperty(mixerUnit,
kAudioUnitProperty_ElementCount,
kAudioUnitScope_Input,
0,
&numChannels,
sizeof(numChannels));
// Connect the sampler node's output 0 to mixer node output 0.
AUGraphConnectNodeInput(_processingGraph, samplerNode, 0, mixerNode, 0);
// Connect the mixer unit to the output unit.
AUGraphConnectNodeInput(_processingGraph, mixerNode, 0, ioNode, 0);
// Obtain reference to the audio unit from its node.
AudioUnit samplerUnit;
AUGraphNodeInfo(_processingGraph, samplerNode, 0, &samplerUnit);
MusicSequenceSetAUGraph(_musicSequence, _processingGraph);
// Set the destination for each track to our single sampler node.
UInt32 trackCount;
MusicSequenceGetTrackCount(_musicSequence, &trackCount);
MusicTrack track;
for (int i = 0; i < trackCount; i++) {
MusicSequenceGetIndTrack(_musicSequence, i, &track);
MusicTrackSetDestNode(track, samplerNode);
}
// You can use either a DLS or an SF2 file bundled with your app; both work in iOS 7.
//NSString *soundBankPath = [[NSBundle mainBundle] pathForResource:#"GeneralUserv1.44" ofType:#"sf2"];
NSString *soundBankPath = [[NSBundle mainBundle] pathForResource:#"gs_instruments" ofType:#"dls"];
NSURL *bankURL = [NSURL fileURLWithPath:soundBankPath];
AUSamplerBankPresetData bpdata;
bpdata.bankURL = (__bridge CFURLRef) bankURL;
bpdata.bankMSB = kAUSampler_DefaultMelodicBankMSB;
bpdata.bankLSB = kAUSampler_DefaultBankLSB;
bpdata.presetID = 0;
UInt8 instrumentNumber = 46; // pick any GM instrument 0-127
bpdata.presetID = instrumentNumber;
AudioUnitSetProperty(samplerUnit,
kAUSamplerProperty_LoadPresetFromBank,
kAudioUnitScope_Global,
0,
&bpdata,
sizeof(bpdata));
}
I have some code, not included here, which polls to see if the sequence is still playing, by calling MusicPlayerGetTime on the MusicPlayer instance. In iOS 7, the result of that call each time is the number of seconds that have elapsed since it started playing. In iOS 8, the call always returns 0, which presumably means the MusicPlayer does not start playing the sequence on the call to MusicPlayerStart.
The code above is highly order-dependent -- you have to make certain calls before others; e.g., opening the graph before calling getInfo on a node, and not loading instruments until you've assigned the tracks to channels. I've followed all the advice in other StackOverflow threads, and have verified that getting the order correct makes error codes disappear.
Any iOS MIDI experts know what might have changed between iOS 7 and iOS 8 to make this code stop working?
In iOS 8 Apple introduced a slick Obj-C abstraction of the core audio API - AVAudioEngine.
You should probably check it out. https://developer.apple.com/videos/wwdc/2014/#502
i'm new in audiounit.
I'm confused to generate binaural tone filter, I was create two sound with left only and right only sound and add filter kAudioUnitSubType_LowPassFilter for each sound. When playing, i'm using UISlider to change kLowPassParam_CutoffFrequency for each player, this is a code :
Float32 value = slider.value; //only 160-190 hz
AEAudioUnitFilter *toneLeft = [self.sound objectForKey:#"binaural_left"];
AEAudioUnitFilter *toneRight = [self.sound objectForKey:#"binaural_right"];
if(toneLeft && toneRight){
Float32 leftFreq = value - self.rangeSlider.value; // i have two slider, for frequency and range
Float32 rightFreq = value + self.rangeSlider.value;
AudioUnitSetParameter(toneLeft.audioUnit,
kLowPassParam_CutoffFrequency,
kAudioUnitScope_Global,
0,
leftFreq,
0);
AudioUnitSetParameter(toneRight.audioUnit,
kLowPassParam_CutoffFrequency,
kAudioUnitScope_Global,
0,
rightFreq,
0);
}
but when sound played, i didn't hear a binaural, only the frequency changes.
I got an idea from : Idea
I'm using theamazingaudioengine.com framework.
Thanks for your help.
I'm trying to use a file player audio unit (kAudioUnitSubType_AudioFilePlayer) to play multiple files (not at the same time, of course). That's on iOS.
So I've successfully opened the files and stored their details in an array of AudioFileID's that I set to the audio unit using kAudioUnitProperty_ScheduledFileIDs. Now I would like to define 2 ScheduledAudioFileRegion's, one per file, and used them with the file player...
But I can't seem to find out:
How to set the kAudioUnitProperty_ScheduledFileRegion property to store these 2 regions (actually, how to define the index of each region)?
How to trigger the playback of a specific region.. My guess is that the kAudioTimeStampSampleTimeValid parameter should enable this but how to define which region you want to play?
Maybe I'm just plain wrong about the way I should use this audio unit, but documentation is very difficult to get and I haven't found any example showing the playback of 2 regions on the same player!
Thanks in advance.
You need to schedule region every time you want play file. In ScheduledAudioFileRegion you must set AudioFileID to play. Playback begins when current time in unit (samples) are equal or greater than sample time in scheduled region.
Example:
// get current unit time
AudioTimeStamp timeStamp;
UInt32 propSize = sizeof(AudioTimeStamp);
AudioUnitGetProperty(m_playerUnit, kAudioUnitProperty_CurrentPlayTime, kAudioUnitScope_Global, 0, &timeStamp, &propSize);
// when to start playback
timeStamp.mSampleTime += 100;
// schedule region
ScheduledAudioFileRegion region;
memset(®ion, 0, sizeof(ScheduledAudioFileRegion));
region.mAudioFile = ...; // your AudioFileID
region.mFramesToPlay = ...; // count of frames to play
region.mLoopCount = 1;
region.mStartFrame = 0;
region.mTimeStamp = timeStamp;
AudioUnitSetProperty(m_playerUnit, kAudioUnitProperty_ScheduledFileRegion, kAudioUnitScope_Global, 0, ®ion,sizeof(region));