We are losing sound (no sound) in our app, but this is somehow causing all other apps to also lose sound. I don't know how it would even be possible for us to block the sound from an external app like the Apple Music app.
We are dumping the contents of our AVAudioSession session and there are no differences that we can see between when sound is working and not. We have verified that the route output is still the iPhone's speaker, even when we lost sound.
This is happening on the iPhone 6s & 6s Plus with the speaker. We can "fix" the audio by changing the output route, such as plug in and unplug headphones.
How is it possible to impact the ability to play sound of other apps, which may help troubleshoot what is happening?
We tracked down the source of the problem to having bad data in the audio buffer that was sent to Core Audio. Specifically, one of the audio processing steps output data that was NaN (Not a Number), instead of a float in the valid range of +/- 1.0.
It appears that on some devices, if the data contains NaN, it kills the audio of the whole device.
We worked around it by looping through the audio data checking for NaN values, and converting them to 0.0 instead. Note that checking if a float is NaN is a strange check (or it seems strange to me). NaN is not equal to anything, including itself.
Some pseudocode to work around the problem until we get new libraries that have a proper fix:
float *interleavedAudio; // pointer to a buffer of the audio data
unsigned int numberOfSamples; // number of left/right samples in the audio buffer
unsigned int numberOfLeftRightSamples = numberOfSamples * 2; // number of float values in the audio buffer
// loop through each float in the audio data
for (unsigned int i = 0; i < numberOfLeftRightSamples; i++)
{
float *sample = interleavedAudio + i;
// NaN is never equal to anything, including itself
if( *sample != *sample )
{
// This sample is NaN - force it to 0.0 so it doesn't corrupt the audio
*sample = 0.0;
}
}
Related
İ need to record pcmaudio per 10 milisecond without playback in swift.
I have tried this code but i can't find how can i stop playback while recording.
RecordAudio Github Repo
and second question: How can i get PCM data from circular buffer for encode-decode process properly. When I convert recorded audio data to signed byte or unsigned byte or anything else the converted data sometimes will corrupt. What is the best practice for this kind of process
In the RecordAudio sample code, the audio format is specified as Float (32-bit floats). When doing a float to integer conversion, you have to make sure your scale and offset results in a value in legal range for the destination type. e.g. check that -1.0 to 1.0 results in 0 to 256 (unsigned byte), and out-of-range values are clipped to legal values. Also pay attention to the number of samples you convert, as an Audio Unit callback can vary the frameCount sent (number of samples returned). You most likely won't get exactly 10 mS in any single RemoteIO callback, but may have to observe a circular buffer filled by multiple callbacks, or a larger buffer that you will have to split.
When RemoteIO is running in play-and-record mode, you can usually silence playback by zeroing the bufferList buffers (after copying, analyzing, or otherwise using the data in the buffers) before returning from the Audio Unit callback.
I downloaded the speakHere example and changed the parameters liked below:
#define kBufferDurationSeconds 0.020
void AQRecorder::SetupAudioFormat(UInt32 inFormatID)
{
memset(&mRecordFormat,0, sizeof(mRecordFormat));
mRecordFormat.mFormatID =kAudioFormatLinearPCM;
mRecordFormat.mSampleRate =8000.0;
mRecordFormat.mFormatFlags =kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;
mRecordFormat.mBitsPerChannel = 16;
mRecordFormat.mFramesPerPacket = mRecordFormat.mChannelsPerFrame =1;
mRecordFormat.mBytesPerFrame = (mRecordFormat.mBitsPerChannel/8) * mRecordFormat.mChannelsPerFrame;
mRecordFormat.mBytesPerPacket = mRecordFormat.mBytesPerFrame ;
}
But I found that it seemed the time interval of the callback function AQRecorder::MyInputBufferHandler() was called but not per 20ms. It called the callback function four times with 1ms interval and after 500ms calls the callback function one time, then four times with 1ms, then 500ms,over and over again. But I set the parameter kBufferDurationSeconds = 0.02
what cause this problem. Please help me.
In iOS, the Audio Session setPreferredIOBufferDuration API (did you even use an OS buffer duration call?) is only a request regarding the app's preference. The OS is free to choose a buffer duration different, but compatible with what iOS thinks is best (for battery life, compatibility with other apps, etc.)
Audio Queues run on top of Audio Units. If the RemoteIO Audio Unit is using 500 mS buffers, it will cut them up into 4 smaller Audio Queue buffers and pass those smaller buffers to the Audio Queue API in a quick burst.
If you use the Audio Unit API instead of the Audio Queue API, and the Audio Session API for a setPreferredIOBufferDuration message, you may be able to request and get shorter, more evenly spaced, audio buffers.
I'm trying to make an accurate timer to analyze an input. I'd like to be able to measure 1% deviation in signals of ~200ms.
My understanding is that using an AudioUnit will be able to get <1ms.
I tried implementing the code from Stefan Popp's example
After updating a few things to get it to work on xcode 6.3, I have the example working, however:
While I do eventually want to capture audio, I thought there should be some way to get a notification, like NSTimer, so I tried an AudioUnitAddRenderNotify, but it does exactly what it says it should - i.e it's tied to the render, not just an arbitrary timer. Is there some way to get a callback triggered without having to record or play?
When I examine mSampleTime, I find that the interval between slices does match the inNumberFrames - 512 - which works out to 11.6ms. I see the same interval for both record and play. I need more resolution than that.
I tried playing with kAudioSessionProperty_PreferredHardwareIOBufferDuration but all the examples I could find use the deprecated AudioSessions, so I tried to convert to AudioUnits:
Float32 preferredBufferSize = .001; // in seconds
status = AudioUnitSetProperty(audioUnit, kAudioSessionProperty_PreferredHardwareIOBufferDuration, kAudioUnitScope_Output, kOutputBus, &preferredBufferSize, sizeof(preferredBufferSize));
But I get OSStatus -10879, kAudioUnitErr_InvalidProperty.
Then I tried kAudioUnitProperty_MaximumFramesPerSlice with values of 128 and 256, but inNumberFrames is always 512.
UInt32 maxFrames = 128;
status = AudioUnitSetProperty(audioUnit, kAudioUnitProperty_MaximumFramesPerSlice, kAudioUnitScope_Global, 0, &maxFrames, sizeof(maxFrames));
[EDIT]
I am trying to compare the timing of an input (user's choice of MIDI or microphone) to when it should be. Specifically, is the instrument being played before or after the beat/metronome and by how much? This is for musicians, not a game, so precision is expected.
[EDIT]
The answers seem re-active to events. i.e. They let me precisely see when something happened, however I don't see how I do something accurately. My fault for not being clear. My app needs to be the metronome as well - synchronize playing a click on the beat and flash a dot on the beat - then I can analyze the user's action to compare timing. But if I can't play the beat accurately, the rest falls apart. Maybe I'm supposed to record audio - even if I don't want it - just to get inTimeStamp from the callback?
[EDIT]
Currently my metronome is:
- (void) setupAudio
{
AVAudioPlayer *audioPlayer;
NSString *path = [NSString stringWithFormat:#"%#/click.mp3", [[NSBundle mainBundle] resourcePath]];
NSURL *soundUrl = [NSURL fileURLWithPath:path];
audioPlayer = [[AVAudioPlayer alloc] initWithContentsOfURL:soundUrl error:nil];
[audioPlayer prepareToPlay];
CADisplayLink *syncTimer;
syncTimer = [CADisplayLink displayLinkWithTarget:self selector:#selector(syncFired:)];
syncTimer.frameInterval = 30;
[syncTimer addToRunLoop:[NSRunLoop mainRunLoop] forMode:NSDefaultRunLoopMode];
}
-(void)syncFired:(CADisplayLink *)displayLink
{
[audioPlayer play];
}
You should be using a circular buffer, and performing your analysis on the signal in chunks that match your desired frame count on your own timer. To do this you set up a render callback, then feed your circular buffer the input audio in the callback. Then you set up your own timer which will pull from the tail of the buffer and do your analysis. This way you could be feeding the buffer 1024 frames every 0.23 seconds, and your analysis timer could fire maybe every 0.000725 seconds and analyze 32 samples. Here is a related question about circular buffers.
EDIT
To get precision timing using a ring buffer, you could also store the timestamp corresponding to the audio buffer. I use TPCircularBuffer for doing just that. TPCircularBufferPrepareEmptyAudioBufferList, TPCircularBufferProduceAudioBufferList, and TPCircularBufferNextBufferList will copy and retrieve the audio buffer and timestamp to and from a ring buffer. Then when you are doing your analysis, there will be a timestamp corresponding to each buffer, eliminating the need to do all of your work in the render thread, and allowing you to pick and choose your analysis window.
If you are using something like cross-correlation and/or a peak detector to find a matched sample vector within an audio sample buffer (or a ring buffer containing samples), then you should be able to count samples between sharp events to within one sample (1/44100 or 0.0226757 milliseconds at a 44.1k Hz sample rate), plus or minus some time estimation error. For events more than one Audio Unit buffer apart, you can sum and add the number of samples within the intervening buffers to get a more precise time interval than just using (much coarser) buffer timing.
However, note that there is a latency or delay between every sample buffer and speaker audio going out, as well as between microphone sound reception and buffer callbacks. That has to be measured, as in you can measure the round trip time between sending a sample buffer out, and when the input buffer autocorrelation estimation function gets it back. This is how long it takes the hardware to buffer, convert (analog to digital and vice versa) and pass the data. That latency might be around the area of 2 to 6 times 5.8 milliseconds, using appropriate Audio Session settings, but might be different for different iOS devices.
Yes, the most accurate way to measure audio is to capture the audio and look at the data in the actual sampled audio stream.
I use ffmpeg to decode video/audio stream and use portaudio to play audio. I encounter a sync problem with portaudio. I have a function like below,
double AudioPlayer::getPlaySec() const
{
double const latency = Pa_GetStreamInfo( mPaStream )->outputLatency;
double const bytesPerSec = mSampleRate * Pa_GetSampleSize( mSampleFormat ) * mChannel;
double const playtime = mConsumedBytes / bytesPerSec;
return playtime - latency;
}
mCousumeBytes is the byte count which written into audio device in the portaudio callback function. I thought I could have got the playing time according to the byte count. Actually, when I execute the other process ( like open firefox ) which make cpu busy, the audio become intermittent, but the callback doesn't stop so that mConsumeBytes is more than expected, and getPlaySec return a time which is larger than playing time.
I have no idea how this happened. Any suggestion is welcome. Thanks!
Latency, in PortAudio is defined a bit vaguely. Something like the average time between when you put data into the buffer and when you can expect it to play. That's not something you want to use for this purpose.
Instead, to find the current playback time of the device, you can actually poll the device using the Pa_GetStreamTime function.
You may want to see this document for more detailed info.
I know this is old. But still; PortAudio v19+ can provide you with its own sample rate. You should use that for audio sync, since actual sample rate playback can differ between different hardware. PortAudio might try to compensate (depending on implementation). If you have drift problems, try using that.
EDIT: In the end I used exactly as I explained below, AVRecorder for recording the speech and openAL for the pitch shift and playback. It worked out quite well.
I got a question regarding recording, modifying and playing back audio. I asked a similar question before ( Record, modify pitch and play back audio in real time on iOS ) but I now have more information and could do with some further advice please.
So firstly this is what I am trying to do (on a separate thread to the main thread):
monitor the iphone mic
check for sound greater than a certain volume
if above threshold start recording e.g. person starts talking
continue to record until volume drops below threshold e.g. person stops talking
modify pitch of recorded sound.
playback sound
I was thinking of using the AVRecorder to monitor and record the sound, good tutorial here: http://mobileorchard.com/tutorial-detecting-when-a-user-blows-into-the-mic/
and I was thinking of using openAL to modify the pitch of the recorded audio.
So my question is, is my thinking correct in the list of points above, am I missing something or is there a better/easier way to do it. Can I avoid mixing audio libraries and just use AVFoundation to change the pitch too?
You can either use AVRecorder or something lower like the realtime IO audio unit.
The concept of 'volume' is pretty vague. You might want to look at the difference between calculating peak and RMS values, and understanding how to integrate an RMS value over a given time (say 300ms which is what a VU meter uses).
Basically you sum all the squares of the values. You would take the square root and convert to dBFS with 10 * log10f(sqrt(sum/num_samples)), but you can do that without the sqrt in one step with 20 * log10f(sum/num_samples).
You'll need to do a lot of adjusting of integration times and thresholds to get it to behave the way you want.
For pitch shifting, I think OpenAL with do the trick, the technique behind it is called band limited interpolation - https://ccrma.stanford.edu/~jos/resample/Theory_Ideal_Bandlimited_Interpolation.html
This example shows a rms calculation as a running average. The circular buffer maintains a history of squares, and eliminates the need to sum the squares every operation. I haven't run it so treat it as pseudo code ;)
Example:
class VUMeter
{
protected:
// samples per second
float _sampleRate;
// the integration time in seconds (vu meter is 300ms)
float _integrationTime;
// these maintain a circular buffer which contains
// the 'squares' of the audio samples
int _integrationBufferLength;
float *_integrationBuffer;
float *_integrationBufferEnd;
float *_cursor;
// this is a sort of accumulator to make a running
// average more efficient
float _sum;
public:
VUMeter()
: _sampleRate(48000.0f)
, _integrationTime(0.3f)
, _sum(0.)
{
// create a buffer of values to be integrated
// e.g 300ms # 48khz is 14400 samples
_integrationBufferLength = (int) (_integrationTime * _sampleRate);
_integrationBuffer = new float[_integrationBufferLength + 1];
bzero(_integrationBuffer, _integrationBufferLength);
// set the pointers for our ciruclar buffer
_integrationBufferEnd = _integrationBuffer + _integrationBufferLength;
_cursor = _integrationBuffer;
}
~VUMeter()
{
delete _integrationBuffer;
}
float getRms(float *audio, int samples)
{
// process the samples
// this part accumulates the 'squares'
for (int i = 0; i < samples; ++i)
{
// get the input sample
float s = audio[i];
// remove the oldest value from the sum
_sum -= *_cursor;
// calculate the square and write it into the buffer
double square = s * s;
*_cursor = square;
// add it to the sum
_sum += square;
// increment the buffer cursor and wrap
++_cursor;
if (_cursor == _integrationBufferEnd)
_cursor = _integrationBuffer;
}
// now calculate the 'root mean' value in db
return 20 * log10f(_sum / _integrationBufferLength);
}
};
OpenAL resampling will change the pitch and the duration inversely. e.g. a sound resampled to a higher pitch will play for a shorter amount of time and thus faster.