I am receiving a stuttered sound when I first start the AUGraph and play a song with a kAudioUnitSubType_AudioFilePlayer component. The stutter is about 3 seconds but its enough to bother me plus I notice that music stops for a split second sometimes while playing(I guess to buffer?). I have tried changing the kAudioUnitProperty_ScheduledFilePrime to random values but notice no change.
What variables or values should I be looking to change to get rid of this flaw? Is this an issue with the stream format?
I am using the YBAudioUnit from https://github.com/ronaldmannak/YBAudioFramework/tree/master/YBAudioUnit
Code:
YBAudioFilePlayer:
- (void)setFileURL:(NSURL *)fileURL typeHint:(AudioFileTypeID)typeHint {
if (_fileURL) {
// Release old file:
AudioFileClose(_audioFileID);
}
_fileURL = fileURL;
if (_fileURL) {
YBAudioThrowIfErr(AudioFileOpenURL((__bridge CFURLRef)fileURL, kAudioFileReadPermission, typeHint, &_audioFileID));
YBAudioThrowIfErr(AudioUnitSetProperty(_auAudioUnit, kAudioUnitProperty_ScheduledFileIDs, kAudioUnitScope_Global, 0, &_audioFileID, sizeof(AudioFileID)));
// Get number of audio packets in the file:
UInt32 propsize = sizeof(_filePacketsCount);
YBAudioThrowIfErr(AudioFileGetProperty(_audioFileID, kAudioFilePropertyAudioDataPacketCount, &propsize, &_filePacketsCount));
// Get file's asbd:
propsize = sizeof(_fileASBD);
YBAudioThrowIfErr(AudioFileGetProperty(_audioFileID, kAudioFilePropertyDataFormat, &propsize, &_fileASBD));
// Get unit's asbd:
propsize = sizeof(_fileASBD);
AudioUnitGetProperty(_auAudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 0, &_unitASBD, &propsize);
if (_fileASBD.mSampleRate > 0 && _unitASBD.mSampleRate > 0) {
_sampleRateRatio = _unitASBD.mSampleRate / _fileASBD.mSampleRate;
} else {
_sampleRateRatio = 1.;
}
}
}
To play I call these methods on the YBAudioFilePlayer:
[player1 setFileURL:item.url typeHint:0];
[player1 scheduleEntireFilePrimeAndStartImmediately];
[graph start];//On a YBAudioUnitGraph which is really just a basic AUGraph
More than an answer this is a comment, but it's rather large, so I'll post it here.
I don't have the time and patience to study the code inside the YB.. API. But a couple of thigns come to my mind.
First I remember experimenting with Audio Units (using Apple's API) and I had a lot of stuttering going on. I solved the problem removing all objective-C calls inside the callback that feeds data to my AUGraph (well, I removed all except one that I couldn't get rid of). I replaced all Objective-c calls with pure C and C++ calls. Example:
... this is the render callback
int i = [myClass someProperty]; // obj-c
int i = myClass->someVarialbe; // c, c++
This was just an example, but it improved dramatically and I got rid of stuttering. Maybe you can take a look at the implementation of the YBXX API and see if there are a lot of obj-c calls in the callback, and if there are, I would not use the API.
Second observation. It seems that you're only trying to play an audio file, for which having an AudioGraph is a lot of overhead, you could use a single IO Audio Unit without the Graph.
There are a large number of questions to ask:
First, are you using a compressed audio file? If so, you may need to take into account padding frames (kAudioFilePropertyPacketTableInfo) to get the real number of audio frames in the file. Perhaps try an AIFF, CAF, or WAV file.
Have you made sure no other audio app are running in the background?
Are there any logging messages?
Have you tried posting to their issue page on github?
The final question is why you are trying to use their framework (which hasn't been updated in two years). I would recommend The Amazing Audio Engine. It is actively developed by some of the best audio folks on iOS.
Related
I've jumped off the deep end, and have decided to figure out low-latency audio on iOS using Audio Units. I've read as much documentation (from Apple and forums galore) as I can find, and the overall concepts make sense, but I'm still scratching my head on some concepts that I need help with:
I saw somewhere that AU Graphs are deprecated and that I should instead connect Audio Units directly. I'm cool with that... but how? Do I just need to use the Connection property of an Audio Unit to connect it to a source AU, and off I go? Initialize and Start the Units, and watch the magic happen? (cause it doesn't for me...)
What's the best Audio Unit setup to use if I simply want to grab audio from my mic, do some processing to the audio data, and then store that audio data without sending it out to the RemoteIO speaker, bus 0 output? I tried hooking up a GenericOutput AudioUnit to catch the data in a callback without any luck...
That's it. I can provide code when requested, but it's way too late, and this has wiped me out. If there's know easy answer, that's cool. I'll send any code snippets at will. Suffice it to say, I can easily get a simple RemoteIO, mic in, speaker out setup working great. Latency seems non-existant (at least to my ears). I just want to do something with the mic data and store it in memory without it going out to the speaker. Eventually hooking in the eq and mixer would be hip, but one step at a time.
FWIW, I'm coding in Xamarin Forms/C# land, but code examples in Objective C, Swift or whatever is fine. I'm stuck on the concepts, not necessarily the exact code.
THANKS!
Working with audio units without a graph is pretty simple and very flexible. To connect two units, you call AudioUnitSetProperty this way :
AudioUnitConnection connection;
connection.sourceAudioUnit = sourceUnit;
connection.sourceOutputNumber = sourceOutputIndex;
connection.destInputNumber = destinationInputIndex;
AudioUnitSetProperty(
destinationUnit,
kAudioUnitProperty_MakeConnection,
kAudioUnitScope_Input,
destinationInputIndex,
&connection,
sizeof(connection)
);
Note that it is required for the units connected this way to have their Stream Format set uniformly and that it must be done before their initialization.
Your question mentions Audio Units, and Graphs. As said in the comments, the graph concept has been replaced with the idea of attaching "nodes" to an AVAudioEngine. These nodes then "connect" to other nodes. Connecting nodes creates signal paths and starting the engine makes it all happen. This may be obvious, but I am trying to respond generally here.
You can do this all in Swift or in Objective-C.
Two high level perspectives to consider with iOS audio are the idea of a "host" and that of a "plugin". The host is an app and it hosts plugins. The plugin is usually created as an "app extension" and you can look up audio unit extensions for more about that as needed. You said you have one doing what you want, so this is all explaining the code used in a host
Attach AudioUnit to an AVaudioEngine
var components = [AVAudioUnitComponent]()
let description =
AudioComponentDescription(
componentType: 0,
componentSubType: 0,
componentManufacturer: 0,
componentFlags: 0,
componentFlagsMask: 0
)
components = AVAudioUnitComponentManager.shared().components(matching: description)
.compactMap({ au -> AVAudioUnitComponent? in
if AudioUnitTypes.codeInTypes(
au.audioComponentDescription.componentType,
AudioUnitTypes.instrumentAudioUnitTypes,
AudioUnitTypes.fxAudioUnitTypes,
AudioUnitTypes.midiAudioUnitTypes
) && !AudioUnitTypes.isApplePlugin(au.manufacturerName) {
return au
}
return nil
})
guard let component = components.first else { fatalError("bugs") }
let description = component.audioComponentDescription
AVAudioUnit.instantiate(with: description) { (audioUnit: AVAudioUnit?, error: Error?) in
if let e = error {
return print("\(e)")
}
// save and connect
guard let audioUnit = audioUnit else {
print("Audio Unit was Nil")
return
}
let hardwareFormat = self.engine.outputNode.outputFormat(forBus: 0)
self.engine.attach(au)
self.engine.connect(au, to: self.engine.mainMixerNode, format: hardwareFormat)
}
Once you have your AudioUnit loaded, you can connect your Athe AVAudioNodeTapBlock below, it has more to it since it need to be a binary or something that other host apps that aren't yours can load.
Recording an AVAudioInputNode
(You can replace the audio unit with the input node.)
In an app, you can record audio by creating an AVAudioInputNode or just reference the 'inputNode' property of the AVAudioEngine, which is going to be connected to the system's selected input device(mic, line in, etc) by default
Once you have the input node you want to process the audio of, next "install a tap" on the node. You can also connect your input node to a mixer node and install a tap there.
https://developer.apple.com/documentation/avfoundation/avaudionode/1387122-installtap
func installTap(onBus bus: AVAudioNodeBus,
bufferSize: AVAudioFrameCount,
format: AVAudioFormat?,
block tapBlock: #escaping AVAudioNodeTapBlock)
The installed tap will basically split your audio stream into two signal paths. It will keep sending the audio to the AvaudioEngine's output device and also send the audio to a function that you define. This function(AVAudioNodeTapBlock) is passed to 'installTap' from AVAudioNode. The AVFoundation subsystem calls the AVAudioNodeTapBlock and passes you the input data one buffer at a time along with the time at which the data arrived.
https://developer.apple.com/documentation/avfoundation/avaudionodetapblock
typealias AVAudioNodeTapBlock = (AVAudioPCMBuffer, AVAudioTime) -> Void
Now the system is sending the audio data to a programmable context, and you can do what you want with it.
To use it elsewhere, you can create a separate AVAudioPCMBuffer and write each of the passed in buffers to it in the AVAudioNodeTapBlock.
I've written my own audio library. (Silly, I know but I enjoy it.) Right now, I'm using it to play stereo files from the iPod library on iOS. It works well except that sometimes the call to ExtAudioFileRead() takes longer than 23ms. Since my AudioUnits output setup is playing audio at 44.1KHz, and it's asking for 1024 frames per callback. My code + ExtAudioFileRead() must take no longer than 23ms to respond.
I'm quite surprised that ExtAudioFileRead() is so slow. What I'm doing seems quite the normal thing to do. I'm thinking there must be some undocumented configuration magic that I'm not doing. Here is my relevant configuration code:
ExtAudioFileRef _extAudioFileRef;
OSStatus result;
result = ::ExtAudioFileOpenURL((__bridge CFURLRef)_url, &_extAudioFileRef);
AudioStreamBasicDescription format;
format.mBitsPerChannel = 8 * sizeof(float);
format.mBytesPerFrame = sizeof(float);
format.mBytesPerPacket = sizeof(float);
format.mChannelsPerFrame = channelCount;
format.mFormatFlags = kAudioFormatFlagIsFloat|kAudioFormatFlagIsNonInterleaved;
format.mFormatID = kAudioFormatLinearPCM;
format.mFramesPerPacket = 1;
format.mSampleRate = framesPerSecond;
result = ::ExtAudioFileSetProperty(_extAudioFileRef,
kExtAudioFileProperty_ClientDataFormat,
sizeof(format),
&format);
I'm not setting any other properties of _extAudioFileRef.
I've traced the heck out of this. I've put time measurement around just my call to ExtAudioFileRead() so I know it's not my code slowing this process down. It's got to be a configuration issue, right?
Thanks so much for any help or even guesses!
Cheers,
Christopher
You shouldn't be reading from the audio file in your audio callback - you should be buffering in another thread and passing the samples across.
You're breaking cardinal audio rule #4:
Don’t do file or network IO on the audio thread.
Like read, write or sendto.
I'm trying to convert a MIDI file to an Audio File (.m4a) in Swift.
Right now I'm using MIKMIDI as a tool to sequence and playback MIDI files, however it does not include the ability to save the playback into a file. MIKMID's creator outlines the process to do this here. In an attempt to capture and save the output to an audio file, I've followed this example to try and replace the MIKMIDI Graph's RemoteIO node with a GeneralIO node in Swift. When I try to save the output to a file using AudioUnitRender and ExtAudioFileWrite, they both return error -50 (kAudio_ParamError).
var channels = 2
var buffFrames = 512
var bufferList = AudioBufferList.allocate(maximumBuffers: 1)
for i in 0...bufferList.count-1{
var buffer = AudioBuffer()
buffer.mNumberChannels = 2
buffer.mDataByteSize = UInt32(buffFrames*sizeofValue(AudioUnitSampleType))
buffer.mData = calloc(buffFrames, sizeofValue(AudioUnitSampleType))
bufferList[i] = buffer
result = AudioUnitRender(generalIOAudioUnit, &flags, &inTimeStamp, busNum, UInt32(buffFrames), bufferList.unsafeMutablePointer)
inTimeStamp.mSampleTime += 1
result = ExtAudioFileWrite(extAudioFile, UInt32(buffFrames), bufferList.unsafeMutablePointer)
}
What is causing error -50, and how can I resolve it to render the MIDI (offline) to .m4a files?
UPDATE: I have resolved the ExtAudioFileWrite error -50 by changing mNumberChannels and channels to = 1. Now I get a one second audio file with noise. AudioUnitRender still returns error -50.
There are a couple of problems with your code:
your AudioBufferList doesn't agree with the client format, try
let bufferList = AudioBufferList.allocate(maximumBuffers: Int(clientFormat.mChannelsPerFrame))
you're replacing the wrong node from the AUGraph, and connecting the remaining node to itself, resulting in an infinite loop on AudioUnitRender.
But the main problem is that you are not implementing the solution that the author suggested. You wish that you could call AudioUnitRender with sample timestamps, faster than realtime, but the author said no, you'll have to manually convert sample time to hosttime and implement the better part of a midi player if you want that.
So you could do that (sounds hard), or file a feature request, or maybe record to file in realtime as you listen to the music by adding a render notification to the graph's remote IO audio unit with AudioUnitAddRenderNotify and writing the samples during the kAudioUnitRenderAction_PostRender phase.
I'm trying to use Dart to get an OGG file to loop using the HTML5 <audio> element. Does anyone have a working example for this. I'm specifically having trouble getting the audio to loop.
I was not able to have a fully controlled loop using the html5 AudioElement; sometimes the loop option was simply not working, sometimes there was a gap, sometimes patterns would overlap.
I had better chance using WebAudio using something like:
source = audioContext.createBufferSource();
source.buffer = buffer;
gainNode = audioContext.createGain();
gainNode.gain.value = 1;
source.connectNode(gainNode);
gainNode.connectNode(audioContext.destination);
// play it now in loop
source.start(audioContext.currentTime);
source.loop = true;
I was not able to load the source buffer from the html audio element which could have been a solution for the CORS issues I had. The samples were loaded using http requests.
I created a dartpad example that demonstrates looping using AudioElement native loop feature and WebAudio
https://dartpad.dartlang.org/879424bca794c63698b0
I'm wanting to load a local .caf audio file and reverse the audio (play it backwards). I've gathered that I basically need to flip an array of buffer data from posts like this
However, I'm not sure how to access this buffer data from a given audio file. I have a little experience playing sounds back with AVaudioPlayer and ObjectAL(an obj-c openAL library), but I don't know how to access something lower level like this buffer data array.
Could I please get an example of how I would go about getting access to that array?
Your problem reduces to the same problem described here, which was linked by P-i in the comment under your question. Kiran answered that question and re-posted his answer for you here. Kiran's answer is accurate, but you may need a few more details to be able to decide how to proceed because you're starting with a CAF file.
The simplest audio file format, linear pulse-code modulation (LPCM), is the easiest to read byte-for-byte or sample-for-sample. This means it's the easiest to reverse. Kiran's solution does just that.
The CAF format is a container/wrapper format, however. While your CAF file could contain a WAV file, it could also contain a compressed file format that cannot be manipulated in the same fashion.
You should consider first converting the CAF file to WAV, then reversing it as shown in the other solution. There are various libraries that will do this conversion for you, but a good place to start might be with the AudioToolbox framework, which includes Audio Converter Services. Alternately, if you can use the WAV file format from the start, you can prevent the need to convert to WAV.
You may need to know more if you find Kiran's sample code gives you an error (Core Audio is finicky). A great place to start is with the Core Audio 'Bible', "Learning Core Audio", written by Chris Adamson and Kevin Avila. That book builds your knowledge of digital sound using great samples. You should also check out anything written by Michael Tyson, who started the Amazing Audio Engine project on github, and wrote AudioBus.
I have worked on a sample app, which records what user says and plays them backwards. I have used CoreAudio to achieve this. Link to app code.
As each sample is 16-bits in size(2 bytes)(mono channel). You can load each sample at a time by copying it into a different buffer by starting at the end of the recording and reading backwards. When you get to the start of the data you have reversed the data and playing will be reversed.
// set up output file
AudioFileID outputAudioFile;
AudioStreamBasicDescription myPCMFormat;
myPCMFormat.mSampleRate = 16000.00;
myPCMFormat.mFormatID = kAudioFormatLinearPCM ;
myPCMFormat.mFormatFlags = kAudioFormatFlagsCanonical;
myPCMFormat.mChannelsPerFrame = 1;
myPCMFormat.mFramesPerPacket = 1;
myPCMFormat.mBitsPerChannel = 16;
myPCMFormat.mBytesPerPacket = 2;
myPCMFormat.mBytesPerFrame = 2;
AudioFileCreateWithURL((__bridge CFURLRef)self.flippedAudioUrl,
kAudioFileCAFType,
&myPCMFormat,
kAudioFileFlags_EraseFile,
&outputAudioFile);
// set up input file
AudioFileID inputAudioFile;
OSStatus theErr = noErr;
UInt64 fileDataSize = 0;
AudioStreamBasicDescription theFileFormat;
UInt32 thePropertySize = sizeof(theFileFormat);
theErr = AudioFileOpenURL((__bridge CFURLRef)self.recordedAudioUrl,kAudioFileReadPermission, 0, &inputAudioFile);
thePropertySize = sizeof(fileDataSize);
theErr = AudioFileGetProperty(inputAudioFile, kAudioFilePropertyAudioDataByteCount, &thePropertySize, &fileDataSize);
UInt32 dataSize = fileDataSize;
void* theData = malloc(dataSize);
//Read data into buffer
UInt32 readPoint = dataSize;
UInt32 writePoint = 0;
while( readPoint > 0 )
{
UInt32 bytesToRead = 2;
AudioFileReadBytes( inputAudioFile, false, readPoint, &bytesToRead, theData );
AudioFileWriteBytes( outputAudioFile, false, writePoint, &bytesToRead, theData );
writePoint += 2;
readPoint -= 2;
}
free(theData);
AudioFileClose(inputAudioFile);
AudioFileClose(outputAudioFile);
I think this sample code could help you.
Mixer Host Sample Code
It will load two caf files from the bundle and play it. It contains a function call readAudioFilesIntoMemory which is loading a caf file to a data array as you said.
The whole program is an example of core audio, I hope it can help you :)
Why not let CoreAudio's AudioConverter do it for you? See this post about "Getting PCM from MP3/AAC/ALAC File", and Apple's Core Audio Essentials
you can use libsox for iphone framework to apply audio effects easily.
it includes a sample project that shows how to do it.
libsox ios