I am trying to develop an iOS app which reads sound from the microphone, apply some effects and play it through the headset instantly, may be with some acceptable delay.
Is this possible? As a first step, i am trying to play the sound received from microphone in my headsets at the same time, but struggling to do so...
I was able to record the sound, save it and then play it easily. Relevant questions, articles couldn't be found easily. Any ideas, links are much appreciated
I did check Apple's aurioTouch. I couldn't find simultaneous record and play of same signal.
Request the shortest buffers possible using audio session APIs (less than 6 mS is possible on most iOS devices). Then feed the raw audio samples you get from RemoteIO recording callbacks to the buffers in the RemoteIO play callbacks, possibly using a lock free circular fifo in between.
Related
I am developing an Karaoke app in which you can record your voice while listening to the music. When user uses headphones, everything is great - he can listen to the music and himself in headphones while singing. Then we have his pure voice recorded and we can mix it with playback.
Problem occurs when user does not use headphones. Then we play music via speakers AVAudioSessionCategoryPlayAndRecord and record simultaneously. In final recording we have user's voice and playback from speakers mixed together. Problem is that playback's volume is very big and it's "covering" user's voice. Firstly I thought that this is normal behaviour because speakers are close to microphone so there is nothing I can do.
However when I tried the same thing on Garage Band it somehow lowers playback from speakers making voice more hearable.
I also tried it with Instagram (you can record while playing music e.g. from Spotify) and I noticed that after ~1 sec. playback's volume is decreasing and we can hear voice more precisely.
I don't think that it's post processing because it would be very complicated so maybe there is an option to let "iOS handle it".
To be clear - it does not lowers playback during recording - it's "done" while listening final video.
I use AVCaptureSession for recording and AudioKit Player for playing.
Thanks in advance for any thoughts/tips/advices!
Regards
Ok so I asked Apple TS and the respond was exactly what I wanted: https://developer.apple.com/documentation/avfoundation/avaudiosession/mode/1616455-voicechat You just have to set this mode in AVAudioSession and system will handle it device’s tonal equalization is optimized for voice
iOS cannot 'just handle' that, there is no "filter out the music" function. The fact that it doesn't do it live, but does so later or with a delay strongly implies they are doing some post processing. I'm not a machine learning expert, but I think if you just used an equalizer and a noise gate you could get this effect. It'd be hard to extract an acapella but you could certainly improve it. Likely Instagram takes that second to identify where the voice frequencies are so it knows how to EQ the signal.
I understand that this question might get a bad rating, but I've been looking at questions which ask how to reroute audio output to the loud speaker on iOS devices.
Every question I looked at the user talked about using your AVAudioSession to reroute it.. However, I'm not using AVAudioSession, I'm using an AVAudioEngine.
So basically my question is, even though I'm using an AVAudioEngine, should I still have an AVAudioSession?
If so, what is the relationship between these two objects? Or is there a way to connect an AVAudioEngine to an AVAudioSession?
If this is not the case, and there is no relation between an AVAudioEngine and an AVAudioSession, than how do you reroute audio so that it plays out of the main speakers on an iOS device rather than the earpiece.
Thank you!
AVAudioSession is specific to iOS and coordinates audio playback between apps, so that, for example, audio is stopped when a call comes in, or music playback stops when the user starts a movie. This API is needed to make sure an app behaves correctly in response to such events
AVAudioEngine is a modern Objective-C API for playback and recording. It provides a level of control for which you previously had to drop down to the C APIs of the Audio Toolbox framework (for example, with real-time audio tasks). The audio engine APIs are built to interface well with lower-level APIs, so you can still drop down to Audio Toolbox if you have to.
The basic concept of this API is to build up a graph of audio nodes, ranging from source nodes (players and microphones) and overprocessing nodes (mixers and effects) to destination nodes (hardware outputs). Each node has a certain number of input and output busses with well-defined data formats. This architecture makes it very flexible and powerful. And it even integrates with audio units.
so there is no inclusive relation between this .
Source Link : https://www.objc.io/issues/24-audio/audio-api-overview/
Yes it is not clearly commented , however, I found this comment from ios developer documentation.
AVFoundation playback and recording classes automatically activate your audio session.
Document Link : https://developer.apple.com/library/content/documentation/Audio/Conceptual/AudioSessionProgrammingGuide/ConfiguringanAudioSession/ConfiguringanAudioSession.html
I hope this may help you.
I'm having some trouble working with the iOS Audio frameworks to create a simple app. I would like to record audio through the Microphone and play it back to the user while recording.
I have tried each of the audio framework layers(AVFoundation, Audio Queue API, and RemoteIO), but have only found old documentation and broken examples.It seems like a simple request that AVFoundation should handle, but I have explored the following other SO questions and still find myself circling for hours to get the hang of this. Here is what I have reviewed:
iOS: Sample code for simultaneous record and playback (Other SO Users also state the accepted answer is not concrete and difficult to implement even with a delay of ~70ms.)
Record and play audio Simultaneously (From 2010 and very high level, I have downloaded the sample code and can't find a working example that does simultaneous playback and recording).
Adjust the length of an AudioUnit Buffer (RemoteIO is so confusing to me right now, is this really required?)
I have also downloaded and reviewed both the SpeakHere and AurioTouch sample projects from Apple. I promise I wouldn't post up without hours of googling and struggling. You can see "record audio and playback iOS simultaneously" returns many dated and non-working examples.I know myself and the community could really benefit from some updated documentation and examples in the audio section. RemoteIO seems to be too advanced for such a simple task. Thanks again for your help and consideration.
The appropriate way to do this is via AudioUnit APIs, even though it seems like a common scenario which should be handled by higher level APIs.
I wrote a small demo app using AudioUnit. You're free to try it our and modify it for suiting your purpose. The demo app does record audio and play it simultaneously, but it's recommended to use a ear phone to see the effect.
The RemoteIO Audio Unit is the only way to play back what is being recorded with low latency. RemoteIO is low latency because it runs audio callbacks in a separate dedicated real-time thread which is why it is fast, but also why it is a bit more complex to code. All the other iOS audio APIs are built on top of RemoteIO and thus add latency.
You will also need to configure the app's Audio Session APIs to request low latency with the appropriate audio session type. The foreground app can request and get audio input and output latencies as low as 5.6 milliseconds on most iOS devices most of the time.
I want to process the stereo output from iOS devices, no matter what application causes them and visualize it in real-time.
Is it possible to use the generic output device (or anything else) to get at the audio data which are currently being played? Maybe as an input to a remoteIO unit?
In other words: I want to do what aurioTouch2 does (FFT only) but instead of using the microphone as input source, I want to process everything which is coming out of the speakers at a given time.
Kind regards
If your own app is playing using the RemoteIO Audio Unit, you can capture that content. You can not capture audio your app is playing using many of the other audio APIs. The iOS security sandbox will prevent your app from capturing audio that any other app is playing (unless that app explicitly exports audio via the Inter-App Audio API or equivalent).
Say you want to playback exactly what the iPhone mic is picking up in real-time. Which framework/class would be used?
You'll need to use the Core Audio framework for this. Specifically, look into audio graphs, audio units, and RemoteIO. Plenty of sample code for those to get you started.