Audio and Recording Reuse on iPhone with Monotouch - ios

I just started testing this very simple audio recording application that was built through Monotouch on actual iPhone devices today. I encountered an issue with what seemed to be the re-use of the AVAudioRecorder and AVPlayer objects after their first use and I am wondering how I might could solve it.
Basic Overview
The application consists of the following three sections :
List of Recordings (TableViewController)
Recording Details (ViewController)
New Recording (ViewController)
Workflow
When creating a recording, the user would click the "Add" button from the List of Recordings area and the application pushes the New Recording View Controller.
Within the New Recording Controller, the following variables are available:
AVAudioRecorder recorder;
AVPlayer player;
each are initialized prior to their usage:
//Initialized during the ViewDidLoad event
recorder = AVAudioRecorder.Create(audioPath, audioSettings, out error);
and
//Initialized in the "Play" event
player = new AVPlayer(audioPath);
Each of this work as intended on the initial load of the New Recording Controller area, however any further attempts do not seem to work (No Audio Playback)
The Details area also has a playback portion to allow the user to playback any recordings, however, much like the New Recording Controller, playback doesn't function there either.
Disposal
They are both disposed as follows (upon exiting / leaving the View) :
if(recorder != null)
{
recorder.Dispose();
recorder = null;
}
if(player != null)
{
player.Dispose();
player = null;
}
I have also attempted to remove any observers that could possible keep any of the objects "alive" in hopes that would solve the issue and have ensured they are each instantiated with each display of the New Recording area, however I still receive no audio playback after the initial Recording session.
I would be happy to provide more code if necessary. (This is using MonoTouch 6.0.6)

After further investigation, I determined that the issue was being caused by the AudioSession as both recording and playback were occurring within the same controller.
The two solutions that I determined were as follows:
Solution 1 (AudioSessionCategory.PlayAndRecord)
//A single declaration of this will allow both AVAudioRecorders and AVPlayers
//to perform alongside each other.
AudioSession.Category = AudioSessionCategory.PlayAndRecord;
//Upon noticing very quiet playback, I added this second line, which allowed
//playback to come through the main phone speaker
AudioSession.OverrideCategoryDefaultToSpeaker = true;
Solution 2 (AudioSessionCategory.RecordAudio & AudioSessionCategory.MediaPlayback)
void YourRecordingMethod()
{
//This sets the session to record audio explicitly
AudioSession.Category = AudioSessionCategory.RecordAudio;
MyRecorder.record();
}
void YourPlaybackMethod()
{
//This sets the session for playback only
AudioSession.Category = AudioSessionCategory.MediaPlayback;
YourAudioPlayer.play();
}
For some additional information on usage of the AudioSession, visit Apple's AudioSession Development Area.

Related

How to record audio and play it back in Safari on iOS in one click?

Problem: Safari will throw exceptions when certain actions are performed to audio elements inside of a callback.
This includes:
Playback
Setting the src property
Playing a URL object that was recorded/generated by the user
I need to record audio and visualize the data in one step. When a button is clicked, the recorded audio should be played back.
My application works like this: The user presses a record button, and the application begins recording audio once the user gives permission for the app to record. When the user presses the stop button, the recording is stopped and then two asynchronous operations happen:
Recording Phase
I. The user stops the recording by pressing a stop button in the UI.
This invokes the stop method on the recording controller class.
II.
Two asynchronous methods fire here: 1. The audio is decoded into an
object URL and set to the src of an HTMLAudioElement that is not
loaded into the DOM. 2. The audio is decoded into a buffer then
visualized on the screen.
//RecordingController
public stopRecording(): Promise
{
return new Promise( (resovle,reject)=>
{
if (this._isRecording)
{
this._isRecording=false;
this._mediaRecorder.stop();
this._mediaRecorder.onstop=()=>{
const blob = new Blob(this._recordBuffer, { 'type' : 'audio/ogg; codecs=opus' });
this._recordBuffer=[];
this.emitter.emit("recording-stopped",blob);
//another view subscribes to recording-stopped and visualizes the data
resovle(blob);
};
}
else {
reject();
}
});
}
//Vue UI - invoked when pressing stop button
protected stopRecording()
{
this.recording=false;
const stopRecording=()=>{
this.controller.stopRecording().then(blob=>{
console.log("RECORDING STOPPED!")
const url = URL.createObjectURL(blob);
if (this._data.item)
{
this._data.item.src=url;
}
this._data.hidePlay=false;
this.emitter.emit("recording-stopped");
});
};
if (!this.isIos)
{
setTimeout(stopRecording,500);
}
else {
stopRecording();
}
}
Everything works perfectly in Firefox and Chrome on Android and Windows. Also works perfectly on Chromium edge. I've devised an unacceptable workaround on Safari iOS that works, partially. When the user presses the play button, it decodes the object URL into a data URL, performs the visualization and then sets the SRC to the data url instead of the object url. Play is also invoked when the data URL is returned, but it fails because it's done in an asynch method. When play is pressed a 2nd time, it plays but is not audible.
Giving an onscreen in-DOM HTMLAudioElement the data URL will play back the audio data and you can hear it. The unacceptable workaround is to use the default HTMLAudioElement.

Audio duration sometimes changes to near-zero in Safari

I am encountering a strange issue on Safari, both on MacOS and iOS. Initially it seemed like audio would sometimes just not play, but not emit any errors. After adding much logging I found that when when I call audio.play() the audio.duration property would accurately reflect the duration of the clip. The pause and then ended events will then almost immediately be emitted and, after that the duration for the same clip is suddenly somewhere between 0.001 and 0.003 seconds.
I am retaining a reference to the audio element and reusing it to play the same audio multiple times. It almost always works the first time, but after the first playthrough, on about 50% of subsequent plays the symptoms described above will present themselves.
The code where I play the audio is below:
// In the constructor for the class managing audio playback:
this.mediaElement.addEventListener('pause', e => {
console.log('Media paused', this.mediaElement.duration); // This shows the very short duration if the audio did not play
});
// In the function in the class that plays the media.
try {
await this.mediaElement.play();
console.log('Media is playing', this.mediaElement.duration); // This shows an accurate duration
this.state = Playable.state.PLAYING;
} catch(e) {
console.error('Playing media failed:', e);
if (e && e.name === 'NotAllowedError') {
ErrorHandler.playbackNotAllowed(e);
this.state = Playable.state.PAUSED;
return;
} else {
this._fail(e);
this._fakePlay();
}
}
As you can see, I'm not doing anything strange or complicated here. So far I haven't been able to figure out why the duration would change in this way only after playing the audio. Is this a known bug, or is there something I may be doing that could cause this behavior. The closest thing I can think of is that sometimes I will set the currentTime to 0 if I need to start playing the audio from the beginning, but that should not change the duration.

Programmatically trigger the action that a headphone pause button would do

I am trying to find a way to pause any playing media on the device, so I was thinking of triggering the same logic that is fired when a user press the headphone "middle button"
I managed to prevent music from resuming (after I pause it within my app, which basically start an AVAudioSession for recording) by NOT setting the AVAudioSession active property to false and leave it hanging, but I am pretty sure thats a bad way to do it. If I deactivate it the music resumes. The other option I am thinking of is playing some kind of silent loop that would "imitate" the silence I need to do. But I think if what I am seeking is doable, it would be the best approach as I understood from this question it cannot be done using the normal means
func stopAudioSession() {
let audioSession = AVAudioSession.sharedInstance(
do {
if audioSession.secondaryAudioShouldBeSilencedHint{
print("someone is playing....")
}
try audioSession.setActive(false, options: .notifyOthersOnDeactivation)
isSessionActive = false
} catch let error as NSError {
print("Unable to deactivate audio session: \(error.localizedDescription)")
print("retying.......")
}
}
In this code snippet as the function name implies I set active to false, tried to find other options but I could not find another way of stopping my recording session and prevent resume of the other app that was already playing
If someone can guide me to which library I should look into, if for example I can tap into the H/W part and trigger it OR if I can find out which library is listening to this button press event and handling the pause/play functionality
A friend of mine who is more experienced in IOS development suggested the following workaround and it worked - I am posting it here as it might help someone trying to achieve a similar behaviour.
In order to stop/pause what is currently being played on a user device, you will need to add a music player into your app. then at the point where you need to pause/stop the current media, you just initiate the player, play and then pause/stop it - simple :)
like so:
let musicPlayer = MPMusicPlayerApplicationController.applicationQueuePlayer
func stopMedia(){
MPMediaLibrary.requestAuthorization({(newPermissionStatus: MPMediaLibraryAuthorizationStatus) in
self.musicPlayer.setQueue(with: .songs())
self.musicPlayer.play()
print("Stopping music player")
self.musicPlayer.pause()
print("Stopped music player")
})
}
the part with MPMediaLibrary.requestAuthorization is needed to avoid an authorisation error when accessing user's media library.
and of course you will need to add the Privacy - Media Library Usage Description
key into your Info.plist file

iOS audio system. Start & stop or just start?

I have an app, where audio recording is the main and the most important part. However user can switch to table view controller where all records are displayed and no recording is performed.
The question is what approach is better: "start & stop audio system or just start it". It may seem obvious that the first one is more correct, like "allocate when you need it, deallocate when used it". I will show my thoughts on this question and I hope to find approval or disapproval with arguments among skilled people.
When I constructed AudioController.m the first time I implemented methods to open/close audio session and to start/stop audio unit. I wanted to stop audio system when recording is not active. I used the following code:
- (BOOL)startAudioSystem {
// open audio session
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
NSError *err = nil;
if (![audioSession setActive:YES error:&err] ) {
NSLog(#"Couldn't activate audio session: %#", err);
}
// start audio unit
OSStatus status;
status = AudioOutputUnitStart([self audioUnit]);
BOOL noErrors = err == nil && status == noErr;
return noErrors;
}
and
- (BOOL)stopAudioSystem {
// stop audio unit
BOOL result;
result = AudioOutputUnitStop([self audioUnit]) == noErr;
HANDLE_RESULT(result);
// close audio session
NSError *err;
HANDLE_RESULT([[AVAudioSession sharedInstance] setActive:NO withOptions:AVAudioSessionSetActiveOptionNotifyOthersOnDeactivation error:&err]);
HANDLE_ERROR(err);
BOOL noErrors = err == nil && result;
return noErrors;
}
I found this approach problematic because of the following reasons:
Audio system starts with delay. That means, recording_callback() not called for some time. I suspect it is AudioOutputUnitStart, which is responsible for that. I tried to comment out the line with this function call and move it to initialization. the delay was gone.
If user performs switching between recording view and table view very very fast (audio system's starts and stops are very fast too), it cause the death of media service (I know that observing AVAudioSessionMediaServicesWereResetNotification could help here, but it is not the point).
To resolve these issues I modified AudioController.m with other approach which I managed to discover: start audio system when application becomes active and do not stop it before the app is terminated In this case there are also several issues:
CPU usage
If audio category is set to recording only, then no other audio could be played when user explores table view controller.
The first one surprisingly is not a big deal, if cancel any kind of processing in recording_callback() like this:
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
AudioController *input = (__bridge AudioController*)inRefCon;
if(!input->shouldPerformProcessing)
return noErr;
// processing
// ...
//
return noErr;
}
By doing this CPU usage equals to 0% on real device, when no recording is needed and no other actions are performed.
And the second issue can be solved by switching audio category to RecordAndPlay and enable mixing or just ignore the problem. For example in my case app requires mini Jack to be used by external device, so no headphones can be used in parallel.
Despite all this, the first approach is more close to me since I like to close/clean every stream/resource when it is no longer needed. And I want to be sure that there is indeed no other option than just start audio system. Please make me sure that I'm not the only one who came to this solution and it is the correct one.
The key to solving this problem is to note that the audio system actually runs in another (real-time) thread. And you can't really stop and deallocate something running in another thread exactly when you (or the app's main UI thread) "don't need it", but have to delay in order to allow the other thread to realize it needs to do something and then finish and clean up itself. This can potentially take up to many 100's of milliseconds for audio.
Given that, strategy 2 (just start) is safer and more realistic.
Or perhaps set a delay of many many seconds of non-use before attempting to stop audio, and possibly another short delay after that before attempting any restart.

recording against a metronome of set length using remote IO

I was able to create the exact functionality I wanted to avaudioplayer and avaudiorecorder but of course experienced latency problems. So after reading pretty much every article on the web and reviewing stacks of sample code, I'm still not sure how to achieve the following:
User chooses to record a sample 2 bars long (4 beats per bar) with a pre-roll/count-in
User clicks record
A metronome starts which counts in 4 beats (accent on the first beat)
The app automatically starts recording on the start of the next bar
The app automatically turns off recording at the end of the 3rd bar (the 2 bars + the pre-roll)
The user can then playback their recording or delete it and start again.
So, with avaudioplayer and avaudiorecorder I simply created a 'caf' using audacity with a metronome set at the correct bpm (bpm is set for the app). I then setup and play the avaudioplayer and using the audiodidfinishsuccessfully delegate method, performed some logic to start the recorder, restart the player, maintain a loop count etc. to turn off recording and audio.
As I mentioned, I was pretty much able to achieve the user experience I am after but the latency problems are not acceptable.
I have been working with audio units and the remote IO and have setup a project with a playback callback and recorder callback etc. but now face the problem of working how to make this work based on the description above. I am trying to work out the following things for starters:
If I create a 1 beat caf file, how could I make use of audio units and remote IO to play x amount of beats and then stop?
How could I do the pre-roll and start the recording callback after 4 beats
Can anyone give me some ideas or point me in the right direction. As I have mentioned, I have already done a stack of research including buying the core audio book, reading every article on atastypixel.com, timbolstad.com etc and trawled through the apple docs.
Thanks in advance for your help.
I start an NSTimer. Use values based on BPM (Beats per Minute) / 60. So if user wants to record a 2 bar file with a count in might do something like this:
//timer interval=100BPM/60secs per minute
timerInterval=100/60;
metroTimer = [NSTimer scheduledTimerWithTimeInterval:timerinterval target:self selector:#selector(blinkMetroLight) userInfo:nil repeats:YES];
- (void)blinkMetroLight
{
if(beatNumber == 0)
{
beatNumber = 1;
}
else if (beatNumber == 5)
{
[self audioProcessorStart];
}
if (beatNumber == 8)
{
[self audioProcessorStop];
[metroTimer invalidate]; metroTimer = nil;
}
beatNumber++
}

Resources