iOS text-to-speech in background - ios

I am having an intermittent (aargh!) problem playing Text-to-Speech in the background, triggered from Apple Watch. I have properly set up the background mode, the AVSession category, and the WatchKitExtensionRequest handler. (See below.) I had this working before, and can't figure out what changed. (Could it be iOS 9 has issues? "Before" means, among other things, iOS 8.)
The problem is this: when the app gets the request from the Watch and the app is either in the background or the phone is sleeping (locked), the speech sometimes plays right away, and other times doesn't play until the app is brought to the foreground. The OS seems to be sometimes queuing the audio, and sometimes not. I can't find any common thread between success and failure cases. I can verify with logging that the call to speakUtterance() is being made in all situations. But its behavior varies, apparently randomly. The only clue is that it might be the case that the longer the app is in the background, the less likely it is to speak right away.
This is making me pull my hair out. Suggestions welcome.
In info.plist:
Required background modes: App plays audio or streams audio/video using AirPlay
In AppDelegate.application:didFinishLaunching:withOptions():
do {
try AVAudioSession.sharedInstance().setCategory(
AVAudioSessionCategoryPlayback,
withOptions:.DuckOthers
)
try AVAudioSession.sharedInstance().setActive(true)
} catch let error as NSError {
// etc...
}
In AppDelegate.application:handleWatchKitExtensionRequest...():
var bgTaskId:UIBackgroundTaskIdentifier = 0
bgTaskId = application.beginBackgroundTaskWithName(
"Prose WKE handler",
expirationHandler: {
application.endBackgroundTask(bgTaskId)
}
)
//... Post notification to call Text-to-Speech
application.endBackgroundTask(bgTaskId)

Here's a workaround: play a second snippet of sound (I used a half-second of silence), using AVAudioPlayer, right after the call to speakUtterance(), This seems to "jog the pipeline".

Related

MPMusicPlayerController fails to play Apple Music songs

I am using an instance of MPMusicPlayerController.systemMusicPlayer to enqueue an array of store IDs. This has worked for months now. Earlier today I updated to iOS 14.3, and the player is now failing to play songs.
The code below is the minimal amount needed to replicate the bug:
// note: repo using any play method you want
let player = MPMusicPlayerController.systemMusicPlayer
let descriptor: MPMusicPlayerStoreQueueDescriptor?
func setup() {
let storeIDs: [String] = ["lorem", "ipsum"] // fetch real IDs from the API
descriptor = MPMusicPlayerStoreQueueDescriptor(queue: storeIDs)
}
func play() {
self.player.setQueue(with: descriptor!)
self.player.play()
}
// Expected: plays song with store ID "lorem"
// Actual: app freezes and I see error logs
When I play a song, instead of playing it, the app completely freezes (meaning it doesn't respond to user interaction), and I see the following logs:
[SDKPlayback] ASYNC-WATCHDOG-1: Attempting to wake up the remote process
[SDKPlayback] SYNC-WATCHDOG-1: Attempting to wake up the remote process
[SDKPlayback] ASYNC-WATCHDOG-2: Tearing down connection
[SDKPlayback] SYNC-WATCHDOG-2: Tearing down connection
The MPMusicPlayerController plays music just fine on iOS 14.2.
Can anybody confirm or shed some light on what's going on here?
I filed a TSI/bug report with Apple in the meantime.
I can confirm the issue is still present, but after doing some testing I found out that what it's actually doing is blocking the main thread from executing. So a workaround that at least worked for me is executing the play function inside the background thread like this:
DispatchQueue.global(qos: .background).async {
player.prepareToPlay()
player.play()
}
Now the issue may still be present sometimes but i found that moving it to the background thread makes it way less tedious and less often. Also adding prepare to play also seems to make it work 99% of the time.

How to create an iOS alarm clock that runs in the background properly

I would like to insert an alarm clock function in an iOS app I am developing, and as a reference, I installed a popular App called "Alarmy."
I managed to keep my app running in the background, just using AVAudioSession properties; however, I noticed that the app consumes a lot of battery during the phone sleep.
After some testing, I think this is due to the app activating the speakers (and keeping them ON) immediately after the AVAudioSession activation.
Even if there is no actual sound playing until the audioPlayer.play(atTime: audioPlayer.deviceCurrentTime + Double(seconds)) is triggered, if I get very very close to my iPhone 7 speakers, I can hear the little buzzing sound that indicates that the speakers are ON. This implicates that the speakers are playing an "empty sound" de facto.
This buzzing sound does not exist when I set the alarm with Alarmy; it just starts playing when it is supposed to.
I did not find any other way to maintain my app in the background and play an alarm sound at a specified time. There are Local Notifications, of course, but they do not allow to play a sound when the phone is silenced.
Going back to "Alarmy," I've seen that they are not only able to play a background alarm without any need to activate the speakers first, but they are also able to put the volume at the max level in the background. Are they maybe triggering some other iOS background mode to achieve those, perhaps using Background Fetch or Processing in some clever way? Is there any known way to replicate those behaviors?
Thanks in advance!
Here is the current code I use to set the alarm:
private func setNewAlarm(audioPlayer: AVAudioPlayer, seconds: Int, ringtone: String) {
do {
self.setNotificationAlarm(audioPlayer: audioPlayer, seconds: seconds, ringtone: ringtone, result: result)
//This calls the method I use to set a secondary alarm using local notifications, just in case the user closes the app
try AVAudioSession.sharedInstance().setActive(false)
try AVAudioSession.sharedInstance().setCategory(.playback, options: [ .mixWithOthers])
try AVAudioSession.sharedInstance().setActive(true)
} catch let error as NSError {
print("AVAudioSession error: \(error.localizedDescription)")
}
audioPlayer.prepareToPlay()
audioPlayer.play(atTime: audioPlayer.deviceCurrentTime + Double(seconds))
result(true)
}

Audio interruption when iOS application is recording in background

My iPad sound application (iOS 7.1) is able to record while in background. Everything is ok as far as recording is not interrupted while in background. For example if one's has the (stupid?) idea to start listening music while recording something.
I tried to manage this interruption in different ways, with no success. The problem is that the
- (void)audioRecorderEndInterruption:(AVAudioPlayer *)p withOptions:(NSUInteger)flags
is never fired when application was in background as the interruption occurred. I then tried, in another solution, to implement the AVAudioSessionInterruptionNotification and a handleInterruption: method as
- (void) viewDidAppear:(BOOL)animated
{
......
AVAudioSession *session=[AVAudioSession sharedInstance];
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(handleInterruption:)
name:AVAudioSessionInterruptionNotification
object:session];
.......
}
// interruption handling
- (void)handleInterruption:(NSNotification *)notification
{
try {
UInt8 theInterruptionType = [[notification.userInfo valueForKey:AVAudioSessionInterruptionTypeKey] intValue];
NSLog(#"Session interrupted > --- %s ---\n", theInterruptionType == AVAudioSessionInterruptionTypeBegan ? "Begin Interruption" : "End Interruption");
.... MANAGE interruption begin
interruptionBeganWhileInBackgroundMode = TRUE;
}
if (theInterruptionType == AVAudioSessionInterruptionTypeEnded) {
.... MANAGE interruption end
}
} catch (CAXException e) {
char buf[256];
fprintf(stderr, "Error: %s (%s)\n", e.mOperation, e.FormatError(buf));
}
}
In this case the handleInterruption: is fired when interruption begins, but it is not when interruption ends (shoulld be fired with theInterruptionType set to AVAudioSessionInterruptionTypeEnded)
To circumvent this problem, I decided to set a flag (interruptionBeganWhileInBackgroundMode) when interruption begins to get the application informed that an interruption has occurred when coming foreground. So that I can manage the end of interruption.
It may seem clever, but it is not! Here is why...
I tried two implementations.
Solution 1. Put a [recorder pause] in handleInterruption: when interruption begins, and manage a [recorder record] in the comingForeground: method when the interruption flag is set.
In this situation, the recording is going on immediately when application comes back to foreground, BUT instead of resuming, the recorder erase the file and restart a new recording, so that everything recorded before interruption is lost.
Solution 2. Put a [recorder stop] in handleInterruption: when interruption begins, to save the file, to be sure to preserve the recorded data. This recording is saved, BUT when application is coming foreground, it stalls for about ten seconds before the user can interact again, as if there was a process (the file saving?) that keep the UI frozen.
Last point: I have exactly the same problems in iPhone version of this application: when application is foreground and a phone call occurs, everything goes OK. But when a phone call occurs as my application is background I see the same bad behaviour as in iPad's version.
I notice that Apple's Voice Memo app in iOS7 correctly manage these background interruptions (it stops and saves the recording) despite it displays a 00:00:00 file length when coming foreground. The iTalk application manages it perfectly, automatically resuming recording when coming foreground.
Did anybody find a workaround for the audio interruption management for backgrounded recording applications? I found plenty of people looking for that in many developer websites, but no answer... Thanks!
I have been going through the same issue myself. It seems as if there is a bug in the iOS7 AVAudioRecorder in how it deals with interrupts. Instead of pausing as I believe the documentation says that is should, it closes the file. I have not been able to figure out what is stalling the app when it comes back to the foreground. In my case, I would see AVAudioRecorder finish (with the success flag set to NO), after 10 seconds.
I ended up rewriting the audio recorder using Audio Queues. I found some sample code here (git#github.com:vecter/Audio-Queue-Services-Example.git) that helped with setting it up in an Objective-C environment and the Apple SpeakHere demo has some code to handle the interrupt notifications.
Essentially, I am stopping the recording on interrupt began and opening an alert for the user to save the file. This alert is deferred until UIApplicationDidBecomeActiveNotification is passed if the interrupt started while the app was in the background.
One other thing to note, there seems to be a minor bug in Audio Queues that the AudioQueueStart method will return -50 sometimes. If you add
AudioSessionInitialize(NULL, NULL,nil,(__bridge void *)(self));
UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord;
AudioSessionSetProperty(kAudioSessionProperty_AudioCategory,
sizeof(sessionCategory),
&sessionCategory
);
AudioSessionSetActive(true);
Before any AudioQueue methods, the error goes away. These methods are marked as deprecated but seem to be necessary.
Don't know / think this is going to be relevant to the original author of the topic, but here is my experience with it:
I ended up here because of the AVAudioSessionInterruptionNotification thing; the .began flavour came as expected, but the .ended one did not.
In my case, it happened because I was using two AVPlayer instances: one to play music, and the other one to play silence while the first struggled to start streaming a new track (otherwise iOS would suspend my app while in background, if next track loading did not happen fast enough).
Turns out it's not the brightest solution, and it somehow messes up the notification system. Giving up the second AVPlayer (the silence playing one) resulted in .ended being triggered as expected.
Of course, I found the answer / solution for the notification problem, but I'm now left with the old one... :-)

MPMoviePlayerController blocking the Main Thread

I am using the MPMoviePlayerController(MPMPC) to stream audio into an application and thats working really fine. Just with one exception, during low network connectivity the app becomes unresponsive.
Now I have even tried to use AVPlayer too but with more or less same experience. And for some reason I cannot find any issues related to this on the internet. So I am not sure if this is from my end or is it how MPMoviePlayerController behaves during low connectivity.
I even tried to log any function that is being called after giving URL to the MPMPC but none of the functions are called.
I have used below three notification to get events of the MPMPC
MPMoviePlayerLoadStateDidChangeNotification
MPMoviePlayerPlaybackDidFinishNotification
MPMoviePlayerPlaybackStateDidChangeNotification
Once the available networking bandwidth is becoming too low to keep up proper playback, MPMoviePlayerController will trigger the MPMoviePlayerLoadStateDidChangeNotification and the loadState will have MPMovieLoadStateStalled set.
You may then mask the load-state within your notification handler and run any actions needed by your app for this state:
if ((movieController_.loadState & MPMovieLoadStateStalled) == MPMovieLoadStateStalled)
{
NSLog(#"playback stalled - make sure we don't block now!");
}
Once the player has recovered, once again the MPMoviePlayerLoadStateDidChangeNotification is triggered and the loadState property will have the bits for MPMovieLoadStatePlaythroughOK set:
if ((movieController_.loadState & MPMovieLoadStatePlaythroughOK) == MPMovieLoadStatePlaythroughOK)
{
NSLog(#"playback should run uninterrupted from now on.");
}
However, I never experienced any interface slowdowns of my app caused by the MPMovieLoadStateStalled state. I'ld say that must be your code acting weird, it is not MPMoviePlayerController as I know it. Additionally, those notifications are always sent, I never experienced scenarios in which they were not properly triggered.
I can only recommend to recreate this issue within a minimal test-case and work your way up from that one towards your app (possibly from both sides, test-case and your app).
For simulating bandwidth breakdowns, I would recommend using Charles proxy.

AVAudioSessionDelegate called at endInterruption, but beginInterruption not called

I'm setting up an AVAudioSession when the app launches and setting the delegate to the appDelegate. Everything seems to be working (playback, etc) except that beginInterruption on the delegate is not being called when the phone receives a call. When the call ends endInterruption is being called though.
The only thought I have is that the audio player code I'm using used to be based on AVAudioPlayer, but is now using AVPlayer. The callbacks for the AVAudioPlayer delegate for handling interrupts are still in there, but it seems odd that they would conflict in any way.
Looking at the header, in iOS6, it looks like AVAudioSessionDelegate is now deprecated.
Use AVAudioSessionInterruptionNotification instead in iOS6.
Update: That didn't work. I think there's a bug in the framework.
Yes, in my experience, beginInterruption, nor the newly documented AVAudioSessionInterruptionNotification work properly. What I had to do was track the status of the player using a local flag, then handle the endInterruption:withFlags: method in order to track recovery from interruptions.
With iOS 6, the resuming from an interruption will at least keep your AudioPlayer in the right place, so there was no need for me to store the last known play time of my AVAudioPlayer, I simply had to hit play.
Here's the solution that I came up with. It seems like iOS 6 kills your audio with a Media Reset if an AVPlayer stays resident too long. What ends up happening, is the AVPlayer plays, but no sound comes out. The rate on the AVPlayer is 1, but there's absolutely no sound. To add pain to the situation, there's no error on either the AVAudioSession setActive, nor the AVPlayer itself that indicates that there's a problem.
Add to the fact that you can't depend on appWillResignActive, because your app may already be in the background if you're depending on remote control gestures at all.
The final solution I implemented was to add a periodic observer on the AVPlayer, and record the last known time. When I receive the event that I've been given back control, I create a new AVPlayer, load it with the AVPlayerItem, and seekToTime to the proper time.
It's quite an annoying workaround, but at least it works, and avoids the periodic crashes that were happening.
I can confirm that using the C api, the interruption method is also not called when the interruption begins; only when it ends
(AudioSessionInitialize (nil, nil, interruptionListenerCallback, (__bridge void *)(self));
I've also filed a bug report with apple for the issue.
Edit: This is fixed in iOS 6.1 (but not iOS 6.0.1)
Just call:
[[AVAudioSession sharedInstance] setDelegate: self];
I just checked on my iPhone 5 (running iOS 6.0) by setting a breakpoint in the AudioSessionInterruptionListener callback function that was declared in AudioSessionInitialize(), and this interrupt callback does, in fact, get called when the app has an active audio session and audio unit and is interrupted with an incoming phone call (Xcode shows the app stopped at the breakpoint at the beginning of the interruption, which I then continue from).
I have the app then stop its audio unit and de-activate its audio session. Then, on the end interruption callback, the app re-activates the audio session and restarts the audio unit without problems (the app is recording audio properly afterwards).
I built a brand new audio streaming (AVPlayer) application atop iOS 6.0.x and found the same problem.
Delegates are now deprecated and we have to use notifications, that's great, however here's my findings:
During an incoming phone call I get only AVAudioSessionInterruptionTypeEnded in my handler, along with AVAudioSessionInterruptionOptionShouldResume. Audio session gets suspended automatically (audio fades) and I just need to resume playback of AVPlayer.
However when attempting to launch a game, such as CSR Racing, I oddly get the dreaded AVAudioSessionInterruptionTypeBegan but no sign when my application can resume playback, not even killing the game.
Now, this may depend on other factors, such as my audio category (in my case AVAudioSessionCategoryPlayback) and the mixing settings of both applications (kAudioSessionProperty_OverrideCategoryMixWithOthers), I'm not sure, but definitely I see something out of place.
Hopefully others reported that on 6.1beta this is fixed and I yet have to upgrade, so we'll see.

Resources