Play Webrtc Audio message when IOS app is in background - ios

Hi this might be duplicate to this question. But I did not find right answer for this. I have developed an app like zello push to talk using react-native. I want to play the audio message automatically without any user interaction when the app is in background or killed. Whenever any user sending real time audio message, I am sending a push notification to ios app and after receiving notification I am invoking a function which establishes socket connection with the webrtc server and then join in the room in which webrtc audio broadcasting is happening. Now I checked that socket is connected and also joining in room successfully done after receiving push notification when app is in background. But there I could not hear any audio and received following error message.
AURemoteIO.cpp:1668 AUIOClient_StartIO failed (561145187)
Then I have set AVAudioSessionCategoryPlayback and that error gone. But this time also did not hear any audio sound.
The app is working fine when it is in foreground. I am using react-native and this is happening for IOS app only. Any help is appreciated. In xcode I have enabled push-notifications, background-fetch, background-airplay etc.

You need to set your app Capabilities Background Modes (Audio and AirPlay).
To Enable this select your iOS project in iOS then go to Signing & Capabilities tab.
Check for Background Modes and select Audio, AirPlay and Picture In Picture option.
Also, set your AVAudioSession category to AVAudioSessionCategoryPlayback and set it active.
Example:
do {
try AVAudioSession.sharedInstance().setCategory(AVAudioSession.Category.playback, options: AVAudioSession.CategoryOptions.mixWithOthers)
NSLog("Playback OK")
try AVAudioSession.sharedInstance().setActive(true)
NSLog("Session is Active")
} catch {
NSLog("ERROR: CANNOT PLAY MUSIC IN BACKGROUND. Message from code: \"\(error)\"")
}

Related

Background Audio is enabled but not needed

following the examples, I enabled the Background Audio capability in my app, even though I do not need it. I want simple playback only, with midi triggering sounds (sf2) from the app bundle only, in foreground only. I got everything working and submitted my first version to the App Store and I get complaints from the reviewer saying that if I don't use background audio (so they can't test it) then I should switch it off in the Capabilities. If I do that my foreground audio triggering stops working. Can anyone help in what is the proper way configure AKSettings and the app in this case, so I could pass the app store review?

MPRemoteCommandCenter and Play Pause Update with the AVSpeechSynthesizer

I am currently using AVSpeechSynthesizer for Text to Speech. Category used for the playback is AVAudioSessionCategoryPlayback and AVAudioSession is set to Active YES.
During the start of the play, [TTS] TTSPlaybackCreate unable to initialize dynamics: -3000 in the Xcode console. When i pause the playback i get [TTS] _BeginSpeaking: couldn't begin playback.
My major issue is MPRemoteCommandCenter doesn't get updated to pause when TTS stopped.
For Stop functionality, I am using this code;
BOOL speechStopped = [self.ttsSpeechSynthesizer stopSpeakingAtBoundary:AVSpeechBoundaryImmediate];
if(!speechStopped) {
[self.ttsSpeechSynthesizer stopSpeakingAtBoundary:AVSpeechBoundaryWord];
}
I had Airplay connected to an Airplay station.
I had a similar issue after updating iOS to the latest version on my phone.
I spent much time trying to understand why my app stopped talking using TextToSpeech while all worked before and code seemed ok.
Siri was talking aloud fine, and the sound in other apps worked as well
Mine was giving no error message in the code and the following in the device log:
Error (730) / LearnByHeart.iOS(TTSSpeechBundle): TTSPlaybackCreate unable to initialize dynamics: -3000
Rebooting the phone did not help.
As funny as it is, all got resolved by turning the physical sound button off and back on.
Hope this saves someone a day

2.16: Multitasking apps may only use background services for their intended purposes: VoIP etc

I created an iBeacon app to detect beacons using phonegap/cordova. How the app works is that once it detects a nearby beacon and the app is closed or running in the background i use local notification which includes an audio (common apple notication sound). I don't know why apple rejects it with the below comment:
2.16: Multitasking apps may only use background services for their intended purposes: VoIP, audio playback, location, task completion, local notifications, etc.
Next Steps
The audio key is intended for use by applications that provide audible content to the user while in the background, such as music player or streaming audio applications. Please revise your app to provide audible content to the user while the app is in the background or remove the "audio" setting from the UIBackgroundModes key.
Your app does not provide continuous streaming audio. For details on delivering an audible alert for a local notification, please refer to the About Local and Remote Notifications Programming Guide.
We hope you will consider making the necessary changes to be in compliance with the App Store Review Guidelines and will resubmit your revised binary.
Apple's comment is quite clear. Your app is incorrectly specifying that it needs the audio background mode. But it doesn't. That mode is only to be used to play audio (such as music) while the app is in the background. It is not to be used just to get audio from a local notification.
Remove the audio background mode setting from your app and Apple will be happy.
Even our App received a rejection last week for the same reason.
We didn't even need to submit a new binary. All we did was explained to the reviewer why we were using each of the background modes which we had listed in our plist. We did this by uploading a video in which we explained the purpose of each background mode. And then provided a link of this video in the resolution centre in iTunes connect. The reviewer had a look at it and later approved our app.
So you have to do a similar thing. If you need the background mode(audio in your case). Just explain it's intended purpose to the reviewer in the resolution centre. Else, just upload a new binary with the audio background mode removed from the plist.

Exporting videos with AVAssetExportSession in background the background

I am working on an iOS 8 app which records a video, modifies it using AVMutableVideoComposition and exports the result with a AVAssetExportSession calling exportAsynchronouslyWithCompletionHandler.
Things work fine as long as the app is in the foreground. When a user pushes the home button and sends the app into the background before the export has finished however, the following error is raised:
Error Domain=AVFoundationErrorDomain Code=-11847 "Operation Interrupted"
UserInfo=0x174271c40 {NSUnderlyingError=0x170246c90 "The operation couldn't
be completed. (OSStatus error -12125.)", NSLocalizedRecoverySuggestion=Stop
other operations and try again., NSLocalizedDescription=Operation
Interrupted}
It looks like others have experienced the same error with Audio: AVAssetExportSession working in background
I have tried all kind of things like requesting more time for the execution using beginBackgroundTaskWithExpirationHandler (does not work because the export is interrupted before the time runs out), adding Audio as a Background Mode in the app capabilities (does not work, maybe because I use AVMutableVideoComposition?), starting the export in a custom queue outside the main thread, but nothing works.
Even trying to restart the export in case it has not completed when the app enters background fails. I tried to listen to UIApplicationDidEnterBackgroundNotification and restart the export after the app is already in the background but I get the same error.
The discussion in this thread hints that it could be because AVMutableVideoComposition can not be used in the background:
Starting AVAssetExportSession in the Background
Is this still true for iOS 8? Is there any workaround or does anyone know how I could accomplish the desired functionality (editing and exporting a video a user has just recorded even if he sends the app to background)?
Any help is highly appreciated, I have spent many hours on this problem.
iOS 13 tested. Play silent sound while exporting.
Audio session should be configured like:
try AVAudioSession.sharedInstance().setCategory(.playback, mode: .default, options: [.mixWithOthers, .allowAirPlay])
try AVAudioSession.sharedInstance().setActive(true)
Background mode required.
Unfortunately, it is still true for iOS 8. You can not export when app is in background but it is possible to start export as soon as it comes to foreground.
Just save the AVMutableCompostion before your app goes to background. This can be done in applicationDidEnterBackground. Then you can start export again in applicationDidBecomeActive.
Note: You have to start the export all over again. It can not be resumed if not finished before entering background.

Playing audio files in background in iOS

I'm trying to find responses to this in Apple's documents, but I don't:
1) Is it technically possible to start playing an audio file when an app goes to background?
2) If technically possible... is it "legal"? Would Apple approve an app that starts playing an audio file without an explicit interaction of the user?
Thanks in advance
An app can start playing an audio file in the background if it is already playing audio before the app goes into the background, has the appropriate audio background mode capabilities key set, and mixes the new audio file into an already playing audio unit, queue or player without stopping. There are plenty of approved apps in the App store that do this (e.g. they sequence through playlists).
An app can't start playing an audio file in the background if it is not already playing or recording audio, as the OS will take away and disable the app's audio session as it goes into the background.
Reports are that apps that play "silence" to get around this restriction are currently being rejected by Apple, although some legacy examples (alarm clocks?) may still exist in the iOS App store. But for private (non-App-store) apps, this trick currently seems to work just fine.
Also, remote notifications can start playing very short audio files contained within an app's bundle, even if the app is in the background without an active audio session.

Resources