Over the past few years I have steadily developed a complete WebRTC based browser Phone using the SIP protocol. The main SIP toolbox is SIPJS (https://sipjs.com/), and it provides all the tools one needs to make and receive calls to a SIP based PBX of your own.
The Browser Phone project: https://github.com/InnovateAsterisk/Browser-Phone/ gives SIPJS it's full functionality and UI. You can simply navigate to the phone in a browser and start using it. Everything will works perfectly.
On Mobile
Apple finally allow WebRTC (getUserMedia()) on WKWebView, so it wasn't long before people started to ask how it would work on mobile. And while the UI is well suited for cellphones and tablets, just the UI isn't enough now days to be a full solution.
The main consideration is that a mobile app is typically one that has a short lifespan, in that you can't or don't leave it running in the background like you can or would with the Browser on a PC. This presents a few challenges to truly making the Browser Phone mobile friendly. iOS is going to want to shutdown the app as soon as its not the front most app - and rightly so. So there are tools for handling that, like Callkit & Push Notifications. This allows the app to be woken up, so that it can accept the call, and notify the user.
Just remember, this app is created by opening a UIViewController, adding a WKWebView, and navigating to the phone page. There is full communication between the app and the html & Javascript, so events can be passed back and forth.
WKWebView & AVAudioSession Issue:
After a LOT of reading unsolved forum posts, it's clear that AVAudioSession.sharedInstance() is simply not connected to the WKWebView, or there is some undocumented connection.
The result is that if the call starts from the app, and is sent to the background, the microphone is disabled. Clearly this isn't an option if you are on a call. Now, I can manage this limitation a little, by putting the call on hold when the app is sent to the background - although this would be confusing to the user and a poor user experience.
However, the real issue is that if the app was woken from Callkit, because the app never goes to the foreground (because Callkit is), the microphone isn't activated in the first place, and even if you do witch to the app, it doesn't activate even after that. This is simply an unacceptable user experience.
What I found interesting is that if you simply open up Safari Browser on iOS (15.x), and navigate to the phone page: https://www.innovateasterisk.com/phone/ (without making an app in xCode and loading it into a WKWebView), the microphone continues to work when the app is sent to the background. So how do Safari manage to do this? Of course this doesn't and can't salve the CallKit issue, but still interesting to see that Safari can make use of the microphone in the background, since Safari is built off WKWebView.
(I was reading about entitlements, and that this may have to be specially granted... im not sure how this works?)
The next problem with AVAudioSession is that since you cannot access the session for WkWebView, you cannot change the output of the <audio> element, so you cannot change it from say speaker to earpiece, or make it use a bluetooth device.
It simply wouldn't be feasible to redevelop the entire application using an outdated WebRTC SDK (Google no long maintain the WebRTC iOS SDK), and then build my own Swift SIP stack like SIPJS and land up with two sets of code to maintain... so my main questions are:
How can I access the AVAudioSession of WKWebView so that I can set the output path/device?
How can I have the microphone stay active when the app is sent to the background?
How can I activate the microphone when Callkit activates the application (while the application is in the background)?
for 1) Maybe someone also is following this approach and can add some insight/correct wrong assumptions: The audio in a WebRTC site is represented as a Mediastream. Maybe it is possible to get that stream from without the WKWebView and play it back within the app somehow ? This code should pass on some Buffers, but they are empty when they arrive over in swift:
//javascript
...
someRecorder = new MediaRecorder(audioStream);
someRecorder.ondataavailable = async (e) =>
{
window.webkit.messageHandlers.callBackMethod.postMessage(await e.data.arrayBuffer());
}
mediaRecorder.start(1000);
and then in swift receive it like
//swift
import UIKit
import WebKit
class ViewController: UIViewController, WKScriptMessageHandler {
...
let config = WKWebViewConfiguration()
config.userContentController = WKUserContentController()
config.userContentController.add(self, name: "callBackMethod")
let webView = WKWebView(frame: CGRect(x: 0, y: 0, width: 10, height: 10), configuration: config)
...
}
func userContentController(_ userContentController: WKUserContentController, didReceive message: WKScriptMessage) {
addToPlayingAudioBuffer(message.body)
//print(message.body) gives the output "{}" every 1000ms.
}
Related
A project I'm working on works mainly around a good UI/UX. The one issue I'm facing is answering calls on a locked iOS device.
Documentation:
Apple introduced the CallKit and PushKit features to allow access to the native call screen which is good, but not ideal in my case.
There is good literature on both of these components, e.g.
Receiving incoming calls
Responding to VoIP Notifications from PushKit
Very useful iOS 13 PushKit restrictions
Previously asked?
There are many questions about this issue, mostly centered around 2 years ago, which is why I am asking again.
Recommended with tutorials - Lock Screen UI with incoming Call
iOS - Can I open my VoIP app on answering call using Callkit
CallKit: Launch app when screen is locked
https://github.com/react-native-webrtc/react-native-callkeep/issues/319#issuecomment-758628836
Some extra CallKit/PushKit tutorials
https://www.nikola-breznjak.com/blog/ios/create-native-ios-app-can-receive-voip-push-notifications/
(Flutter Specific, can be used for native iOS too) https://github.com/masashi-sutou/flutter_ios_webrtc_kit
https://www.raywenderlich.com/1276414-callkit-tutorial-for-ios
https://agostini.tech/2019/06/23/receiving-incoming-calls-with-pushkit/
https://learn.vonage.com/blog/2021/01/28/handling-voip-push-notifications-with-callkit/
https://medium.com/#ykawanabe/system-calling-screen-with-callkit-77004b1224e5
The issue is non of these, as far as I have read, provide a mechanism to open an app directly after answering.
Viable solutions
The only way to do this with the current implementation is to use the last of the 6 buttons on the CallKit screen (optionally with an AppIcon), see image [Masked Image Icon]:
Examples:
GitHub example with App Icon & custom ring sound
Question:
I can't find any solution to open my iOS (Flutter app) when answering a VoIP call from a locked state - is this at all possible?
I don't know if you still need the solution, but for the sake of others finding this question: I found the answer in another Stackoverflow post.
Link: iOS CallKit. Jump directly to application
I am copying the answer here for clarity (credits go to 0awawa0) Note that the code example is in Swift.
So apparently, if you set. hasVideo property to true for your
CXCallUpdate when reporting the call, system will automatically open
your application when call is accepted. Anyway, this feature is not
mentioned anywhere in the documentation. Or, at least, I can not find
anything about it.
Here is the part of my code that reports new call now:
let update = CXCallUpdate()
update.remoteHandle = CXHandle(type: .generic, value: handle)
update.hasVideo = true // <- was false earlier, setting to true did the trick
update.supportsDTMF = false
update.supportsHolding = true
update.supportsGrouping = false
update.supportsUngrouping = false
callProvider.reportNewIncomingCall(with: uuid, update: update, completion { error in /*...*/ }
I am working on automating testing for Callkit interaction.
I managed to call from client A to B.
When B receives the call he is shown the Callkit screen.
Manually, I can click on accept and the call is established.
However when
I try to record the Callkit interaction, XCode is crashing and the application on the phone as well.
Am I breaking any security limitation by trying to automate the "Accept call" interaction?
As I am not able to access the Callkit screen element via interaction recording;
I am wondering if someone as any experience with that kind of automation or if someone could point me to the documentation that would describe the button ids for the Callkit screen.
You should determine the application of this screen. Probably, Springboard. Try inspecting with the Console app.
Then, initialise interaction with this app with let app = XCUIApplication(bundleIdentifier: )
Inspect app hierarchy with breakpoint and po app command to get labels and accessibility identifiers. Or use Accessibility Inspector.
Do required action is code (taps and/or swipes)
Thank you #Smart Monkey for putting me on the good track
I managed to accept the call using the Springboard app.
let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard")
springboard.buttons["Accept"].tap()
The call is then properly established.
Step 01: I can run my app (APP NAME : TextDetectAPP)
Step 02: User press Home button
Step 03:User open the Skype or any third party app (APP NAME : TextDetectAPPMove to Background state )
Spep04: User copy text (I need to get copy text from Skype or or any third party app )
Note :
UIpasteboard is working fine only user developed apps in iOS swift
Shall we have access rights to read text messages from Skype , whats up or any thirds parts apps in iOS swift?
Thanks in Advance
If I understand your question right, you want to get texts that were copied in other apps (Skype, FB etc.). You can do this using the UIPasteboard class. Usually texts are copied to the general pasteboard and you can access it without any problems (I've tried with Skype and Message (iOS) applications, it is working). When you open your app back, you can get the copied text from third party apps like this in the applicationDidBecomeActive delegate method:
func applicationDidBecomeActive(_ application: UIApplication) {
if let clipboardString = UIPasteboard.general.string {
print(clipboardString)
}
}
You can also use UIPasteboardChanged notification to listen to changes in pasteboard, however you cannot receive change notifications from other apps. Also your app may not be in background state all the time executing code (unless you enable some specific background modes like Audio, Airplay etc.). So there is no way for you to get the copied texts when you are in background. You either need to use the method above, (or) if your app supports background execution, you can have an NSTimer fire every n seconds that gets the pasteboard content.
I'm implementing an iOS app that handles a custom protocol.
Writing the method application(openURL:sourceApplication:annotation:) in my AppDelegate was easy but I'm left with a problem: I want that - once the user have done with the request - my app move to the background and send the user back to the caller sourceApplication (e.g. a browser, a QRCode reader, or any another app).
This is just like the difference between "tel:" and "telprompt:" url calls: in the former case the phone app remains active, in the latter case, after the call, the user is send back to the sourceApplication.
In order to let my app handle my custom protocol like "telprompt:" does, the only way I can think about is terminate the app once the user action is completed... but this is against iOS Human Interface Guidelines (they say "Don’t Quit Programmatically") and my app can be rejected by Apple.
On Android it is easy: you respond to an Intent with an Activity and when you call finish() on that activity the user is back to his previous app/browser/whatever.
Anyone knows a good way to achieve this on iOS?
Just to clarify:
my app don't call openUrl, it responds to openUrl requests from browser, QRCode reader, other apps;
I don't have to make phone calls;
when I handle a request I ask the user for some data, contact a server, and that's it: the interaction is finished and it would be very nice to drive the user back to previous app without let him use the home button.
I believe you should call openUrl when you are done, with the source app url in param.
That's what facebook does when you use the "connect with facebook" API.
I override my app's openURL-method to know when we're about to leave the app from an ABPersonViewController, the reason being that that class doesn't notify its delegate on all of the actions it presents to the user. If it did everything would be fine.
This worked flawlessly in iOS 7, but when trying this in iOS 8.1 it turns out that the ABPersonViewController doesn't call openURL for all its actions anymore. For instance, tapping a phone number gets me to the phone app without calling openURL. Tapping the SMS bubble on the other hand will call openURL.
When tapping the facebook profile entry (with the URL "fb://profile/1234567890") the log says that "Launch Services" doesn't have a registered handler for the scheme "fb". So I'm assuming that calls to Launch Services have replaced the calls to openURL. I can't find out much more about it other than that it's "a private API".
Is there a way to detect these calls? Or, is it possible to override handlers for known schemes like "tel" and "mailto" internally for my app? I basically just need to know when we're leaving the app and where we're going.