My web application is failing to gather WebRTC relay ICE candidates via a CoTURN server when using Safari 11 on iOS 11 (iPhone 5s & iPhone 7) or desktop. The web application (which establishes a one-way audio only WebRTC peer connection) works fine between the real browsers (Chrome and Firefox) either direct or via CoTURN relay, and I normally get 6-15 ICE candidates on these browsers.
I have a (frankly, unnecessary) call to getUserMedia on the receiving side, which allows host ICE candidates to be produced by Safari. (Note... the user must approve audio and/or video access before Safari will provide host Ice Candidates, even if on a receiving-only end. I'm past that hurdle, but just so you won't hit it too... This is out of "privacy" concerns.). Before I added the allow getUserMedia, I received no ICE. Now I receive two candidates. One with a private IPv4 and another with an IPv6. This is enough to get the app working properly when on the same machine or local network. So I'm pretty confident with other parts of the application code. I am not sure if my problem is with the application code or the CoTURN server.
Example of the ICE candidates received:
{"candidate":{"candidate":"candidate:622522263 1 udp 2113937151 172.27.0.65 56182 typ host generation 0 ufrag r23H network-cost 50","sdpMid":"audio","sdpMLineIndex":0,"usernameFragment":"r23H"}}
I pretty sure the RTCIceServer Dictionary for my RTCPeerConnection is inline with the following standards:
https://w3c.github.io/webrtc-pc/webrtc.html
https://www.rfc-editor.org/rfc/rfc7064
https://www.rfc-editor.org/rfc/rfc7065
And I've tried multiple variations of parameters:
// For Example:
var RPCconfig = {
iceServers: [{
urls: "turn:Example.live",
username: "un",
credential: "pw"
}]
};
// Or:
var RPCconfig = {
iceServers: [{
urls: "turns:Example.live",
username: "un",
credential: "pw",
credentialType: "password"
}, {
urls: "stun:Example.live"
}]
};
// And even more desperate attempts...
var RPCconfig = {
iceServers: [{
urls: "turn:Example.live?transport=tcp",
username: "un",
credential: "pw",
credentialType: "password"
}]
};
Here's an example of the signaling process log for an idea of what is going on. This is from the receiving side, which is Safari 11. The other browser was Chrome (compare 6 vs 2 ICE candidates). The state change refers to oniceconnectionstatechange.
SDP Offer received.
Sending signal SDP
Sending signal IceCandidate
Sending signal IceCandidate
ICE Candidate Received
4:08:25 AM State Change -> checking
ICE Candidate Received
ICE Candidate Received
ICE Candidate Received
ICE Candidate Received
ICE Candidate Received
4:08:40 AM State Change -> failed
CoTURN is configured quite liberally in terms of accepting every possible transport method as far as I am aware. It works well for providing ICE Candidates and as a relay for the other browsers.
Any direction would be greatly appreciated. Even if it is just a sample RTCIceServer Dictionary code that works or a proven TURN server to try.
Related
I am attempting to create an application that uses the new M1 chip's UWB sensor to identify distance and bearing of connected devices in 3-D space. I have gotten to interact with the sensor locally, and generate a token for connection. When another device enables it's own sensor though, there is an error that prevents connection. The relevant console output is below:
## requestPermision
## start NearbyInteractionManager
## session = NISession
## session?.delegate = self <NISession: isSupported: YES, Token: E30B090973E326020A8033FB63EE9C803E68C14A3BB5786E2DBB, Configuration: null>
## MultipeerConnectivityManager.instance.delegate
## startBrowsingForPeers
## MultipeerConnectivityManager.instance.startBrowsingForPeers
...
## connected devices changed []
## session peer didChange
## discoveryTokenData
## connected to device : 307 bytes
## discoveryTokenData
## shareDiscoveryToken
2022-04-18 11:54:06.175774-0500 Proximity[7502:4557553] send data to 1 peers
2022-04-18 11:54:06.176141-0500 Proximity[7502:4557553] peer <MCPeerID: 0x283c205c0 DisplayName = Taylor’s iPhone> didChangeState: 2
## connected devices changed ["Taylor’s iPhone"]
## session didReceive
2022-04-18 11:54:06.178287-0500 Proximity[7502:4557553] didReceiveData: 307 bytes from Taylor’s iPhone
## data received
## Delegate session didInvalidateWithError : Error Domain=com.apple.NearbyInteraction Code=-5884 "User declined session." UserInfo={NSLocalizedRecoverySuggestion=Create a new session to prompt the user to allow., NSLocalizedDescription=User declined session., NSLocalizedFailureReason=The user declined to authorize the session.}
## session didInvalidateWithError : Error Domain=com.apple.NearbyInteraction Code=-5884 "User declined session." UserInfo={NSLocalizedRecoverySuggestion=Create a new session to prompt the user to allow., NSLocalizedDescription=User declined session., NSLocalizedFailureReason=The user declined to authorize the session.}
2022-04-18 11:54:15.910274-0500 Proximity[7502:4557743] [GCKSession] Not in connected state, so giving up for participant [7A766541] on channel [0].
2022-04-18 11:54:15.921590-0500 Proximity[7502:4557743] [GCKSession] Not in connected state, so giving up for participant [7A766541] on channel [1].
2022-04-18 11:54:15.927949-0500 Proximity[7502:4557743] [GCKSession] Not in connected state, so giving up for participant [7A766541] on channel [2].
2022-04-18 11:54:15.932892-0500 Proximity[7502:4557743] [GCKSession] Not in connected state, so giving up for participant [7A766541] on channel [3].
As you see, a relevant token is generated for my own session, but the connection to another device appears to be blocked. Permissions are requested at the beginning of the app and both phones accept as well.
I am new to the iOS framework and XCode since I primarily develop Unity products, as is the case for this project (it will be used in a mobile application) so I am hoping this is caused by something simple that I can research once I have a lead to go on. Google has not been helpful so far, but if you guys have any idea as to why this would be blocking permissions, please let me know. I can provide code as well if necessary, thanks!
Intro:
was created a Google Smart Home project
was configured a proxy server via ngrok to redirect the Google request to my local machine
I develop an IoT project that has the ability to open/close a lock. I need to implement Google integration to use the Google Assistant to control the user locks. I have implemented OAuth Server for Google. Also I have implemented some controllers to handle Google Action Intents: SYNC, QUERY and EXECUTE. Google send a request with the SYNC intent and App response a payload that contain devices list with specific settings. Instance:
{
requestId: 'requestIdOfGoogle', // contains in the request body
payload: {
agentUserId: 'userId123', // matches user id inside app system
devices: [
{
id: 1,
type: 'action.devices.types.LOCK', // device type
traits: ['action.devices.traits.LockUnlock'], // feature that has a device
name: {
name: 'Kos Lock'
},
willReportState: true,
roomHint: 'Main Door',
deviceInfo: { // Test data
manufacturer: 'smart-home-inc',
model: 'hs1234',
hwVersion: '3.2',
swVersion: '11.4'
}
}
]
}
}
Then Google send requests to my server with QUERY intent to get info about state of a devices, instance
{
requestId: 'requestIdOfGoogle', // contains in the request body
payload: {
devices: {
1: {
status: 'SUCCESS',
online: true,
isLocked: true,
// isJammed - Boolean. Whether the device is currently jammed and therefore its
// locked state cannot be determined.
isJammed: false
}
}
}
}
But after sending a response a test lock isn't configured and a user can't control one with Google Assistant.
enter image description here
I have tried to add other traits for a lock but it didn't help me. Also I have the same problem when I try to configure a Door device. But when I send to Google a Light device it works successfully. When you use the LockUnlock trait then Google Doc recommends to setup secondary user verification but it's optional.
I don't understand that do incorrect. If someone faced such a problem and solved it then could you help me, please
Prerequisites:
use node ^14.0.0
programming language - js
Touch controls are not supported for every device, and locks are not a device type that can be controlled directly. But they will still respond to voice commands.
I am trying to submit a form but I get an "internal" error after submit on IOS Safari. Happened on two separate devices. I'm using Firebase functions onCall function. Client code:
var contactForm =
window.firebase.functions().httpsCallable('contactForm');
let result = await contactForm({ accountUID, foldersFilter,
firstName, lastName, email, cellNumber, dobDay, dobMonth })
And server code:
exports.contactForm = functions.https.onCall(( data, context ) => {
return contactForm.contactForm( data, context )
});
This function is called via form. The form works great on chrome, safari desktop, but for some reason it gets an internal errror sometimes when testing on IOS device. At first I thought it only happened when I was using autofill, but I've tested more and I get the same error when not using autofill too.
The confusing thing is my function code is actually never being called (I don't see any firebase function logs). Here is my console in safari:
The network connection was lost.
Fetch API cannot load https://us-central1-projectId.cloudfunctions.net/contactForm due to access control checks
Failed to load resource: The network connection was lost.
internal
Why won't this form submit on ios safari?
I fixed the issue. Turns out it has something to do with Google Cloud Functions being IPv4, and Safari requiring IPv6. I suspect this will become a bigger issue moving forward. I'm having to move all onCall Firebase functions to https triggers. In order to make https triggers work, you have to use a custom domain in Firebase hosting and rewrite to your function endpoint.
{
"hosting": {
...
"rewrites": [
{
"source": "/api/contactForm",
"function": "contactForm"
}
}
and so now instead of calling https://us-central1-projectId.cloudfunctions.net/contactForm to trigger my api. I call https://customdomain.com/api/contactForm
As I recently found this paper describing a sniffing mechanism for iOS using Apple's NEPacketTunnelProvider Extension, I got curious and it made me want to understand it from a technical point of view. As I usually don't work at a deep network layer like that, I'm not able to comprehend it in the detail I'd like to. As Charles Proxy for iOS must do something very similar without requiring supervised devices, I assume the approach which the author of the paper presented in 2016 might be still working nowadays.
The author claimed that "Everything like IP packet parsing, building
an IP packet or parsing a DNS response had to be implemented ourselves." As I want to fully understand that, I tried to build it myself. I build a NetworkExtension and a message loop for the packetFlow of the NEPacketTunnelProvider. I was able to obtain the ip datagrams and tried to parse them. I used unsigned integers of the corresponding size for the source and target ip, the transport protocol and ip version, but I'm unsure how to handle the treat the payload. My parser uses the ptr.load(fromByteOffset: <offset>, as:<DataType>.self) where ptr is a UnsafeRawPointer to access the packet flow information. Since the data might exceed the storage of UInt64, I don't know how to access and store the payload in a proper way.
Furthermore, I figured that the source IP is always 192.168.20.1 (set as my interface's NEIPv4Settings address) and my target ip is always 192.168.2.1 (my dummy NEDNSSettings server). This leads me to my first questions: Are those DNS queries? Will the datagram packet claim any further information about the actual target? Would that mean that I have to somehow execute the request to the DNS server and reroute the packet to the target which I will obtain from that DNS query?
The next step would be to implement a TCP / UDP handling, right? My current parsing approach is able to distinguish between UDP, TCP and ICMP (even though I don't have investigated in the last one yet). Therefore, I'd iterate over the datagrams and lookup whether they require a UPD or TCP session/connection and transfer the datagram. The problem I currently see their from a conceptional point of view: How do I know which source/target port to use for TCP/UPD connections/sessions? As far as I know, this information is not part of the IP Packet itself (since it's rather some information we need on transport layer level, not on network layer level).
Additionally, I found a project called Specht on github. It uses a self-written library called NEKit which somehow also uses the NEPacketTunnelProvider approach. When I understand their approach correctly, they managed to somehow build a local proxy server by writing some observer mechanisms in order to handle the requests, but since I'm relatively new to networking and swift, I'm not sure whether I understand that completely correct or whether I just haven't find all those TCP/UDP and/or DNS logic. Is this project comparable to the approach of the paper and charles proxy?
One last question: Charles proxy is in most cases able to show the hostname of the target. I'm currently just able to see destination ip addresses (which aren't real destination ip addresses, but the address of my DNS server). How am I able to see the hostname as human readable text? Does Charles do a nslookup somehow? Does Charles obtain that information out of the datagrams?
I know it's quite ambitious of me with much missing knowledge in this topic, to build something similar for test reasons, but I'm still motivated to look deeper into that topic and also have the feeling that I have understand already some key points, but unfortunately not enough to solve the puzzle... Maybe you're able to give me some more hints to get a better understanding. If there might be even an easier way to archive a similar behavior (to see outgoing connections on hostname level), I'd be interested in these as well :-)
I've published a Beta Proxyman iOS (website) - a Network Sniffer by using NEPacketTunnelProvider and Network Extension, so I might have experienced to answer some of your questions.
IP Package, IP Diagram, DNS, How to parse it?
Luckily, there is another way to set up a NEPacketTunnelProvider to provide you with an HTTP Message, not IP Package (it's too low-level, and you have to deal with the Parser, DNS, ...)
HTTP Message is easier to parse because there are plenty of reliable libraries (e.g. http-parser from nodeJS)
How to build a Network Sniffer on iOS?
It's a complicated question to answer, I would break it into small chunks:
MitM / Proxy Server
Firstly, you need a working MitM Proxy Server, which is capable of proxying and intercepting the HTTP/HTTPS Traffic. You can implement it by using SwiftNIO or CocoaAsyncSocket.
How does it work?
In general, the data flow might look like this:
The Internet -> iPhone -> Your Network Extension (VPN) -> Forward to your Local Proxy Server (in the Network Extension) -> Mitm/Proxy Server starts intercepting or monitoring the traffic -> Save to a local database (in Shared Container Group) -> Forward again to the destination server.
From the main app, you can receive the data by reading the local database.
The reason why we need a local database is that the Network Extension and the Main app are two different processes, so they could not communicate directly like a normal app.
Show me the code?
In the Network extension, let start a Proxy Server at Host:Port, then init the NetworkSetting, like the sample:
private func initTunnelSettings(proxyHost: String, proxyPort: Int) -> NEPacketTunnelNetworkSettings {
let settings: NEPacketTunnelNetworkSettings = NEPacketTunnelNetworkSettings(tunnelRemoteAddress: "127.0.0.1")
/* proxy settings */
let proxySettings: NEProxySettings = NEProxySettings()
proxySettings.httpServer = NEProxyServer(
address: proxyHost,
port: proxyPort
)
proxySettings.httpsServer = NEProxyServer(
address: proxyHost,
port: proxyPort
)
proxySettings.autoProxyConfigurationEnabled = false
proxySettings.httpEnabled = true
proxySettings.httpsEnabled = true
proxySettings.excludeSimpleHostnames = true
proxySettings.exceptionList = [
"192.168.0.0/16",
"10.0.0.0/8",
"172.16.0.0/12",
"127.0.0.1",
"localhost",
"*.local"
]
settings.proxySettings = proxySettings
/* ipv4 settings */
let ipv4Settings: NEIPv4Settings = NEIPv4Settings(
addresses: [settings.tunnelRemoteAddress],
subnetMasks: ["255.255.255.255"]
)
ipv4Settings.includedRoutes = [NEIPv4Route.default()]
ipv4Settings.excludedRoutes = [
NEIPv4Route(destinationAddress: "192.168.0.0", subnetMask: "255.255.0.0"),
NEIPv4Route(destinationAddress: "10.0.0.0", subnetMask: "255.0.0.0"),
NEIPv4Route(destinationAddress: "172.16.0.0", subnetMask: "255.240.0.0")
]
settings.ipv4Settings = ipv4Settings
/* MTU */
settings.mtu = 1500
return settings
}
Then start a VPN,
let networkSettings = initTunnelSettings(proxyHost: ip, proxyPort: port)
// Start
setTunnelNetworkSettings(networkSettings) { // Handle success }
Then forward the package to your local proxy server:
let endpoint = NWHostEndpoint(hostname: proxyIP, port: proxyPort)
self.connection = self.createTCPConnection(to: endpoint, enableTLS: false, tlsParameters: nil, delegate: nil)
packetFlow.readPackets {[weak self] (packets, protocols) in
guard let strongSelf = self else { return }
for packet in packets {
strongSelf.connection.write(packet, completionHandler: { (error) in
})
}
// Repeat
strongSelf.readPackets()
}
From that, your local server can receive the packages then forwarding to the destination server.
Don't forget to save all traffic log to the local database, then notifying the main app to reload it.
One last question: Charles proxy is in most cases able to show the hostname of the target. I'm currently just able to see destination ip addresses (which aren't real destination ip addresses, but the address of my DNS server). How am I able to see the hostname as human readable text? Does Charles do a nslookup somehow? Does Charles obtain that information out of the datagrams?
Since we don't deal with IP Package, we don't need to implement the DNS Resolver. If you need a DNS, you can config like the following code:
let dnsSettings = NEDNSSettings(servers: ["8.8.8.8", "1.1.1.1"])
settings.dnsSettings = dnsSettings
As we receive the HTTP Message package, you can get hostname for free (From the Request's URL or Host Header)
Hope that my answer could help you.
What is the best way to test if a Worklight app is off-line?
After I use the WL.Device.startAcquisition( ... ) api to start stuff off, I am currently using:
WL.Device.Geo.acquirePosition(function(pos) {
console.log("***** Aquired position ***** " + JSON.stringify(pos));
}, function(error) {
console.log(JSON.stringify("***** Unable to aquire position ***** " + error.code + ' : ' + error.message));
// call method to asynchronously - periodicallyCheckIfOnline( ... );
}, {timeout: 5000});
And if I determine that I am offline, I then use the watchPosition api to periodically test for a new connection.
navHandle = navigator.geolocation.watchPosition(onSuccess, onError, { timeout: 5000 });
Once I get the connection back I then clear the watch.
navigator.geolocation.clearWatch(navHandle);
Is this the best way of doing it or are there better Worklight APIs to use for this.
Note: I am trying to test this in a Mobile Browser Simulator scenario hence the short timeouts.
On startup, use something like
WL.Client.connect({onSuccess:onConnectSuccess,onFailure:onConnectFailure,timeout:number_of_ms});
to check if you have initial connectivity.
To detect any further changes in connectivity, you can use the
WL.Client.setHeartBeatInterval(number_of_s) API.
This will 'ping' the worklight server every number_of_s seconds and fire the WL.Events.WORKLIGHT_IS_DISCONNECTED and WL.Events.WORKLIGHT_IS_CONNECTED events, to which you attach callbacks to as described in the reading-worthy tutorial linked to by #Leandro David.
NOTE : if you need to use network to transfer heavy data, do a double check : once you know you have connectivity to the worklight server, use the WL.Device.getNetworkInfo API to check the connection quality before sending/receiving data.
There is a tutorial about dealing with online/offline mode in the worklight geting started material ("Working Offline" link):
http://www.ibm.com/developerworks/mobile/worklight/getting-started.html#GS_work_offline
It tells the best way to use Worklight API to deal with online/offline connection
To summarize, I believe this is the most important part:
Active detection of connectivity
Connectivity loss can be detected in two locations in your application code:
– Application initialization – WL.Client.init() method, typically called from initOptions.js file
– Adapter procedure invocation – WL.Client.invokeProcedure() method
-To add connectivity failure detection in either location, add the onConnectionFailure property and specify a callback function to be invoked if connectivity fails
var wlInitOptions = {
onConnectionFailure: function (data){
connectionFailure(data);
},
or
WL.Client.invokeProcedure(invocationData, {
onSuccess: successHandlerFunction,
onConnectionFailure: connectionFailure,
timeout: 1000
});
Passive detection – Offline and online events
Each time the Worklight framework attempts to access the Worklight Server, it might detect that the application switched from offline to online status or vice versa.
In both cases, JavaScript events are fired:
– WL.Events.WORKLIGHT_IS_DISCONNECTED event is fired when connectivity to the Worklight Server fails
– WL.Events.WORKLIGHT_IS_CONNECTED event is fired when connectivity to the Worklight Server is restored
You can add event listeners to these events and specify the callback functions to handle them.
document.addEventListener(WL.Events.WORKLIGHT_IS_CONNECTED, connectDetected, false);
document.addEventListener(WL.Events.WORKLIGHT_IS_DISCONNECTED, disconnectDetected, false);
Note: WL.Events.WORKLIGHT_IS_DISCONNECTED and WL.Events.WORKLIGHT_IS_CONNECTED are namespace constants, not strings
There are more details available in the tutorial above