WebRTC + IOS + Freeswitch : Can't hear audio - ios

Question

I'm trying to implement mod_verto on IOS (calling from iPhone to Desktop). I'm using Google's libjingle library for the RTC side, got it up and running using this excellent tutorial.
When making a call from my iPhone, I get the call on the desktop browser using the Verto Communicator (downloaded and running on my local machine).
On the iPhone side, I can hear the audio from my desktop, but I hear nothing on the desktop side.
If I'm making the call using 2 browser windows (with the Verto Communicator), all works fine.
Full Disclosure, I'm using ws:// unsecure websocket to connect to
FreeSwitch
Here is my JSONRPC log:
Sending login request:
{"jsonrpc":"2.0","method":"login","id":1,"params":{"login":"1000#MY-IP-ADDRESS","loginParams":{},"userVariables":{},"passwd":"1234","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}
Login Response:
{"jsonrpc":"2.0","id":1,"result":{"message":"logged in","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}
verto.invite (including the iPhone sdp):
{"jsonrpc":"2.0","method":"verto.invite","id":2,"params":{"dialogParams":{"remote_caller_id_number":"1008","useVideo":false,"useMic":"any","useStereo":false,"tag":"webcam","login":"1000#159.203.164.7","useCamera":"any","videoParams":{"minFrameRate":30,"minWidth":"1280","minHeight":"720"},"destination_number":"1008","screenShare":false,"caller_id_name":"FreeSWITCH User","caller_id_number":"1000","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800","remote_caller_id_name":"Outbound Call","useSpeak":"any"},"sessid":"53FB0781-B586-4CDA-98C6-558680663B46","sdp":"v=0\r\no=- 8564086442942257834 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio video\r\na=msid-semantic: WMS\r\nm=audio 58157 UDP\/TLS\/RTP\/SAVPF 111 103 104 9 102 0 8 106 105 13 127 126\r\nc=IN IP4 82.166.93.197\r\na=rtcp:52576 IN IP4 82.166.93.197\r\na=candidate:3168280865 1 udp 2122260223 11.0.0.244 58157 typ host generation 0\r\na=candidate:1260196625 1 udp 2122194687 10.134.172.254 58951 typ host generation 0\r\na=candidate:3168280865 2 udp 2122260222 11.0.0.244 52576 typ host generation 0\r\na=candidate:1260196625 2 udp 2122194686 10.134.172.254 58945 typ host generation 0\r\na=candidate:4066106833 1 tcp 1518280447 11.0.0.244 60562 typ host tcptype passive generation 0\r\na=candidate:94302177 1 tcp 1518214911 10.134.172.254 60563 typ host tcptype passive generation 0\r\na=candidate:4066106833 2 tcp 1518280446 11.0.0.244 60564 typ host tcptype passive generation 0\r\na=candidate:94302177 2 tcp 1518214910 10.134.172.254 60565 typ host tcptype passive generation 0\r\na=candidate:1610196941 1 udp 1686052607 82.166.93.197 58157 typ srflx raddr 11.0.0.244 rport 58157 generation 0\r\na=candidate:1610196941 2 udp 1686052606 82.166.93.197 52576 typ srflx raddr 11.0.0.244 rport 52576 generation 0\r\na=candidate:2274372738 2 udp 1685987070 176.13.15.205 5834 typ srflx raddr 10.134.172.254 rport 58945 generation 0\r\na=candidate:2274372738 1 udp 1685987071 176.13.15.205 5840 typ srflx raddr 10.134.172.254 rport 58951 generation 0\r\na=ice-ufrag:g8lHDtPwH7m5xRex\r\na=ice-pwd:Q6jcBJNTWAyu0JTuIaQAeNI3\r\na=fingerprint:sha-256 0F:A1:68:51:87:3E:B4:C1:0D:33:97:40:78:22:2A:8C:D2:B6:46:23:F5:99:C9:88:5D:34:DB:E2:C5:94:B3:DD\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:3 http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus\/48000\/2\r\na=fmtp:111 minptime=10; useinbandfec=1\r\na=rtpmap:103 ISAC\/16000\r\na=rtpmap:104 ISAC\/32000\r\na=rtpmap:9 G722\/8000\r\na=rtpmap:102 ILBC\/8000\r\na=rtpmap:0 PCMU\/8000\r\na=rtpmap:8 PCMA\/8000\r\na=rtpmap:106 CN\/32000\r\na=rtpmap:105 CN\/16000\r\na=rtpmap:13 CN\/8000\r\na=rtpmap:127 red\/8000\r\na=rtpmap:126 telephone-event\/8000\r\na=maxptime:60\r\nm=video 61966 UDP\/TLS\/RTP\/SAVPF 100 101 116 117 96\r\nc=IN IP4 82.166.93.197\r\na=rtcp:63816 IN IP4 82.166.93.197\r\na=candidate:3168280865 1 udp 2122260223 11.0.0.244 61966 typ host generation 0\r\na=candidate:1260196625 1 udp 2122194687 10.134.172.254 50435 typ host generation 0\r\na=candidate:3168280865 2 udp 2122260222 11.0.0.244 63816 typ host generation 0\r\na=candidate:1260196625 2 udp 2122194686 10.134.172.254 63396 typ host generation 0\r\na=candidate:4066106833 1 tcp 1518280447 11.0.0.244 60566 typ host tcptype passive generation 0\r\na=candidate:94302177 1 tcp 1518214911 10.134.172.254 60567 typ host tcptype passive generation 0\r\na=candidate:4066106833 2 tcp 1518280446 11.0.0.244 60568 typ host tcptype passive generation 0\r\na=candidate:94302177 2 tcp 1518214910 10.134.172.254 60569 typ host tcptype passive generation 0\r\na=candidate:1610196941 1 udp 1686052607 82.166.93.197 61966 typ srflx raddr 11.0.0.244 rport 61966 generation 0\r\na=candidate:1610196941 2 udp 1686052606 82.166.93.197 63816 typ srflx raddr 11.0.0.244 rport 63816 generation 0\r\na=candidate:2274372738 1 udp 1685987071 176.13.15.205 5879 typ srflx raddr 10.134.172.254 rport 50435 generation 0\r\na=candidate:2274372738 2 udp 1685987070 176.13.15.205 5860 typ srflx raddr 10.134.172.254 rport 63396 generation 0\r\na=ice-ufrag:g8lHDtPwH7m5xRex\r\na=ice-pwd:Q6jcBJNTWAyu0JTuIaQAeNI3\r\na=fingerprint:sha-256 0F:A1:68:51:87:3E:B4:C1:0D:33:97:40:78:22:2A:8C:D2:B6:46:23:F5:99:C9:88:5D:34:DB:E2:C5:94:B3:DD\r\na=setup:actpass\r\na=mid:video\r\na=extmap:2 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:3 http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time\r\na=extmap:4 urn:3gpp:video-orientation\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:100 VP8\/90000\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtpmap:101 VP9\/90000\r\na=rtcp-fb:101 ccm fir\r\na=rtcp-fb:101 nack\r\na=rtcp-fb:101 nack pli\r\na=rtcp-fb:101 goog-remb\r\na=rtcp-fb:101 transport-cc\r\na=rtpmap:116 red\/90000\r\na=rtpmap:117 ulpfec\/90000\r\na=rtpmap:96 rtx\/90000\r\na=fmtp:96 apt=100\r\n"}}
Call Created response:
{"jsonrpc":"2.0","id":2,"result":{"message":"CALL CREATED","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}
verto.media invoked:
{"jsonrpc":"2.0","method":"verto.media","id":637,"params":{"sdp":"v=0\no=FreeSWITCH 1457232832 1457232833 IN IP4 159.203.164.7\ns=FreeSWITCH\nc=IN IP4 159.203.164.7\nt=0 0\na=msid-semantic: WMS TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\nm=audio 30784 UDP/TLS/RTP/SAVPF 111 126\na=rtpmap:111 opus/48000/2\na=fmtp:111 useinbandfec=1; minptime=10\na=rtpmap:126 telephone-event/8000\na=silenceSupp:off - - - -\na=ptime:20\na=sendonly\na=fingerprint:sha-256 FE:CD:54:3E:2A:D7:DB:00:57:B7:D4:55:A8:EB:79:08:16:BB:B0:EA:43:44:42:9A:90:01:49:37:7B:31:48:F8\na=setup:active\na=rtcp-mux\na=rtcp:30784 IN IP4 159.203.164.7\na=ice-ufrag:qLh1zzclxONPNyQO\na=ice-pwd:G7g4Drkist37beYsP5jfvlqS\na=candidate:9922185636 1 udp 659136 159.203.164.7 30784 typ host generation 0\na=ssrc:1323504502 cname:bhqCyFkpPbjUPSk0\na=ssrc:1323504502 msid:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub a0\na=ssrc:1323504502 mslabel:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\na=ssrc:1323504502 label:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Uba0\nm=video 31380 UDP/TLS/RTP/SAVPF 100\na=rtpmap:100 VP8/90000\na=sendonly\na=fingerprint:sha-256 FE:CD:54:3E:2A:D7:DB:00:57:B7:D4:55:A8:EB:79:08:16:BB:B0:EA:43:44:42:9A:90:01:49:37:7B:31:48:F8\na=setup:active\na=rtcp-mux\na=rtcp:31380 IN IP4 159.203.164.7\nb=AS:1024\na=rtcp-fb:100 ccm fir\na=rtcp-fb:100 nack\na=rtcp-fb:100 nack pli\na=ssrc:594893571 cname:bhqCyFkpPbjUPSk0\na=ssrc:594893571 msid:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub v0\na=ssrc:594893571 mslabel:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\na=ssrc:594893571 label:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ubv0\na=ice-ufrag:2KDK4wDMYuAuVdAZ\na=ice-pwd:YTpxObqpLuBEfig7TKHN6bqU\na=candidate:7508673635 1 udp 659136 159.203.164.7 31380 typ host generation 0\n","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800"}}
verto.answer invoked:
{"jsonrpc":"2.0","method":"verto.answer","id":638,"params":{"callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800"}}
Q:What am I missing in order to hear audio on the browser side ?
Any information is appreciated :)
Update, added the freeswitch log
Update 2 IOS : audio stream code
...
let audioTrack = self.factory.audioTrackWithID("Local-Audio")
self.localMediaStream?.addAudioTrack(audioTrack);
self.peerConnection!.addStream(self.localMediaStream)
...
Update 3 - Partial solution
While inspecting my code, I found old code that used to add a video track to my local media stream, disabling this section solves the audio problem, but why? what is wrong with that code?
P.S the Promise class was created by a friend, and mimic the JS Promise approach.
func getUserMedia(mediaOptions:Dictionary<String , Any>? = nil) -> Promise<RTCMediaStream>{
return Promise<RTCMediaStream>(executor: { (resolve, reject) -> () in
var cameraID:String?
self.localMediaStream = self.factory.mediaStreamWithLabel("Local-Meida")
//if video option is enabled (default true)
//-------------- Disabling this section solves the audio issues --------------
if(mediaOptions?["video"] as? Bool ?? true){
for captureDevice in AVCaptureDevice.devicesWithMediaType(AVMediaTypeVideo){
if (captureDevice.position == mediaOptions?["devicePosition"] as? AVCaptureDevicePosition ?? AVCaptureDevicePosition.Front){
cameraID = captureDevice.localizedName
break
}
}
if(cameraID == nil){
reject(NSError(domain: "No cammera detected", code: 0, userInfo: nil))
}
let capturer = RTCVideoCapturer.init(deviceName: cameraID)
let videoSource = self.factory.videoSourceWithCapturer(capturer, constraints: mediaOptions?["constraints"] as? RTCMediaConstraints ?? nil)
if let localVideoTrack = self.factory.videoTrackWithID("Local-Video", source: videoSource){
//!!!! THIS IS THE PROBLEMATIC LINE !!!!
self.localMediaStream?.addVideoTrack(localVideoTrack)
}else{
reject(NSError(domain: "No Video track", code: 0, userInfo: nil))
}
}
//-------------- Disabling this section solves the audio issues --------------
if(mediaOptions?["audio"] as? Bool ?? true){
let audioTrack = self.factory.audioTrackWithID("Local-Audio")
self.localMediaStream?.addAudioTrack(audioTrack);
}
self.peerConnection!.addStream(self.localMediaStream)
resolve(self.localMediaStream!)
})
}
Debug at problematic line

Answer 1

Update
It is very difficult to understand what is going wrong with the WebRTC implementation unless you have check Media Server, Web Client and iOS Client in our case.
Your case is an Audio Call, so no video streams need to be included on your localStream, but if you take a look closely you see that you are actually adding a videoTrack on your mobile stream:
if(mediaOptions?["video"] as? Bool ?? true){
for captureDevice in AVCaptureDevice.devicesWithMediaType(AVMediaTypeVideo){
if (captureDevice.position == mediaOptions?["devicePosition"] as? AVCaptureDevicePosition ?? AVCaptureDevicePosition.Front){
cameraID = captureDevice.localizedName
break
}
}
if(cameraID == nil){
reject(NSError(domain: "No cammera detected", code: 0, userInfo: nil))
}
let capturer = RTCVideoCapturer.init(deviceName: cameraID)
let videoSource = self.factory.videoSourceWithCapturer(capturer, constraints: mediaOptions?["constraints"] as? RTCMediaConstraints ?? nil)
if let localVideoTrack = self.factory.videoTrackWithID("Local-Video", source: videoSource){
//!!!! THIS IS THE PROBLEMATIC LINE !!!!
self.localMediaStream?.addVideoTrack(localVideoTrack)
}else{
reject(NSError(domain: "No Video track", code: 0, userInfo: nil))
}
}
So the line which causes the trouble is: self.localMediaStream?.addVideoTrack(localVideoTrack), because you are attaching the video to the localStream.
Opinions
We may have different scenarios of trouble as I mentioned, here I am listing some opinions based on my experience when we built a similar system:
Your MediaServer may not have an implementation which can redirect and handle your calls in a successful state, because there are additional things added when you attach a video (Please see your session description what you actually send there), and it simply refuses to create a call.
Even if your MediaServer handles the scenario, this will include the right implementation of the Client (both Desktop and Mobile) to conform to signaling of its protocols.
You passed all the tests, and you are now adding video and audio, so you are initiating localStream from mobile, and the same need to be created other way. Then you need to handle events when you add streams, remove stream and other stuff via websockets.
Solution in this case
Remove the part that adds localTrack inside localStream, and then even if you have errors, are not caused by creating your localStream, so this step is currently solved.
Original Answer
Here I have a working version of mine, but adapted for your needs as you use only audio.
Creating and setting up the peerConnection (localSide)
// Connecting to the socket
.........
// Create PeerConnectionFactory
self.peerConnectionFactory = [[RTCPeerConnectionFactory alloc] init];
RTCMediaConstraints *constraints = [self defaultPeerConnectionConstraints];
// Initialize peerConnection based on a list of ICE Servers
self.peerConnection = [self.peerConnectionFactory peerConnectionWithICEServers:[self getICEServers] constraints:constraints delegate:self];
// Create the localStram which contains the audioTrack
RTCMediaStream *localStream = [self createLocalMediaStream];
// Add this stream to the peerConnection
[self.peerConnection addStream:localStream];
// Please be aware here that I am using blocks, as I created a wrapper for easier maintenance, but you can use createOfferWithDelegate: which will go back at your delegation
NSLog(#"Creating peer offer");
RTCManager *strongSelf = self;
[strongSelf.peerConnection createOfferWithCallback:^(RTCSessionDescription *sdp, NSError *error) {
if (!error) {
dispatch_async(dispatch_get_main_queue(), ^{
NSLog(#"Success at creating offer, now setting local description");
[strongSelf.peerConnection setLocalDescriptionWithCallback:^(NSError *error) {
if (!error) {
dispatch_async(dispatch_get_main_queue(), ^{
NSLog(#"Success at setting local description");
// On my type of signalization here I am connected, but yours is based on what type of signalization requires
});
}
} sessionDescription:sdp];
});
}
} constraints:[strongSelf defaultPeerConnectionConstraints]];
Helpers
// Now here we create the stream which contains the audio (Please note the ID)
- (RTCMediaStream *)createLocalMediaStream {
RTCMediaStream *localStream = [self.peerConnectionFactory mediaStreamWithLabel:#"ARDAMS"];
[localStream addAudioTrack:[self.peerConnectionFactory audioTrackWithID:#"ARDAMSa0"]];
return localStream;
}
- (RTCMediaConstraints *)defaultPeerConnectionConstraints {
// DtlsSrtpKeyAgreement is required for Chrome and Firefox to interoperate.
NSArray *optionalConstraints = #[[[RTCPair alloc] initWithKey:#"DtlsSrtpKeyAgreement" value:#"true"]];
RTCMediaConstraints *constraints = [[RTCMediaConstraints alloc] initWithMandatoryConstraints:nil optionalConstraints:optionalConstraints];
return constraints;
}
Please be aware that your problem might be caused also for not calling stuff on the main thread.