I have been working on a Twilio project and i have set everything up using this guide: Stream TwiML
i have already resolved a common issue with the help of Alan here Answer
I'm unable to send back the processed bytes I'm using.
ws.send(bytes)
ws.receive()
These two methods i assume are required to send and receive back and forth data using the web socket.
But im receving a constant data stream from twilio.
That's:
b'\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xf
Which does not makes any sense. The guide officially says you can get bytes from real time call and then send them. So let's say i receive bytes and process them using my custom logic maybe storing it somewhere. My processing outputs the same PCM/16Bit Mono audio data in bytes. Which i think twilio is also send me raw bytes as it says in the documentation that it gives out raw bytes (which i assume are without any kind of encoding). So all i want here is to receive audio data sent from the caller to twilio to receive and send bytes to an endpoint which i have setup ed earlier in flask to handle and then i can process it a little bit and return PCM bytes to it which can be played back at the caller side.
All i want is to caller send his voice and gets processed and receives back maybe our voices added to it as well. It wont matter how the byte is treated here my concept is to add some audio data to it and send back.
But since verb is new so there's no where i can find how to properly handle the bytes and send them back.
It would be nice if someone can point me what twilio expects in the web socket return as a audio data that it can be played on the caller side. And what i should be expecting from twilio. But since in the documentation it says that payload sends out raw bytes so i assume its a plain PCM data in bytes.
Please let me know so i can receive bytes from twilio and send them back which can be played at the caller side and caller's voice can be heard from here.I have players and clients setup ed as well to play the byte data but since the data is plain or recursive i cannot play it.
Thanks in advance.
Twilio developer evangelist here.
As far as I know right now Media Streams only supports streaming the data to your server one way. You cannot stream media back to Twilio via the websocket connection.
If you were able to save the bytes you want to send back as an audio file, you can play it back to the call by redirecting the call to some TwiML that <Play>s the audio to the call.
Let me know if that helps at all.
I am using the Twilio Network Traversal Service as part of a native application I am working on to perform peer-to-peer remote desktop connections. We implement a subset of the WebRTC protocol stack that is equivalent to the WebRTC data channels (not the WebRTC video and audio protocols). When using a TURN relay, the TURN allocation seems to be invalidated randomly somewhere between a few minutes and a maximum of 12 minutes from the session start. This issue looks very similar to this one, but the proposed workaround (sending silent audio) is not acceptable in my case, since I do not implement the WebRTC audio/video protocols.
I have been pulling my hair on this problem for the last two weeks, and isolated the issue as being the Twilio service itself. To compare, I have used a web based WebRTC data channel demo using firefox and the Xirsys TURN server cloud. I have wireshark captures showing firefox getting disconnected with Twilio just like my native application, while the exact same firefox demo doesn't get disconnected when using the Xirsys servers.
I was using Xirsys originally, but I experienced some instability with their service that made me switch to Twilio, which is why I would rather have Twilio fix this issue instead of going back with Xirsys. At the bare minimum, I would rather have two WebRTC hosting providers I can choose from that I know should work fine. This is why I am taking the time to explain the issue in detail so it can get fixed.
Here are two wireshark captures (with the peer-to-peer data messages filtered out) showing firefox using WebRTC data channels and the Twilio TURN relay servers:
The traffic stops being relayed after 4 minutes in the first capture, and after about 11 minutes in the second capture. In both captures, firefox detects that traffic stops being relayed (at the data channel level) and attempts a graceful disconnection by sending a Refresh request packet with a lifetime of zero. Both graceful disconnections result in a 437 Allocation Mismatch error, indicating that the server doesn't even know about the allocation firefox is trying to close gracefully.
With my native application, this would often take the form of a CreatePermission Request message that fails with a 438 "Wrong nonce" error, which is basically what should happen if a client tries to update the permission on an allocation that no longer exists. The error code 438 usually means "Stale nonce", which is not really an error, but an indication that the nonce has expired and the client should try again using the new nonce contained in the "error" message. It took me a while to figure out, but even if the error code is 438, the error string is not the same. I have observed a true stale nonce error with Xirsys and successfully updated my permission with the new nonce from the error response, so I know I can properly handle this case in my implementation.
Here is the source code for the WebRTC data channel demo I have used:
https://github.com/devolutions/webrtc-demo
For comparison, here is the same firefox data channel demo using the Xirsys TURN server cloud:
In this capture, I have let the demo run for about 16 minutes (it works for much longer than that, the longest I have tried is two hours). We can see that the traffic keeps getting relayed for the entire duration of the session, and CreatePermission requests keep getting sent by firefox with success. At the end, the graceful disconnection is triggered by firefox closing the WebRTC data channel (instead of being closed due to traffic no longer being relayed). As opposed to the Twilio captures, the Refresh request with a lifetime of zero is successful: the Xirsys TURN server still knows about the allocation and sends back a success response, as expected.
It should be noted that the ICMP unreachable errors are normal because I think in this case firefox is no longer listening on the given port when the response comes back. In other words, it sends the Refresh request with a lifetime of zero and doesn't wait for the answer to come back.
For the time being, I have no other choice but to go back with Xirsys, but I would really like if the Twilio Network Traversal Service could be fixed. Let me know if you have more questions regarding the issue.
I have uploaded the wireshark captures here for reference.
EDIT: I have modified the webrtc demo page such that it doesn't close the connection when the ice connection state is set to 'disconnected'. Now I get the real disconnection when the ice connection state goes to 'failed'. However, it effectively didn't change anything, since in this case it takes just a few seconds more for the state to go from 'disconnected' to 'failed'.
Since I have new relevant screenshots and captures, I am updating the original question to clarify certain problems pointed out by Philipp Hancke:
First, here is a new capture with the ice connection state fix (the browser closes the connection only when the state goes to 'failed'):
It's interesting to see that this time, the session stayed up for a whole 18 minutes. This was taken on a saturday morning, so I'm guessing that the issue could be related to the current workload on the twilio servers. However, it failed in the exact same way as it always does so far for me. As a bonus, we even have a valid stale nonce response that is correctly handled by firefox.
However, if we take a different view of the same capture, we can see that the traffic stops being relayed for a solid 30 seconds before firefox considers the connection as being dropped and sends the Refresh request with a lifetime of zero. As in previous captures, the server responds with an Allocation Mismatch error, indicating it doesn't know which allocation firefox is talking about.
The last eight packets being sent are of the same size, so my guess is that they are retransmissions. After 30 seconds of retransmissions, it is likely that SCTP considers the transport as being dropped.
With regards to the refresh request with a lifetime of zero, I did a test where I close the connection early on, from the browser. In this case, the server recognizes the allocation and returns a success response:
The allocation mismatch is the easiest symptom to observe, but in my testing with my native application, I have seen similar errors with Refresh requests for non-zero lifetimes, and with CreatePermission requests (438 "Wrong nonce" error). However, since the browser closes the connection after 30 seconds of data not being relayed, it is hard to observe these errors with the current webrtc demo. If we could change that timeout to 10 minutes, we would see those errors as well.
Excellent problem description!
Without the server log this is hard to determine what goes wrong. I tried with the appear.in TURN servers which run an up-to-date version of coturn and show the same behaviour as the Twilio servers. Xirsys seems to be running a custom version of coturn (Coturn-0.5 'Xirsys Turn Services' from the software field but coturn never had such a version).
In both captures, firefox detects that traffic stops being relayed (at the data channel level) and attempts a graceful disconnection by sending a Refresh request packet with a lifetime of zero.
Not quite. A refresh request with a lifetime of 0 is used to discard an allocation. At that point it does not matter what the server returns as the connection is beyond repair anyway.
This is caused by peerjs closing the peerconnection if the iceconnectionstate changes to disconnected, here in your bundled library version.
This is overly aggressive (and does not even fix things) and we've had a discussion about what the specification should do wrt to trying to fix things with an ice restart here which also links to a great explanation of the disconnected state.
The disconnected state probably happens because a few packets get lost. But this is something that can happen when there is minor congestion. I'd recommend removing the pc.close() in the disconnected case.
If you are looking for other TURN providers, Tokbox provides the same service. For datachannels the latency of a properly run distributed TURN network does not matter as much as for VoIP so you might run your own servers in a single location instead.
I want to develop an android app which uses push to talk or Voip to communicate between 2 mobile devices using a Wi-Fi connection without using a data or the internet.
Is it possible to develop this?
I am looking for a 'free' Wi-Fi Walkie Talkie but I don't know how to start doing it?
For a very simple, initial version, I'd do the following:
Assign a static IP address on each phone
Record the audio and packetise it into a UDP stream that you'll send to the remote IP address
Repackage again the UDP stream and play the audio.
Microphone ---> Samples ----> UDP (over WiFi) ----> Samples ----> Speaker.
There are plenty of enhancements that you can add over the time:
Device discovery
Signalling separate (SIP or custom messages via TCP or UDP to indicate when to start the audio transmission and negotiate the media channel that will be used).
Support for third party devices (SIP)
Use standard paketization using RTP and encode the audio using G711alaw/G711ulaw...
There are plenty options, but you should start by capturing the audio, and sending it over the network, even make packets of 2 seconds with the incurring delay, but that's a point to start. Then, you should lower the packet length to contain 20ms of audio to avoid delays in the transmission.
Hope this helps.
I've been reading about MQTT and I understand it uses TCP for network transport. So if I have a mobile app that will send subscribe request, I presume this will be an full-duplex connection so the client can be notified for incoming pushed data.
How then is this more battery and network efficient? I mean you still established an open TCP connection. Also how does it handle disconnection, does it auto-reconnect to the broker?
Taking into account my comment on the question, assuming you want to compare to HTTP Long polling these 2 links may help to answer your question:
https://www.ibm.com/developerworks/community/blogs/sowhatfordevs/entry/using_mqtt_protocol_advantages_over_http_in_mobile_application_development5?lang=en
http://stephendnicholas.com/archives/1217
TL;DR version:
The message sizes tend to be significantly smaller with MQTT vs HTTP (especially when you take into account all the http headers that get sent), this saves on network usage and in turn battery usage.
As for the reconnect side of things, the client libraries do not automatically reconnect but they do trigger a call back when the connection drops so you can handle reconnecting as required.
We are working on an iOS application for which we need to learn how VoIP works on 2 devices/machines which are on same local network. When transferring files between 2 Skype users on same network, the file transfer is very fast. So there must be something different when 2 devices are connected which are on same network.
Maybe file sharing is different than VoIP and at least the VoIP part works the same way for users on same network or on different networks. But if its not the case then I would like to know whats the difference between connecting machines on same and different networks via Skype.
Generaly VoIP is based on 3 protocols. SIP, RTP and RTCP.
Try to read some articles about SIP signalisation or SIP messages flow. The two devices within the same network can communicate directly or via SIP proxy. Basically the app has to support at least:
REGISTER
INVITE
CANCEL
BYE
100 Trying
180 Ringing
200 OK
486 Busy here
and few others SIP messages and status codes. Your app has to be able to handle SDP messages where the codec is negotiated - choose some free and open codec such as G.711 or iLBC. Within the SDP header RTP and RTCP port are negotiated as well, try to read something about SDP as well.
As this standard model uses multiple ports, it may cause issues when NAT is implemented, but this is not your case as you are within the same LAN.
Regardin the speed of transfer - if you are within the same network you shall not experience any impairments while calling - voice is transmitted in very small packets (i.e. 273 bytes) sent every 20 ms. If you are on wired 100 Mbit or 1 Gbit network, this shall work flawlessly. When on Wi-Fi some impairments may occur as it is not that much reliable.
Further information can be found here:
http://www.ietf.org/rfc/rfc3261.txt
https://www.rfc-editor.org/rfc/rfc4566.html
http://www.ietf.org/rfc/rfc3550.txt
I hope I answered your question at least a bit, if you have any further questions regarding VoIP feel free to ask.