I'm developing an iOS app with screensharing. Since screenshsring on iOS is not supported by Apple, the only way is to take screenshots and transmit them.
I'm using WebRTC DataChannel to share the screenshots to a browser. It works decent, but requires a good and stable upload speed to work.
My problem is when there is slower network or when it goes up and down in speed. Because I want the screensharing to be relatively real-time. There needs to be some kind of flow control implemented. Now there will be a delay on the receiver side when the connection slows down / lag spikes.
So when the speed is lower, I want to slow down the screenshots send per second. And vice versa. I'm syncing clocks with ntp, and have a naive implementation at the moment sending the time the screenshot was taken and with that does some naive algorithm I made up. However it doesn't work as well as I need.
Is there some real time flow control algorithm you can recommend me, or any other ideas?
Related
This is my first question posted on stackoverflow.
I'm trying to make screen cast app using BroadcastExtension and WebRTC protocol. But broadcast extension's memory limit(50mb) is so tight that if an application tries to send the original video(886 x 1918 30fps) without any processing, it immediately dies after receiving a memory usage warning. After lowering the resolution and frame rate of the video, there is no problem. Investigating the application using the profiler does not seem to cause any problems with memory leaks. I guess it is because of the frames allocated during the encoding process inside WebRTC framework.
So my question is, is it possible to send the original video using WebRTC without any other processing, such as down scaling or lowering the frame rate?
Possible.
I forgot to mention in the question, but the library I used is Google WebRTC. I made two mistakes. One is to build the modified framework in debug mode, and the other is to use a software encoder(default is VP8). Because of this, it seems that the processing of the video frames was delayed and accumulated in the memory. DefaultEncoderFactory basically provides an encoder that operates in SW. (At least on iOS. Android seems to support HW-based decoder encoders automatically.) Fortunately, the iOS version google WebRTC framework supports the H264 hardware encoder(EncoderFactoryH264). In other cases you have to implement it yourself.
However, when using H264 to transmit, there is a problem that some platforms cannot play, for example, Android. The Google webrtc group seems to be aware of this problem, but at least it seems to me that it has not been resolved properly. Additional work is needed to solve this.
I need to realize a source-synchronous receiver in a Virtex 6 that receives data and a clock from a high speed ADC.
For the SERDES Module I need two clocks, that are basically the incoming clock, buffered by BUFIO and BUFR (recommended). I hope my picture makes the situation clear.
Clock distribution
My problem is, that I have some IOBs, that cannot be reached by the BUFIO because they are in a different, not adjacent clock region.
A friend recommended using the MMCM and connecting the output to a BUFG, which can reach all IOBs.
Is this a good idea? Can't I connect my LVDS clock buffer directly to a BUFG, without using the MMCM before?
My knowledge about FPGA Architecture and clocking regions is still very limited, so it would be nice if anybody has some good ideas, wise words or has maybe worked out a solution to a similar problem in the past.
It is quite common to use a MMCM for external inputs, if only to cleanup the signal and realize some other nice features (like 90/180/270 degree phase shift for quad-data rate sampling).
With the 7-series they introduced the multi-region clock buffer (BUFMR) that might help you here. Xilinx has published a nice answer record on which clock buffer to use when: 7 Series FPGA Design Assistant - Details on using different clocking buffers
I think your friends suggestion is correct.
Also check this application note for some suggestions: LVDS Source Synchronous 7:1
Serialization and Deserialization Using
Clock Multiplication
I am developing the application in which every things depends on internet.my Requirement is that when the speed of internet is low,app should display alert to user that "your internet speed is low, that's why this feature is not available to you."
I want to know that is there any feature available in ios from which we can know that our internet connection is low or speed of internet, I have found 2-3 answers about this but not get any feasible solution.
There is no magic call to know whether the Internet connection is fast or slow. There's only one solution - transfer data and time how long it takes to transfer a certain amount of data.
The problem is that the next chunk of data can be much slower, much faster, or about the same.
So you really need some sort of threshold where if the app is unable to transfer at least X number of bytes in Y seconds, then you stop the transfer and alert the user.
In other words, there's no simple way to ask "Is the connection fast or slow?".
I would like to know if it's still necessary to throttle the bandwidth when transferring multiples large files (PDF) over the cellular network.
There is no information about this in the guidelines but some old threads points out this was necessary.
http://forums.macrumors.com/archive/index.php/t-1130677.html
iPhone app rejected for "transferring excessive volumes of data"
Thanks in advance.
I believe this only applies for streaming like mp3 data.
If you have a large single file which you need to download, throttling the bandwidth is an especially bad idea. The opposite would be recommend: download as fast as possible in order to safe battery and increase reliability of the connection (the shorter the duration the less the probability that it gets interrupted).
You should however check whether the connection uses the cellular network and notify the User and ask whether your app should defer the download when Wifi is available, possibly automatically in the background without starting the app. This approach is feasible (utilizing NSURLSession).
We have a critical need to lower the latency of our UDP listener on iOS.
We're implementing an alternative to RTP-MIDI that runs on iOS but relies on a simple UDP server to receive MIDI data. The problem we're having is that RTP-MIDI is able receive and process messages around 20ms faster than our simple UDP server on iOS.
We wrote 3 different code bases in order to try and eliminate the possibility that something else in the code was causing the unacceptable delays. In the end we concluded that there is a lag between time when the iPAD actually receives a packet and when that packet is actually presented to our application for reading.
We measured this with with a scope. We put a pulse on one of the probes from the sending device every time it sent a Note-On command. We put another probe attached to the audio output of the ipad. We triggered on the pulse and measured the amount of time it took to hear the audio. The resulting timing was a reliable average of 45ms with a minimum of 38 and maximum around 53 in rare situations.
We did the exact same test with RTP-MIDI (a far more verbose protocol) and it was 20ms faster. The best hunch I have is that, being part of CoreMIDI, RTPMIDI could possibly be getting higher priority than our app, but simply acknowledging this doesn't help us. We really need to figure out how fix this. We want our app to be just as fast, if not faster, than RTPMIDI and I think this should be theoretically possible since our protocol will not be as messy. We've declared RTPMIDI to be unacceptable for our application due to the poor design of its journal system.
The 3 code bases that were tested were:
Objective-C implementation derived from the PGMidi example which would forward data received on UDP verbatim via virtual midi ports to GarageBand etc.
Objective-C source base written by an experienced audio engine developer with a built-in low-latency sine wave generator for output.
Unity3D application with Mono-based UDP listener and built-in sound-font synthesizer plugns.
All 3 implementations showed identical measurements on the scope test.
Any insights on how we can get our messages faster would be greatly appreciated.
NEWER INFORMATION in the search for answers:
I was digging around for answers, and I found this question which seems to suggest that iOS might respond more quickly if the communication were TCP instead of UDP. This would take some effort to test on our part because our embedded system lacks TCP capabilities, only UDP. I am curious as to whether maybe I could hold open a TCP connection for the sole purpose of keeping the Wifi responsive. Crazy Idea? I dunno. Has anyone tried this? I need this to be as real-time as possible.
Answering my own question here:
In order to keep the UDP latency down, it turns out, all I had to do was to make sure the Wifi doesn't go silent for more than 150ms (or so). The exact timing requirements are unknown to me at this time, however the initial tests I was running were with packets 500ms apart and that was too long. When I increased the packet rate to 1 every 150ms, the UDP latency was on par with RTPMIDI giving us total lag time of around 18ms average (vs. 45ms) using the same techniques I described in the original question. This was on par with our RTPMIDI measurements.