This is my first question posted on stackoverflow.
I'm trying to make screen cast app using BroadcastExtension and WebRTC protocol. But broadcast extension's memory limit(50mb) is so tight that if an application tries to send the original video(886 x 1918 30fps) without any processing, it immediately dies after receiving a memory usage warning. After lowering the resolution and frame rate of the video, there is no problem. Investigating the application using the profiler does not seem to cause any problems with memory leaks. I guess it is because of the frames allocated during the encoding process inside WebRTC framework.
So my question is, is it possible to send the original video using WebRTC without any other processing, such as down scaling or lowering the frame rate?
Possible.
I forgot to mention in the question, but the library I used is Google WebRTC. I made two mistakes. One is to build the modified framework in debug mode, and the other is to use a software encoder(default is VP8). Because of this, it seems that the processing of the video frames was delayed and accumulated in the memory. DefaultEncoderFactory basically provides an encoder that operates in SW. (At least on iOS. Android seems to support HW-based decoder encoders automatically.) Fortunately, the iOS version google WebRTC framework supports the H264 hardware encoder(EncoderFactoryH264). In other cases you have to implement it yourself.
However, when using H264 to transmit, there is a problem that some platforms cannot play, for example, Android. The Google webrtc group seems to be aware of this problem, but at least it seems to me that it has not been resolved properly. Additional work is needed to solve this.
Related
I have an app that broadcasts the screen of the device. I am using OpenTok library to send the frames. Their library does not handle the compressing process so every time I send a frame using this code to consume Buffer:
self.videoCaptureConsumer?.consumeImageBuffer(pixelBuffer, orientation:sample.orientation.openTokOrientation,timestamp:CMSampleBufferGetPresentatioTimeStamp(sample) ,metadata:nil)
the Broadcast extension crashes because of the memory limitation(50MB).
I have been searching in SO as well as repos in GitHub finally I ended up using a CPU image processing using Accelerate. I created an extension of CVPixelBuffer to resize the buffer. Here is the extension
I can resize the CVPixelBuffer and then send the new buffer to the OpenTok library. But the problem is since it is done by CPU, on iPhones X and below, the broadcast extension is stopped by the high CPU usage by the system.
So, I thought I have to find a way to compress-resize the buffer in a faster and memory-safer way with GPU acceleration.
Then I ended up checking Telegram's iOS app. I discovered the Telegram`s Broadcast Extension which actually works like a charm and it is what I need but I lack information on how it works since it uses C libraries.
My question is how I can compress-resize the CVPixelBuffer in a similar way that Telegram does but at least written with Swift language without passing memory limit and using GPU acceleration?
So I'm planning to build an app that at the very least let's me use the mic on an iphone be converted into a balanced audio signal through the headphone jack. The problem is I'm not sure if getting mic input to the output is possible without a delay. I've looked into CoreAudio and AVFoundation, but it looks like one is getting deprecated soon and the other might be too high level to ever do what I need. I'm testing out AudioKit, but I've only run it in a simulator that's running on a virtual machine inside windows, so I might get much better results on an actual device (although I'm skeptical because the audio delay is about the same as when I monitor my microphone through windows).
Does anyone know any frameworks or literally anything that might make it possible to do real time audio processing without too noticeable of a delay?
Is it even possible on iOS or is the OS overhead too big?
Literally any answer is appreciated.
I'm doing real-time audio processing using AudioKit. There were a few hiccups, but I've been able to manage to add processing nodes to real-time mic recordings and output them to the speaker with virtually no delay.
A notable hiccup I ran into was the difference between a 'debug' build and a 'release' build in xcode. The release build takes longer to compile, but runs faster, thus reduces delay in the audiobuffer processing. My testing platform is an old ipad2 though, so you may not run into those issues if you're using more recent hardware.
I am currently struggling with memory management inside a webgl application on the web, on iOS only.
I keep getting the following error message :
Application 'UIKitApplication:com.apple.Preferences[0xa7c1]' was killed by jetsam.
I understand that Jetsam is the system process responsible for memory management, but the crash occurs while the browser uses around 25% of the device's RAM, which is not that high. I don't have any other open application while running my webGL content. What I don't understand is what Taun Chapman said :
Jetsam monitors memory use and kills applications which are not good memory citizens. A good memory citizen is an application which is willing to give back memory when asked and does not keep asking for more memory.
Well, the app needs more memory in a short time (when unzipping 3D models using workers), I can't continue my app without it! And it crashed at these particular times.
Moreover, I think I have some memory leaks in my app, according to Chrome DevTools I am currently trying to fix. But the browser itself seems to have some leaks too. Thus, fixing mine will just delay the inevitable.
I know the following question is odd, or inappropriate, but do you know if the jetsam "limit" can be increased ? Or if you can add an exception on the currently running WebGL app ?
For your information, I use the Three.js WebGL library and the zip.js library to compress my 3D models.
Yes, I've already read the following question : Why does simple website crash on mobile (iOS Safari and Chrome, at least)?, but the problem does not come from my CSS.
The 10.3.2 version of iOS (released the 15th of May) made Jetsam less aggressive, or at least, the memory is better handled.
https://support.apple.com/en-gb/HT207798
I'm developing an iOS app with screensharing. Since screenshsring on iOS is not supported by Apple, the only way is to take screenshots and transmit them.
I'm using WebRTC DataChannel to share the screenshots to a browser. It works decent, but requires a good and stable upload speed to work.
My problem is when there is slower network or when it goes up and down in speed. Because I want the screensharing to be relatively real-time. There needs to be some kind of flow control implemented. Now there will be a delay on the receiver side when the connection slows down / lag spikes.
So when the speed is lower, I want to slow down the screenshots send per second. And vice versa. I'm syncing clocks with ntp, and have a naive implementation at the moment sending the time the screenshot was taken and with that does some naive algorithm I made up. However it doesn't work as well as I need.
Is there some real time flow control algorithm you can recommend me, or any other ideas?
I'm looking for a way to read the whole video memory that a video card outputs to a display. That includes also hardware accelerated output, video playback and output in fullscreen mode (that somehow I feel could be different from windowed mode).
In short: I want to be able to capture everything that is going to be represented on a display.
I suppose that IF that's possible it would be os-dependant. The targets I'm interested in are Windows OSX and Linux.
Do you have any hint?
For windows I guess you could take CamStudio, strip it down and use it to record the screen then do whatever you want with the output, other than that you could look into forensic kernel drivers for accessing RAM. It's not exactly as simple as a pointer pointing to the video memory anymore, haha.
Digital Rights Management, requested feature of Windows, attempts to block your access to blocks of graphics-card frame buffer memory. Using an open-source driver under Linux would seem to be the only way to access this memory, or as mentioned earlier, some 3rd party software that knows some back doors or hacks or ways to locate other program's frame buffer space.
Unless of course, you are trying to capture output from your own program (ie you are calling the video/graphics creation functions yourself), there are APIs to manipulate display frames in DirectX and OpenGL.
I think I found some resources that can help to capture the display memory in Windows
Fastest method of screen capturing
How to save backbuffer to file in DirectX 10?
http://betterlogic.com/roger/2010/07/fast-screen-capture/