Video streaming via NSInputStream and NSOutputStream - ios

Right now I'm investigating possibility to implement video streaming through MultipeerConnectivity framework. For that purpose I'm using NSInputStream and NSOutputStream.
The problem is: I can't receive any picture so far. Right now I'm trying to pass simple picture and show it on the receiver. Here's a little snippet of my code:
Sending picture via NSOutputStream:
- (void)sendMessageToStream
{
NSData *imgData = UIImagePNGRepresentation(_testImage);
int img_length = (int)[imgData length];
NSMutableData *msgData = [[NSMutableData alloc] initWithBytes:&img_length length:sizeof(img_length)];
[msgData appendData:imgData];
int msg_length = (int)[msgData length];
uint8_t *readBytes = (uint8_t *)[msgData bytes];
uint8_t buf[msg_length];
(void)memcpy(buf, readBytes, msg_length);
int stream_len = [_stream writeData:(uint8_t*)buf maxLength:msg_length];
//int stream_len = [_stream writeData:(uint8_t *)buf maxLength:data_length];
//NSLog(#"stream_len = %d", stream_len);
_tmpCounter++;
dispatch_async(dispatch_get_main_queue(), ^{
_lblOperationsCounter.text = [NSString stringWithFormat:#"Sent: %ld", (long)_tmpCounter];
});
}
The code above works totally fine. stream_len parameter after writing equals to 29627 bytes which is expected value, because image's size is around 25-26 kb.
Receiving picture via NSinputStream:
- (void)readDataFromStream
{
UInt32 length;
if (_currentFrameSize == 0) {
uint8_t frameSize[4];
length = [_stream readData:frameSize maxLength:sizeof(int)];
unsigned int b = frameSize[3];
b <<= 8;
b |= frameSize[2];
b <<= 8;
b |= frameSize[1];
b <<= 8;
b |= frameSize[0];
_currentFrameSize = b;
}
uint8_t bytes[1024];
length = [_stream readData:bytes maxLength:1024];
[_frameData appendBytes:bytes length:length];
if ([_frameData length] >= _currentFrameSize) {
UIImage *img = [UIImage imageWithData:_frameData];
NSLog(#"SETUP IMAGE!");
_imgView.image = img;
_currentFrameSize = 0;
[_frameData setLength:0];
}
_tmpCounter++;
dispatch_async(dispatch_get_main_queue(), ^{
_lblOperationsCounter.text = [NSString stringWithFormat:#"Received: %ld", (long)_tmpCounter];
});
}
As you can see I'm trying to receive picture in several steps, and here's why. When I'm trying to read data from stream, it's always reading maximum 1095 bytes no matter what number I put in maxLength: parameter. But when I send the picture in the first snippet of code, it's sending absolutely ok (29627 bytes . Btw, image's size is around 29 kb.
That's the place where my question come up - why is that? Why is sending 29 kb via NSOutputStream works totally fine when receiving is causing problems? And is there a solid way to make video streaming work through NSInputStream and NSOutputStream? I just didn't find much information about this technology, all I found were some simple things which I knew already.

Here's an app I wrote that shows you how:
https://app.box.com/s/94dcm9qjk8giuar08305qspdbe0pc784
Build the project with Xcode 9 and run the app on two iOS 11 devices.
To stream live video, touch the Camera icon on one of two devices.
If you don't have two devices, you can run one app in the Simulator; however, you can only use the camera on the real device (the Simulator will display the video broadcasted).
Just so you know: this is not the ideal way to stream real-time video between devices (it should probably be your last choice). Data packets (versus streaming) are way more efficient and faster.
Regardless, I'm really confused by your NSInputStream-related code. Here's something that makes a little more sense, I think:
case NSStreamEventHasBytesAvailable: {
// len is a global variable set to a non-zero value;
// mdata is a NSMutableData object that is reset when a new input
// stream is created.
// displayImage is a block that accepts the image data and a reference
// to the layer on which the image will be rendered
uint8_t * buf[len];
len = [aStream read:(uint8_t *)buf maxLength:len];
if (len > 0) {
[mdata appendBytes:(const void *)buf length:len];
} else {
displayImage(mdata, wLayer);
}
break;
}
The output stream code should look something like this:
// data is an NSData object that contains the image data from the video
// camera;
// len is a global variable set to a non-zero value
// byteIndex is a global variable set to zero each time a new output
// stream is created
if (data.length > 0 && len >= 0 && (byteIndex <= data.length)) {
len = (data.length - byteIndex) < DATA_LENGTH ? (data.length - byteIndex) : DATA_LENGTH;
uint8_t * bytes[len];
[data getBytes:&bytes range:NSMakeRange(byteIndex, len)];
byteIndex += [oStream write:(const uint8_t *)bytes maxLength:len];
}
There's a lot more to streaming video than setting up the NSStream classes correctly—a lot more. You'll notice in my app, I created a cache for the input and output streams. This solved a myriad of issues that you would likely encounter if you don't do the same.
I have never seen anyone successfully use NSStreams for video streaming...ever. It's highly complex, for one reason.
There are many different (and better) ways to stream video; I wouldn't go this route. I just took it on because no one else has been able to do it successfully.

I think that the problem is in your assumption that all data will be available in NSInputStream all the time while you are reading it. NSInputStream made from NSURL object has an asynchronous nature and it should be accessed accordingly using NSStreamDelegate. You can look at example in the README of POSInputStreamLibrary.

Related

Store the ADC stream on µSD Card without loss on STM32H743ZI

I am working on a project in which I have to store the datas of an ADC Stream on a µSD card. However even if I use a 16 bits buffer, I lose data from the ADC stream. My ADC is used with DMA and I use FATFS (WITHOUT DMA) and the SDMMC1 peripheral to fill a .bin file with the datas.
Do you have an idea to avoid this loss ?
Here is my project : https://github.com/mathieuchene/STM32H743ZI
I use a nucleo-h743zi2 Board, CubeIDE, and CubeMx in their last version.
EDIT 1
I tried to implement Colin's solution, it's better but I have a strange things in the middle of my acquisition. However when I increase the maximal count value or try to debug, the HardFault_Handler appears. I modified main.c file by creating 2 blocks (uint16_t blockX[BUFFERLENGTH/2]) and 2 flags for when adcBuffer is half filled or completely filled.
I also changed the while(1) part in main function like this
if (flagHlfCplt){
//flagCplt=0;
res = f_write(&SDFile, block1, strlen((char*)block1), (void *)&byteswritten);
memcpy(block2, adcBuffer, BUFFERLENGTH/2);
flagHlfCplt = 0;
count++;
}
if (flagCplt){
//flagHlfCplt=0;
res = f_write(&SDFile, block2, strlen((char*)block2), (void *)&byteswritten);
memcpy(block1, adcBuffer[(BUFFERLENGTH/2)-1], BUFFERLENGTH/2);
flagCplt = 0;
count++;
}
if (count == 10){
f_close(&SDFile);
HAL_ADC_Stop_DMA(&hadc1);
while(1){
HAL_GPIO_TogglePin(LD1_GPIO_Port, LD1_Pin);
HAL_Delay(1000);
}
}
}
EDIT 2
I modified my program. I set block 1 and block 2 with the length of BUFFERLENGTH and I added a pointer (*idx) to change the buffer which is filled. I don't have HardFault_Handler anymore but I still loose some datas from my adc's stream.
Here are the modification I made:
// my pointer and buffers
uint16_t block1[BUFFERLENGTH], block2[BUFFERLENGTH], *idx;
// init of pointer and adc start
idx=block1;
HAL_ADC_Start_DMA(&hadc1, (uint32_t*)idx, BUFFERLENGTH);
// while(1) part
while (1)
{
if (flagCplt){
if (flagToChangeBuffer) {
idx=block1;
res = f_write(&SDFile, block2, strlen((char*)block2), (void *)&byteswritten);
flagCplt = 0;
flagToChangeBuffer=0;
count++;
}
else {
idx=block2;
res = f_write(&SDFile, block1, strlen((char*)block1), (void *)&byteswritten);
flagCplt = 0;
flagToChangeBuffer=1;
count++;
}
}
if (count == 150){
f_close(&SDFile);
HAL_ADC_Stop_DMA(&hadc1);
while(1){
HAL_GPIO_TogglePin(LD1_GPIO_Port, LD1_Pin);
HAL_Delay(1000);
}
}
}
Does someone know how to solve my matter with these loss?
Best Regards
Mathieu

libyuv crash on iPhone

I get a libyuv crash recently.
I try a lot, but no use.
Please help or try to give some ideas how to achieve this. Thanks!
I have a iOS project(Objective C). One of the functions is encode the video stream.
My idea is
Step 1: Start a timer(20 FPS)
Step 2: Copy and get the bitmap data
Step 3: Transfer the bitmap data to YUV I420 (libyuv)
Step 4: Encode to the h264 format (Openh264)
Step 5: Send the h264 data with RTSP
All of function run on the foreground.
It works well for 3~4hr.
BUT it always will be crashed after 4hr+.
Check the CPU(39%), Memory(140MB), it is stable(No memory leak, CPU busy, etc.).
I try a lot, but no use ( Include add try-catch in my project, detect the data size before run in this line )
I figure out it will run more if decrease the FPS time(20FPS -> 15FPS)
Does it need to add something after encode each frame?
Could someone help me or give some idea for this? Thanks!
// This function runs in a GCD timer
- (void)processSDLFrame:(NSData *)_frameData {
if (mH264EncoderPtr == NULL) {
[self initEncoder];
return;
}
int argbSize = mMapWidth * mMapHeight * 4;
NSData *frameData = [[NSData alloc] initWithData:_frameData];
if ([frameData length] == 0 || [frameData length] != argbSize) {
NSLog(#"Incorrect frame with size : %ld\n", [frameData length]);
return;
}
SFrameBSInfo info;
memset(&info, 0, sizeof (SFrameBSInfo));
SSourcePicture pic;
memset(&pic, 0, sizeof (SSourcePicture));
pic.iPicWidth = mMapWidth;
pic.iPicHeight = mMapHeight;
pic.uiTimeStamp = [[NSDate date] timeIntervalSince1970];
#try {
libyuv::ConvertToI420(
static_cast<const uint8 *>([frameData bytes]), // sample
argbSize, // sample_size
mDstY, // dst_y
mStrideY, // dst_stride_y
mDstU, // dst_u
mStrideU, // dst_stride_u
mDstV, // dst_v
mStrideV, // dst_stride_v
0, // crop_x
0, // crop_y
mMapWidth, // src_width
mMapHeight, // src_height
mMapWidth, // crop_width
mMapHeight, // crop_height
libyuv::kRotateNone, // rotation
libyuv::FOURCC_ARGB); // fourcc
} #catch (NSException *exception) {
NSLog(#"libyuv::ConvertToI420 - exception:%#", exception.reason);
return;
}
pic.iColorFormat = videoFormatI420;
pic.iStride[0] = mStrideY;
pic.iStride[1] = mStrideU;
pic.iStride[2] = mStrideV;
pic.pData[0] = mDstY;
pic.pData[1] = mDstU;
pic.pData[2] = mDstV;
if (mH264EncoderPtr == NULL) {
NSLog(#"OpenH264Manager - encoder not initialized");
return;
}
int rv = -1;
#try {
rv = mH264EncoderPtr->EncodeFrame(&pic, &info);
} #catch (NSException *exception) {
NSLog( #"NSException caught - mH264EncoderPtr->EncodeFrame" );
NSLog( #"Name: %#", exception.name);
NSLog( #"Reason: %#", exception.reason );
[self deinitEncoder];
return;
}
if (rv != cmResultSuccess) {
NSLog(#"OpenH264Manager - encode failed : %d", rv);
[self deinitEncoder];
return;
}
if (info.eFrameType == videoFrameTypeSkip) {
NSLog(#"OpenH264Manager - drop skipped frame");
return;
}
// handle buffer data
int size = 0;
int layerSize[MAX_LAYER_NUM_OF_FRAME] = { 0 };
for (int layer = 0; layer < info.iLayerNum; layer++) {
for (int i = 0; i < info.sLayerInfo[layer].iNalCount; i++) {
layerSize[layer] += info.sLayerInfo[layer].pNalLengthInByte[i];
}
size += layerSize[layer];
}
uint8 *output = (uint8 *)malloc(size);
size = 0;
for (int layer = 0; layer < info.iLayerNum; layer++) {
memcpy(output + size, info.sLayerInfo[layer].pBsBuf, layerSize[layer]);
size += layerSize[layer];
}
// alloc new buffer for streaming
NSData *newData = [NSData dataWithBytes:output length:size];
// Send the data with RTSP
sendData( newData );
// free output buffer data
free(output);
}
[Jan/08/2020 Update]
I report this ticket on the Google Issue Report
https://bugs.chromium.org/p/libyuv/issues/detail?id=853
The Googler give me a feedback.
ARGBToI420 does no allocations. Its similar to a memcpy with a source and destination and number of pixels to convert.
The most common issues with it are
1. the destination buffer has been deallocated. Try adding validation that the YUV buffer is valid. Write to the first and last byte of each layer.
This often occurs on shutdown and threads dont shut down in the order you were hoping. A mutex to guard the memory could help.
2. the destination is an odd size and the allocator did not allocate enough memory. When alllocating the UV plane, use (width + 1) / 2 for width/stride and (height + 1) / 2 for height of UV. Allocate stride * height bytes. You could also use an allocator that verifies there are no overreads or overwrites, or a sanitizer like asan / msan.
When screen casting, usually windows are a multiple of 2 pixels on Windows and Linux, but I have seen MacOS use odd pixel count.
As a test you could wrap the function with temporary buffers. Copy the ARGB to a temporary ARGB buffer.
Call ARGBToI420 to a temporary I420 buffer.
Copy the I420 result to the final I420 buffer.
That should give you a clue which buffer/function is failing.
I will try them.

How can I receive simple integer values over bluetooth in ios

I am trying to learn how to get some sensors plugged into an Arduino board to talk to an iPhone over Bluetooth with a Red Bear Labs mini board but have hit a brick wall.
The sensors get a reading and this is sent to the phone over BLE. So far I've connected to the device and I get back what appears to be data but I can't make sense of it.
I've written a little sketch that looks like this, to simulate the sensor data.
#include <SoftwareSerial.h>
SoftwareSerial bluetooth(5, 6);
void setup() {
bluetooth.begin(57600);
}
void loop() {
//int reading = analogRead(2);
int reading = 123; // fake reading
byte lowerByte = (byte) reading & 0xFF;
byte upperByte = (byte) (reading >> 8) & 0xFF;
bluetooth.write(reading);
bluetooth.write(upperByte);
bluetooth.write(lowerByte);
delay(1000);
}
In iOS I send a call to read the data and then the data is received by a piece of code that looks something like:
- (void)peripheral:(CBPeripheral *)peripheral
didUpdateValueForCharacteristic:(CBCharacteristic *)characteristic
error:(NSError *)error
{
Byte data[20];
static unsigned char buf[512];
static int len = 0;
NSInteger data_len;
if (!error && [characteristic.UUID isEqual:[CBUUID UUIDWithString:#RBL_CHAR_TX_UUID]]){
data_len = characteristic.value.length;
[characteristic.value getBytes:data length:data_len];
if (data_len == 20){
memcpy(&buf[len], data, 20);
len += data_len;
if (len >= 64){
[[self delegate] bleDidReceiveData:buf length:len];
len = 0;
}
} else if (data_len < 20) {
memcpy(&buf[len], data, data_len);
len += data_len;
[[self delegate] bleDidReceiveData:buf length:len];
len = 0;
}
}
}...
}
But when I look at the data that comes back it just makes no sense to me at all.. (I'll dig out an example as soon as I can).
Does anyone know a simple step I'm missing or a good example I could look at to try and better understand this?
I finally realised that the data was correct, I had to 'pull out' the data by bit shifting it.
UInt16 value;
UInt16 pin;
for (int i = 0; i < length; i+=3) {
pin = data[i];
value = data[i+2] | data[i+1] << 8;
NSLog(#"Pin: %d", pin);
NSLog(#"Value %d",value);
}

NSMutableData encryption in place using NSInputStream

I am trying to use CommonCrypto to encrypt an NSMutableData object in place (copying the resulting bytes to itself, without duplicating it). Previously, I was using CCCrypt() "one-shot" method, mainly because it seemed simple. I noticed that my data object got duplicated in memory.
To avoid this, I tried using an NSInputStream object with a buffer size of 2048 bytes. I am reading my NSMutableData object, and continuously call CCCryptorUpdate(), to handle the encryption. The problem is, that it still seems to be duplicated. Here's my current code (please note that it's a category on NSMutableData - mainly because of historical reasons - thus the "self" references):
- (BOOL)encryptWithKey:(NSString *)key
{
// Key creation - not relevant to the dercribed problem
char * keyPtr = calloc(1, kCCKeySizeAES256+1);
[key getCString: keyPtr maxLength: sizeof(keyPtr) encoding: NSUTF8StringEncoding];
// Create cryptographic context for encryption
CCCryptorRef cryptor;
CCCryptorStatus status = CCCryptorCreate(kCCEncrypt, kCCAlgorithmAES128, kCCOptionECBMode, keyPtr, kCCKeySizeAES256, NULL, &cryptor);
if (status != kCCSuccess)
{
MCLog(#"Failed to create a cryptographic context (%d CCCryptorStatus status).", status);
}
// Initialize the input stream
NSInputStream *inStream = [[NSInputStream alloc] initWithData:self];
[inStream open];
NSInteger result;
// BUFFER_LEN is a define 2048
uint8_t buffer[BUFFER_LEN];
size_t bytesWritten;
while ([inStream hasBytesAvailable])
{
result = [inStream read:buffer maxLength:BUFFER_LEN];
if (result > 0)
{
// Encryption goes here
status = CCCryptorUpdate(
cryptor, // Previously created cryptographic context
&result, // Input data
BUFFER_LEN, // Length of the input data
[self mutableBytes], // Result is written here
[self length], // Size of result
&bytesWritten // Number of bytes written
);
if (status != kCCSuccess)
{
MCLog(#"Error during data encryption (%d CCCryptorStatus status)", status);
}
}
else
{
// Error
}
}
// Cleanup
[inStream close];
CCCryptorRelease(cryptor);
free(keyPtr);
return ( status == kCCSuccess );
}
I am definitely missing something obvious here, encryption, and even using input streams is a bit new to me..
As long as you only call CCUpdate() one time, you can encrypt into the same buffer you read from without using a stream. See RNCryptManager.m for an example. Study applyOperation:fromStream:toStream:password:error:. I did use streams here, but there's no requirement that you do that if you already have an NSData.
You must ensure that CCUpdate() is only called one time, however. If you call it multiple times it will corrupt its own buffer. This is an open bug in CommonCryptor (radar://9930555).
As a side note: your key generation is extremely insecure, and use of ECB mode for this kind of data barely qualifies as encryption. It leaves patterns in the ciphertext which can be used to decrypt the data, in some cases just by looking at it. I do not recommend this approach if you actually intend to secure this data. If you want to study how to use these tools well, see Properly Encrypting With AES With CommonCrypto. If you want a prepackaged solution, see RNCryptor. (RNCryptor does not currently have a convenient method for encrypting in-place, however.)
In the line:
result = [inStream read:buffer maxLength:BUFFER_LEN];
the data is read into buffer and result is set to the outcome of the execution.
in lines:
status = CCCryptorUpdate(cryptor, &result, ...
You should be using buffer for the input data, not the status
status = CCCryptorUpdate(cryptor, buffer, ...
Using better names would help eliminate the simple error. If instead of result the variable had been named readStatus the error would most likely not occurred. Likewise instead of naming rthe data variable buffer it had been named streamData things would also have been more clear. Poor naming really can cause errors.

RTMP Streaming from ios to server?

Right now, I am using a static librtmp.a library to open a RTMP connection between my iphone and server. When the record button is pressed, the camera starts taking input, and on captureoutput, uses AVAssetWriter's in different threads to encode the videos to h264/AAC. The videos are then saved to a specific URL. I'm trying to take these processed frames and send them over my RTMP client using librtmp.
-(void)writeURL:(NSURL*)segmentURL {
NSLog(#"Utilities are writing...");
NSData *segmentData = [NSData dataWithContentsOfURL:segmentURL];
const char* body = [segmentData bytes];
NSLog(#"%i", [segmentData length]);
NSLog(#"%s", body);
RTMPPacket packet = _rtmp->m_write;
RTMPPacket_Alloc(&packet, [segmentData length]);
packet.m_headerType = RTMP_PACKET_SIZE_MEDIUM;
packet.m_packetType = RTMP_PACKET_TYPE_VIDEO;
packet.m_body = (char*)body;
RTMPPacket_Dump(&packet);
RTMP_SendPacket(_rtmp, &packet, TRUE);
//RTMP_Write(_rtmp, packet.m_body, packet.m_nBodySize);
}
crashes everytime on the RTMPPacket_alloc call and I'm unsure what to do. Is this the right way to approach sending the data over the network?
EDIT:
sample output
2013-03-31 22:53:16.163 videoAppPrototype[2567:907] Switching encoders
2013-03-31 22:53:16.179 videoAppPrototype[2567:1703] Encoder switch finished
2013-03-31 22:53:16.220 videoAppPrototype[2567:1703] Upload public.mpeg-4
2013-03-31 22:53:16.223 videoAppPrototype[2567:1703] Utilities are writing...
2013-03-31 22:53:16.230 videoAppPrototype[2567:1703] 339171
DEBUG: RTMP PACKET: packet type: 0x09. channel: 0x00. info 1: 0 info 2: 0. Body size: 0. body: 0x00
EDIT 2:
I changed my code to use RTMP_Write() instead of RTMP_SendPacket().
New Method:
-(void)writeURL:(NSURL*)segmentURL {
NSLog(#"Utilities are writing...");
NSData *segmentData = [NSData dataWithContentsOfURL:segmentURL];
NSUInteger len = [segmentData length] / sizeof(unsigned char);
Byte *byteData = (Byte*)malloc(len);
memcpy(byteData, [segmentData bytes], len);
free(byteData);
NSLog(#"%i", [segmentData length]);
NSLog(#"First write attempt...");
RTMP_Write(_rtmp, (char *)byteData, len);
NSLog(#"Successful?");
}
Which now crashes at RTMP_Write, as shown in stack trace:
If someone has any idea or needs anymore information, lemme know!

Resources