I use to read AVAssetReaderTrackOutput video.
Setting "kCVPixelBufferPixelFormatTypeKey" - "kCVPixelFormatType_32BGRA" work!
But I need a 16 bit video.
If set setting value "kCVPixelFormatType_16...." does not work.
[asset_reader_output copyNextSampleBuffer] - always nil =(
Why is this happening?
How do I change a bit color?
UPD:
`code:
[videoWriterInput requestMediaDataWhenReadyOnQueue:queueVideo usingBlock:^
{
while([videoWriterInput isReadyForMoreMediaData])
{
CMSampleBufferRef sampleBuffer=[video_asset_reader_output copyNextSampleBuffer];
if(sampleBuffer)
{
NSLog(#"write video");
[videoWriterInput appendSampleBuffer:sampleBuffer];
CFRelease(sampleBuffer);
} else
{
[videoWriterInput markAsFinished];
dispatch_release(queueVideo);
videoFinished=YES;
break;
}
}
}];
`
Core Video doesn't support all the pixel formats. BGRA is guaranteed to work though. You have to perform your own conversion. What are you using the buffer for?
UPDATE: To access the pixels, use something like this:
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void* bufferAddress = CVPixelBufferGetBaseAddress(pixelBuffer);
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
// Read / modify the pixel data with bufferAddress, height & bytesPerRow
// For BGRA format, it's 4-byte per pixel in that order
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
Related
I have a computer vision app that takes grayscale images from sensor and processes them. The image acquisition for iOS is written in Obj-C and the image processing is performed in C++ using OpenCV. As I only need the luminance data, I acquire the image in YUV (or Yp Cb Cr) 420 bi-planar full range format and just assign the buffer's data to an OpenCV Mat object (see aquisition code below). This worked great so far, until the brand new iOS 13 came out... For some reason, on iOS 13 the image I obtain is misaligned, resulting in diagonal stripes. By looking at the image I obtain, I suspect this is the consequence of a change in ordering of the buffer's Y Cb an Cr components or a change in the buffer's stride. Does anyone know if iOS 13 introduces this kind of changes and how I could update my code to avoid this, preferably in a backward-compatible manner?
Here is my image acquisition code:
//capture config
- (void)initialize {
AVCaptureDevice *frontCameraDevice;
NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
for (AVCaptureDevice *device in devices) {
if (device.position == AVCaptureDevicePositionFront) {
frontCameraDevice = device;
}
}
if (frontCameraDevice == nil) {
NSLog(#"Front camera device not found");
return;
}
_session = [[AVCaptureSession alloc] init];
_session.sessionPreset = AVCaptureSessionPreset640x480;
NSError *error = nil;
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:frontCameraDevice error: &error];
if (error != nil) {
NSLog(#"Error getting front camera device input: %#", error);
}
if ([_session canAddInput:input]) {
[_session addInput:input];
} else {
NSLog(#"Could not add front camera device input to session");
}
AVCaptureVideoDataOutput *videoOutput = [[AVCaptureVideoDataOutput alloc] init];
// This is the default, but making it explicit
videoOutput.alwaysDiscardsLateVideoFrames = YES;
if ([videoOutput.availableVideoCVPixelFormatTypes containsObject:
[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]]) {
OSType format = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
videoOutput.videoSettings = [NSDictionary dictionaryWithObject:[NSNumber numberWithUnsignedInt:format]
forKey:(id)kCVPixelBufferPixelFormatTypeKey];
} else {
NSLog(#"YUV format not available");
}
[videoOutput setSampleBufferDelegate:self queue:dispatch_queue_create("extrapage.camera.capture.sample.buffer.delegate", DISPATCH_QUEUE_SERIAL)];
if ([_session canAddOutput:videoOutput]) {
[_session addOutput:videoOutput];
} else {
NSLog(#"Could not add video output to session");
}
AVCaptureConnection *captureConnection = [videoOutput connectionWithMediaType:AVMediaTypeVideo];
captureConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
}
//acquisition code
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
if (_listener != nil) {
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
OSType format = CVPixelBufferGetPixelFormatType(pixelBuffer);
NSAssert(format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, #"Only YUV is supported");
// The first plane / channel (at index 0) is the grayscale plane
// See more infomation about the YUV format
// http://en.wikipedia.org/wiki/YUV
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
CGFloat width = CVPixelBufferGetWidth(pixelBuffer);
CGFloat height = CVPixelBufferGetHeight(pixelBuffer);
cv::Mat frame(height, width, CV_8UC1, baseaddress, 0);
[_listener onNewFrame:frame];
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
}
}
I found the solution to this problem. It was a row stride issue: appearently, in iOS 13, the row stride of the Yp Cb Cr 4:2:0 8 bit bi-planar buffer was changed. Maybe for it to always be a power of 2. Therefore in some cases, the row stride is no longer the same as the width. It was the case for me. The fix is easy, just get the row stride from the buffer's info and pass it to the OpenCV Mat's constructor as shown below.
void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
size_t width = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0);
size_t height = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0);
cv::Mat frame(height, width, CV_8UC1, baseaddress, bytesPerRow);
Note that I also changed how I get the width and height by using the dimensions of the plane instead of the ones of the buffer. For the Y plane, it should always be the same. I am not sure that this makes a difference.
Also be careful: after the Xcode update to support the iOS 13 SDK, I had to uninstall my app from the test device because otherwise, Xcode kept running the old version instead of the newly compiled one.
this is not an answer, but we have a similar problem. I tried with different resolution used in photo capture, only one resolution (2592 x 1936) does not work, other resolutions does work. I think change the resolution to 1440x1920 for example may be a workaround to your problem.
I am developing an iOS application and trying to get a still image snapshot from the camera using capture session but I'm unable to convert it successfully to an OpenCV Mat.
The still image output is created using this code:
- (void)createStillImageOutput;
{
// setup still image output with jpeg codec
self.stillImageOutput = [[AVCaptureStillImageOutput alloc] init];
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:AVVideoCodecJPEG, AVVideoCodecKey, nil];
[self.stillImageOutput setOutputSettings:outputSettings];
[self.captureSession addOutput:self.stillImageOutput];
for (AVCaptureConnection *connection in self.stillImageOutput.connections) {
for (AVCaptureInputPort *port in [connection inputPorts]) {
if ([port.mediaType isEqual:AVMediaTypeVideo]) {
self.videoCaptureConnection = connection;
break;
}
}
if (self.videoCaptureConnection) {
break;
}
}
NSLog(#"[Camera] still image output created");
}
And then attempting to capture a still image using this code:
[self.stillImageOutput captureStillImageAsynchronouslyFromConnection:self.videoCaptureConnection
completionHandler:
^(CMSampleBufferRef imageSampleBuffer, NSError *error)
{
if (error == nil && imageSampleBuffer != NULL)
{
NSData *jpegData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageSampleBuffer];
}
I need a way to create an OpenCV Mat based on the pixel data in the buffer.
I have tried creating a Mat using this code which I have taken from OpenCV video camera class camera here:
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void* bufferAddress;
size_t width;
size_t height;
size_t bytesPerRow;
CGColorSpaceRef colorSpace;
CGContextRef context;
int format_opencv;
OSType format = CVPixelBufferGetPixelFormatType(imageBuffer);
if (format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
format_opencv = CV_8UC1;
bufferAddress = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
width = CVPixelBufferGetWidthOfPlane(imageBuffer, 0);
height = CVPixelBufferGetHeightOfPlane(imageBuffer, 0);
bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0);
} else { // expect kCVPixelFormatType_32BGRA
format_opencv = CV_8UC4;
bufferAddress = CVPixelBufferGetBaseAddress(imageBuffer);
width = CVPixelBufferGetWidth(imageBuffer);
height = CVPixelBufferGetHeight(imageBuffer);
bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
}
cv::Mat image(height, width, format_opencv, bufferAddress, bytesPerRow);
but it fails to get the actual height and width of the image on the CVPixelBufferGetWidth or CVPixelBufferGetHeight call and so the creation of Mat fails.
I'm aware that I can create a UIImage based on the pixel data using this code:
UIImage* newImage = [UIImage imageWithData:jpegData];
But I prefer to construct a CvMat directly as it is the case in the OpenCV CvVideoCamera class because I'm interested in handling the image in OpenCV only and I don't want to consume time converting again and not risk losing quality or having issues with orientation (anyway the UIImagetoCV conversion function provided by OpenCV is causing me memory leaks and not freeing the memory).
Please advise how can I get the image as an OpenCV Mat.
Thanks in advance.
I have found the solution for my problem.
the Solution is :
by overriding this method in the OpenCv Camera Class : "createVideoPreviewLayer"
it should look like this :
- (void)createVideoPreviewLayer;
{
self.parentView.layer.sublayers = nil;
if (captureVideoPreviewLayer == nil) {
captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc]
initWithSession:self.captureSession];
}
if (self.parentView != nil) {
captureVideoPreviewLayer.frame = self.parentView.bounds;
captureVideoPreviewLayer.videoGravity =
AVLayerVideoGravityResizeAspectFill;
[self.parentView.layer addSublayer:captureVideoPreviewLayer];
}
NSLog(#"[Camera] created AVCaptureVideoPreviewLayer");
}
you should add this line to the "createVideoPreviewLayer" method will solve
the problem :
self.parentView.layer.sublayers = nil;
and you need to use the pause method instead of the stop method
I'm still learning about AVFoundation, so I'm unsure how best I should approach the problem of needing to capture a high quality still image, but provide a low-quality preview video stream.
I've got an app that needs to take high quality images (AVCaptureSessionPresetPhoto), but process the preview video stream using OpenCV - for which a much lower resolution is acceptable. Simply using the base OpenCV Video Camera class is no good, as setting the defaultAVCaptureSessionPreset to AVCaptureSessionPresetPhoto results in the full resolution frame being passed to processImage - which is very slow indeed.
How can I have a high-quality connection to the device that I can use for capturing the still image, and a low-quality connection that can be processed and displayed? A description of how I need to set up sessions/connections would be very helpful. Is there an open-source example of such an app?
I did something similar - I grabbed the pixels in the delegate method, made a CGImageRef of them, then dispatched that to the normal priority queue, where it was modified. Since AVFoundation must be using a CADisplayLink for the callback method it has highest priority. In my particular case I was not grabbing all pixels so it worked on an iPhone 4 at 30fps. Depending on what devices you want to run you have number of pixels, fps, etc trade offs.
Another idea is to grab a power of 2 subset of pixels - for instance every 4th in each row and every 4th row. Again I did something similar in my app at 20-30fps. You can then further operate on this smaller image in dispatched blocks.
If this seems daunting offer a bounty for working code.
CODE:
// Image is oriented with bottle neck to the left and the bottle bottom on the right
- (void)captureOutput:(AVCaptureVideoDataOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
#if 1
AVCaptureDevice *camera = [(AVCaptureDeviceInput *)[captureSession.inputs lastObject] device];
if(camera.adjustingWhiteBalance || camera.adjustingExposure) NSLog(#"GOTCHA: %d %d", camera.adjustingWhiteBalance, camera.adjustingExposure);
printf("foo\n");
#endif
if(saveState != saveOne && saveState != saveAll) return;
#autoreleasepool {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
//NSLog(#"PE: value=%lld timeScale=%d flags=%x", prStamp.value, prStamp.timescale, prStamp.flags);
/*Lock the image buffer*/
CVPixelBufferLockBaseAddress(imageBuffer,0);
NSRange captureRange;
if(saveState == saveOne) {
#if 0 // B G R A MODE !
NSLog(#"PIXEL_TYPE: 0x%lx", CVPixelBufferGetPixelFormatType(imageBuffer));
uint8_t *newPtr = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
NSLog(#"ONE VAL %x %x %x %x", newPtr[0], newPtr[1], newPtr[2], newPtr[3]);
}
exit(0);
#endif
[edgeFinder setupImageBuffer:imageBuffer];
BOOL success = [edgeFinder delineate:1];
if(!success) {
dispatch_async(dispatch_get_main_queue(), ^{ edgeFinder = nil; [delegate error]; });
saveState = saveNone;
} else
bottleRange = edgeFinder.sides;
xRange.location = edgeFinder.shoulder;
xRange.length = edgeFinder.bottom - xRange.location;
NSLog(#"bottleRange 1: %# neck=%d bottom=%d", NSStringFromRange(bottleRange), edgeFinder.shoulder, edgeFinder.bottom );
//searchRows = [edgeFinder expandRange:bottleRange];
rowsPerSwath = lrintf((bottleRange.length*NUM_DEGREES_TO_GRAB)*(float)M_PI/360.0f);
NSLog(#"rowsPerSwath = %d", rowsPerSwath);
saveState = saveIdling;
captureRange = NSMakeRange(0, [WLIPBase numRows]);
dispatch_async(dispatch_get_main_queue(), ^
{
[delegate focusDone];
edgeFinder = nil;
captureOutput.alwaysDiscardsLateVideoFrames = YES;
});
} else {
NSInteger rows = rowsPerSwath;
NSInteger newOffset = bottleRange.length - rows;
if(newOffset & 1) {
--newOffset;
++rows;
}
captureRange = NSMakeRange(bottleRange.location + newOffset/2, rows);
}
//NSLog(#"captureRange=%u %u", captureRange.location, captureRange.length);
/*Get information about the image*/
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
// Note Apple sample code cheats big time - the phone is big endian so this reverses the "apparent" order of bytes
CGContextRef newContext = CGBitmapContextCreate(NULL, width, captureRange.length, 8, bytesPerRow, colorSpace, kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrder32Little); // Video in ARGB format
assert(newContext);
uint8_t *newPtr = (uint8_t *)CGBitmapContextGetData(newContext);
size_t offset = captureRange.location * bytesPerRow;
memcpy(newPtr, baseAddress + offset, captureRange.length * bytesPerRow);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
OSAtomicIncrement32(&totalImages);
int32_t curDepth = OSAtomicIncrement32(&queueDepth);
if(curDepth > maxDepth) maxDepth = curDepth;
#define kImageContext #"kImageContext"
#define kState #"kState"
#define kPresTime #"kPresTime"
CMTime prStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer); // when it was taken?
//CMTime deStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer); // now?
NSDictionary *dict = [NSDictionary dictionaryWithObjectsAndKeys:
[NSValue valueWithBytes:&saveState objCType:#encode(saveImages)], kState,
[NSValue valueWithNonretainedObject:(__bridge id)newContext], kImageContext,
[NSValue valueWithBytes:&prStamp objCType:#encode(CMTime)], kPresTime,
nil ];
dispatch_async(imageQueue, ^
{
// could be on any thread now
OSAtomicDecrement32(&queueDepth);
if(!isCancelled) {
saveImages state; [(NSValue *)[dict objectForKey:kState] getValue:&state];
CGContextRef context; [(NSValue *)[dict objectForKey:kImageContext] getValue:&context];
CMTime stamp; [(NSValue *)[dict objectForKey:kPresTime] getValue:&stamp];
CGImageRef newImageRef = CGBitmapContextCreateImage(context);
CGContextRelease(context);
UIImageOrientation orient = state == saveOne ? UIImageOrientationLeft : UIImageOrientationUp;
UIImage *image = [UIImage imageWithCGImage:newImageRef scale:1.0 orientation:orient]; // imageWithCGImage: UIImageOrientationUp UIImageOrientationLeft
CGImageRelease(newImageRef);
NSData *data = UIImagePNGRepresentation(image);
// NSLog(#"STATE:[%d]: value=%lld timeScale=%d flags=%x", state, stamp.value, stamp.timescale, stamp.flags);
{
NSString *name = [NSString stringWithFormat:#"%d.png", num];
NSString *path = [[wlAppDelegate snippetsDirectory] stringByAppendingPathComponent:name];
BOOL ret = [data writeToFile:path atomically:NO];
//NSLog(#"WROTE %d err=%d w/time %f path:%#", num, ret, (double)stamp.value/(double)stamp.timescale, path);
if(!ret) {
++errors;
} else {
dispatch_async(dispatch_get_main_queue(), ^
{
if(num) [delegate progress:(CGFloat)num/(CGFloat)(MORE_THAN_ONE_REV * SNAPS_PER_SEC) file:path];
} );
}
++num;
}
} else NSLog(#"CANCELLED");
} );
}
}
In AVCaptureSessionPresetPhoto it use small video preview(about 1000x700 for iPhone6) and high resolution photo(about 3000x2000).
So I use modified 'CvPhotoCamera' class to process small preview and take photo of full-size picture. I post this code here: https://stackoverflow.com/a/31478505/1994445
I need to obtain the UIImage from uncompressed image data from CMSampleBufferRef. I'm using the code:
captureStillImageOutput captureStillImageAsynchronouslyFromConnection:connection
completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error)
{
// that famous function from Apple docs found on a lot of websites
// does NOT work for still images
UIImage *capturedImage = [self imageFromSampleBuffer:imageSampleBuffer];
}
http://developer.apple.com/library/ios/#qa/qa1702/_index.html is a link to imageFromSampleBuffer function.
But it does not work properly. :(
There is a jpegStillImageNSDataRepresentation:imageSampleBuffer method, but it gives the compressed data (well, because JPEG).
How can I get UIImage created with the most raw non-compressed data after capturing Still Image?
Maybe, I should specify some settings to video output? I'm currently using those:
captureStillImageOutput = [[AVCaptureStillImageOutput alloc] init];
captureStillImageOutput.outputSettings = #{ (id)kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_32BGRA) };
I've noticed, that output has a default value for AVVideoCodecKey, which is AVVideoCodecJPEG. Can it be avoided in any way, or does it even matter when capturing still image?
I found something there: Raw image data from camera like "645 PRO" , but I need just a UIImage, without using OpenCV or OGLES or other 3rd party.
The method imageFromSampleBuffer does work in fact I'm using a changed version of it, but if I remember correctly you need to set the outputSettings right. I think you need to set the key as kCVPixelBufferPixelFormatTypeKey and the value as kCVPixelFormatType_32BGRA.
So for example:
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary* outputSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[newStillImageOutput setOutputSettings:outputSettings];
EDIT
I am using those settings to take stillImages not video.
Is your sessionPreset AVCaptureSessionPresetPhoto? There may be problems with that
AVCaptureSession *newCaptureSession = [[AVCaptureSession alloc] init];
[newCaptureSession setSessionPreset:AVCaptureSessionPresetPhoto];
EDIT 2
The part about saving it to UIImage is identical with the one from the documentation. That's the reason I was asking for other origins of the problem, but I guess that was just grasping for straws.
There is another way I know of, but that requires OpenCV.
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image = [UIImage imageWithCGImage:quartzImage];
// Release the Quartz image
CGImageRelease(quartzImage);
return (image);
}
I guess that is of no help to you, sorry. I don't know enough to think of other origins for your problem.
Here's a more efficient way:
UIImage *image = [UIImage imageWithData:[self imageToBuffer:sampleBuffer]];
- (NSData *) imageToBuffer:(CMSampleBufferRef)source {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(source);
CVPixelBufferLockBaseAddress(imageBuffer,0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer);
NSData *data = [NSData dataWithBytes:src_buff length:bytesPerRow * height];
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
return data;
}
I am trying to encode a single YUV420P image gathered from a CMSampleBuffer to an AVPacket so that I can send h264 video over the network with RTMP.
The posted code example seems to work as avcodec_encode_video2 returns 0 (Success) however got_output is also 0 (AVPacket is empty).
Does anyone have any experience with encoding video on iOS devices that might know what I am doing wrong?
- (void) captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
// sampleBuffer now contains an individual frame of raw video frames
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
// access the data
int width = CVPixelBufferGetWidth(pixelBuffer);
int height = CVPixelBufferGetHeight(pixelBuffer);
int bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0);
unsigned char *rawPixelBase = (unsigned char *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
// Convert the raw pixel base to h.264 format
AVCodec *codec = 0;
AVCodecContext *context = 0;
AVFrame *frame = 0;
AVPacket packet;
//avcodec_init();
avcodec_register_all();
codec = avcodec_find_encoder(AV_CODEC_ID_H264);
if (codec == 0) {
NSLog(#"Codec not found!!");
return;
}
context = avcodec_alloc_context3(codec);
if (!context) {
NSLog(#"Context no bueno.");
return;
}
// Bit rate
context->bit_rate = 400000; // HARD CODE
context->bit_rate_tolerance = 10;
// Resolution
context->width = width;
context->height = height;
// Frames Per Second
context->time_base = (AVRational) {1,25};
context->gop_size = 1;
//context->max_b_frames = 1;
context->pix_fmt = PIX_FMT_YUV420P;
// Open the codec
if (avcodec_open2(context, codec, 0) < 0) {
NSLog(#"Unable to open codec");
return;
}
// Create the frame
frame = avcodec_alloc_frame();
if (!frame) {
NSLog(#"Unable to alloc frame");
return;
}
frame->format = context->pix_fmt;
frame->width = context->width;
frame->height = context->height;
avpicture_fill((AVPicture *) frame, rawPixelBase, context->pix_fmt, frame->width, frame->height);
int got_output = 0;
av_init_packet(&packet);
avcodec_encode_video2(context, &packet, frame, &got_output)
// Unlock the pixel data
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
// Send the data over the network
[self uploadData:[NSData dataWithBytes:packet.data length:packet.size] toRTMP:self.rtmp_OutVideoStream];
}
Note: It is known that this code has memory leaks because I am not freeing the memory that is dynamically allocated.
UPDATE
I updated my code to use #pogorskiy method. I only try to upload the frame if got output returns 1 and clear the buffer once I am done encoding video frames.