Why AVAssetWriter inflates video file? - ios

Strange problem. I take frames from a video file (.mov) and write them with AVAssetWriter to another file without any explicit processing. Actually I just copy the frame from one memory buffer to another and them flush them through PixelbufferAdaptor. Then I take the resulting file, delete the original file, put the resulting file instead the original and do the same operation. Interesting thing is that the size of the file constantly grows! Can somebody explain why?
if(adaptor.assetWriterInput.readyForMoreMediaData==YES) {
CVImageBufferRef cvimgRef=nil;
CMTime lastTime=CMTimeMake(fcounter++, 30);
CMTime presentTime=CMTimeAdd(lastTime, frameTime);
CMSampleBufferRef framebuffer=nil;
CGImageRef frameImg=nil;
if ( [asr status]==AVAssetReaderStatusReading ){
framebuffer = [asset_reader_output copyNextSampleBuffer];
frameImg = [self imageFromSampleBuffer:framebuffer withColorSpace:rgbColorSpace];
}
if(frameImg && screenshot){
//CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(framebuffer);
CVReturn stat= CVPixelBufferLockBaseAddress(screenshot, 0);
pxdata=CVPixelBufferGetBaseAddress(screenshot);
bufferSize = CVPixelBufferGetDataSize(screenshot);
// Get the number of bytes per row for the pixel buffer.
bytesPerRow = CVPixelBufferGetBytesPerRow(screenshot);
// Get the pixel buffer width and height.
width = CVPixelBufferGetWidth(screenshot);
height = CVPixelBufferGetHeight(screenshot);
// Create a Quartz direct-access data provider that uses data we supply.
CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, pxdata, bufferSize, NULL);
CGImageAlphaInfo ai=CGImageGetAlphaInfo(frameImg);
size_t bpx=CGImageGetBitsPerPixel(frameImg);
CGColorSpaceRef fclr=CGImageGetColorSpace(frameImg);
// Create a bitmap image from data supplied by the data provider.
CGImageRef cgImage = CGImageCreate(width, height, 8, 32, bytesPerRow,rgbColorSpace, kCGImageAlphaNoneSkipLast | kCGBitmapByteOrder32Big,dataProvider, NULL, true, kCGRenderingIntentDefault);
CGDataProviderRelease(dataProvider);
stat= CVPixelBufferLockBaseAddress(finalPixelBuffer, 0);
pxdata=CVPixelBufferGetBaseAddress(finalPixelBuffer);
bytesPerRow = CVPixelBufferGetBytesPerRow(finalPixelBuffer);
CGContextRef context = CGBitmapContextCreate(pxdata, imgsize.width,imgsize.height, 8, bytesPerRow, rgbColorSpace, kCGImageAlphaNoneSkipLast);
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(frameImg), CGImageGetHeight(frameImg)), frameImg);
//CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
//CGImageRef myMaskedImage;
const float myMaskingColors[6] = { 0, 0, 0, 1, 0, 0 };
CGImageRef myColorMaskedImage = CGImageCreateWithMaskingColors (cgImage, myMaskingColors);
//CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(myColorMaskedImage), CGImageGetHeight(myColorMaskedImage)), myColorMaskedImage);
[adaptor appendPixelBuffer:finalPixelBuffer withPresentationTime:presentTime];}

well, the mystery seems to be solved. The problem was in inappropriate codec configuration.
This is set of configuration options I use now and it seems to do the work:
NSDictionary *codecSettings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:1100000], AVVideoAverageBitRateKey,
[NSNumber numberWithInt:5],AVVideoMaxKeyFrameIntervalKey,
nil];
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:[SharedApplicationData sharedData].overlayView.frame.size.width], AVVideoWidthKey,
[NSNumber numberWithInt:[SharedApplicationData sharedData].overlayView.frame.size.height], AVVideoHeightKey,
codecSettings,AVVideoCompressionPropertiesKey,
nil];
AVAssetWriterInput* writerInput = [AVAssetWriterInput
assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings];
Now the file size still grows but at much slower pace. There is a tradeoff between the file size and the video quality - size reduction affects the quality.

Related

Image from CMSampleBufferRef is always white

I am trying to get each frame from the replaykit using startCaptureWithHandler.
startCaptureWithHandler returns a CMSampleBufferRef which i need to convert to an image.
Im using this method to convert to UIImage but its always white.
- (UIImage *) imageFromSampleBuffer3:(CMSampleBufferRef) sampleBuffer
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey, [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil];
CVPixelBufferRef pxbuffer = NULL;
CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer);
CVPixelBufferLockFlags flags = (CVPixelBufferLockFlags)0;
CVPixelBufferLockBaseAddress(pxbuffer, flags);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
// CGContextRef context = CGBitmapContextCreate(pxdata, width, height, 8, CVPixelBufferGetBytesPerRow(pxbuffer), rgbColorSpace, kCGImageAlphaPremultipliedFirst);
CGContextRef context = CGBitmapContextCreate(pxdata, width, height, 8, CVPixelBufferGetBytesPerRow(pxbuffer), rgbColorSpace, kCGImageAlphaPremultipliedFirst);
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, flags);
UIImage *image = [UIImage imageWithCGImage:quartzImage scale:1.0f orientation:UIImageOrientationRight];
CGImageRelease(quartzImage);
return image;
}
Can anyone tell me where im going wrong?
sampleBuffer is 420f format and it has 2 planar.
For locking memory, CVPixelBufferLockBaseAddress(imageBuffer, 0 then 1).
For getting Y plane data, CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0).
For UV plane data, CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1).
Do not forget unlock memory. I am not sure how to convert YUV to RGB.
On your code, you do not read image data from imageBuffer.
You only work on pxbuffer and pxdata which has no image data.

How to convert BGRA bytes to UIImage for saving?

I want to capture raw pixel data for manipulation using GPUImage framework. I capture the data like this:
CVImageBufferRef cameraFrame = CMSampleBufferGetImageBuffer(imageSampleBuffer);
CVPixelBufferLockBaseAddress(cameraFrame, 0);
GLubyte *rawImageBytes = CVPixelBufferGetBaseAddress(cameraFrame);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(cameraFrame);
NSData *dataForRawBytes = [NSData dataWithBytes:rawImageBytes length:bytesPerRow * CVPixelBufferGetHeight(cameraFrame)];
//raw values
UInt32 *values = [dataForRawBytes bytes];//, cnt = [dataForRawBytes length]/sizeof(int);
//test out dropbox upload here
[self uploadDropbox:dataForRawBytes];
//end of dropbox upload
// Do whatever with your bytes
// [self processImages:dataForRawBytes];
CVPixelBufferUnlockBaseAddress(cameraFrame, 0); }];
I am using the following settings for camera:
NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:AVVideoCodecJPEG, AVVideoCodecKey,[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];
For testing purposes I want to save the image i capture to dropbox, to do that I need to save it to a tmp directory, how would i save dataForRawBytes?
Any help would be very appreciated!
So i was able to figure out how to gain a UIImage from the raw data, here is my modified code:
CVImageBufferRef cameraFrame = CMSampleBufferGetImageBuffer(imageSampleBuffer);
CVPixelBufferLockBaseAddress(cameraFrame, 0);
Byte *rawImageBytes = CVPixelBufferGetBaseAddress(cameraFrame);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(cameraFrame);
size_t width = CVPixelBufferGetWidth(cameraFrame);
size_t height = CVPixelBufferGetHeight(cameraFrame);
NSData *dataForRawBytes = [NSData dataWithBytes:rawImageBytes length:bytesPerRow * CVPixelBufferGetHeight(cameraFrame)];
// Do whatever with your bytes
// create suitable color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
//Create suitable context (suitable for camera output setting kCVPixelFormatType_32BGRA)
CGContextRef newContext = CGBitmapContextCreate(rawImageBytes, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
// release color space
CGColorSpaceRelease(colorSpace);
//Create a CGImageRef from the CVImageBufferRef
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
UIImage *FinalImage = [[UIImage alloc] initWithCGImage:newImage];
//is the image captured, now we can test saving it.
I needed to create properties such as colourspace and generate a CDContexyRef and work with that to finally get a UIImage, and when debugging I can properly see the image i captured.

How to go from vImage_Buffer to CVPixelBufferRef

I'm recording live video in my iOS app. On another Stack Overflow page, I found that you can use vImage_Buffer to work on my frames.
The problem is that I have no idea how to get back to a CVPixelBufferRef from the outputted vImage_buffer.
Here is the code that is given in the other article:
NSInteger cropX0 = 100,
cropY0 = 100,
cropHeight = 100,
cropWidth = 100,
outWidth = 480,
outHeight = 480;
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
vImage_Buffer inBuff;
inBuff.height = cropHeight;
inBuff.width = cropWidth;
inBuff.rowBytes = bytesPerRow;
int startpos = cropY0 * bytesPerRow + 4 * cropX0;
inBuff.data = baseAddress + startpos;
unsigned char *outImg = (unsigned char*)malloc(4 * outWidth * outHeight);
vImage_Buffer outBuff = {outImg, outHeight, outWidth, 4 * outWidth};
vImage_Error err = vImageScale_ARGB8888(&inBuff, &outBuff, NULL, 0);
if (err != kvImageNoError) NSLog(#" error %ld", err);
And now I need to convert outBuff to a CVPixelBufferRef.
I assume I need to use vImageBuffer_CopyToCVPixelBuffer, but I'm not sure how.
My first attempts failed with an EXC_BAD_ACCESS: CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
CVPixelBufferRef pixelBuffer;
CVPixelBufferCreate(kCFAllocatorSystemDefault, 480, 480, kCVPixelFormatType_32BGRA, NULL, &pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
vImage_CGImageFormat format = {
.bitsPerComponent = 8,
.bitsPerPixel = 32,
.bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst, //BGRX8888
.colorSpace = NULL, //sRGB
};
vImageBuffer_CopyToCVPixelBuffer(&outBuff,
&format,
pixelBuffer,
NULL,
NULL,
kvImageNoFlags); // Here is the crash!
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
Any idea?
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool : YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool : YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
[NSNumber numberWithInt : 480], kCVPixelBufferWidthKey,
[NSNumber numberWithInt : 480], kCVPixelBufferHeightKey,
nil];
status = CVPixelBufferCreateWithBytes(kCFAllocatorDefault,
480,
480,
kCVPixelFormatType_32BGRA,
outImg,
bytesPerRow,
NULL,
NULL,
(__bridge CFDictionaryRef)options,
&pixbuffer);
You should generate a new pixelBuffer like above.
Just in case... if you want a cropped live video feed into your interface, use an AVPlayerLayer, AVCaptureVideoPreviewLayer and/or other CALayer subclasses, use the layer bounds, frame and position for your 100x100 pixel area to 480x480 area.
Notes for vImage for your question (different circumstances may differ):
CVPixelBufferCreateWithBytes will not work with vImageBuffer_CopyToCVPixelBuffer() because you need to copy the vImage_Buffer data into a "clean" or "empty" CVPixelBuffer.
No need for locking/unlocking - make sure you know when to lock & when not to lock pixel buffers.
Your inBuff vImage_Buffer just needs to be initialized from the pixel buffer data, not manually (unless you know how to use CGContexts etc, to init the pixel grid)
use vImageBuffer_InitWithCVPixelBuffer()
vImageScale_ARGB8888 will scale the entire CVPixel data to a smaller/larger rectangle. It won't SCALE a portion/crop area of the buffer to another buffer.
When you use vImageBuffer_CopyToCVPixelBuffer(),
vImageCVImageFormatRef & vImage_CGImageFormat need to be filled out correctly.
CGColorSpaceRef dstColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceITUR_709);
vImage_CGImageFormat format = {
.bitsPerComponent = 16,
.bitsPerPixel = 64,
.bitmapInfo = (CGBitmapInfo)kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder16Big ,
.colorSpace = dstColorSpace
};
vImageCVImageFormatRef vformat = vImageCVImageFormat_Create(kCVPixelFormatType_4444AYpCbCr16,
kvImage_ARGBToYpCbCrMatrix_ITU_R_709_2,
kCVImageBufferChromaLocation_Center,
format.colorSpace,
0);
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
480,
480,
kCVPixelFormatType_4444AYpCbCr16,
NULL,
&destBuffer);
NSParameterAssert(status == kCVReturnSuccess && destBuffer != NULL);
err = vImageBuffer_CopyToCVPixelBuffer(&sourceBuffer, &format, destBuffer, vformat, 0, kvImagePrintDiagnosticsToConsole);
NOTE: these are settings for 64 bit ProRes with Alpha - adjust for 32 bit.

UIImages exported as movie error

Problem
My AVAssetWriter is failing after appending 5 or so images to it using a AVAssetWriterInputPixelBufferAdaptor, and I have no idea why.
Details
This popular question helped but isn't working for my needs:
How do I export UIImage array as a movie?
Everything works as planned, I even delay the assetWriterInput until it can handle more media.
But for some reason, it always fails after 5 or so images. The images I'm using are extracted frames from a GIF
Code
Here is my iteration code:
-(void)writeImageData
{
__block int i = 0;
videoQueue = dispatch_queue_create("com.videoQueue", DISPATCH_QUEUE_SERIAL);
[self.writerInput requestMediaDataWhenReadyOnQueue:dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0) usingBlock:^{
while (self.writerInput.readyForMoreMediaData) {
if (i >= self.imageRefs.count){
[self endSession];
videoQueue = nil;
[self saveToLibraryWithCompletion:^{
NSLog(#"Saved");
}];
break;
}
if (self.writerInput.readyForMoreMediaData){
CGImageRef imageRef = (__bridge CGImageRef)self.imageRefs[i];
CVPixelBufferRef buffer = [self pixelBufferFromCGImageRef:imageRef];
CGFloat timeScale = (CGFloat)self.imageRefs.count / self.originalDuration;
BOOL accepted = [self.adaptor appendPixelBuffer:buffer withPresentationTime:CMTimeMake(i, timeScale)];
CVBufferRelease(buffer);
if (!accepted){
NSLog(#"Buffer did not add %#, index %d, timescale %f", self.writer.error, i, timeScale);
}else{
NSLog(#"Buffer did nothing wrong");
}
i++;
}
}
}];
}
My other bits of code match the code from the Link above. This is only slightly different:
-(CVPixelBufferRef)pixelBufferFromCGImageRef:(CGImageRef)image
{
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CGFloat width = 640;
CGFloat height = 640;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, width,
height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
&pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, width,
height, 8, 4*width, rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextDrawImage(context, CGRectMake(0, 0, width,
height), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
One thing that stands out to me is your use of CMTimeMake(adjustedTime, 1).
You need to calculate the time of each frame properly. Note that CMTime takes two integers, and passing them as floating point values in truncates them.
The second issue is that you weren't using your serial dispatch queue :)

IOS 5: UIView converted to video and the resulting video is corrupted

I want to grab the UIView, convert to image and than store that image in a video file (.mp4).
I use the next portion of code which grabs the image and puts it to pixel buffer:
BOOL appended;
if(input.readyForMoreMediaData==YES){
//grab the view and convert it into image
CGSize imgsize=self.imageSource.frame.size;
UIGraphicsBeginImageContext(imgsize);
[self.imageSource.layer renderInContext:UIGraphicsGetCurrentContext()];
UIImage* grabbedImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
CVReturn cvErr = kCVReturnSuccess;
CGImageRef image = (CGImageRef) [grabbedImage CGImage];
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, imgsize.width,
imgsize.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
&pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, imgsize.width,
imgsize.height, 8, 4*imgsize.width, rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextConcatCTM(context, CGAffineTransformMakeRotation(0));
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image),
CGImageGetHeight(image)), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
appended = [pxlBufAdaptor appendPixelBuffer:pxbuffer withPresentationTime:presentationTime];
CVBufferRelease(pxbuffer );
}
The problem is that the resulting video contains the corrupted image - all pixels are offset . It looks like the memory is filled with bytes with some offset and that offset corrupts the presentation.
How can this be fixed?
I would like to have any glue or direction.
Thanks in advance.
This looks suspicious:
CGContextRef context = CGBitmapContextCreate(pxdata, imgsize.width,
imgsize.height, 8, 4*imgsize.width, rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
You are computing the bytesPerRow parameter based on the image width, instead of asking pxbuffer for its bytes-per-row. Try this:
CGContextRef context = CGBitmapContextCreate(pxdata, imgsize.width,
imgsize.height, 8, CVPixelBufferGetBytesPerRow(pxbuffer),
rgbColorSpace, kCGImageAlphaNoneSkipFirst);
Also, it seems inefficient to create a bitmap graphics context with UIGraphicsGetCurrentContext, render the layer into the context, get an image from the context, destroy the context, create a pixel buffer, create a bitmap graphics context using the pixel buffer, and draw the image of the layer into the new context. Why not replace your CGContextDrawImage call with [self.imageSource.layer renderInContext:context]?

Resources