iOS drawing screen video capture not smooth - ios

I am creating an application in which we can able to draw using our finger in an imageView , the same time we can record the screen also.
I have done these features so far , but the problem is once the video recording is completed , if we play the recorded video the finger drawing is not smooth in video.
I am not using opengl , the drawing is on UIImageView and on every 0.01 sec we capture th image from UIImageView and append the pixel buffer to the AVAssetWriterInputPixelBufferAdaptor object .
Here is the code I used for converting the UIImage into buffer
- (CVPixelBufferRef) pixelBufferFromCGImage:(CGImageRef) image {
CGSize frameSize = CGSizeMake(976, 667);
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CVPixelBufferCreate(kCFAllocatorDefault, frameSize.width,
frameSize.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
&pxbuffer);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
CGColorSpaceRef rgbColorSpace = CGImageGetColorSpace(image);
CGContextRef context = CGBitmapContextCreate(pxdata, frameSize.width,
frameSize.height, 8, 4*frameSize.width, rgbColorSpace,
kCGImageAlphaPremultipliedFirst);
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image),
CGImageGetHeight(image)), image);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
the below method is calling on 0.01 sec timeinterval
CVPixelBufferRef pixelBufferX = (CVPixelBufferRef)[self pixelBufferFromCGImage:theIM];
bValue = [self.avAdaptor appendPixelBuffer:pixelBufferX withPresentationTime:presentTime];
Can any one guide for the improvement in video capture ?
Thanks in advance

You shouldn't display things by calling them every 0.01 seconds. If you want to stay in sync with video, see AVSynchronizedLayer, which is explicitly for this. Alternately, see CADisplayLink, which is for staying in sync with screen refreshes. 0.01 seconds doesn't line up with anything in particular, and you're probably getting beats where you're out of sync with the video and with the display. In any case, you should be doing your drawing in some callback from your player, not with a timer.
You are also leaking your pixel buffer in every loop. Since you called CVPixelBufferCreate(), you're responsible for eventually calling CFRelease() on the resulting pixel buffer. I would expect your program to eventually crash by running out of memory if this ran for awhile.
Make sure you've studied the AV Foundation Programming Guide so you know how all the pieces fit together in media playback.

Related

Get upright frames from AVPlayerItemVideoOutput when video has rotated exif orientation

I'm using an AVPlayerItemVideoOutput to get video frames from an AVPlayer and upload them to a GL texture for display. The issue is that AVPlayerItemVideoOutput seems to ignore the video's rotation exif data, so the CVPixelBufferRef it returns isn't upright.
Options:
I could edit my GL code to counter rotate the texture when displaying it, but i'd kind of prefer to get the frames upright in the first place so I don't have to transform the texture coordinates.
Some magic to get AVPlayerItemVideoOutput to give me the frames upright in the first place. Solution must be hardware accelerated.
code:
//
// Setup
//
AVPlayer *player = [AVPlayer playerWithURL:fileURL];
AVPlayerItemVideoOutput *output = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:#{
(id)kCVPixelBufferPixelFormatTypeKey: #(kCVPixelFormatType_32ARGB),
(id)kCVPixelBufferOpenGLCompatibilityKey: #YES,
}];
[player.currentItem addOutput:output];
//
// getting the frame data into a GL texture
//
CMTime currentTime = player.currentTime;
if ([output hasNewPixelBufferForItemTime:currentTime]){
CVPixelBufferRef frame = [output copyPixelBufferForItemTime:currentTime itemTimeForDisplay:NULL];
CVPixelBufferLockBaseAddress(frame, kCVPixelBufferLock_ReadOnly);
GLsizei height = (GLsizei)CVPixelBufferGetHeight(frame);
GLsizei bpr = (GLsizei)CVPixelBufferGetBytesPerRow(frame);
void *data = CVPixelBufferGetBaseAddress(frame);
glBindTexture(GL_TEXTURE_2D, gltexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bpr/4, height, 0, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8, data);
CVPixelBufferUnlockBaseAddress(frame, kCVPixelBufferLock_ReadOnly);
CVPixelBufferRelease(frame);
}

Fast way to create openGL texture from JPEG-2000?

I need to load large-ish (5 megapixel) jpeg images and create openGL texture from them. They are non-power-of-two, and cannot be pre-processed for this application. Loading is extremely slow, about one second per image on an iPad Air 2. I need to load a dozen or two such images and create a GL texture for each, as quickly as I can.
Profiling shows the bottleneck to be CGContextDrawImage. Previous answers suggest this is a common problem.
This previous answer seems most relevant and (unfortunately) does not leave me hopeful. I haven't tried lib-jpeg (suggested in another answer) yet - trying to keep third party code out for several reasons.
But - that answer was 2014 and things change. Does anybody know of a faster way to create textures from jpegs? Either by changing the arguments to CGContextDrawImage (as in this answer- I've tried the suggested changes with no noticeable speed change) or using a different approach entirely?
The current texture creation block (called asynchronously):
UIImage *image = [UIImage imageWithData:jpegImageData];
if (image) {
GLuint textureID;
glGenTextures(1, &textureID);
glBindTexture( GL_TEXTURE_2D, textureID);
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR);
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
GLsizei width = (GLsizei)CGImageGetWidth(image.CGImage);
GLsizei height = (GLsizei)CGImageGetHeight(image.CGImage);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
void *imageData = malloc( height * width * 4 );
CGContextRef imgcontext = CGBitmapContextCreate( imageData, width, height, 8, 4 * width, colorSpace, kCGImageAlphaNoneSkipLast | kCGBitmapByteOrder32Big );
CGColorSpaceRelease( colorSpace );
CGContextDrawImage( imgcontext, CGRectMake( 0, 0, width, height ), image.CGImage );
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, imageData);
CGContextRelease(imgcontext);
free(imageData);
// ... store the textureID for use by the caller
// ...
}
(edited to add)
I tried GLKTextureLoader. I kept getting a nil return value, with error theError NSError * domain: "GLKTextureLoaderErrorDomain" - code: 12.
I've realized that the JPEGs I need to load are JPEG 2000; and that may be the problem. I've played with the GLKTextureLoader approach; I can get it to work non-J2K jpegs, but not the J2K ones I need to load. (FWIW, the files I need to load are packed inside larger files, thus I extract a data subrange from within the file, as such:
NSData *jpegImageData = [data subdataWithRange:NSMakeRange(offset, dataLength)];
GLKTextureInfo *jpegTexture;
NSError *theError;
jpegTexture = [GLKTextureLoader textureWithContentsOfData:jpegImageData options:nil error:&theError];
but, as mentioned, jpegImageData comes back as nil with the aforementioned error. This works on small jpegs, even using the subdataWithRange approach.
Likewise,
UIImage *image = [UIImage imageWithData:jpegImageData];
jpegTexture = [GLKTextureLoader textureWithCGImage:image.CGImage options:nil error:&theError];
returns nil with the same "code 12" error.
This iOS Developer page (Table 1-1) suggests that JPEG-2000 is supported on OS X only, but when I try the
CFArrayRef mySourceTypes = CGImageSourceCopyTypeIdentifiers();
CFShow(mySourceTypes);
approach for showing supported formats, JPEG-2000 is among them (running on my iOS device):
33 : <CFString 0x19d721bf8 [0x1a1da0150]>{contents = "public.jpeg-
Any suggestions for using the faster GLKTextureLoader methods on JPEG-2000?
Did you try the GLKit Framework method?
GLKTexGtureInfo *spriteTexture;
NSError *theError;
NSString *filePath = [[NSBundle mainBundle] pathForResource:#"Sprite" ofType:#"jpg"]; // 1
spriteTexture = [GLKTextureLoader textureWithContentsOfFile:filePath options:nil error:&theError]; // 2
glBindTexture(spriteTexture.target, spriteTexture.name); // 3

Capture 120/240 fps using AVCaptureVideoDataOutput into frame buffer using low resolution

Currently, using the iPhone 5s/6 I am able to capture 120(iPhone 5s) or 240(iPhone 6) frames/second into a CMSampleBufferRef. However, the AVCaptureDeviceFormat that is returned to me only provides these high speed frame rates with a resolution of 1280x720.
I would like to capture this in lower resolution (640x480 or lower) since I will be putting this into a circular buffer for storage purpose. While I am able to reduce the resolution in the didOutputSampleBuffer delegate method, I would like to know if there is any way for the CMSampleBufferRef to provide me a lower resolution directly by configuring the device or setting, instead of taking the 720p image and lowering the resolution manually using CVPixelBuffer.
I need to store the images in a buffer for later processing and want to apply minimum processing necessary or else I will begin to drop frames. If I can avoid resizing and obtain a lower resolution CMSampleBuffer from the didOutputSampleBuffer delegate method directly, that would be ideal.
At 240fps, I would need to process each image within 5ms and the resizing routine cannot keep up with downscaling the image at this rate. However, I would like to store it into a circular buffer for later processing (e.g. writing out to a movie using AVAssetWriter) but require a lower resolution.
It seems that the only image size supported in high frame rate recording is 1280x720. Putting multiple images of this resolution into the frame buffer will generate memory pressure so I'm looking to capture a lower resolution image directly from didOutputSampleBuffer if it is at all possible to save on memory and to keep up with the frame rate.
Thank you for your assistance.
// core image use GPU to all image ops, crop / transform / ...
// --- create once ---
EAGLContext *glCtx = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
CIContext *ciContext = [CIContext contextWithEAGLContext:glCtx options:#{kCIContextWorkingColorSpace:[NSNull null]}];
// use rgb faster 3x
CGColorSpaceRef ciContextColorSpace = CGColorSpaceCreateDeviceRGB();
OSType cvPixelFormat = kCVPixelFormatType_32BGRA;
// create compression session
VTCompressionSessionRef compressionSession;
NSDictionary* pixelBufferOptions = #{(__bridge NSString*) kCVPixelBufferPixelFormatTypeKey:#(cvPixelFormat),
(__bridge NSString*) kCVPixelBufferWidthKey:#(outputResolution.width),
(__bridge NSString*) kCVPixelBufferHeightKey:#(outputResolution.height),
(__bridge NSString*) kCVPixelBufferOpenGLESCompatibilityKey : #YES,
(__bridge NSString*) kCVPixelBufferIOSurfacePropertiesKey : #{}};
OSStatus ret = VTCompressionSessionCreate(kCFAllocatorDefault,
outputResolution.width,
outputResolution.height,
kCMVideoCodecType_H264,
NULL,
(__bridge CFDictionaryRef)pixelBufferOptions,
NULL,
VTEncoderOutputCallback,
(__bridge void*)self,
&compressionSession);
CVPixelBufferRef finishPixelBuffer;
// I'm use VTCompressionSession pool, you can use AVAssetWriterInputPixelBufferAdaptor
CVReturn res = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, VTCompressionSessionGetPixelBufferPool(compressionSession), &finishPixelBuffer);
// -------------------
// ------ scale ------
// new buffer comming...
// - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
CIImage *baseImg = [CIImage imageWithCVPixelBuffer:pixelBuffer];
CGFloat outHeight = 240;
CGFloat scale = 1 / (CVPixelBufferGetHeight(pixelBuffer) / outHeight);
CGAffineTransform transform = CGAffineTransformMakeScale(scale, scale);
// result image not changed after
CIImage *resultImg = [baseImg imageByApplyingTransform:transform];
// resultImg = [resultImg imageByCroppingToRect:...];
// CIContext applies transform to CIImage and draws to finish buffer
[ciContext render:resultImg toCVPixelBuffer:finishPixelBuffer bounds:resultImg.extent colorSpace:ciContextColorSpace];
CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
// [videoInput appendSampleBuffer:CMSampleBufferCreateForImageBuffer(... finishPixelBuffer...)]
VTCompressionSessionEncodeFrame(compressionSession, finishPixelBuffer, CMSampleBufferGetPresentationTimeStamp(sampleBuffer), CMSampleBufferGetDuration(sampleBuffer), NULL, sampleBuffer, NULL);
// -------------------

OpenGL format texture

I am currently trying to display a video on screen using OpenGL ES 2 on iOS.
I will sum up a bit what I am doing to playback and display the video on screen :
First I have a .mov file recorded using a GPUImageMovieWriter object. When the recording is completed I am going to playback the video using AVPlayer. Therefore I set a AVPlayerItemVideoOutput to be able to retrieve frame from the video :
NSDictionary *test = [NSDictionary dictionaryWithObject: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey: (id)kCVPixelBufferPixelFormatTypeKey];
self.videoOutput = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:test];
I then use the copyPixelBufferForItemTime function from the AVPlayerItemVideoOutput and receive the CVImageBufferRef corresponding to the frame of the initial video at a specific time.
Finally, here is the function I created to create an OpenGL texture from the buffer :
- (void)setupTextureFromBuffer:(CVImageBufferRef)imageBuffer {
CVPixelBufferLockBaseAddress(imageBuffer, 0);
int bufferHeight = CVPixelBufferGetHeight(imageBuffer);
int bufferWidth = CVPixelBufferGetWidth(imageBuffer);
CVPixelBufferGetPixelFormatType(imageBuffer);
glBindTexture(GL_TEXTURE_2D, m_videoTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(imageBuffer));
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
By doing this (and also using some non related algorithms to do some augmented reality things) I got a very strange result as if the video has been put in slices(I can't show you because I don't have enough reputation to do so).
It looks like the data are not well interpreted by OpenGL (wrong format ? type ?)
I checked whether it could be a corrupted buffer error by using this function :
- (void)saveImage:(CVPixelBufferRef)pixBuffer
{
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixBuffer];
CIContext *temporaryContext = [CIContext contextWithOptions:nil];
CGImageRef videoImage = [temporaryContext
createCGImage:ciImage
fromRect:CGRectMake(0, 0,
CVPixelBufferGetWidth(pixBuffer),
CVPixelBufferGetHeight(pixBuffer))];
UIImage *uiImage = [UIImage imageWithCGImage:videoImage];
UIImageWriteToSavedPhotosAlbum(uiImage, self, #selector(image:didFinishSavingWithError:contextInfo:), nil);
}
-> The saved image appeared properly in the photo album.
It may come from the .mov file but what can I do to check if there's something wrong with this file ?
Thanks a lot for your help, I'm really stuck on this problem for hours/days !
You need to use kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange.
Then transfer them to separate chroma and luma OpenGLES textures. Example at https://developer.apple.com/library/ios/samplecode/AVBasicVideoOutput/Listings/AVBasicVideoOutput_APLEAGLView_m.html
I tried using several RGB based options but could not make it work.

Zxing zoom functionality in ios AVFoundation

I have been scouring the internet and have been looking high and low for any type of code to help me zoom in on barcodes using ZXing.
I started with the code from their git site here
https://github.com/zxing/zxing
Since then I have been able to increase the default resolution to 1920x1080.
self.captureSession.sessionPreset = AVCaptureSessionPreset1920x1080;
This would be fine but the issue is that I am scanning very small barcodes and even though 1920x1080 would work it doesnt give me any kind of zoom to capture closer to a smaller barcode without losing focus. Now the resolution did help me quite a bit but its simply not close enough.
Im thinking what I need to do is to set the capture session to a scroll view that is 1920x1080 and then set the actual image capture to take from the bounds of my screen so i can zoom in and out of the scroll view itself to achieve a "zoom" kind of affect.
The problem with that is im really not sure where to start...any ideas?
Ok since I have seen this multiple time on here and no one seems to have an answer. I thought I would share my own answer.
There are 2 properties NO ONE seems to know about. Ill cover both.
Now the first one is good for iOS 6+. Apple added a property called setVideoScaleAndCropfactor.
This setting this returns a this is a CGFloat type. The only downfall in this is that you have to set the value to your AVConnection and then set the connection to a stillImageCapture. It will not work with anything else in iOS 6. Now in order to do this you have to set it up to work Asynchronously and you have to loop your code for the decoder to work and take pictures at that scale.
Last thing is that you have to scale your preview layer yourself.
This all sounds like a lot of work. And it really really is. However, This sets your original scan picture to be taken at 1920x1080 or whatever you have it set as. Rather than scaling a current image which will stretch pixels causing the decoder to miss the barcode.
Sp this will look something like this
stillImageConnection = [stillImageOutput connectionWithMediaType:AVMediaTypeVideo];
[stillImageConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
[stillImageConnection setVideoScaleAndCropFactor:effectiveScale];
[stillImageOutput setOutputSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCMPixelFormat_32BGRA]
forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[stillImageOutput captureStillImageAsynchronouslyFromConnection:stillImageConnection
completionHandler:^(CMSampleBufferRef imageDataSampleBuffer, NSError *error)
{
if(error)
return;
NSString *path = [NSString stringWithFormat:#"%#%#",
[[NSBundle mainBundle] resourcePath],
#"/blank.wav"];
SystemSoundID soundID;
NSURL *filePath = [NSURL fileURLWithPath:path isDirectory:NO];
AudioServicesCreateSystemSoundID(( CFURLRef)filePath, &soundID);
AudioServicesPlaySystemSound(soundID);
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(imageDataSampleBuffer);
/*Lock the image buffer*/
CVPixelBufferLockBaseAddress(imageBuffer,0);
/*Get information about the image*/
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
uint8_t* baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
void* free_me = 0;
if (true) { // iOS bug?
uint8_t* tmp = baseAddress;
int bytes = bytesPerRow*height;
free_me = baseAddress = (uint8_t*)malloc(bytes);
baseAddress[0] = 0xdb;
memcpy(baseAddress,tmp,bytes);
}
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef newContext =
CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace,
kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst);
CGImageRef capture = CGBitmapContextCreateImage(newContext);
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
free(free_me);
CGContextRelease(newContext);
CGColorSpaceRelease(colorSpace);
Decoder* d = [[Decoder alloc] init];
[self decoding:d withimage:&capture];
}];
}
Now the second one that is coming in iOS 7 that will change EVERYTHING I just said. You have a new property called videoZoomFactor. this is a CGFloat. However it changes everything at the TOP of the stack rather than just affecting like the stillimagecapture.
In Otherwords you wont have to manually zoom your preview layer. You wont have to go through some stillimagecaptureloop and you wont have to set it to an AVConnection. You simply set the CGFloat and it scales everything for you.
Now I know its going to be a while before you can publish iOS 7 applications. So I would seriously consider figuring out how to do this the hard way. Quick tips. I would use a pinch and zoom gesture to set your CGFloat for setvideoscaleandcropfactor. Dont forget to set the value to 1 in your didload and you can scale from there. At the same time in your gesture you can use it to do your CATransaction to scale the preview layer.
Heres a sample of how to do the gesture capture and preview layer
- (IBAction)handlePinchGesture:(UIPinchGestureRecognizer *)recognizer
{
effectiveScale = recognizer.scale;
if (effectiveScale < 1.0)
effectiveScale = 1.0;
if (effectiveScale > 25)
effectiveScale = 25;
stillImageConnection = [stillImageOutput connectionWithMediaType:AVMediaTypeVideo];
[stillImageConnection setVideoScaleAndCropFactor:effectiveScale];
[CATransaction begin];
[CATransaction setAnimationDuration:0];
[prevLayer setAffineTransform:CGAffineTransformMakeScale(effectiveScale, effectiveScale)];
[CATransaction commit];
}
Hope this helps someone out! I may go ahead and just to a video tutorial on this. Depends on what kind of demand there is for it I guess.

Resources