Recording and merging video for screen capture - ios

Question

I am currently grabbing the frames of a video using AVPlayerItemVideoOutput. I use CADisplayLink to grab frames from the output. Then i pass the pixel buffer off to the assetwriter. I do it like this:
- (void)displayLinkCallback:(CADisplayLink *)sender
{
CMTime outputItemTime = kCMTimeInvalid;
// Calculate the nextVsync time which is when the screen will be refreshed next.
CFTimeInterval nextVSync = (sender.timestamp + sender.duration);
outputItemTime = [self.videoOutput itemTimeForHostTime:nextVSync];
if (self.playerOne.playerAsset.playable) {
if ([[self videoOutput] hasNewPixelBufferForItemTime:outputItemTime] && self.newSampleReady) {
dispatch_async(self.captureSessionQueue, ^{
CVPixelBufferRelease(self.lastPixelBuffer);
self.lastPixelBuffer = [self.videoOutput copyPixelBufferForItemTime:outputItemTime itemTimeForDisplay:NULL];
CMTime fpsTime = CMTimeMake(1, 24);
self.currentVideoTime = CMTimeAdd(self.currentVideoTime, fpsTime);
[_assetWriterInputPixelBufferAdaptor appendPixelBuffer:self.lastPixelBuffer withPresentationTime:self.currentVideoTime];
self.newSampleReady = NO;
});
}
}
}
This allows me to switch videos real time and keep making a screen recording. But I also want to switch to a split view with two players, grab each of the frames from the players and merge them into a single video. AVComposition would work except that you have to know in advance what tracks and times you want to merge. My screen capture program lets the user switch freely between single and split view and back. Is there a way to get the pixel buffer's and use those to merge the recordings into a single video?
I have tried doing the following by just taking the first pixel buffer and creating two images, combining them and then creating a new pixel buffer that I pass back to the assetwriter but I just get a black screen video. Here's my code for this:
-(CVPixelBufferRef)pixelBufferToCGImageRef:(CVPixelBufferRef)pixelBuffer withSecond:(CVPixelBufferRef)pixelBuffer2
{
CIContext *temporaryContext = [CIContext contextWithOptions:nil];
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
UIImage *im1 = [UIImage imageWithCIImage:ciImage];
UIImage *im2 = [UIImage imageWithCIImage:ciImage];
CGSize newSize = CGSizeMake(640, 480);
UIGraphicsBeginImageContext( newSize );
[im1 drawInRect:CGRectMake(0,0,newSize.width/2,newSize.height)];
[im2 drawInRect:CGRectMake(newSize.width/2,0,newSize.width/2,newSize.height) blendMode:kCGBlendModeNormal alpha:1.0];
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
CIImage *newCIImage = [newImage CIImage];
UIGraphicsEndImageContext();
CVPixelBufferRef pbuff = NULL;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
640,
480,
kCVPixelFormatType_32ARGB,
(__bridge CFDictionaryRef)(options),
&pbuff);
if (status == kCVReturnSuccess) {
[temporaryContext render:newCIImage
toCVPixelBuffer:pbuff
bounds:CGRectMake(0, 0, 640, 480)
colorSpace:nil];
} else {
NSLog(#"Failed create pbuff");
}
return pbuff;
}
Any suggestions?

Answer 1

The black screen I was getting was because ciImage becomes nil right after I get it on the simulator. If I run the code on a device then it works.