I have an AR app where the view is constantly showing what the back camera is seeing and sending each frame for analysis to VisionRequest.
When the object was identified, I would like to capture that particular last frame and save it as a regular UIImage and send it down the segue chain to the final view controller where I display that last frame. I have issues capturing that last frame and showing it.
Here is what I tried so far:
When the image is recognized with a high-enough confidence, I attempt to retrieve the current last frame from the CVPixelBuffer and save it in a local variable that is later passed in a segue to subsequent view controllers.
Is this the correct way of doing it? or do I have to add a second output to the session (a photo output in addition to a video data output) ?
//attempting to get the current last frame of captured video
let attachments = CMCopyDictionaryOfAttachments(allocator: kCFAllocatorDefault, target: self.currentlyAnalyzedPixelBuffer!, attachmentMode: kCMAttachmentMode_ShouldPropagate)
let ciImage = CIImage(cvImageBuffer: self.currentlyAnalyzedPixelBuffer!, options: attachments as? [CIImageOption : Any])
self.image = UIImage(ciImage: ciImage)
Actually, there are more chances that you get not exact output you needed. Because You never know that last frame captured has exact same you wanted. There might be possibilities where you can have false results like the camera is in motion and frame you got is blurred or not properly as per your need.
May be I am wrong with it. But my suggestion or solution would keep array of 10 images or pixel buffers and store last 10 Frames or pixel buffers. When you get your object identified from vision check that array again and get the highest quality (confidence) frame or you may show the user a collection view as an option to choose the correct image.
Hope it may helpful
The current last frame may not be the one that triggered the successful image recognition, so you may want to hold to the pixelBuffer that triggered it.
Then you can get the UIImage from the pixelBuffer like so:
import VideoToolbox
var cgImage: CGImage?
VTCreateCGImageFromCVPixelBuffer(matchingPixelBuffer, options: nil, imageOut: &cgImage)
let uiImage = UIImage(cgImage: cgImage)
Related
I get from AR session, the current frame with: self.sceneView.session.currentFrame?.capturedImage so I get a CVPixelBuffer with my image information.
I followed this link to convert my CVPixelBuffer to CGImage. (I use previously createCGImage method but this method create a memory leak in my app.)
But I have an issue now, the CGImage received at the end is on the landscape mode and not portrait (my app allows only portrait mode) like you can see on the image below.
So I'm trying to fix this rotation.
I saw vImageRotate90_ARGB8888(_:_:_:_:_:) exist but I have trouble to use it because my pointer for the backColor is always nil.
Do you have any method to rotate it without a memory leak because I have to rotate it couple times per seconde?
Thanks
I have access to the CVPixelBufferRef for each frame and want to apply the ChromaKey filter to it before rendering it.
So far, the only solution I can think of is to first convert the pixel buffer to an image. Here is my barebones solution just for PoC.
var cgImage: CGImage?
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, nil, &cgImage)
let image = UIImage.init(cgImage: cgImage!).filterWithOperation(filter!)
Once I get the filtered image, I pass it to an MTKView to draw.
So my specific question is, can I avoid converting the pixel buffer to an image and still use GPUImage2 for the filter?
In my project i need to draw 2D image in realtime corresponding with UIGestureRecognizer updates .
The image would be the same UIImage , drawing on various positions.
let arrayOfPositions = [pos1,pos2,pos3]
And i need to transfer the result image on MetalLayer into Single UIImage , the result image will have the same size as device's screen.
something similar to
let resultImage = UIGraphicsGetImageFromCurrentImageContext()
I'm new to Metal and after watching realm's video and apple documentation my sanity went to chaos . Most tutorials focus on 3D rendering which is beyond my need (and my knowledge)
If anyone would help me a simple code how to draw UIImage in to MetalLayer , then convert the whole into single UIImage as a result ? thanks
I have managed to setup a basic AVCaptureSession which records a video and saves it on device by using AVCaptureFileOutputRecordingDelegate. I have been searching through docs to understand how we can add statistics overlays on top of the video which is being recorded.
i.e.
As you can see in the above image. I have multiple overlays on top of video preview layer. Now, when I save my video output I would like to compose those views onto the video as well.
What have I tried so far?
Honestly, I am just jumping around on internet to find a reputable blog explaining how one would do this. But failed to find one.
I have read few places that one could render text layer overlays as described in following post by creating CALayer and adding it as a sublayer.
But, what about if I want to render MapView on top of the video being recorded. Also, I am not looking for screen capture. Some of the content on the screen will not be part of the final recording so I want to be able to cherry pick view that will be composed.
What am I looking for?
Direction.
No straight up solution
Documentation link and class names I should be reading more about to create this.
Progress So Far:
I have managed to understand that I need to get hold of CVImageBuffer from CMSampleBuffer and draw text over it. There are things still unclear to me whether it is possible to somehow overlay MapView over the video that is being recorded.
The best way that helps you to achieve your goal is to use a Metal framework. Using a Metal camera is good for minimising the impact on device’s limited computational resources. If you are trying to achieve the lowest-overhead access to camera sensor, using a AVCaptureSession would be a really good start.
You need to grab each frame data from CMSampleBuffer (you're right) and then to convert a frame to a MTLTexture. AVCaptureSession will continuously send us frames from device’s camera via a delegate callback.
All available overlays must be converted to MTLTextures too. Then you can composite all MTLTextures layers with over operation.
So, here you'll find all necessary info in four-part Metal Camera series.
And here's a link to a blog: About Compositing in Metal.
Also, I'd like to publish code's excerpt (working with AVCaptureSession in Metal):
import Metal
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
// Handle an error here.
}
// Texture cache for converting frame images to textures
var textureCache: CVMetalTextureCache?
// `MTLDevice` for initializing texture cache
var metalDevice = MTLCreateSystemDefaultDevice()
guard
let metalDevice = metalDevice
where CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textureCache) == kCVReturnSuccess
else {
// Handle an error (failed to create texture cache)
}
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
var imageTexture: CVMetalTexture?
let result = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache.takeUnretainedValue(), imageBuffer, nil, pixelFormat, width, height, planeIndex, &imageTexture)
// `MTLTexture` is in the `texture` variable now.
guard
let unwrappedImageTexture = imageTexture,
let texture = CVMetalTextureGetTexture(unwrappedImageTexture),
result == kCVReturnSuccess
else {
throw MetalCameraSessionError.failedToCreateTextureFromImage
}
And here you can find a final project on a GitHub: MetalRenderCamera
Before stackoverflow members answer with "You shouldn't. It's a privacy violation" let me counter with why there is a legitimate need for this.
I have a scenario where a user can change the camera device by swiping left and right. In order to make this animation not look like absolute crap, I need to grab a freeze frame before making this animation.
The only sane answer I have seen is capturing the buffer of AVCaptureVideoDataOutput, which is fine, but now I can't let the user take the video/photo with kCVPixelFormatType_420YpCbCr8BiPlanarFullRange which is a nightmare trying to get a CGImage from with CGBitmapContextCreate See How to convert a kCVPixelFormatType_420YpCbCr8BiPlanarFullRange buffer to UIImage in iOS
When capturing a still photo are there any serious quality considerations when using AVCaptureVideoDataOutput instead of AVCaptureStillImageOutput? Since the user will be taking both video and still photos (not just freeze-frame preview stills) Also, can some one "Explain it to me like I'm five" about the differences between kCVPixelFormatType_420YpCbCr8BiPlanarFullRange/kCVPixelFormatType_32BGRA besides one doesn't work on old hardware?
I don't think there is a way to directly capture a preview image using AVFoundation. You could however take a capture the preview layer by doing the following:
UIGraphicsBeginImageContext(previewView.frame.size);
[previewLayer renderInContext:UIGraphicsGetCurrentContext()];
UIImage *image = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
Where previewView.layer is the
previewLayer is the AVCaptureVideoPreviewLayer added to the previewView. "image" is rendered from this layer and can be used for your animation.