My camera app captures a photo, enhances it in a certain way, and saves it.
To do so, I get the input image from the camera in the form of a CVPixelBuffer (wrapped in a CMSampleBuffer). I perform some modifications on the pixel buffer, and I then want to convert it to a Data object. How do I do this?
Note that I don't want to convert the pixel buffer / image buffer to a UIImage or CGImage since those don't have metadata (like EXIF). I need a Data object. How do I get one from a CVPixelBuffer / CVImageBuffer?
PS: I tried calling AVCapturePhotoOutput.jpegPhotoDataRepresentation() but that fails saying "Not a JPEG sample buffer". Which makes sense since the CMSampleBuffer contains a pixel buffer (a bitmap), not a JPEG.
As you said that you are able to get the CMSampleBuffer, then you can get it using
NSData *myata = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:<your_cmsample_buffer_obj>];
Related
Hi I'm working on getting a raw image data from AVCaptureVideoDataoutput.
I have lots of experience in using avfoundation and worked lots of projects but this time I'm working on image processing project which I don't have any experience.
public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
}
I know I'm getting CMSampleBuffer right here in delegate callback function.
MY Questions.
is CMSampleBuffer is 'RAW' data which is from image sensor?
I want to know whole flow of making CMSampleBuffer to here. But I could not find any detail flow.
I found some possible solutions searching on stackoverflow
(Raw image data from camera like "645 PRO")
in this question, the author tried using opencv and got what he wanted.
but.. what is exactly different the result by opencv from CMSamplebuffer?(why the opencv result is real raw data the author said)
And also (How to get the Y component from CMSampleBuffer resulted from the AVCaptureSession?)
if i set as below,
if self.session.canAddOutput(self.captureOutput) {
self.session.addOutput(self.captureOutput)
captureOutput.videoSettings = [
kCVPixelBufferPixelFormatTypeKey : kCVPixelFormatType_32BGRA
] as [String : Any]
captureOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "capture"))
captureOutput.alwaysDiscardsLateVideoFrames = true
}
by setting format key _32BGRA,, AM i now getting raw data from samplebuffer?
It's not RAW in your case. All modern sensors build on Bayer filter, so you get an image converted from the Bayer format. You can't get raw image with this api. There is a format called kCVPixelFormatType_14Bayer_BGGR, but the camera probably won't support it.
Maybe on WWDC 419 session you will find answer. I don't know
It's the same; cv::Mat is just a wrapper around the image data from CMSampleBuffer. If you save your data as PNG, you will not lose any quality. The TIFF format saves without any compression, but you can also use PNG without compression.
If you use RGBA format, behind the scenes it is converted from Bayer to RGBA. To get the Y channel, you need to additionally apply an RGBA to YUV conversion and take the Y channel. Or you can use the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format and get the Y channel from the first plane. Also note that VideoRange has different chroma output range.
// ObjC code
int width = CVPixelBufferGetWidth(imageBuffer);
int height = CVPixelBufferGetHeight(imageBuffer);
uint8_t *yBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
size_t yPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0);
cv::Mat yChannel(height, width, CV_8UC1, yBuffer, yPitch);
I have an AR app where the view is constantly showing what the back camera is seeing and sending each frame for analysis to VisionRequest.
When the object was identified, I would like to capture that particular last frame and save it as a regular UIImage and send it down the segue chain to the final view controller where I display that last frame. I have issues capturing that last frame and showing it.
Here is what I tried so far:
When the image is recognized with a high-enough confidence, I attempt to retrieve the current last frame from the CVPixelBuffer and save it in a local variable that is later passed in a segue to subsequent view controllers.
Is this the correct way of doing it? or do I have to add a second output to the session (a photo output in addition to a video data output) ?
//attempting to get the current last frame of captured video
let attachments = CMCopyDictionaryOfAttachments(allocator: kCFAllocatorDefault, target: self.currentlyAnalyzedPixelBuffer!, attachmentMode: kCMAttachmentMode_ShouldPropagate)
let ciImage = CIImage(cvImageBuffer: self.currentlyAnalyzedPixelBuffer!, options: attachments as? [CIImageOption : Any])
self.image = UIImage(ciImage: ciImage)
Actually, there are more chances that you get not exact output you needed. Because You never know that last frame captured has exact same you wanted. There might be possibilities where you can have false results like the camera is in motion and frame you got is blurred or not properly as per your need.
May be I am wrong with it. But my suggestion or solution would keep array of 10 images or pixel buffers and store last 10 Frames or pixel buffers. When you get your object identified from vision check that array again and get the highest quality (confidence) frame or you may show the user a collection view as an option to choose the correct image.
Hope it may helpful
The current last frame may not be the one that triggered the successful image recognition, so you may want to hold to the pixelBuffer that triggered it.
Then you can get the UIImage from the pixelBuffer like so:
import VideoToolbox
var cgImage: CGImage?
VTCreateCGImageFromCVPixelBuffer(matchingPixelBuffer, options: nil, imageOut: &cgImage)
let uiImage = UIImage(cgImage: cgImage)
I have access to the CVPixelBufferRef for each frame and want to apply the ChromaKey filter to it before rendering it.
So far, the only solution I can think of is to first convert the pixel buffer to an image. Here is my barebones solution just for PoC.
var cgImage: CGImage?
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, nil, &cgImage)
let image = UIImage.init(cgImage: cgImage!).filterWithOperation(filter!)
Once I get the filtered image, I pass it to an MTKView to draw.
So my specific question is, can I avoid converting the pixel buffer to an image and still use GPUImage2 for the filter?
I am cropping an opencv Mat:
cv::Size size = img.size();
cv::Rect roi(size.width*/4., size.height/4.,size.width/2., size.height/.2);
img= img(roi);
I then use img.data pointer to create a vtkImageData (via vtkImageImport):
vtkSmartPointer<vtkImageImport> importer = vtkSmartPointer<vtkImageImport>::New();
importer->SetImportVoidPointer(img.data);
...
importer->Update();
vtkImageData* vtkImg = importer->GetOutput();
I don't get the expected result when I display the vtkImg. I've digged into opencv's code and the problem is that when creating the cropped data, opencv does not allocate a new pointer that is 4 times smaller but instead keeps the same already allocated block, advances the pointer upstream and flags the new img variable as not continuous. Therefore my vtk image still imports data from the original uncropped Mat. I know I could import the full image to vtkImageData and then do the cropping with a vtk filter but I would prefer not to.
Is there a way with opencv to obtain a cropped image that is "physically" cropped (with a newly allocated data pointer)?
Thank you
I believe you are looking for cv::Mat::clone(). It makes a deep copy of the underlying image data and returns a cv::Mat object which contains said data.
You would then change the line
img= img(roi);
to
img = img(roi).clone();
After which img contains only the cropped data.
I'm confused as to why so much converting between image formats is needed in iOS. For example, if I load a jpg into a UIImage and then want to do face detection on it, I need to create a CIImage to pass to the CIDetector. Doesn't this represent a hit in both memory and performance?
Is this some legacy thing between Core Graphics, Core Image and UIKit (and probably openGL ES but I don't work with that)? Is the hit trivial overall?
I'll do what I need to do but I'd like to understand more about this is needed. Also, I've run into issues sometimes doing conversions and get tangled in the differences between the formats.
Update
Ok - so I just got dinged again by my confusion over these formats (or the confusion OF these formats...). Wasted a half hour. Here is what I was doing:
Testing for faces in a local image, I created the needed CIImage with:
CIImage *ciImage = [image CIImage];
and was not getting any features back no matter what orientation I passed in. I know this particular image has worked with the CIDetectorTypeFace before and that I have run into trouble with the CIImage format. The tried creating the CCImage like this:
CIImage *ciImage = [CIImage imageWithCGImage:image.CGImage];
and Face Detection works fine. Arrgh! I made sure with [image CIImage] that the resulting CIImage was not nil. So I'm confused. The first approach just gets a pointer while the second creates a new CIImage. Does that make the difference?
Digging into the UIImage.h file I see the following:
// returns underlying CGImageRef or nil if CIImage based
#property(nonatomic,readonly) CGImageRef CGImage;
// returns underlying CIImage or nil if CGImageRef based
#property(nonatomic,readonly) CIImage *CIImage;
So I guess that is the key - Developer Beware: test for nil...
The reason is in the conception. All of UIKit, CoreGraphics and CoreImage does three such fundamentally different things that there can't be a 'grand central unified image format'. Also, these frameworks cooperate well; that said, the conversion should be as optimized and as fast as possible, but of course image processing is always a relatively computationally expensive operation.