So I'm trying to get a jpeg/png representation of the grayscale depth maps that are typically used in iOS image depth examples. The depth data is stored in each jpeg as aux data. I've followed some tutorials and I have no problem rendering this gray scale data to the screen, but I can find no way to actually save it as a jpeg/png representation. I'm pretty much using this code: https://www.raywenderlich.com/168312/image-depth-maps-tutorial-ios-getting-started
The depth data is put into a cvpixelbuffer and manipulated accordingly. I believe it's in the format kCVPixelFormatType_DisparityFloat32.
While I'm able to see this data represented accordingly on the screen, I'm unable to use UIImagePNGRepresentation or UIImageJPGRepresentation. Sure, I could manually capture a screenshot, but that's not really ideal.
I have a suspicion that the cvpixelbuffer data format is not compatible with these UIImage functions and that's why I can't get them to spit out an image.
Does anyone have any suggestions?
// CVPixelBuffer to UIImage
let ciImageDepth = CIImage(cvPixelBuffer: cvPixelBufferDepth)
let contextDepth:CIContext = CIContext.init(options: nil)
let cgImageDepth:CGImage = contextDepth.createCGImage(ciImageDepth, from: ciImageDepth.extent)!
let uiImageDepth:UIImage = UIImage(cgImage: cgImageDepth, scale: 1, orientation: UIImage.Orientation.up)
// Save UIImage to Photos Album
UIImageWriteToSavedPhotosAlbum(uiImageDepth, nil, nil, nil)
I figured it out. I had to convert to a CGImage first. Ended up doing cvPixelBuffer to CIImage to CGImage to UIImage.
Posting a swift code sample in case anyone wants to use it.
let ciimage = CIImage(cvPixelBuffer: depthBuffer) // depth cvPixelBuffer
let depthUIImage = UIImage(ciImage: ciimage)
Related
I am using Swift's Vision Framework for Deep Learning and want to upload the input image to backend using REST API - for which I am converting my UIImage to MultipartFormData using jpegData() and pngData() function that swift natively offers.
I use session.sessionPreset = .vga640x480 to specify the image size in my app for processing.
I was seeing different size of image in backend - which I was able to confirm in the app because UIImage(imageData) converted from the image is of different size.
This is how I convert image to multipartData -
let multipartData = MultipartFormData()
if let imageData = self.image?.jpegData(compressionQuality: 1.0) {
multipartData.append(imageData, withName: "image", fileName: "image.jpeg", mimeType: "image/jpeg")
}
This is what I see in Xcode debugger -
The following looks intuitive, but manifests the behavior you describe, whereby one ends up with a Data representation of the image with an incorrect scale and pixel size:
let ciImage = CIImage(cvImageBuffer: pixelBuffer) // 640×480
let image = UIImage(ciImage: ciImage) // says it is 640×480 with scale of 1
guard let data = image.pngData() else { ... } // but if you extract `Data` and then recreate image from that, the size will be off by a multiple of your device’s scale
However, if you create it via a CGImage, you will get the right result:
let ciImage = CIImage(cvImageBuffer: pixelBuffer)
let ciContext = CIContext()
guard let cgImage = ciContext.createCGImage(ciImage, from: ciImage.extent) else { return }
let image = UIImage(cgImage: cgImage)
You asked:
If my image is 640×480 points with scale 2, does my deep learning model would still take the same to process as for a 1280×960 points with scale 1?
There is no difference, as far as the model goes, between 640×480pt # 2× versus 1280×960pt # 1×.
The question is whether 640×480pt # 2× is better than 640×480pt # 1×: In this case, the model will undoubtedly generate better results, though possibly slower, with higher resolution images (though at 2×, the asset is roughly four times larger/slower to upload; on 3× device, it will be roughly nine times larger).
But if you look at the larger asset generated by the direct CIImage » UIImage process, you can see that it did not really capture a 1280×960 snapshot, but rather captured 640×480 and upscaled (with some smoothing), so you really do not have a more detailed asset to deal with and is unlikely to generate better results. So, you will pay the penalty of the larger asset, but likely without any benefits.
If you need better results with larger images, I would change the preset to a higher resolution but still avoid the scale based adjustment by using the CIContext/CGImage-based snippet shared above.
While trying to apply a simple vignette filter to the raw camera feed of an iPhone6, with the help of Metal and Core Image, I see a lot of lag between the frames being processed and rendered in an MTKView
The approach which I have followed is (MetalViewController.swift):
Get raw camera output using AVCaptureVideoDataOutputSampleBufferDelegate
Convert CMSampleBuffer > CVPixelBuffer > CGImage
Create an MTLTexture with this CGImage.
Point no. 2 and 3 are inside the method named: fillMTLTextureToStoreTheImageData
Apply a CIFilter to the CIImage fetched from the MTLTexture in the MTKViewDelegate
func draw(in view: MTKView) {
if let currentDrawable = view.currentDrawable {
let commandBuffer = self.commandQueue.makeCommandBuffer()
if let myTexture = self.sourceTexture{
let inputImage = CIImage(mtlTexture: myTexture, options: nil)
self.vignetteEffect.setValue(inputImage, forKey: kCIInputImageKey)
self.coreImageContext.render(self.vignetteEffect.outputImage!, to: currentDrawable.texture, commandBuffer: commandBuffer, bounds: inputImage!.extent, colorSpace: self.colorSpace)
commandBuffer?.present(currentDrawable)
commandBuffer?.commit()
}
}
}
The performance is not at all what Apple mentioned in this doc: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_tasks/ci_tasks.html#//apple_ref/doc/uid/TP30001185-CH3-TPXREF101
Am I missing something?
Your step 2 is way too slow to support real-time rendering... and it looks like you're missing a couple of steps. For your purpose, you would typically:
Setup:
create a pool of CVPixelBuffer - using CVPixelBufferPoolCreate
create a pool of metal textures using CVMetalTextureCacheCreate
For each frame:
convert CMSampleBuffer > CVPixelBuffer > CIImage
Pass that CIImage through your filter pipeline
render the output image into a CVPixelBuffer from the pool created in step 1
use CVMetalTextureCacheCreateTextureFromImage to create a metal texture with your filtered CVPixelBuffer
If setup correctly, all these steps will make sure your image data stays on the GPU, as opposed to travelling from GPU to CPU and back to GPU for display.
The good news is all this is demoed in the AVCamPhotoFilter sample code from Apple https://developer.apple.com/library/archive/samplecode/AVCamPhotoFilter/Introduction/Intro.html#//apple_ref/doc/uid/TP40017556. In particular see the RosyCIRenderer class and its superclass FilterRenderer.
This question already has answers here:
UIImageWriteToSavedPhotosAlbum() doesn't save cropped image
(2 answers)
Closed 4 years ago.
I'm desperately trying to create a qr-code in Swift and convert the image to a (NS)Data-String. It's supposed to act as an image in a HTML file later.
Although the qr-code is created perfectly, the conversion to a data string nevertheless produces nil. Does anyone have an idea what's wrong with my code?
let dataString = "some text or code or whatever"
let data = dataString.data(using: .utf8)
if let filter = CIFilter(name: "CIQRCodeGenerator") {
filter.setValue(data, forKey: "inputMessage")
filter.setValue("Q", forKey: "inputCorrectionLevel")
let transform = CGAffineTransform(scaleX: 3, y: 3)
if let output = filter.outputImage?.transformed(by: transform) {
let bild = UIImage(ciImage: output) // <-- works quite well, image is shown in ImageView
let bildData = UIImageJPEGRepresentation(bild, 1.0) // <-- produces NIL
let bildString = bildData?.base64EncodedString(options: .lineLength64Characters)
//also tried: let bildString: String = String(data: bildData, encoding: .utf8)!
print("QRCODE-String: \(bildString)") // NIL
}
}
I also tried UIImagePNGRepresentation() with the same result.
CIImage is filter instructions. CGImage is bitmap data.
UIImage is a wrapper. A UIImage wrapping a CIImage merely contains filter instructions. A UIImage wrapping a CGImage contains bitmap data.
So the problem you are having has nothing to do with NSData. It has to do with UIImage. You are saying:
let bild = UIImage(ciImage: output)
let bildData = UIImageJPEGRepresentation(bild, 1.0)
bild is not a "real" image; it is merely a kind of wrapper for a CIImage. There is no data in the image — all there is is the instructions for the CIImage filter. You can't see anything until you render the image into a bitmap. UIImageView might be able to do that for you, but UIImageJPEGRepresentation cannot.
If you want to save the image as data, first draw the image into an image graphics context, to get the bitmap. Extract the resulting image, and now you have a real UIImage backed by CGImage. You can now save its data, because now it has data.
After applying a CIFilter to a photo captured with the camera the image taken shrinks and repositions itself.
I was thinking that if I was able to get the original images size and orientation that it would scale accordingly and pin the imageview to the corners of the screen. However nothing is changed with this approach and not aware of a way I can properly get the image to scale to the full size of the screen.
func applyBloom() -> UIImage {
let ciImage = CIImage(image: image) // image is from UIImageView
let filteredImage = ciImage?.applyingFilter("CIBloom",
withInputParameters: [ kCIInputRadiusKey: 8,
kCIInputIntensityKey: 1.00 ])
let originalScale = image.scale
let originalOrientation = image.imageOrientation
if let image = filteredImage {
let image = UIImage(ciImage: image, scale: originalScale, orientation: originalOrientation)
return image
}
return self.image
}
Picture Description:
Photo Captured and screenshot of the image with empty spacing being a result of an image shrink.
Try something like this. Replace:
func applyBloom() -> UIImage {
let ciInputImage = CIImage(image: image) // image is from UIImageView
let ciOutputImage = ciInputImage?.applyingFilter("CIBloom",
withInputParameters: [kCIInputRadiusKey: 8, kCIInputIntensityKey: 1.00 ])
let context = CIContext()
let cgOutputImage = context.createCGImage(ciOutputImage, from: ciInputImage.extent)
return UIImage(cgImage: cgOutputImage!)
}
I remained various variables to help explain what's happening.
Obviously, depending on your code, some tweaking to optionals and unwrapping may be needed.
What's happening is this - take the filtered/output CIImage, and using a CIContext, write a CGImage the size of the input CIImage.
Be aware that a CIContext is expensive. If you already have one created, you should probably use it.
Pretty much, a UIImage size is the same as a CIImage extent. (I say pretty much because some generated CIImages can have infinite extents.)
Depending on your specific needs (and your UIImageView), you may want to use the output CIImage extent instead. Usually though, they are the same.
Last, a suggestion. If you are trying to use a CIFilter to show "near real-time" changes to an image (like a photo editor), consider the major performance improvements you'll get using CIImages and a GLKView over UIImages and a UIImageView. The former uses a devices GPU instead of the CPU.
This could also happen if a CIFilter outputs an image with dimensions different than the input image (e.g. with CIPixellate)
In which case, simply tell the CIContext to render the image in a smaller rectangle:
let cgOutputImage = context.createCGImage(ciOutputImage, from: ciInputImage.extent.insetBy(dx: 20, dy: 20))
I'm making a simple filter app. I've found that if you load an image from the camera roll that is a PNG (PNGs have no orientation data flag) and the height is greater than the width, upon applying certain distortion filters to said image it will rotate and present it self as if it were a landscape image.
I found the below technique online somewhere in the many tabs i had open and it seems to do exactly what i want. It uses the original scale and orientation of the image when it was first loaded.
let newImage = UIImage(CIImage:(output), scale: 1.0, orientation: self.origImage.imageOrientation)
but this is the warning i get when i try to use it:
Ambiguous use of 'init(CIImage:scale:orientation:)'
Here's the entire thing I'm trying to get working:
//global variables
var image: UIImage!
var origImage: UIImage!
func setFilter(action: UIAlertAction) {
origImage = image
// make sure we have a valid image before continuing!
guard let image = self.imageView.image?.cgImage else { return }
let openGLContext = EAGLContext(api: .openGLES3)
let context = CIContext(eaglContext: openGLContext!)
let ciImage = CIImage(cgImage: image)
let currentFilter = CIFilter(name: "CIBumpDistortion")
currentFilter?.setValue(ciImage, forKey: kCIInputImageKey)
if let output = currentFilter?.value(forKey: kCIOutputImageKey) as? CIImage{
//the line below is the one giving me errors which i thought would work.
let newImage = UIImage(CIImage:(output), scale: 1.0, orientation: self.image.imageOrientation)
self.imageView.image = UIImage(cgImage: context.createCGImage(newImage, from: output.extent)!)}
The filters all work, they unfortunately turn images described above by 90 degrees for the reasons I suspect.
I've tried some other methods like using an extension that checks orientation of UIimages and converting the CIimage to a Uiimage, using the extension, then trying to convert it back to a Ciimage or just load the UIimage to the imageView for output. I ran into snag after snag with that process. I started to seem really convoluted just to get certain images to their default orientation as well.
Any advice would be greatly appreciated!
EDIT: heres where I got the method I was trying: When applying a filter to a UIImage the result is upside down
I found the answer. My biggest issue was the "Ambiguous use of 'init(CIImage:scale:orientation:)' "
it turned out that Xcode was auto populating the code as 'CIImage:scale:orientation' when it should have been ciImage:scale:orientation' The very vague error left a new dev like my scratching my head for 3 days over this. (This was true for CGImage and UIImage inits as well, but my original error was with CIImage so I used that to explain.)
with that knowledge I was able to formulate the code below for my new output:
if let output = currentFilter?.value(forKey: kCIOutputImageKey) as? CIImage{
let outputImage = UIImage(cgImage: context.createCGImage(output, from: output.extent)!)
let imageTurned = UIImage(cgImage: outputImage.cgImage!, scale: CGFloat(1.0), orientation: origImage.imageOrientation)
centerScrollViewContents()
self.imageView.image = imageTurned
}
This code replaces the if let output in the OP.