I'm attempting to crop an UIImage in iOS using Saliency via the VNGenerateObjectnessBasedSaliencyImageRequest().
I'm following the documentation provided by Apple here https://developer.apple.com/documentation/vision/2908993-vnimagerectfornormalizedrect and
working off of this tutorial https://betterprogramming.pub/cropping-areas-of-interest-using-vision-in-ios-e83b5e53440b.
I'm also referencing this project https://developer.apple.com/documentation/vision/highlighting_areas_of_interest_in_an_image_using_saliency.
This is the code I currently have in place.
static func cropImage(_ image: UIImage, completionHandler:#escaping(UIImage?, String?) -> Void) -> Void {
guard let originalImage = image.cgImage else { return }
let saliencyRequest = VNGenerateObjectnessBasedSaliencyImageRequest()
let requestHandler = VNImageRequestHandler(cgImage: originalImage, orientation: .right, options: [:])
DispatchQueue.global(qos: .userInitiated).async {
do {
try requestHandler.perform([saliencyRequest])
guard let results = saliencyRequest.results?.first else{return}
if let observation = results as VNSaliencyImageObservation?
{
let salientObjects = observation.salientObjects
if let ciimage = CIImage(image: image)
{
let salientRect = VNImageRectForNormalizedRect((salientObjects?.first!.boundingBox)!,
Int(ciimage.extent.size.width),
Int(ciimage.extent.size.height))
let croppedImage = ciimage.cropped(to: salientRect)
let cgImage = iOSVisionHelper.convertCIImageToCGImage(inputImage: croppedImage)
if cgImage != nil {
let thumbnail = UIImage(cgImage: cgImage!)
completionHandler(thumbnail, nil)
}else{
completionHandler(nil, "Unable to crop image")
}
}
}
} catch {
completionHandler(nil, error.localizedDescription)
}
}
}
static func convertCIImageToCGImage(inputImage: CIImage) -> CGImage? {
let context = CIContext(options: nil)
if let cgImage = context.createCGImage(inputImage, from: inputImage.extent) {
return cgImage
}
return nil
}
This is working pretty well, except it seems like it's not adjusting the height of the image. It crops in the sides perfectly, but not the top or bottom.
Here are examples of the original image and it being cropped.
This is what the iOS demo app found at https://developer.apple.com/documentation/vision/highlighting_areas_of_interest_in_an_image_using_saliency generates.
Any help would be very much appreciated.
Related
My app will crush showing "Terminating Due to Memory Issue" .When I am use GPUImage filter many times..How can I memory free form GPUImage. My code is bellow:
func caartonImage(ciImage:CIImage)->UIImage
{
autoreleasepool{
let uiImg = self.convert(cmage: ciImage)
var toonFilter:SmoothToonFilter!
toonFilter = SmoothToonFilter()
toonFilter.blurRadiusInPixels = 1.9
let cartoonImage = uiImg.filterWithOperation(toonFilter)
toonFilter = nil
if cartoonImage != nil
{
return cartoonImage
}
else
{
return uiImg
}
}
}
func convert(cmage:CIImage) -> UIImage
{
var context:CIContext!
var cgImage:CGImage!
context = CIContext.init(options: nil)
cgImage = context.createCGImage(cmage, from: cmage.extent)!
let image:UIImage = UIImage.init(cgImage: cgImage)
context = nil
return image
}
I got two solution converting CVPixelBuffer to UIImage.
Using CIImage and its context
First solution is using CIImage and CIContext.
func readAssetAndCache(completion: #escaping () -> Void) {
/** Setup AVAssetReader **/
reader.startReading()
let context = CIContext()
while let sampleBuffer = generator.trackoutput.copyNextSampleBuffer() {
autoreleasepool {
guard let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let ciImage = CIImage(cvImageBuffer: imageBuffer)
guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { return }
if let image = UIImage(cgImage: cgImage) {
CacheManager.default.cache(image: image, forKey: "CachedImages") {
completion()
}
}
}
}
}
Using VTCreateCGImageFromCVPixelBuffer
Code Looks like this.
func readAssetAndCache(completion: #escaping () -> Void) {
/** Setup AVAssetReader **/
reader.startReading()
while let sampleBuffer = generator.trackoutput.copyNextSampleBuffer() {
autoreleasepool {
guard let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
if let image = UIImage(pixelBuffer: imageBuffer,
scale: 1.0,
orientation: self.imageOrientation) {
CacheManager.default.cache(image: image, forKey: "CachedImages") {
completion()
}
}
}
}
}
}
In this solution I use this extension method.
import UIKit
import VideoToolbox
extension UIImage {
public convenience init?(pixelBuffer: CVPixelBuffer, scale: CGFloat, orientation: UIImageOrientation) {
var image: CGImage?
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, nil, &image)
guard let cgImage = image else { return nil }
self.init(cgImage: cgImage, scale: scale, orientation: orientation)
}
}
Whole process related to extracting images from video is as follows.
Whole Process
Extract All frames serially.
Cache each image in disk concurrently, using OperationQueue.
Question
The problem occurs when images are cached concurrently, using CIContext and its createCGImage(ciImage:rect:). Memory footprints gets higher until the end of all caching process in first scenario. But, Second way, it releases its memory when each workItem is finished.
I really don't know why.
Here is my Caching method related to this problem.
func cache(image: UIImage, forKey key: String, completion: #escaping () -> Void) {
let operation = BlockOperation(block: { [weak self] in
guard let `self` = self else { return }
self.diskCache(image: image)
})
operation.completionBlock = completion
operationQueue.addOperation(operation)
}
func diskCache(image: UIImage) {
let data = UIImageJPEGRepresentation(image, 1.0)
let key = "\(self.identifierPrefix).jpg"
FileManager.default.createFile(atPath: self.cachePath.appendingPathComponent(key).path,
contents: data, attributes: nil)
}
I get nil on an iOS device (works ok on Simulator or Playground) when I create create a barcode UIImage with generateBarcode("ABCDEF") and call UIImageJPEGRepresentation or UIImagePNGRepresentation.
There is clearly something still wrong in the UIImage as the debugger cannot display the UIImage either. The image exists, the cgImage property is set but UIImageJPEGRepresentation doesn't like it.
I have tried to solve it as per: UIImageJPEGRepresentation returns nil
class func generateBarcode(from string: String) -> UIImage? {
let data = string.data(using: String.Encoding.ascii)
if let filter = CIFilter(name: "CICode128BarcodeGenerator") {
filter.setValue(data, forKey: "inputMessage")
if let ciImage = filter.outputImage {
if let cgImange = convertCIImageToCGImage(inputImage: ciImage) {
return UIImage(cgImage: cgImange)
}
}
}
return nil
}
class func convertCIImageToCGImage(inputImage: CIImage) -> CGImage? {
let context = CIContext(options: nil)
if let cgImage = context.createCGImage(inputImage, from: inputImage.extent) {
return cgImage
}
return nil
}
I am making image form QR Code by using following code:
func createQRFromString(str: String) -> CIImage? {
let stringData = str.dataUsingEncoding(NSUTF8StringEncoding)
let filter = CIFilter(name: "CIQRCodeGenerator")
filter?.setValue(stringData, forKey: "inputMessage")
filter?.setValue("H", forKey: "inputCorrectionLevel")
return filter?.outputImage
}
And Then I am adding to UIImageView Like this:
if let img = createQRFromString(strQRData) {
let somImage = UIImage(CIImage: img, scale: 1.0, orientation: UIImageOrientation.Down)
imgviewQRcode.image = somImage
}
Now I need to save this to a JPEG or PNG file. But when I am doing so my app crashes:
#IBAction func btnSave(sender: AnyObject) {
// // Define the specific path, image name
let documentsDirectoryURL = try! NSFileManager().URLForDirectory(.DocumentDirectory, inDomain: .UserDomainMask, appropriateForURL: nil, create: true)
// create a name for your image
let fileURL = documentsDirectoryURL.URLByAppendingPathComponent("image.jpg")
if let image = imgviewQRcode.image // imgviewQRcode is UIImageView
{
if let path = fileURL?.path
{
if !NSFileManager.defaultManager().fileExistsAtPath(fileURL!.path!)
{
if UIImageJPEGRepresentation(image, 1.0)!.writeToFile(path, atomically: true)
{
print("file saved")
}
}//Checking existing file
}//Checking path
}//CHecking image
}
Crash Point
UIImageJPEGRepresentation(image, 1.0)!.writeToFile(path, atomically: true)
Reason
fatal error: unexpectedly found nil while unwrapping an Optional value
Debug Tests:
func convert(cmage:CIImage) -> UIImage
{
let context:CIContext = CIContext.init(options: nil)
let cgImage:CGImage = context.createCGImage(cmage, from: cmage.extent)!
let image:UIImage = UIImage.init(cgImage: cgImage)
return image
}
Use this function to convert CIImage to UIImage . It works .
func convert(image:CIImage) -> UIImage
{
let image:UIImage = UIImage.init(ciImage: image)
return image
}
Perhaps, this was unavailable before, but it is now possible to create UIImages directly from CIImage.
My final code
func generateQRCode(from string: String) -> UIImage? {
let data = string.data(using: String.Encoding.ascii)
if let filter = CIFilter(name: "CIQRCodeGenerator") {
filter.setValue(data, forKey: "inputMessage")
let transform = CGAffineTransform(scaleX: 3, y: 3)
if let output = filter.outputImage?.transformed(by: transform) {
let context:CIContext = CIContext.init(options: nil)
let cgImage:CGImage = context.createCGImage(output, from: output.extent)!
let image:UIImage = UIImage.init(cgImage: cgImage)
return image
}
}
return nil
}
I want to detect qrcode from still image
Here is the code
let ciimage = <Load image from asset and covert to CIImage>
let detector = CIDetector(ofType: CIDetectorTypeQRCode, context:nil, options:nil)
let features = detector.featuresInImage(ciimage)
print(features.count)
The image is correct CIImage instance which has QRCode but I always get no features, Have any one had the same issue?
Thank you very much.
I think I found answer
That code only works in Playground or real iPhone, it never works on simulator
I generated QRCode image using CIQRCodeGenerator CoreImage filter.
I hope someone can check my answer here and give me advice.
Below is source code what I used
enum QRCodeUtilError:ErrorType{
case GenerationFailed
}
extension UIImage {
/**
This function trys to return CIImage even the image is CGImageRef based.
- Returns: CIImage equal to current image
*/
func ciImage() -> CoreImage.CIImage?{
// if CIImage is not nil then return it
if self.CIImage != nil {
return self.CIImage
}
guard let cgImage = self.CGImage else {
return nil
}
return CoreImage.CIImage(CGImage: cgImage)
}
}
/**
QRCode utility class
*/
class QRCodeUtil{
/**
Generates 'H' mode qrcode image
- Parameter qrCode: QRCode String
- Returns: Optional UIImage instance of QRCode
*/
class func imageForQrCode(qrCode:String) throws -> UIImage{
guard let filter = CIFilter(name: "CIQRCodeGenerator") else {
throw QRCodeUtilError.GenerationFailed
}
guard let data = qrCode.dataUsingEncoding(NSISOLatin1StringEncoding) else {
throw QRCodeUtilError.GenerationFailed
}
filter.setValue(data, forKey: "inputMessage")
filter.setValue("H", forKey: "inputCorrectionLevel")
guard let outputImage = filter.outputImage else {
throw QRCodeUtilError.GenerationFailed
}
/*
Convert the image to CGImage based because
it crashes when saving image using
UIImagePNGRepresentation or UIImageJPEGRepresentation
if image is CIImage based.
*/
let context = CIContext(options: nil)
let cgImage = context.createCGImage(outputImage, fromRect: outputImage.extent)
return UIImage(CGImage: cgImage)
}
/**
Detects QRCode from string
- Parameter qrImage: UIImage which has QRCode
- Returns: QRCode detected
*/
class func qrCodeFromImage(qrImage:UIImage) -> String {
let detector = CIDetector(ofType: CIDetectorTypeQRCode, context: nil, options: [CIDetectorAccuracy:CIDetectorAccuracyHigh])
guard let ciImage = qrImage.ciImage() else {
return ""
}
guard let feature = detector.featuresInImage(ciImage).last as? CIQRCodeFeature else {
return ""
}
return feature.messageString
}
}