I'm converting the image buffer (CVPixelBuffer) to UIImage and, in doing so, I'm changing the orientation as well as the size.
let image = CIImage(cvImageBuffer: imageBuffer)
let imageSize = CGSize(width: CVPixelBufferGetWidth(imageBuffer), height: CVPixelBufferGetHeight(imageBuffer))
let normalizeTransform = CGAffineTransform(scaleX: 1.0 / imageSize.width, y: 1.0 / imageSize.height)
let flipTransform = orientation.isPortrait ? CGAffineTransform(scaleX: -1, y: -1).translatedBy(x: -1, y: -1) : .identity
let viewPortSize: CGSize = viewPort.size
let displayTransform: CGAffineTransform = arFrame.displayTransform(for: orientation, viewportSize: CGSize(width: viewPortSize.width, height: viewPortSize.height))
let scaleX: CGFloat = viewPortSize.width
let scaleY: CGFloat = viewPortSize.height
let viewPortTransform = CGAffineTransform(scaleX: scaleX, y: scaleY)
let scaledImage: CIImage = image
.transformed(by: normalizeTransform
.concatenating(flipTransform)
.concatenating(displayTransform)
.concatenating(viewPortTransform)
)
.cropped(to: viewPort)
guard let uiImage: UIImage = self.convert(cmage: scaledImage) else {
return nil
}
(arFrame is ARFrame from ARKit and displayTransform is for creating the CGAffineTransform for transforming a normalized image.)
The breakdown of above code is something like this:
Scale down the image to normalize the coordinates.
Flip the image according to the orientation (some ARKit quirk)
Transform the image suited for rendering the camera image onscreen.
Scale up the image to fit the camera screen.
One problem I'm facing is that since I'm scaling down the image #1 and enlarging it back up #4, the image quality seems to be severely impacted. #1 has to come before #4 and can't be combined since #3 has to take in a normalized image.
Update
When I try to prevent drastically scaling down the size of the image by doing the following:
let flipTransform: CGAffineTransform = metadata.orientation.isPortrait ? CGAffineTransform(scaleX: -1, y: -1).translatedBy(x: -1, y: -1) : .identity
let scaleTransform = CGAffineTransform(scaleX: scaleX / imageSize.width, y: scaleY / imageSize.height)
let img = image.oriented(cgImagePropertyOrientation)
let scaledImage = img
.transformed(by: flipTransform
.concatenating(scaleTransform))
the resulting image is cut off by half and the ratio skewed.
Yeah. #rapiddevice has it right. Scaling your image down is destructive. You need to do the scaling from source image to target resolution once, and keep the pixel count as high as possible to preserve image detail.
Related
I implemented addFaceLandmarksToImage function to crop the outer lips of the image. addFaceLandmarksToImage function first detects the face on the image using vision, convert the face bounding box size and origin to image size and origin. then I used face landmark outer lips to get the normalized point of the outer lips and draw the line on the outer lips of the image by connecting all the normalized points.
Then I implemented logic to crop the image. I first converted the normalized point of the outer lips into the image coordinate and used the find point method to get left, right, top and bottom-most points and extracted outer lips by cropping it and showed in processed image view. The issue with this function is output following is not as expected. Also if I use an image other then the one that is in the following link, it crops images other then outer lips. I could not figure out where have I gone wrong, is it on calculation of cropping rect or should I use another approach(using OpenCV and region of interest (ROI)) to extract the outer lips from the image?
video of an application
func addFaceLandmarksToImage(_ face: VNFaceObservation)
{
UIGraphicsBeginImageContextWithOptions(image.size, true, 0.0)
let context = UIGraphicsGetCurrentContext()
// draw the image
image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
context?.translateBy(x: 0, y: image.size.height)
context?.scaleBy(x: 1.0, y: -1.0)
// draw the face rect
let w = face.boundingBox.size.width * image.size.width
let h = face.boundingBox.size.height * image.size.height
let x = face.boundingBox.origin.x * image.size.width
let y = face.boundingBox.origin.y * image.size.height
let cropFace = self.image.cgImage?.cropping(to: CGRect(x: x, y: y, width: w, height: h))
let ii = UIImage(cgImage: cropFace!)
// outer lips
context?.saveGState()
context?.setStrokeColor(UIColor.yellow.cgColor)
if let landmark = face.landmarks?.outerLips {
var actualCordinates = [CGPoint]()
print(landmark.normalizedPoints)
for i in 0...landmark.pointCount - 1 {
// last point is 0,0
let point = landmark.normalizedPoints[i]
actualCordinates.append(CGPoint(x: x + CGFloat(point.x) * w, y: y + CGFloat(point.y) * h))
if i == 0 {
context?.move(to: CGPoint(x: x + CGFloat(point.x) * w, y: y + CGFloat(point.y) * h))
} else {
context?.addLine(to: CGPoint(x: x + CGFloat(point.x) * w, y: y + CGFloat(point.y) * h))
}
}
// Finding left,right,top,buttom point from actual coordinates points[CGPOINT]
let leftMostPoint = self.findPoint(points: actualCordinates, position: .leftMost)
let rightMostPoint = self.findPoint(points: actualCordinates, position: .rightMost)
let topMostPoint = self.findPoint(points: actualCordinates, position: .topMost)
let buttonMostPoint = self.findPoint(points: actualCordinates, position: .buttonMost)
print("actualCordinates:",actualCordinates,
"leftMostPoint:",leftMostPoint,
"rightMostPoint:",rightMostPoint,
"topMostPoint:",topMostPoint,
"buttonMostPoint:",buttonMostPoint)
let widthDistance = -(leftMostPoint.x - rightMostPoint.x)
let heightDistance = -(topMostPoint.y - buttonMostPoint.y)
//Cropping the image.
// self.image is actual image
let cgCroppedImage = self.image.cgImage?.cropping(to: CGRect(x: leftMostPoint.x,y: leftMostPoint.x - heightDistance,width:1000,height: topMostPoint.y + heightDistance + 500))
let jj = UIImage(cgImage: cgCroppedImage!)
self.processedImageView.image = jj
}
context?.closePath()
context?.setLineWidth(8.0)
context?.drawPath(using: .stroke)
context?.saveGState()
// get the final image
let finalImage = UIGraphicsGetImageFromCurrentImageContext()
// end drawing context
UIGraphicsEndImageContext()
imageView.image = finalImage
}}
normalized points of the outer lips of an image:-
[(0.397705078125, 0.3818359375),
(0.455322265625, 0.390625),
(0.5029296875, 0.38916015625),
(0.548828125, 0.40087890625),
(0.61279296875, 0.3984375),
(0.703125, 0.37890625),
(0.61474609375, 0.21875),
(0.52294921875, 0.1884765625),
(0.431640625, 0.20166015625),
(0.33203125, 0.34423828125)]
actual coordinate points of the outer lips of an image: -
[(3025.379819973372, 1344.4951847679913),
(3207.3986613331363, 1372.2607707381248),
(3357.7955853380263, 1367.633173076436),
(3502.7936454042792, 1404.6539543699473),
(3704.8654099646956, 1396.9412916004658),
(3990.2339324355125, 1335.2399894446135),
(3711.035540180281, 829.2893117666245),
(3421.039420047775, 733.6522934250534),
(3132.5858324691653, 775.3006723802537),
(2817.9091914743185, 1225.7201781179756)]
I also tried using the following method that uses CIDetector to get the mouth position and extracting the outer lips through cropping. The output wasn't good.
func focusonMouth() {
let ciimage = CIImage(cgImage: image.cgImage!)
let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options)!
let faces = faceDetector.features(in: ciimage)
if let face = faces.first as? CIFaceFeature {
if face.hasMouthPosition {
let crop = image.cgImage?.cropping(to: CGRect(x: face.mouthPosition.x, y: face.mouthPosition.y, width: face.bounds.width - face.mouthPosition.x , height: 200))
processedImageView.image = imageRotatedByDegrees(oldImage: UIImage(cgImage: crop!), deg: 90)
}
}
}
There are two problems:
input image on iOS can be rotated with orientation property marking how it is rotated. Vision Framework will do the job, but coordinates will be rotated. The simplest solution is to supply image that is up-oriented (normal rotation).
position and size of located landmarks are relative to position and size of the detected face landmarks. So found position and size should be scaled by size and offset of the face found, not whole image.
I am working to add support for wide color photos in iOS 10. When the user takes a photo from the camera, I need to use the new API that supports the new color space to save the photo data - UIGraphicsImageRenderer's jpegData instead of UIImageJPEGRepresentation.
I'm running into some troubles with image orientations. Taking a photo on my iPad in portrait, the image isn't being drawn correctly. See the comments below:
Old API:
let image = info[UIImagePickerControllerOriginalImage] as! UIImage
let imageData = UIImageJPEGRepresentation(image, 1)
New API:
let image = info[UIImagePickerControllerOriginalImage] as! UIImage
let cgImage = image.cgImage!
let ciImage = CIImage(cgImage: cgImage)
let format = UIGraphicsImageRendererFormat()
format.scale = 1
format.prefersExtendedRange = true
let renderer = UIGraphicsImageRenderer(bounds: ciImage.extent, format: format)
let imageData = renderer.jpegData(withCompressionQuality: 1, actions: { context in
context.cgContext.draw(cgImage, in: ciImage.extent) //draws flipped horizontally
//image.draw(at: .zero) //draws rotated 90 degrees leaving black at bottom
//image.draw(in: ciImage.extent) //draws rotated 90 degrees stretching and compressing the image to fill the rect
})
What's the correct way to replace UIImageJPEGRepresentation with UIGraphicsImageRenderer's jpegData?
UIImage can have different orientation depending on camera rotation.
You can dynamically resolve the transformation needed to be applied to the image depending on that orientation, like this:
let renderer = UIGraphicsImageRenderer(size: image.size, format: format)
let imageData = renderer.jpegData(withCompressionQuality: 1, actions: { context in
var workSize = image.size;
workSize.width = floor(workSize.width / image.scale)
workSize.height = floor(workSize.height / image.scale)
// No-op if the orientation is already correct
// if image.imageOrientation == .up { draw image }
// We need to calculate the proper transformation to make the image upright.
// We do it in 2 steps: Rotate if Left/Right/Down, and then flip if Mirrored.
var transform = CGAffineTransform.identity
switch image.imageOrientation
{
case .down, .downMirrored:
transform = transform.translatedBy(x: workSize.width, y: workSize.height)
transform = transform.rotated(by: CGFloat(Double.pi))
break
case .left, .leftMirrored:
transform = transform.translatedBy(x: workSize.width, y: 0.0)
transform = transform.rotated(by: CGFloat(Double.pi / 2.0))
break
case .right, .rightMirrored:
transform = transform.translatedBy(x: 0.0, y: workSize.height)
transform = transform.rotated(by: CGFloat(-Double.pi / 2.0))
break
case .up, .upMirrored:
break
}
switch image.imageOrientation
{
case .upMirrored, .downMirrored:
transform = transform.translatedBy(x: workSize.width, y: 0.0)
transform = transform.scaledBy(x: -1.0, y: 1.0)
break
case .leftMirrored, .rightMirrored:
transform = transform.translatedBy(x: workSize.height, y: 0.0);
transform = transform.scaledBy(x: -1.0, y: 1.0);
break
case .up, .down, .left, .right:
break
}
// Now we draw the underlying CGImage into a new context, applying the transform
// calculated above.
let ctx = context.cgContext
ctx.concatenate(transform)
switch image.imageOrientation {
case .left, .leftMirrored, .right, .rightMirrored:
ctx.draw(image.cgImage!, in: CGRect(x: 0.0, y:0.0, width: workSize.height, height: workSize.width))
break;
default:
ctx.draw(image.cgImage!, in: CGRect(origin: .zero, size: workSize))
break;
}
})
Answer based on UIImage+fixOrientation
The image.draw() method should correct the orientation automatically for you:
This method draws the entire image in the current graphics context, respecting the image’s orientation setting. In the default coordinate system, images are situated down and to the right of the origin of the specified rectangle. This method respects any transforms applied to the current graphics context, however.
I'm not sure why you need to use ci.extent. The extent seems to be (I make no claims to be an expert in Core Image APIs) the dimensions of the "raw" image data, which is stored without any image orientation correction. If you raw data needs rotation (non-Up orientations like Left, Right), the extent will still be the original rect that the data is stored.
I use the following code for an image with imageOrientation.rawValue = 3 and my data turns out perfectly fine at the right size with "corrected" imageOrientation.rawValue = 0 when reloaded.
let imageSize = image.Size
let renderer = UIGraphicsImageRenderer(size: renderedSize, format: UIGraphicsImageRendererFormat())
return renderer.jpegData(withCompressionQuality: 0.95) { (rendererContext) in
let rect = CGRect(x: 0, y: 0, width: imageSize.width, height: imageSize.height)
image.draw(in: rect)
}
My input image "renders" its data correctly with my code, taking into account the image orientation. On the right, I use draw(in: extent). The image isn't rotated though. Just stretched.
I've created Swift version of this class: https://github.com/bennythemink/ZoomRotatePanImageView/blob/master/ZoomRotatePanImageView.m Works nice. Now I want to save modified image to file. The thing is I want to save it in full resolution and also I want to save area which is only visible to user.
Let me show you simple example:
This is how it looks in my app. Image is one a few samples in iOS simulator. Most of it is out of screen. I want only visible part.
After saving without cropping it looks like this:
So far so good after clipping it'd be nice.
But now let's make some changes:
After saving:
Looks like it's transformed by wrong pivot. How can I fix it?
Here's my code for saving:
UIGraphicsBeginImageContextWithOptions(image.size, false, 0)
let context = UIGraphicsGetCurrentContext()
let transform = imageView.transform
let imageRect = CGRectMake(0, 0, image.size.width, image.size.height)
CGContextSetFillColorWithColor(context, UIColor.blueColor().CGColor) //for debugging
CGContextFillRect(context, imageRect)
CGContextConcatCTM(context, transform)
image.drawInRect(imageRect)
let newImage = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
There's a simpler method to achieve it:
UIGraphicsBeginImageContextWithOptions(imageContainer.bounds.size, false, 0)
self.imageContainer.drawViewHierarchyInRect(imageContainer.bounds, afterScreenUpdates: true)
let screenshot = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
However output image has size of the view not an actual image and I want it in full resolution.
Updated code to work with image of any size :
let boundsScale = imageView.bounds.size.width / imageView.bounds.size.height
let imageScale = image!.size.width / image!.size.height
let size = (image?.size)!
var canvasSize = size
if boundsScale > imageScale {
canvasSize.width = canvasSize.height * boundsScale
}else{
canvasSize.height = canvasSize.width / boundsScale
}
let xScale = canvasSize.width / imageView.bounds.width
let yScale = canvasSize.height / imageView.bounds.height
let center = CGPointApplyAffineTransform(imageView.center, CGAffineTransformScale(CGAffineTransformIdentity, xScale, yScale))
let xCenter = center.x
let yCenter = center.y
UIGraphicsBeginImageContextWithOptions(canvasSize, false, 0);
let context = UIGraphicsGetCurrentContext()!
//Apply transformation
CGContextTranslateCTM(context, xCenter, yCenter)
CGContextConcatCTM(context, imageView.transform)
CGContextTranslateCTM(context, -xCenter, -yCenter)
var drawingRect : CGRect = CGRectZero
drawingRect.size = canvasSize
//Transaltion
drawingRect.origin.x = (xCenter - size.width*0.5)
drawingRect.origin.y = (yCenter - size.height*0.5)
//Aspectfit calculation
if boundsScale > imageScale {
drawingRect.size.width = drawingRect.size.height * imageScale
}else{
drawingRect.size.height = drawingRect.size.width / imageScale
}
image!.drawInRect(drawingRect)
let newImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
Simulator screen shot
Saved image
imageView is the photo. I created a copy of the photo called tempImageView and I tried to draw on the tempImageView. However, when I try to draw tempImageView, it enlarges to fill the entire screen.
func createTempImageView(){
tempImageView = UIImageView(frame:CGRect(x: 0, y: 0, width: imageView.image!.size.width, height: imageView.image!.size.height))
tempImageView.center = CGPoint(x: imageView.center.x, y: imageView.center.y)
view.insertSubview(tempImageView, aboveSubview: imageView)
}
Because I set imageView.contentMode = .ScaleAspectFit, the image on screen has a smaller width and height than the original image. However, the imageView.image!.size.width and imageView.image!.size.height refers the original image. Therefore, when I tried to draw on the tempImageView, the image enlarged because tempImageView refers to the original image, which is much larger than the image on screen.
To get the dimensions of the image on screen and to stop the image from enlarging, I did this:
func createTempImageView(){
let widthRatio = imageView.bounds.size.width / imageView.image!.size.width
let heightRatio = imageView.bounds.size.height / imageView.image!.size.height
let scale = min(widthRatio, heightRatio)
let imageWidth = scale * imageView.image!.size.width
let imageHeight = scale * imageView.image!.size.height
tempImageView = UIImageView(frame:CGRect(x: 0, y: 0, width: imageWidth, height: imageHeight))
tempImageView.center = CGPoint(x: imageView.center.x, y: imageView.center.y)
tempImageView.backgroundColor = UIColor.clearColor()
view.insertSubview(tempImageView, aboveSubview: imageView)
}
I am analyzing the images captured by the camera, I am going through each pixel by pixel and storing the rgb value of each pixel in an dictionary, obviously it's taking a lot of time. Can I scale down an image without loosing the percentage of color ratio in the image...
Example: Original image has RGB color and count as
Color count
[0xF0F0F0] 100
[0xB0B0B0] 50
[0x909090] 20
Then after scaling the image by half the color count in scaled image is like this:
Color count
[0xF0F0F0] 50
[0xB0B0B0] 25
[0x909090] 10
So 2 questions:
1. Is this doable in Swift?
2. If yes, then how do I scale an image in swift
Scaling down an image will definitely lose some pixels.
To scale an image in Swift:
var scaledImage // your scaled-down image
var origImage // your original image
let itemSize = CGSizeMake(origImage.size.width/2, origImage.size.height/2)
UIGraphicsBeginImageContextWithOptions(itemSize, false, UIScreen.mainScreen().scale)
let imageRect = CGRectMake(0.0, 0.0, itemSize.width, itemSize.height)
origImage.drawInRect(imageRect)
scaledImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
This is my code for resizing a image taken by the camera, after this process I send it to our servers. I recommend it to resize it instead of eliminating colors.
Also if you don't want to resize it, you can't convert the image to JPEG and assign a quality.
Use UIImageJPEGRepresentation instead of UIImagePNGRepresentation
if let img = value as? UIImage {
var newWidth = 0
var newHeight = 0
var sizeLimit = 700 //in px
let originalWidth = Int(img.size.width)
let originalHeight = Int(img.size.height)
//Get new size with max w/h = 700
if originalWidth > originalHeight {
//Max width
newWidth = sizeLimit
newHeight = (originalHeight*sizeLimit)/originalWidth
}else{
newWidth = (originalWidth*sizeLimit)/originalHeight
newHeight = sizeLimit
}
let newSize = CGSizeMake(CGFloat(newWidth), CGFloat(newHeight))
UIGraphicsBeginImageContext(newSize)
img.drawInRect(CGRectMake(0, 0, CGFloat(newWidth), CGFloat(newHeight)))
let newImg = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
//This is a UIIMAGE
let imageData:UIImage = UIImagePNGRepresentation(newImg)
}
i would say this probably isnt possible, because resizing the image would change the value of the actual colours so it would give you different results.
but maybe you can look into speeding it up using the full sized image. Look into maybe using metal for GPU accelerated calculations
Is this doable in Swift?
Sure
If yes, then how do I scale an image in swift
If you don't mind low level functions, since this is something you want to have done fast and on the CPU, not GPU, you might want to look at the Accelerate framework and vImage_scale in particular. It provides direct access to the RGB data in memory as well.
When I did something similar in the past, I noticed that the scaling actually cuts off some colors and does not give accurate representation of the original. Unfortunately, I don't have the examples to show anymore, but I'd suggest that you try this technique with different scaling ratio to see if it gives satisfactory data in your case.
func scaleAndRotateImage(image: UIImage, MaxResolution iIntMaxResolution: Int) -> UIImage {
let kMaxResolution = iIntMaxResolution
let imgRef = image.cgImage!
let width: CGFloat = CGFloat(imgRef.width)
let height: CGFloat = CGFloat(imgRef.height)
var transform = CGAffineTransform.identity
var bounds = CGRect.init(x: 0, y: 0, width: width, height: height)
if Int(width) > kMaxResolution || Int(height) > kMaxResolution {
let ratio: CGFloat = width / height
if ratio > 1 {
bounds.size.width = CGFloat(kMaxResolution)
bounds.size.height = bounds.size.width / ratio
}
else {
bounds.size.height = CGFloat(kMaxResolution)
bounds.size.width = bounds.size.height * ratio
}
}
let scaleRatio: CGFloat = bounds.size.width / width
let imageSize = CGSize.init(width: CGFloat(imgRef.width), height: CGFloat(imgRef.height))
var boundHeight: CGFloat
let orient = image.imageOrientation
// The output below is limited by 1 KB.
// Please Sign Up (Free!) to remove this limitation.
switch orient {
case .up:
//EXIF = 1
transform = CGAffineTransform.identity
case .upMirrored:
//EXIF = 2
transform = CGAffineTransform.init(translationX: imageSize.width, y: 0.0)
transform = transform.scaledBy(x: -1.0, y: 1.0)
case .down:
//EXIF = 3
transform = CGAffineTransform.init(translationX: imageSize.width, y: imageSize.height)
transform = transform.rotated(by: CGFloat(Double.pi / 2))
case .downMirrored:
//EXIF = 4
transform = CGAffineTransform.init(translationX: 0.0, y: imageSize.height)
transform = transform.scaledBy(x: 1.0, y: -1.0)
case .leftMirrored:
//EXIF = 5
boundHeight = bounds.size.height
bounds.size.height = bounds.size.width
bounds.size.width = boundHeight
transform = CGAffineTransform.init(translationX: imageSize.height, y: imageSize.width)
transform = transform.scaledBy(x: -1.0, y: 1.0)
transform = transform.rotated(by: CGFloat(Double.pi / 2) / 2.0)
break
default: print("Error in processing image")
}
UIGraphicsBeginImageContext(bounds.size)
let context = UIGraphicsGetCurrentContext()
if orient == .right || orient == .left {
context?.scaleBy(x: -scaleRatio, y: scaleRatio)
context?.translateBy(x: -height, y: 0)
}
else {
context?.scaleBy(x: scaleRatio, y: -scaleRatio)
context?.translateBy(x: 0, y: -height)
}
context?.concatenate(transform)
context?.draw(imgRef, in: CGRect.init(x: 0, y: 0, width: width, height: height))
let imageCopy = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return imageCopy!
}