I have some doubts about Core Image coordinate system, way transforms are applied and extent is determined. I couldn't find much in documentation or on internet so I tried the following code to rotate CIImage and display it in UIImageView.
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view.
imageView.contentMode = .scaleAspectFit
let uiImage = UIImage(contentsOfFile: imagePath)
ciImage = CIImage(cgImage: (uiImage?.cgImage)!)
imageView.image = uiImage
}
private var currentAngle = CGFloat(0)
private var ciImage:CIImage!
private var ciContext = CIContext()
#IBAction func rotateImage() {
let extent = ciImage.extent
let translate = CGAffineTransform(translationX: extent.midX, y: extent.midY)
let uiImage = UIImage(contentsOfFile: imagePath)
currentAngle = currentAngle + CGFloat.pi/10
let rotate = CGAffineTransform(rotationAngle: currentAngle)
let translateBack = CGAffineTransform(translationX: -extent.midX, y: -extent.midY)
let transform = translateBack.concatenating(rotate.concatenating(translate))
ciImage = CIImage(cgImage: (uiImage?.cgImage)!)
ciImage = ciImage.transformed(by: transform)
NSLog("Extent \(ciImage.extent), Angle \(currentAngle)")
let cgImage = ciContext.createCGImage(ciImage, from: ciImage.extent)
imageView.image = UIImage(cgImage: cgImage!)
}
So as I rotate the image every time by the push of a button, the image is rotated by angle pi/10 each time. But I see the image shrinking in UIImageView. The NSLogs show the extent is growing with some rotations with the origin x and y becoming negative.
2021-09-24 14:43:29.280393+0400 CoreImagePrototypes[65817:5175194] Metal API Validation Enabled
2021-09-24 14:43:31.094877+0400 CoreImagePrototypes[65817:5175194] Extent (-105.0, -105.0, 1010.0, 1010.0), Angle 0.3141592653589793
2021-09-24 14:43:41.426371+0400 CoreImagePrototypes[65817:5175194] Extent (-159.0, -159.0, 1118.0, 1118.0), Angle 0.6283185307179586
2021-09-24 14:43:42.244703+0400 CoreImagePrototypes[65817:5175194] Extent (-159.0, -159.0, 1118.0, 1118.0), Angle 0.9424777960769379
2021-09-24 14:43:42.644446+0400 CoreImagePrototypes[65817:5175194] Extent (-105.0, -105.0, 1010.0, 1010.0), Angle 1.2566370614359172
2021-09-24 14:43:43.037312+0400 CoreImagePrototypes[65817:5175194] Extent (0.0, 0.0, 800.0, 800.0), Angle 1.5707963267948966
2021-09-24 14:43:43.478774+0400 CoreImagePrototypes[65817:5175194] Extent (-105.0, -105.0, 1010.0, 1010.0), Angle 1.8849555921538759
2021-09-24 14:43:44.045811+0400 CoreImagePrototypes[65817:5175194] Extent (-159.0, -159.0, 1118.0, 1118.0), Angle 2.199114857512855
My questions:
How exactly do I determine scale factor to rescale the image so that the extent does not cross the original image rectangle?
What exactly does negative extent origin means? Relative to what it is negative? I understand coordinate system in Core Image is relative assuming bottom left corner of image to be (0,0), not with respect to some superview like in UIKit.
It's unclear what the question is, but what you seem to be focussed on is the meaning of the extent. This is like the frame, and, just like the frame, it loses its meaning if you have applied a transform to the CIImage. After a rotation, the extent is now based on the bounding box of the transformed image. So if you have a horizontally wider image and you rotate it a little bit counterclockwise, the extent becomes taller and its top becomes negative.
Related
I have an app where users can add an image to a map. This is pretty straightforward. It becomes much more difficult when I want to add rotation (taken from the current map heading). The code I use to create an image is pretty straightforward:
let imageAspectRatio = Double(image.size.height / image.size.width)
let mapAspectRatio = Double(visibleMapRect.size.height / visibleMapRect.size.width)
var mapRect = visibleMapRect
if mapAspectRatio > imageAspectRatio {
// Aspect ratio of map is bigger than aspect ratio of image (map is higher than the image), take away height from the rectangle
let heightChange = mapRect.size.height - mapRect.size.width * imageAspectRatio
mapRect.size.height = mapRect.size.width * imageAspectRatio
mapRect.origin.y += heightChange / 2
} else {
// Aspect ratio of map is smaller than aspect ratio of image (map is higher than the image), take away width from the rectangle
let widthChange = mapRect.size.width - mapRect.size.height / imageAspectRatio
mapRect.size.width = mapRect.size.height / imageAspectRatio
mapRect.origin.x += widthChange / 2
}
photos.append(ImageOverlay(image: image, boundingMapRect: mapRect, rotation: cameraHeading))
The ImageOverlay class inherits from MKOverlay, which I can easily draw on the map. Here's the code for that class:
class ImageOverlay: NSObject, MKOverlay {
let image: UIImage
let boundingMapRect: MKMapRect
let coordinate: CLLocationCoordinate2D
let rotation: CLLocationDirection
init(image: UIImage, boundingMapRect: MKMapRect, rotation: CLLocationDirection) {
self.image = UIImage.fixedOrientation(for: image) ?? image
self.boundingMapRect = boundingMapRect
self.coordinate = CLLocationCoordinate2D(latitude: boundingMapRect.midX, longitude: boundingMapRect.midY)
self.rotation = rotation
}
}
I have figured out by how much I need to scale and translate the context to fit on the correct location on the map (from which it was added). I can't figure out how to rotate the context to make the image render in the correct location.
I figured out that the map rotation was in degrees, and the rotate method takes radians (took longer than I dare to admit), but the image moves around when I apply the rotation.
I use the following code to render the overlay:
override func draw(_ mapRect: MKMapRect, zoomScale: MKZoomScale, in context: CGContext) {
guard let overlay = overlay as? ImageOverlay else {
return
}
let rect = self.rect(for: overlay.boundingMapRect)
context.scaleBy(x: 1.0, y: -1.0)
context.translateBy(x: 0.0, y: -rect.size.height)
context.rotate(by: CGFloat(overlay.rotation * Double.pi / 180))
context.draw(overlay.image.cgImage!, in: rect)
}
How do I need to rotate this context to get the image to be aligned properly?
This is an open source project with the code here
Edit: I have tried (and failed) to use some kind of trig function. If I scale by a factor of 3 * sin(rotation) / 4 (no clue where the 3/4 comes from), I get a correct scale for some rotations, but not for others.
It sounds like you trying to rotate the object in it's local coordinates, but are actually rotating in the world coordinates. I admit I'm not familiar with this library, but the moral of the story is that order of operation on transformations matter. But it looks like you have "TranslateBy" and you are sending in zero, which might not be moving it at all? If you are trying to translate back to local you'd need to translate to local by subtracting it's current coordinates in the CLLocationCoordinate2D struct.
Translate to local coordinates if not already there (which might be X:0, y:0, you might need to subtract the current coordinate values instead of trying to set them to a specific number like 0)
Apply rotation
Translate back to world coordinates (where you had it before, originally defined as CLLocationCoordinate2D)
This should allow the image to be in the correct position but now rotated to align with the heading.
Here is a paper which explains what you are probably encountering, although more in depth and specific to the matrix/opengl, but the first slide illustrates your issue.
Transforms PDF
I'm taking images with AVCapturePhotoOutput and then using their JPEG representation as the texture on a SceneKit SCNPlane that is the same aspect ratio as the image:
let image = UIImage(data: dataImage!)
let rectangle = SCNPlane(width:9, height:12)
let rectmaterial = SCNMaterial()
rectmaterial.diffuse.contents = image
rectmaterial.isDoubleSided = true
rectangle.materials = [rectmaterial]
let rectnode = SCNNode(geometry: rectangle)
let pos = sceneSpacePosition(inFrontOf: self.pictCamera, atDistance: 16.5) // 16.5 is arbitrary, but makes the rectangle the same size as the camera
rectnode.position = pos
rectnode.orientation = self.pictCamera.orientation
pictView.scene?.rootNode.addChildNode(rectnode)
sceneSpacePosition is a bit of code that can be found here on SO that maps CoreMotion into SceneKit orientation. It is used to place the rectangle, which does indeed appear at the right location with the right size. All very cool.
The problem is that the image is rotated 90 degrees to the rectangle. So I did the obvious:
rectmaterial.diffuse.contentsTransform = SCNMatrix4MakeRotation(Float.pi / 2, 0, 0, 1)
This does not work property; the resulting image is unrecognizable. It appears that one small part of the image has been stretched to a huge size. I thought it might be the axis, but I tried all three with the same result.
Any ideas?
You are rotating on the upper left corner as suggested by Alain T.
If you move your image down, you may get the rotation you were expecting.
Try this:
let translation = SCNMatrix4MakeTranslation(0, -1, 0)
let rotation = SCNMatrix4MakeRotation(Float.pi / 2, 0, 0, 1)
let transform = SCNMatrix4Mult(translation, rotation)
rectmaterial.diffuse.contentsTransform = transform
I have a class that takes an UIImage, initializes a CIImage with it like so:
workingImage = CIImage.init(image: baseImage!)
Then the image is used to cut out 9 neighbouring squares in a 3x3 pattern out of it - in a loop:
for x in 0..<3
{
for y in 0..<3
{
croppingRect = CGRect(x: CGFloat(Double(x) * sideLength + startPointX),
y: CGFloat(Double(y) * sideLength + startPointY),
width: CGFloat(sideLength),
height: CGFloat(sideLength))
let tmpImg = (workingImage?.cropping(to: croppingRect))!
}
}
Those tmpImgs are inserted into a table and later used, but thats besides the point.
This code works on IOS 9, and on IOS 10 simulators, but not on an actual IOS 10 device. The images produced are either all empty, or one of them is like a half of what its supposed to be, with the rest being, again, empty.
Is this not how its supposed to be done in IOS 10?
The heart of the matter is that passing through CIImage is not the way to crop a UIImage. For one thing, coming back from CIImage to UIImage is a complicated business. For another, the whole round-trip is unnecessary.
How To Crop
To crop an image, make an image graphics context of the desired cropped size and call draw(at:) on the UIImage to draw it at the desired point relative to the graphics context, so that the desired portion of the image falls into the context. Now extract the resulting new image and close the context.
To demonstrate, I'll crop to one of the thirds you are trying to crop to, namely the lower right third:
let sz = baseImage.size
UIGraphicsBeginImageContextWithOptions(
CGSize(width:sz.width/3.0, height:sz.height/3.0),
false, 0)
baseImage.draw(at:CGPoint(x: -sz.width/3.0*2.0, y: -sz.height/3.0*2.0))
let tmpImg = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
Original image (baseImage):
Cropped image (tmpImg):
The other sections are completely parallel.
Core Image's coordinate system mismatches with UIKit, so the rect needs to be mirrored.
So in your specific case, you want:
var ciRect = croppingRect
ciRect.origin.y = workingImage!.extent.height - ciRect.origin.y - ciRect.height
let tmpImg = workingImage!.cropped(to: ciRect)
This definitely works for iOS 10+.
In a more general case, we would make a UIImage extension that covers both possible coordinate systems, and that's way faster than draw(at:):
extension UIImage {
/// Return a new image cropped to a rectangle.
/// - parameter rect:
/// The rectangle to crop.
open func cropped(to rect: CGRect) -> UIImage {
// a UIImage is either initialized using a CGImage, a CIImage, or nothing
if let cgImage = self.cgImage {
// CGImage.cropping(to:) is magnitudes faster than UIImage.draw(at:)
if let cgCroppedImage = cgImage.cropping(to: rect) {
return UIImage(cgImage: cgCroppedImage)
} else {
return UIImage()
}
}
if let ciImage = self.ciImage {
// Core Image's coordinate system mismatch with UIKit, so rect needs to be mirrored.
var ciRect = rect
ciRect.origin.y = ciImage.extent.height - ciRect.origin.y - ciRect.height
let ciCroppedImage = ciImage.cropped(to: ciRect)
return UIImage(ciImage: ciCroppedImage)
}
return self
}
}
I've made a pod for it, so the source code is at https://github.com/Coeur/ImageEffects/blob/master/SwiftImageEffects/ImageEffects%2Bextensions.swift
Given a CGRect, I want to use GPUImage to crop a video. For example, if the rect is (0, 0, 50, 50), the video would be cropped at (0,0) with a length of 50 on each side.
What's throwing me is that GPUImageCropFilter doesn't take a rectangle, rather a normalized crop region with values ranging from 0 to 1. My intuition was to to this:
let assetSize = CGSizeApplyAffineTransform(videoTrack.naturalSize, videoTrack.preferredTransform)
let cropRect = CGRect(x: frame.minX/assetSize.width,
y: frame.minY/assetSize.height,
width: frame.width/assetSize.width,
height: frame.height/assetSize.height)
to calculate the crop region based on the size of the incoming asset. Then:
// Filter
let cropFilter = GPUImageCropFilter(cropRegion: cropRect)
let url = NSURL(fileURLWithPath: "\(NSTemporaryDirectory())\(String.random()).mp4")
let movieWriter = GPUImageMovieWriter(movieURL: url, size: assetSize)
movieWriter.encodingLiveVideo = false
movieWriter.shouldPassthroughAudio = false
// add targets
movieFile.addTarget(cropFilter)
cropFilter.addTarget(movieWriter)
cropFilter.forceProcessingAtSize(frame.size)
cropFilter.setInputRotation(kGPUImageRotateRight, atIndex: 0)
What should the movie writer size be? Shouldn't it be the size of the frame I want to crop with? And should I be using forceProcessingAtSize with the size value of my crop frame?
A complete code example would be great; I've been trying for hours and I can't seem to get the section of the video that I want.
FINAL:
if let videoTrack = self.asset.tracks.first {
let movieFile = GPUImageMovie(asset: self.asset)
let transformedRegion = CGRectApplyAffineTransform(region, videoTrack.preferredTransform)
// Filters
let cropFilter = GPUImageCropFilter(cropRegion: transformedRegion)
let url = NSURL(fileURLWithPath: "\(NSTemporaryDirectory())\(String.random()).mp4")
let renderSize = CGSizeApplyAffineTransform(videoTrack.naturalSize, CGAffineTransformMakeScale(transformedRegion.width, transformedRegion.height))
let movieWriter = GPUImageMovieWriter(movieURL: url, size: renderSize)
movieWriter.transform = videoTrack.preferredTransform
movieWriter.encodingLiveVideo = false
movieWriter.shouldPassthroughAudio = false
// add targets
// http://stackoverflow.com/questions/37041231/gpuimage-crop-to-cgrect-and-rotate
movieFile.addTarget(cropFilter)
cropFilter.addTarget(movieWriter)
movieWriter.completionBlock = {
observer.sendNext(url)
observer.sendCompleted()
}
movieWriter.failureBlock = { _ in
observer.sendFailed(.VideoCropFailed)
}
disposable.addDisposable {
cropFilter.removeTarget(movieWriter)
movieWriter.finishRecording()
}
movieWriter.startRecording()
movieFile.startProcessing()
}
As you note, the GPUImageCropFilter takes in a rectangle in normalized coordinates. You're on the right track, in that you just need to convert your CGRect in pixels to normalized coordinates by dividing the X components (origin.x and size.width) by the width of the image and the Y components by the height.
You don't need to use forceProcessingAtSize(), because the crop will automatically output an image of the appropriate cropped size. The movie writer's size should be matched to this cropped size, which you should know from your original CGRect.
The one complication you introduce is the rotation. If you need to apply a rotation in addition to your crop, you might want to check and make sure that you don't need to swap your X and Y for your crop region. This should be apparent in the output if the two need to be swapped.
There were some bugs with applying rotation at the same time as a crop a while ago, and I can't remember if I fixed all those. If I didn't, you could insert a dummy filter (gamma or brightness set to default values) before or after the crop and apply the rotation at that stage.
I'm using the CIPerspectiveCorrection Filter and my problem is that my returned image results are mirrored, upside down, and the points used for the perspective correction seem to be referencing the wrong axis, or axis direction.
In order to isolate the issue I have been working with a test image that is 1024 x 1024 and I am passing in a perfectly rectangular area. I'm still ending up with images flipped vertically and horizontally.
Here is my function that returns a cropped CIImage instance given an image and set of points:
private func _getCroppedImageWithImage(image:CIImage, topLeft:CGPoint, topRight:CGPoint, botLeft:CGPoint, botRight:CGPoint) -> CIImage {
var rectCoords = NSMutableDictionary(capacity: 4)
rectCoords["inputTopLeft"] = CIVector(CGPoint:topLeft)
rectCoords["inputTopRight"] = CIVector(CGPoint:topRight)
rectCoords["inputBottomLeft"] = CIVector(CGPoint:botLeft)
rectCoords["inputBottomRight"] = CIVector(CGPoint:botRight)
return image.imageByApplyingFilter("CIPerspectiveCorrection", withInputParameters: rectCoords)
}
And here is where I am calling this function:
func testCrop() {
let ciInputImage = CIImage(image:UIImage(named:"test-pattern.jpg")!)
println("source image is \(ciInputImage)") //<CIImage: 0x170212290 extent [0 0 1024 1024]>
let ptBotLeft = CGPointMake(32.0,992.0)
let ptBotRight = CGPointMake(992.0,992.0)
let ptTopRight = CGPointMake(992.0,32.0)
let ptTopLeft = CGPointMake(32.0,32.0)
let croppedImage = _getCroppedImageWithImage(ciInputImage, topLeft: ptTopLeft, topRight: ptTopRight, botLeft: ptBotLeft, botRight: ptBotRight)
println("cropped image \(croppedImage)") //<CIImage: 0x174204a60 extent [0 0 960 960]>
let croppedImageCG = CIContext(options: nil).createCGImage(croppedImage, fromRect: croppedImage.extent())
let imageVC = ImageViewController(image: UIImage(CGImage: croppedImageCG))
presentViewController(imageVC, animated: true, completion: nil)
}
Has anyone encountered problems like this before?
Here is the source image
And here is the final image displayed in a UIImageView with contentMode set to scaleAspectFit
OK, my issue, I am pretty sure, is that CoreImage uses the Cartesian coordinate system. Y is up. (zero, zero) is at the bottom left.