Cropping UIImage Using VNRectangleObservation - ios

Question

I'm using the Vision framework to detect rectangular documents in a captured photo. Detecting and drawing a path around the document is working perfectly. I then want to crop the image to be only the detected document. I'm successfully cropping the image, but it seems the coordinates don't line up and the cropped image is only part of the detected document and the rest is just the desk behind the document. I'm using the following cropping code:
private UIImage CropImage(UIImage image, CGRect rect, float scale)
{
var drawRect = new CGRect(rect.X, rect.Y, rect.Size.Width, rect.Size.Height);
using (var cgImage = image.CGImage.WithImageInRect(drawRect))
{
var croppedImage = UIImage.FromImage(cgImage);
return croppedImage;
};
}
Using the following parameters:
image is the same UIImage that i successfully drew the rectangle path on.
rect is the VNRectangleObservation.BoundingBox. This is normalized so i'm scaling it using the image.size. it's the same scaling i do when drawing the rectangle path.
scale is 1f, but i'm currently ignoring this.
The cropped image generally seems to be the right size, but it is shifted up and to the left which cuts off the lower and right side of the document. Any help would be appreciated.

Answer 1

for anyone else that finds this, the issue seemed to be CGImage rotating when cropping the image which caused the VNRectangleObservation to not line up anymore. I used this article, Tracking and Altering Images, to get a working solution using CIFilter. Cropping code follows:
var ciFilter = CIFilter.FromName("CIPerspectiveCorrection");
if (ciFilter == null) continue;
var width = inputImage.Extent.Width;
var height = inputImage.Extent.Height;
var topLeft = new CGPoint(observation.TopLeft.X * width, observation.TopLeft.Y * height);
var topRight = new CGPoint(observation.TopRight.X * width, observation.TopRight.Y * height);
var bottomLeft = new CGPoint(observation.BottomLeft.X * width, observation.BottomLeft.Y * height);
var bottomRight = new CGPoint(observation.BottomRight.X * width, observation.BottomRight.Y * height);
ciFilter.SetValueForKey(new CIVector(topLeft), new NSString("inputTopLeft"));
ciFilter.SetValueForKey(new CIVector(topRight), new NSString("inputTopRight"));
ciFilter.SetValueForKey(new CIVector(bottomLeft), new NSString("inputBottomLeft"));
ciFilter.SetValueForKey(new CIVector(bottomRight), new NSString("inputBottomRight"));
var ciImage = inputImage.CreateByApplyingOrientation(CGImagePropertyOrientation.Up);
ciFilter.SetValueForKey(ciImage, CIFilterInputKey.Image);
var outputImage = ciFilter.OutputImage;
var uiImage = new UIImage(outputImage);
imageList.Add(uiImage);
imageList is a List<UImage> since i'm handling multiple detected rectangles.
observation is a single observation of type VNRectangleObservation.

Answer 2

The cropped image generally seems to be the right size, but it is shifted up and to the left which cuts off the lower and right side of the document.
From Apple documentation CGImageCreateWithImageInRect , there is a discussion about the cropped size .
CGImageCreateWithImageInRect performs the following tasks to create the subimage:
It calls the CGRectIntegral function to adjust the rect parameter to integral bounds.
It intersects the rect with a rectangle whose origin is (0,0) and size is equal to the size of the image specified by the image parameter.
It reads the pixels within the resulting rectangle, treating the first pixel within as the origin of the subimage.
If W and H are the width and height of image, respectively, then the point (0,0) corresponds to the first pixel of the image data. The point (W–1, 0) is the last pixel of the first row of the image data, while (0, H–1) is the first pixel of the last row of the image data and (W–1, H–1) is the last pixel of the last row of the image data.
Then you can check in your local project with an image (size is : 1920 * 1080) as follow:
UIImageView imageView = new UIImageView(new CGRect(0, 400, UIScreen.MainScreen.Bounds.Size.Width, 300));
UIImage image = new UIImage("th.jpg");
imageView.Image = CropImage(image, new CGRect(0, 0, 1920, 1080), 1);
View.AddSubview(imageView);
The CropImage Method :
private UIImage CropImage(UIImage image, CGRect rect, float scale)
{
var drawRect = new CGRect(rect.X, rect.Y, rect.Size.Width, rect.Size.Height);
using (var cgImage = image.CGImage.WithImageInRect(drawRect))
{
if(null != cgImage)
{
var croppedImage = UIImage.FromImage(cgImage);
return croppedImage;
}
else
{
return image;
}
};
}
This will show the Original Size of Image :
Now you can modify the cropped size as follow :
UIImageView imageView = new UIImageView(new CGRect(0, 400, UIScreen.MainScreen.Bounds.Size.Width, 300));
UIImage image = new UIImage("th.jpg");
imageView.Image = CropImage(image, new CGRect(0, 0, 1920, 100), 1);
View.AddSubview(imageView);
Here I set x = 0 , y = 0 , that means from (0,0) to start , and width is 1920 ,height is 100 . I just crop the height of the original Image . The effect as follow :
Then if you modify the x/y ,the cropped image will move to other area to crop .As follow:
UIImageView imageView = new UIImageView(new CGRect(0, 400, UIScreen.MainScreen.Bounds.Size.Width, 300));
UIImage image = new UIImage("th.jpg");
imageView.Image = CropImage(image, new CGRect(0, 100, 1920, 100), 1);
View.AddSubview(imageView);
Then you will see it's different with the second effect :
So when cropping an image , you should understand the drawRect of image.CGImage.WithImageInRect(drawRect) clearly .
Note from doc :
Be sure to specify the subrectangle's coordinates relative to the original image's full size, even if the UIImageView shows only a scaled version.