I rotated the image from the left to the image on the right using
image = cv2.warpAffine(image, R, dst_size, flags=cv2.INTER_LINEAR)
where dst_size is (547, 363), and I've modified R so that the original image should fit in the new dimensions.
I show the image with cv2.imshow('rect post-rotation', img) to show the image on the right. You can see that it appears clipped to a size of (363, 547), even though when I print out the img shape, I get (547, 363, 3).
Why does the shape of the image not reflect the shape of the displayed image?
you need to set the destination size to the rotated size ie:
cv::Size dst_size(imOrig.size().height,imOrig.size().width);
alternatively you can use transpose and flip for 90 degree rotation:
cv::Mat imRot90 = imOrig.t();
cv::flip(imRot90 ,imRot90 ,1);
cheers
Related
I'm using the iOS Vision framework to detect rectangles in real-time with the camera on an iPhone and it works well. The live preview displays a moving yellow rectangle around the detected shape.
However, when the same code is run on an iPad, the yellow rectangle tracks accurately along the X axis, but on the Y it is always slightly offset from the centre and it is not correctly scaled. The included image shows both devices tracking the same test square to better illustrate. In both cases, after I capture the image and plot the rectangle on the full camera frame (1920 x 1080), everything looks fine. It's just the live preview on the iPad that does not track properly.
I believe the issue is caused by how the iPad screen has a 4:3 aspect ratio. The iPhone's full screen preview scales its 1920 x 1080 raw frame down to 414 x 718, where both X and Y dims are scaled down by the same factor (about 2.6). However, the iPad scales the 1920 x 1080 frame down to 810 x 964, which warps the image and causes the error along the Y axis.
A rough solution could be to set a preview layer size smaller than the full screen and have it be scaled down uniformly in a 16:9 ratio matching 1920 x 1080, but I would prefer to use the full screen. Has anyone here come across this issue and found a transform that can properly translate and scale the rect observation onto the iPad screen?
Example test images and code snippet are below.
let rect: VNRectangleObservation
//Camera preview (live) image dimensions
let previewWidth = self.previewLayer!.bounds.width
let previewHeight = self.previewLayer!.bounds.height
//Dimensions of raw captured frames from the camera (1920 x 1080)
let frameWidth = self.frame!.width
let frameHeight = self.frame!.height
//Transform to change detected rectangle from Vision framework's coordinate system to SwiftUI
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -(previewHeight))
let scale = CGAffineTransform.identity.scaledBy(x: previewWidth, y: previewHeight)
//Convert the detected rectangle from normalized [0, 1] coordinates with bottom left origin to SwiftUI top left origin
//and scale the normalized rect to preview window dimensions.
var bounds: CGRect = rect.boundingBox.applying(scale).applying(transform)
//Rest of code draws the bounds CGRect in yellow onto the preview window, as shown in the image.
In case it helps anyone else, based on the info posted by Mr.SwiftOak's comment, I was able to resolve the problem through a combination of changing the preview layer to scale as .resizeAspect, rather than .resizeAspectFill, preserving the ratio of the raw frame in the preview. This led to the preview no longer taking up the full iPad screen, but made it a lot simpler to overlay accurately.
I then drew the rectangles as a .overlay to the preview window, so that the drawing coords are relative to the origin of the image (top left) rather than the view itself, which has an origin at (0, 0) top left of the entire screen.
To clarify on how I've been drawing the rects, there are two parts:
Converting the detect rect bounding boxes into paths on CAShapeLayers:
let boxPath = CGPath(rect: bounds, transform: nil)
let boxShapeLayer = CAShapeLayer()
boxShapeLayer.path = boxPath
boxShapeLayer.fillColor = UIColor.clear.cgColor
boxShapeLayer.strokeColor = UIColor.yellow.cgColor
boxLayers.append(boxShapeLayer)
Appending the layers in the updateUIView of the preview UIRpresentable:
func updateUIView(_ uiView: VideoPreviewView, context: Context)
{
if let rectangles = self.viewModel.rectangleDrawings {
for rect in rectangles {
uiView.videoPreviewLayer.addSublayer(rect)
}
}
}
This question already has answers here:
How to crop a UIImageView to a new UIImage in 'aspect fill' mode?
(2 answers)
Closed 4 years ago.
The problem I am facing is that the image taken from the camera is larger then the one shown in the live view. I have the camera view setup as Aspect Fill.
So the image that I get from the camera is about 4000x3000 and the view that shows the live feed from the camera is 375x800 (fullscreen iPhoneX size) so how do I transform/cut out part of the image from the image gotten from the camera to be the same as the one shown in the live view, so I can further manipulate the image (draw over it).
As far as I understand the Aspect Fill property clips the image that cannon't be shown in the view. But that clip does not happen on X = 0 and y = 0 it happens somewhere in the middle of the image. So how do i get that X and Y on the original image so that i can crop out exactly that part out.
I hope I explained well enough.
EDIT:
To give more context and some code snipets to make it easier to understand the issue.
Setting up my camera with the .resizeAspectFill gravity.
cameraPreviewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
cameraPreviewLayer?.videoGravity = AVLayerVideoGravity.resizeAspectFill
cameraPreviewLayer?.connection?.videoOrientation = AVCaptureVideoOrientation.portrait
cameraPreviewLayer?.frame = self.captureView.frame
self.captureView.layer.addSublayer(cameraPreviewLayer!)
which is displayed in the live view (captureView) that has the size of
375x818 (width: 375 and height: 818).
Then I get the image from that camera on button click and the size of that image is:
3024x4032 (width: 3024 and height: 4032)
So what i want to do is crop the image from the camera to be the same as the one in the live view (captureView) that is set to AspectFill type.
As you already state, content mode option Aspect fill tries to fill up the live view and you are also right that it crops some rectangle from center (cropping top-bottom or left-right depending upon the image size and the image view size)
For generic solution there are two possible case
The image needed to be cropped along the height to fit the image view (proportional drawing height is smaller)
The image needed to be cropped along the width to fit the image view (proportional drawing width is smaller)
Considering your size notation is 4000x3000 (height = 4000, width = 3000 a portrait image) and your drawing canvas size is 375X800 (height = 375, width = 800), then your cropping would be height wise while setting the content mode Aspect Fill.
So cropping will be done from X=0 but the Y would be somewhat positive. So lets calculate the Y
let propotionalHeight = 4000 / 3000 * 800
let allowedHeight = 375
let topBottomCroppedHeight = proportionalHeight - allowedHeight
let croppedYPosition = topBottomCroppedHeight / 2
So here you got your Y value. and the height would be the height of the canvas / live view where you are rendering. Please replace these values with your variables.
If you are interested in how all the contentMode works can dive into here. All the contentMode supported by UIImageView is simulated here.
Happy coding.
UPDATE
one thing i forgot to mention that, this calculated croppedYPosition is for smaller proportion image. If you want to use this value for the original 4000X3000 image you have to scale this up for the original value as following
let originalYPosition = (croppedYPosition / 375) * 4000
Use originalYPosition to crop from the original image of size 4000X3000.
I am doing some operation where I need to replace the bigger rectangle with the smaller rectangle.
Most answers suggested to use smallerRectMat.copyTo(biggerRectMat) but it didn't give me the require output. The submat is changed but the original image is as it is.
And when I try to see the submat both became same of same smaller rectangle size.
Mat rectNose = testBuffer.submat(rectA.y,rectA.y+rectA.height,rectA.x,rectC.x+rectC.width);
Rect biggerRect = getHeadContour(testBuffer);
Mat rectHead = testBuffer.submat(biggerRect.y+1,biggerRect.y+biggerRect.height,biggerRect.x+1,biggerRect.x+biggerRect.width);
rectNose.copyTo(rectHead);
Imgcodecs.imwrite("/Users/test.jpg",rectHead);
Imgcodecs.imwrite("/Users/test1.jpg",rectNose);
Imgcodecs.imwrite("/Users/test1.jpg",testBuffer);
Basically I want to copy the rectangle near the nose region to the rectangle with blue boundary at forehead.
You can try ROI(Region Of Image) scaling
smallRect = img[rectA.y:rectA.y+rectA.height, rectA.x:rectC.x+rectC.width]
upscaledRegion = cv2.resize(smallRect , (biggerRect.width, biggerRect.height), interpolation=cv2.INTER_AREA)
img[biggerRect.y:biggerRect.y+biggerRect.height, biggerRect.x:biggerRect.x+biggerRect.width] = upscaledRegion
I use an image view:
#IBOutlet weak var imageView: UIImageView!
to paint an image and also another image which has been rotated. It turns out that the rotated image has very bad quality. In the following image the glasses in the yellow box are not rotated. The glasses in the red box are rotated by 4.39 degrees.
Here is the code I use to draw the glasses:
UIGraphicsBeginImageContext(imageView.image!.size)
imageView.image!.drawInRect(CGRectMake(0, 0, imageView.image!.size.width, imageView.image!.size.height))
var drawCtxt = UIGraphicsGetCurrentContext()
var glassImage = UIImage(named: "glasses.png")
let yellowRect = CGRect(...)
CGContextSetStrokeColorWithColor(drawCtxt, UIColor.yellowColor().CGColor)
CGContextStrokeRect(drawCtxt, yellowRect)
CGContextDrawImage(drawCtxt, yellowRect, glassImage!.CGImage)
// paint the rotated glasses in the red square
CGContextSaveGState(drawCtxt)
CGContextTranslateCTM(drawCtxt, centerX, centerY)
CGContextRotateCTM(drawCtxt, 4.398 * CGFloat(M_PI) / 180)
var newRect = yellowRect
newRect.origin.x = -newRect.size.width / 2
newRect.origin.y = -newRect.size.height / 2
CGContextAddRect(drawCtxt, newRect)
CGContextSetStrokeColorWithColor(drawCtxt, UIColor.redColor().CGColor)
CGContextSetLineWidth(drawCtxt, 1)
// draw the red rect
CGContextStrokeRect(drawCtxt, newRect)
// draw the image
CGContextDrawImage(drawCtxt, newRect, glassImage!.CGImage)
CGContextRestoreGState(drawCtxt)
How can I rotate and paint the glasses without losing quality or get a distorted image?
You should use UIGraphicsBeginImageContextWithOptions(CGSize size, BOOL opaque, CGFloat scale) to create the initial context. Passing in 0.0 as the scale will default to the scale of the current screen (e.g., 2.0 on an iPhone 6 and 3.0 on an iPhone 6 Plus).
See this note on UIGraphicsBeginImageContext():
This function is equivalent to calling the UIGraphicsBeginImageContextWithOptions function with the opaque parameter set to NO and a scale factor of 1.0.
As others have pointed out, you need to set up your context to allow for retina displays.
Aside from that, you might want to use a source image that is larger than the target display size and scale it down. (2X the pixel dimensions of the target image would be a good place to start.)
Rotating to odd angles is destructive. The graphics engine has to map a grid of source pixels onto a different grid where they don't line up. Perfectly straight lines in the source image are no longer straight in the destination image, etc. The graphics engine has to do some interpolation, and a source pixel might be spread over several pixels, or less than a full pixel, in the destination image.
By providing a larger source image you give the graphics engine more information to work with. It can better slice and dice those source pixels into the destination grid of pixels.
I have an image I want to partially mask (wallSprite), an image to act as a mask over it (wallMaskBox), and a node to hold both (wallCropNode). When I simply add both images as children of wallCropNode, both image display correctly:
var wallSprite = SKSpriteNode(imageNamed: "wall.png")
var wallCropNode = SKCropNode()
var wallMaskBox = SKSpriteNode(imageNamed: "blacksquaretiny.png")
wallMaskBox.zPosition = 100
wallCropNode.addChild(wallSprite)
wallCropNode.addChild(wallMaskBox)
gameplayContainerNode.addChild(wallCropNode)
But when I set the mask image as a maskNode property of the crop node:
var wallSprite = SKSpriteNode(imageNamed: "wall.png")
var wallCropNode = SKCropNode()
var wallMaskBox = SKSpriteNode(imageNamed: "blacksquaretiny.png")
wallMaskBox.zPosition = 100
wallCropNode.addChild(wallSprite)
wallCropNode.maskNode = wallMaskBox
gameplayContainerNode.addChild(wallCropNode)
the wallSprite image disappears entirely, instead of being partly cropped. Any ideas?
The issue is your black square image is completely opaque. Some (or all) of its pixels should be transparent (i.e., alpha = 0). The pixels that correspond to the mask node's transparent pixels will be masked out (i.e., not rendered) in the cropped node. To demonstrate this, I used your code to create the following.
Here's the original image:
Here's the mask image that I used for the maskNode. Note that the white regions are transparent (i.e., alpha = 0). From Apple's documentation,
When rendering its children, each pixel is verified against the
corresponding pixel in the mask. If the pixel in the mask has an alpha
value of less than 0.05, the image pixel is masked out. Any pixel not
rendered by the mask node is automatically masked out.
and here's the cropped node. I took a screenshot of the scene from the iPhone 6 simulator.