Convert VNRectangleObservation points to other coordinate system - ios

Question

I need to convert the VNRectangleObservation received CGPoints (bottomLeft,
bottomRight, topLeft, topRight) to another coordinate system (e.g. a view's coordinate on screen).
I define a request:
// Rectangle Request
let rectangleDetectionRequest = VNDetectRectanglesRequest(completionHandler: handleRectangles)
rectangleDetectionRequest.minimumSize = 0.5
rectangleDetectionRequest.maximumObservations = 1
I get the sampleBuffer from camera in delegate call, and perform a detection request:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {return}
var requestOptions:[VNImageOption:Any] = [:]
if let cameraIntrinsicData = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {
requestOptions = [.cameraIntrinsics:cameraIntrinsicData]
}
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: CGImagePropertyOrientation(rawValue:6)!, options: requestOptions)
do {
try imageRequestHandler.perform(self.requests)
} catch {
print(error)
}
}
Later in completionHandler I receive the results:
func handleRectangles (request:VNRequest, error:Error?) {
guard let results = request.results as? [VNRectangleObservation] else { return }
let flipTransform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -self.previewView.frame.height)
let scaleTransform = CGAffineTransform.identity.scaledBy(x: self.previewView.frame.width, y: self.previewView.frame.height)
for rectangle in results {
let rectangleBounds = rectangle.boundingBox.applying(scaleTransform).applying(flipTransform)
// convertedTopLeft = conversion(rectangle.topLeft)
// convertedTopRight = conversion(rectangle.topRight)
// convertedBottomLeft = conversion(rectangle.bottomLeft)
// convertedBottomRight = conversion(rectangle.bottomRight)
}
}
This works for boundingBox which is CGRect, but I need to transform the CGPoints instead to a coordinate system of another view.
The problem is that I don't know how to get the transformation from the sampleBuffer: CMSampleBuffer's coordinate system to the previewView coordinate system.
Thanks!

Answer 1

That was simply a matter of applying the transform to the CGPoint itself where size is the CGSize of the destination view for which I need transpose the four points.
let transform = CGAffineTransform.identity
.scaledBy(x: 1, y: -1)
.translatedBy(x: 0, y: -size.height)
.scaledBy(x: size.width, y: size.height)
let convertedTopLeft = rectangle.topLeft.applying(transform)
let convertedTopRight = rectangle.topRight.applying(transform)
let convertedBottomLeft = rectangle.bottomLeft.applying(transform)
let convertedBottomRight = rectangle.bottomRight.applying(transform)

Answer 2

#mihaicris answer works, but only in portrait mode. In landscape, we need to do it a little different.
if UIApplication.shared.statusBarOrientation.isLandscape {
transform = CGAffineTransform.identity
.scaledBy(x: -1, y: 1)
.translatedBy(x: -size.width, y: 0)
.scaledBy(x: size.width, y: size.height)
} else {
transform = CGAffineTransform.identity
.scaledBy(x: 1, y: -1)
.translatedBy(x: 0, y: -size.height)
.scaledBy(x: size.width, y: size.height)
}
let convertedTopLeft = rectangle.topLeft.applying(transform)
let convertedTopRight = rectangle.topRight.applying(transform)
let convertedBottomLeft = rectangle.bottomLeft.applying(transform)
let convertedBottomRight = rectangle.bottomRight.applying(transform)

Answer 3

I assume you use layer for the camera, and the layer is AVCaptureVideoPreviewLayer. (https://developer.apple.com/documentation/avfoundation/avcapturevideopreviewlayer).
So if you want to convert single point, use this function:layerPointConverted (https://developer.apple.com/documentation/avfoundation/avcapturevideopreviewlayer/1623502-layerpointconverted). Please notices that the y is inverted because of the VNRectangleObservation coordinates system.
let convertedTopLeft: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.topLeft.x, y: 1 - rectangle.topLeft.y))
let convertedTopRight: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.topRight.x, y: 1 - rectangle.topRight.y))
let convertedBottomLeft: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.bottomLeft.x, y: 1 - rectangle.bottomLeft.y))
let convertedBottomRight: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.bottomRight.x, y: 1 - rectangle.bottomRight.y))
Hope it helped