AVCaptureVideoPreviewLayer converts capture device coordinates incorrectly - ios

Question

I'm working on implementing a camera barcode scanner with Vision framework. My setup seems fairly simple: a preview layer for video from the device's rear camera, frames from the video stream are processed using VNDetectBarcodesRequest, and if a barcode is detected, I would like to show a rectangle around it in the preview.
Vision framework coordinates are normalized to the video frame size, so I have to convert them back to the layer coordinates to be able to draw the rectangle. The recommended way to do this is to use AVCaptureVideoPreviewLayer.layerPointConverted(fromCaptureDevicePoint:). There is also a difference between the point of origin (it's in the lower left corner for Vision framework), so I have to take care of that as well. Here's what the result looks like:
let convertPoint = { (point: CGPoint) in
let coordinateFlip = CGAffineTransform.identity
.translatedBy(x: 0, y: 1)
.scaledBy(x: 1, y: -1)
return videoLayer.layerPointConverted(fromCaptureDevicePoint: point.applying(coordinateFlip))
}
The problem is that this conversion is incorrect: the rectangle ends up being rotated 90º clockwise:
I have no idea why, I set video orientation to .portrait on both AVCaptureVideoPreviewLayer.connection and AVCaptureVideoDataOutput.connection(with: .video). The Vision coordinates seem to be correct, if I convert them manually like so:
extension VNBarcodeObservation {
func verticesConvertedToCoordinatesOf(_ targetLayer: CALayer) -> [CGPoint] {
let layerSize = targetLayer.bounds.size
return [topLeft, topRight, bottomRight, bottomLeft]
.map { CGPoint(x: $0.x * layerSize.width, y: $0.y * layerSize.height) }
}
}
the result is correct:
Why is AVCaptureVideoPreviewLayer.layerPointConverted(fromCaptureDevicePoint:) returning incorrect results?
It seems like somehow AVCaptureVideoPreviewLayer is stuck in the .landscapeRight orientation.
Here's how I set up video capture:
let captureSessionQueue = DispatchQueue.global(qos: .userInteractive)
let camera = AVCaptureDevice.default(for: .video)
let session = AVCaptureSession()
session.beginConfiguration()
session.setCamera(camera, preferredPresets: [.hd1280x720])
let request = VNDetectBarcodesRequest(symbologies: VNDetectBarcodesRequest.supportedSymbologies)
request.regionOfInterest = CGRect(x: 0.15, y: 0.45, width: 0.7, height: 0.4)
let delegate = CaptureVideoDataOutputDelegate()
delegate.vnRequests = [request]
let deviceOutput = AVCaptureVideoDataOutput()
deviceOutput.setSampleBufferDelegate(delegate, queue: captureSessionQueue)
deviceOutput.alwaysDiscardsLateVideoFrames = true
session.addOutput(deviceOutput)
deviceOutput.connection(with: .video)?.videoOrientation = .portrait
session.commitConfiguration()
captureSessionQueue.async {
session.startRunning()
}
and CaptureVideoDataOutputDelegate:
public class CaptureVideoDataOutputDelegate: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
public var vnRequests: [VNRequest]?
private let vnHandler = VNSequenceRequestHandler()
private var frameCounter: Int64 = 0
public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
defer { frameCounter += 1 }
guard
// Try to reduce CPU usage and energy consumption by processing only every 10th frame
frameCounter.isMultiple(of: 10),
let vnRequests = vnRequests,
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else { return }
do {
let size = CGSize(width: CVPixelBufferGetWidth(pixelBuffer), height: CVPixelBufferGetHeight(pixelBuffer))
print(">>> will scan a \(size) frame #\(frameCounter) for barcodes (connection: \(connection), orientation: \(connection.videoOrientation))")
try vnHandler.perform(vnRequests, on: pixelBuffer, orientation: .up)
} catch {
print(">>> failed to scan frame #\(frameCounter) for barcodes: \(error)")
}
}
}
I have verified that both deviceOutput.connection(with: .video) and videoLayer.connection are not nil when I try to configure video orientation. Examining the size of buffer received by CaptureVideoDataOutputDelegate confirms that orientation is correct (the size is 720x1280, not 1280x720). Orientation of the connection is also .portrait at this point.