So I am using AVCaptureSession to take pictures with front camera. I am also creating previewLayer from this session to display current image on screen.
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill
It all works like it should.
But now I have a problem because I need to implement a button which will flip / mirror (transform) this preview layer - so users have a choice to take normal selfie picture or take mirrored one.
I have already tried transforming previewLayer and it KINDA works. The problem is that if you rotate device, preview picture rotates in the other way since it is transformed. (in the default or any other camera app picture rotates with camera). Anyone has any idea how to achieve that?
Mirroring preview layer: (I tried transforming layer and even view later, same result).
#IBAction func mirrorCamera(_ sender: AnyObject) {
cameraMirrored = !cameraMirrored
if cameraMirrored {
// TRANSFORMING VIEW
self.videoPreviewView.transform = CGAffineTransform(scaleX: -1, y: 1);
// OR LAYER
self.previewLayer.transform = CATransform3DMakeScale(-1, 1, 1);
} else {
self.videoPreviewView.transform = CGAffineTransform(scaleX: 1, y: 1);
self.videoPreviewView.transform = CATransform3DMakeScale(1, 1, 1);
}
}
Nowadays, if you use the mirrored property of the preview layer directly, you will get a deprecation warning at runtime. The current way to do it is using directly the connection from the camera. You must do something like this (code below is not real code, property names probably will differ, but you get the idea)
if (cameraPreviewLayer.connection.SupportsVideoMirroring) {
cameraPreviewLayer.connection.automaticallyAdjustsVideoMirroring = false
cameraPreviewLayer.connection.videoMirrored = true
}
AVCaptureVideoPreviewLayer has a property mirrored. Set this true or false, as required.
Related
This is a long question so I wanted to put a TL;DR on top:
I want to track QR codes via on of two methods: image tracking by cropping them upon detection, or placing anchors with raycasting. Both of these methods fail when the phone is in portrait mode. Camera source is an ARSession, SceneKit and RealityKit not used. There's only ARKit. What to do?
I am currently working on an application with Swift in which I try to render some stuff on a server, transmit the video to iPhone and display it on screen using a MTKView. I only needed a custom Meal shader to apply some complex calculations to received frames, so I did not use SceneKit or RealityKit. I only have ARSession from ARKit and a Metal view here, and up to this point everything works fine.
I am able to do image tracking at this point. However, I want to apply this behaviour to QR codes. What I want is to detect a QR code (multiple if possible) and then track it just like images. Since I don't have the QR code as ARReferenceImages beforehand like normal image tracking, I was left with two options:
Option 1: Using raycast(_:) on ARSession
This is probably the right way to do it. However, for this I need to activate both plane tracking options on ARSession, which then creates many anchors and managing them with image tracking becomes harder. This is not the actual problem though. Actual problem is that when the phone is in landscape mode, raycasting works as intended. When phone goes into portrait mode, even if I pass the frame in correct orientation it misses everything and hit test results return empty. I am not using hitTest(_:) because it is deprecated.
I want to explain the "correct orientation" thing here before going into second option. ARSession is capturing frames and I am able to check each frame through didUpdate delegate function of the session. When I read the pixel buffer out of the frame using frame.capturedImage and turn it into a CIImage, the image is always in landscape mode (width > height). Doesn't matter if the phone is in portrait mode or not. So whenever I want to pass this image, I am using oriented(.right) for portrait and oriented(.up) for landscape. I got that idea from another question asked about QR bounding box, and so far it is the best option (but not good enough). Just want to note that when I tried raycasting, I tried it with the image size, not screen size (screen size = my Metal view size because it is fullscreen) since the image is larger than the screen in reality. I am able to see this if I put a breakpoint and quicklook my CIImage created from current camera frame.
Option 2: Cropping the QR and treating it as image tracking
This is another approach which I am currently working on. Algorithm is simple: check every frame with Vision. If there are detected QR codes, read their data first. If that data matches with an existing QR, then re-read it if the cropped QR size is larger than existing one. If not, do nothing. Then use this cropped QR image for tracking QR as an image. At this point we would have the data already so no problems here.
However, I tried many times to do the proper transformation explained here in the answer. Again, I think I am able to transform normalized bounding box into a real rect which can correctly crop the image. Yet, as it is in raycasting, works perfectly only if the phone is in landscape position. When in portrait it works good enough ONLY IF the phone is really close to QR code and it is centered on the screen.
For related code, I have this in my View controller:
private var ciContext: CIContext = CIContext.init(options: nil)
private var sequenceHandler: VNImageRequestHandler?
And then I have this code to extract QR codes from CIImage:
func extractQrCode(image: CIImage) -> [VNBarcodeObservation]? {
self.sequenceHandler = VNImageRequestHandler(ciImage: image)
let barcodeRequest = VNDetectBarcodesRequest()
barcodeRequest.symbologies = [.QR]
try? self.sequenceHandler?.perform([barcodeRequest])
guard let results = barcodeRequest.results else {
return nil
}
return results
}
An this is the delegate that checks and operates on every frame (code currently for Option 2):
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let rotImg = self.renderer?.getInterfaceOrientation() == .portrait ? CIImage(cvPixelBuffer: frame.capturedImage).oriented(.right) : CIImage(cvPixelBuffer: frame.capturedImage)
if let barcodes = self.extractQrCode(image: rotImg) {
for barcode in barcodes {
guard let payload = barcode.payloadStringValue else { continue }
var rect = CGRect()
rect = VNImageRectForNormalizedRect(barcode.boundingBox.botToTop(), Int(rotImg.extent.width), Int(rotImg.extent.height))
let existingQR = TrackedImagesManager.imagesToTrack.filter{ $0.isQR && $0.QRData == payload}.first
if ((rect.size.width < 800 || rect.size.height < 800 || abs(rect.size.height - rect.size.width) > 32) && existingQR == nil) {
DispatchQueue.main.async {
self.showToastMessage(message: "Please get closer to the QR code and try centering it on your screen.", font: UIFont.systemFont(ofSize: 18), duration: 3)
}
continue
} else if (existingQR != nil) {
if (rect.width > existingQR?.originalImage?.size.width ?? 999) {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
existingQR?.originalImage = trackImg
existingQR?.image = ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1)
} else {
continue
}
} else if rect.width != 0 {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
TrackedImagesManager.imagesToTrack.append(TrackedImage(id: 9, type: 1, image: ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1), originalImage: trackImg, isQR: true, QRData: payload))
print("qr norm rect: \(barcode.boundingBox) \n qr rect: \(rect) \nqr data: \(payload) \nqr hittestres: ")
}
}
}
}
Finally, for the transformation, I have this extension (tried various ways, this is the best so far):
extension CGRect {
func botToTop() -> CGRect {
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -1)
return self.applying(transform)
}
}
So for both options I need some advice to make things right. Android side of the same thing is implemented as in Option 2, but Android returns a nicely cropped QR code upon detection. We don't have that. What do I do now?
When I record videos with the front-facing "selfie" camera with my app that I am developing in iOS, and save it to my camera roll, the orientation appears correct. It also appears correct if share the video to my Mail app an open it in Mac OS. As soon as I share this video from my app, or from the camera roll to Facebook or Messenger, the orientation gets flipped upside down. It also gets flipped upside down if I send it to Messenger or post it on Facebook on MacOS.
What is it about this video that makes it appear correct orientation on my device or in a web browser, but never on the Facebook app, Messenger or Facebook website?
If there's any extra information I need to write to the video to make it appear correct, how is that done in Swift?
Even without knowing how to change it in Swift, it would really help to know what about the video is responsible.
UPDATE
Upon another look at EXIF data using exiftool, I found two vital differences.
The video that flips incorrectly taken from the front-facing camera when posting to Facebook or Messenger has the two following different properties:
Matrix Structure: -1 0 0 0 1 0 0 0 1
Rotation: 180
The video that doesn't flip (correct) taken from the back-facing camera when posting to Facebook or Messenger has the two following different properties:
Matrix Structure: 1 0 0 0 1 0 0 0 1
Rotation: 0
As of now, I don't know what my method will be of changing these properties, or to change one or both. Since it appears correct in places other than Facebook or Messenger, I'm guessing it's only one property.
UPDATE 2
I'm fairly certain it's fixed now. I am totally mystified by why I was transforming the AVAssetWriterInput. I'm already compensating for it flipping after the video is captured. I commented everything within the isFrontCamera if statement:
let assetWriterVideoInput = AVAssetWriterInput(mediaType: AVMediaType.video, outputSettings: videoSettings)
if isFrontCamera {
var transform: CGAffineTransform = .identity
if selectedInterfaceOrientation == interfaceOrientations.portrait {
transform = CGAffineTransform(scaleX: -1.0, y: 1.0)
}
else if selectedInterfaceOrientation == interfaceOrientations.landscapeLeft {
transform = CGAffineTransform(scaleX: 1.0, y: -1.0)
}
else if selectedInterfaceOrientation == interfaceOrientations.landscapeRight {
transform = CGAffineTransform(scaleX: 1.0, y: -1.0)
}
else if selectedInterfaceOrientation == interfaceOrientations.portraitUpsideDown {
transform = CGAffineTransform(scaleX: -1.0, y: 1.0)
}
assetWriterVideoInput.transform = transform
}
Is there a way to display camera images without using AVCaptureVideoPreviewLayer?
I want to do screen capture, but I can not do it.
session = AVCaptureSession()
camera = AVCaptureDevice.default(
AVCaptureDevice.DeviceType.builtInWideAngleCamera,
for: AVMediaType.video,
position: .front) // position: .front
do {
input = try AVCaptureDeviceInput(device: camera)
} catch let error as NSError {
print(error)
}
if(session.canAddInput(input)) {
session.addInput(input)
}
let previewLayer = AVCaptureVideoPreviewLayer(session: session)
cameraView.backgroundColor = UIColor.red
previewLayer.frame = cameraView.bounds
previewLayer.videoGravity = AVLayerVideoGravity.resizeAspect
cameraview.layer.addSublayer(previewLayer)
session.startRunning()
I am currently trying to broadcast a screen capture. It is to synthesize the camera image and some UIView. However, if you use AVCaptureVideoPreviewLayer screen capture can not be done and the camera image is not displayed. Therefore, I want to display the camera image so that screen capture can be performed.
Generally the views that are displayed using GPU directly may not be redrawn on the CPU. This includes situations like openGL content or these preview layers.
The "screen capture" redraws the screen on a new context on CPU which obviously ignores the GPU part.
You should try and play around with adding some outputs on the session which will give you images or rather CMSampleBuffer shots which may be used to generate the image.
There are plenty ways in doing this but you will most likely need to go a step lower. You can add output to your session to receive samples directly. Doing this is a bit of a code so please refer to some other posts like this one. The point in this you will have a didOutputSampleBuffer method which will feed you CMSampleBufferRef objects that may be used to construct pretty much anything in terms of images.
Now in your case I assume you will be aiming to get UIImage from sample buffer. To do so you may again need a bit of code so refer to some other post like this one.
To put it all together you could as well simply use an image view and drop the preview layer. As you get the sample buffer you can create image and update your image view. I am not sure what the performance of this would be but I discourage you on doing this. If image itself is enough for your case then you don't need a view snapshot at all.
But IF you do:
On snapshot create this image. Then overlay your preview layer with and image view that is showing this generated image (add a subview). Then create the snapshot and remove the image view all in a single chunk:
func snapshot() -> UIImage? {
let imageView = UIImageView(frame: self.previewPanelView.bounds)
imageView.image = self.imageFromLatestSampleBuffer()
imageView.contentMode = .aspectFill // Not sure
self.previewPanelView.addSubview(imageView)
let image = createSnapshot()
imageView.removeFromSuperview()
return image
}
Let us know how things turn and you tried, what did or did not work.
I'm facing an issue which is driving my crazy rightnow.
I have an app which has a AVCaptureVideoPreviewLayer to show the camera, analog to snapchat.
When I take the still picture through AVCaptureStillImageOutput, it is way wider than the preview layer.
What I want to achieve is, that after taking the picture, the taken picture looks 1:1 same as in the preview, based on size.
func takePicture(){
if let videoConnection = stillImageOutput.connectionWithMediaType(AVMediaTypeVideo) {
// make sure we have the proper device orientation set
videoConnection.videoOrientation = previewLayer!.connection!.videoOrientation
stillImageOutput.captureStillImageAsynchronouslyFromConnection(videoConnection) {
(imageDataSampleBuffer, error) -> Void in
// take the picture from output
let imageData = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageDataSampleBuffer)
let imageOriginal:UIImage = UIImage(data: imageData)!
// imageOriginal must have the same aspect and size as the previewLayer!
}
}
}
I tried already with cropping based on calculated aspects, but this does not work.
The Previewlayer has the gravity allready set
previewLayer?.videoGravity = AVLayerVideoGravityResizeAspectFill
This doesn't change anything, and I think the fault lies within the stillImageOutput and not in the preview layer
If somebody knows help would be great, it looks like the whole AVFoundation-topic is very little covered across the www.
If somebody does not get what I mean, check those two screenshots from snapchat
I'm applying list of filters to detect shape during camera capturing process. Once shape is detected - want to save it to photos for review. Googled for imageFromCurrentFramebuffer, but is always saves black picture.
// camera init
var videoCamera = GPUImageVideoCamera(sessionPreset: AVCaptureSessionPreset1920x1080, cameraPosition: .Back)
videoCamera!.outputImageOrientation = .Portrait;
videoCamera?.startCameraCapture()
// filter init
var houghTransformFilter = GPUImageHoughTransformLineDetector()
houghTransformFilter!.lineDetectionThreshold = 0.3
houghTransformFilter!.useNextFrameForImageCapture() //without this crashes
houghTransformFilter!.linesDetectedBlock = {
// my custom shape detection logic
if (found) {
self.videoCamera?.pauseCameraCapture()
var capturedImage:UIImage? = self.houghTransformFilter!.imageFromCurrentFramebuffer()
UIImageWriteToSavedPhotosAlbum(capturedImage, nil, nil, nil);
}
}
I had a similar problem. You probably need to add this line before the call to grab the frame:
self.houghTransformFilter?.useNextFrameForImageCapture()