Is there a way to display camera images without using AVCaptureVideoPreviewLayer?
I want to do screen capture, but I can not do it.
session = AVCaptureSession()
camera = AVCaptureDevice.default(
AVCaptureDevice.DeviceType.builtInWideAngleCamera,
for: AVMediaType.video,
position: .front) // position: .front
do {
input = try AVCaptureDeviceInput(device: camera)
} catch let error as NSError {
print(error)
}
if(session.canAddInput(input)) {
session.addInput(input)
}
let previewLayer = AVCaptureVideoPreviewLayer(session: session)
cameraView.backgroundColor = UIColor.red
previewLayer.frame = cameraView.bounds
previewLayer.videoGravity = AVLayerVideoGravity.resizeAspect
cameraview.layer.addSublayer(previewLayer)
session.startRunning()
I am currently trying to broadcast a screen capture. It is to synthesize the camera image and some UIView. However, if you use AVCaptureVideoPreviewLayer screen capture can not be done and the camera image is not displayed. Therefore, I want to display the camera image so that screen capture can be performed.
Generally the views that are displayed using GPU directly may not be redrawn on the CPU. This includes situations like openGL content or these preview layers.
The "screen capture" redraws the screen on a new context on CPU which obviously ignores the GPU part.
You should try and play around with adding some outputs on the session which will give you images or rather CMSampleBuffer shots which may be used to generate the image.
There are plenty ways in doing this but you will most likely need to go a step lower. You can add output to your session to receive samples directly. Doing this is a bit of a code so please refer to some other posts like this one. The point in this you will have a didOutputSampleBuffer method which will feed you CMSampleBufferRef objects that may be used to construct pretty much anything in terms of images.
Now in your case I assume you will be aiming to get UIImage from sample buffer. To do so you may again need a bit of code so refer to some other post like this one.
To put it all together you could as well simply use an image view and drop the preview layer. As you get the sample buffer you can create image and update your image view. I am not sure what the performance of this would be but I discourage you on doing this. If image itself is enough for your case then you don't need a view snapshot at all.
But IF you do:
On snapshot create this image. Then overlay your preview layer with and image view that is showing this generated image (add a subview). Then create the snapshot and remove the image view all in a single chunk:
func snapshot() -> UIImage? {
let imageView = UIImageView(frame: self.previewPanelView.bounds)
imageView.image = self.imageFromLatestSampleBuffer()
imageView.contentMode = .aspectFill // Not sure
self.previewPanelView.addSubview(imageView)
let image = createSnapshot()
imageView.removeFromSuperview()
return image
}
Let us know how things turn and you tried, what did or did not work.
Related
This is a long question so I wanted to put a TL;DR on top:
I want to track QR codes via on of two methods: image tracking by cropping them upon detection, or placing anchors with raycasting. Both of these methods fail when the phone is in portrait mode. Camera source is an ARSession, SceneKit and RealityKit not used. There's only ARKit. What to do?
I am currently working on an application with Swift in which I try to render some stuff on a server, transmit the video to iPhone and display it on screen using a MTKView. I only needed a custom Meal shader to apply some complex calculations to received frames, so I did not use SceneKit or RealityKit. I only have ARSession from ARKit and a Metal view here, and up to this point everything works fine.
I am able to do image tracking at this point. However, I want to apply this behaviour to QR codes. What I want is to detect a QR code (multiple if possible) and then track it just like images. Since I don't have the QR code as ARReferenceImages beforehand like normal image tracking, I was left with two options:
Option 1: Using raycast(_:) on ARSession
This is probably the right way to do it. However, for this I need to activate both plane tracking options on ARSession, which then creates many anchors and managing them with image tracking becomes harder. This is not the actual problem though. Actual problem is that when the phone is in landscape mode, raycasting works as intended. When phone goes into portrait mode, even if I pass the frame in correct orientation it misses everything and hit test results return empty. I am not using hitTest(_:) because it is deprecated.
I want to explain the "correct orientation" thing here before going into second option. ARSession is capturing frames and I am able to check each frame through didUpdate delegate function of the session. When I read the pixel buffer out of the frame using frame.capturedImage and turn it into a CIImage, the image is always in landscape mode (width > height). Doesn't matter if the phone is in portrait mode or not. So whenever I want to pass this image, I am using oriented(.right) for portrait and oriented(.up) for landscape. I got that idea from another question asked about QR bounding box, and so far it is the best option (but not good enough). Just want to note that when I tried raycasting, I tried it with the image size, not screen size (screen size = my Metal view size because it is fullscreen) since the image is larger than the screen in reality. I am able to see this if I put a breakpoint and quicklook my CIImage created from current camera frame.
Option 2: Cropping the QR and treating it as image tracking
This is another approach which I am currently working on. Algorithm is simple: check every frame with Vision. If there are detected QR codes, read their data first. If that data matches with an existing QR, then re-read it if the cropped QR size is larger than existing one. If not, do nothing. Then use this cropped QR image for tracking QR as an image. At this point we would have the data already so no problems here.
However, I tried many times to do the proper transformation explained here in the answer. Again, I think I am able to transform normalized bounding box into a real rect which can correctly crop the image. Yet, as it is in raycasting, works perfectly only if the phone is in landscape position. When in portrait it works good enough ONLY IF the phone is really close to QR code and it is centered on the screen.
For related code, I have this in my View controller:
private var ciContext: CIContext = CIContext.init(options: nil)
private var sequenceHandler: VNImageRequestHandler?
And then I have this code to extract QR codes from CIImage:
func extractQrCode(image: CIImage) -> [VNBarcodeObservation]? {
self.sequenceHandler = VNImageRequestHandler(ciImage: image)
let barcodeRequest = VNDetectBarcodesRequest()
barcodeRequest.symbologies = [.QR]
try? self.sequenceHandler?.perform([barcodeRequest])
guard let results = barcodeRequest.results else {
return nil
}
return results
}
An this is the delegate that checks and operates on every frame (code currently for Option 2):
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let rotImg = self.renderer?.getInterfaceOrientation() == .portrait ? CIImage(cvPixelBuffer: frame.capturedImage).oriented(.right) : CIImage(cvPixelBuffer: frame.capturedImage)
if let barcodes = self.extractQrCode(image: rotImg) {
for barcode in barcodes {
guard let payload = barcode.payloadStringValue else { continue }
var rect = CGRect()
rect = VNImageRectForNormalizedRect(barcode.boundingBox.botToTop(), Int(rotImg.extent.width), Int(rotImg.extent.height))
let existingQR = TrackedImagesManager.imagesToTrack.filter{ $0.isQR && $0.QRData == payload}.first
if ((rect.size.width < 800 || rect.size.height < 800 || abs(rect.size.height - rect.size.width) > 32) && existingQR == nil) {
DispatchQueue.main.async {
self.showToastMessage(message: "Please get closer to the QR code and try centering it on your screen.", font: UIFont.systemFont(ofSize: 18), duration: 3)
}
continue
} else if (existingQR != nil) {
if (rect.width > existingQR?.originalImage?.size.width ?? 999) {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
existingQR?.originalImage = trackImg
existingQR?.image = ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1)
} else {
continue
}
} else if rect.width != 0 {
let croppedImg = rotImg.cropped(to: rect)
let croppedCgImage = self.ciContext.createCGImage(croppedImg, from: croppedImg.extent)!
let trackImg = UIImage(cgImage: croppedCgImage)
TrackedImagesManager.imagesToTrack.append(TrackedImage(id: 9, type: 1, image: ARReferenceImage(croppedCgImage, orientation: .up, physicalWidth: 0.1), originalImage: trackImg, isQR: true, QRData: payload))
print("qr norm rect: \(barcode.boundingBox) \n qr rect: \(rect) \nqr data: \(payload) \nqr hittestres: ")
}
}
}
}
Finally, for the transformation, I have this extension (tried various ways, this is the best so far):
extension CGRect {
func botToTop() -> CGRect {
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -1)
return self.applying(transform)
}
}
So for both options I need some advice to make things right. Android side of the same thing is implemented as in Option 2, but Android returns a nicely cropped QR code upon detection. We don't have that. What do I do now?
I'm taking a snapshot of every frame, applying a filter, and updating the background contents of the ARSCNView with the filtered image. Everything is working fine, but there is a lot of latency with all the UI elements on the screen. No latency on the ARSCNView.
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let image = CIImage(image: sceneView.snapshot()) else { return }
// I'm setting a filter to each image here. Which has no effect on the latency.
sceneView.scene.background.contents = context.createCGImage(image, from: image.extent)
}
I know I can use frame.capturedImage, which makes latency go away. However, I also place AR objects on the screen which frame.capturedImage ignores for some reason, and sceneView.scene.background.contents cannot be reset to its original source. So, I cannot turn off the image filter. That's why I need to take a snapshot.
Is there anything I can do that will reduce latency on the UI elements? I have a few UIScrollViews on the screen that have tremendous lag.
I'm also in the middle of looking for a way to do this with no lag, but I was able to at least reduce the lag by rendering the view into an image manually:
extension ARSCNView {
/// Performs screen snapshot manually, seems faster than built in snapshot() function, but still somewhat noticeable
var snapshot: UIImage? {
let renderer = UIGraphicsImageRenderer(size: self.bounds.size)
let image = renderer.image(actions: { context in
self.drawHierarchy(in: self.bounds, afterScreenUpdates: true)
})
return image
}
}
It's frustrating that this is faster than the built-in snapshot function, but it seems to be, and also still captures all the SceneKit graphics in the snapshot. (Doing this every frame will still be expensive though, FYI, and the only real solution for that would likely be a custom Metal shader.)
I'm also trying to work with ARSCNView.snapshotView(afterScreenUpdates: Bool) because that seems to have essentially no lag for my purposes, but whenever I try to turn the resulting View into a UIImage, it's totally blank. Either way, the above method cut the lag in about half for me, so you might have some luck with that.
I'm creating a blur effect using this below function in viewDidLoad of viewController
func applyBlurEffect(image: UIImage){
let imageToBlur = CIImage(image: image)!
let blurfilter = CIFilter(name: "CIGaussianBlur")!
blurfilter.setValue(10, forKey: kCIInputRadiusKey)
blurfilter.setValue(imageToBlur, forKey: "inputImage")
let resultImage = blurfilter.value(forKey: "outputImage") as! CIImage
let croppedImage: CIImage = resultImage.cropping(to: CGRect(x:0,y: 0,width: imageToBlur.extent.size.width,height: imageToBlur.extent.size.height))
let context = CIContext(options: nil)
let blurredImage = UIImage (cgImage: context.createCGImage(croppedImage, from: croppedImage.extent)!)
self.backImage.image = blurredImage
}
But this piece of code blocks the UI and the viewController opens after 3-4 seconds of lag. I don't want to present the UI without the blurEffect as well as i don't want the user to wait for 3-4 seconds while opening the viewController.
Please provide with a optimum solution for this problem.
GPUImage (https://github.com/BradLarson/GPUImage) blur works really much faster than CoreImage one:
extension UIImage {
func imageWithGaussianBlur() -> UIImage? {
let source = GPUImagePicture(image: self)
let gaussianFilter = GPUImageGaussianBlurFilter()
gaussianFilter.blurRadiusInPixels = 2.2
source?.addTarget(gaussianFilter)
gaussianFilter.useNextFrameForImageCapture()
source?.processImage()
return gaussianFilter.imageFromCurrentFramebuffer()
}
}
However small delay is still possible (depends on image size), so if you can't preprocess the image until view loads, I'd suggest to resize the image first, blur and display the resulted thumbnail, and then after the original image is processed in background queue, replace the thumbnail with the blurred original.
Core Image Programming Guide
Performance Best Practices
Follow these practices for best performance:
Don’t create a CIContext object every time you render. Contexts store a lot of state information; it’s more efficient to reuse
them.
Evaluate whether you app needs color management. Don’t use it unless you need it. See Does Your App Need Color Management?. Avoid
Core Animation animations while rendering CIImage objects with a
GPU context. If you need to use both simultaneously, you can set up
both to use the CPU.
Make sure images don’t exceed CPU and GPU limits. Image size limits for CIContext objects differ depending on whether Core Image uses the
CPU or GPU. Check the limit by using the methods
inputImageMaximumSize and outputImageMaximumSize.
User smaller images when possible. Performance scales with the number of output pixels. You can have Core Image render into a
smaller view, texture, or framebuffer. Allow Core Animation to
upscale to display size.
Use Core Graphics or Image I/O functions to crop or downsample, such as the functions CGImageCreateWithImageInRect or
CGImageSourceCreateThumbnailAtIndex.
The UIImageView class works best with static images. If your app needs to get the best performance, use lower-level APIs.
Avoid unnecessary texture transfers between the CPU and GPU. Render to a rectangle that is the same size as the source image before
applying a contents scale factor.
Consider using simpler filters that can produce results similar to algorithmic filters. For example, CIColorCube can produce output
similar to CISepiaTone, and do so more efficiently.
Take advantage of the support for YUV image in iOS 6.0 and later. Camera pixel buffers are natively YUV but most image processing
algorithms expect RBGA data. There is a cost to converting between
the two. Core Image supports reading YUB from CVPixelBuffer objects
and applying the appropriate color transform.
Have a look at Brad Larson's GPUImage also. You might want to use it. see this answer. https://stackoverflow.com/a/12336118/1378447
Can you present the view controller with the original image and perform the blur on a background thread and do a nice effect to replace the image once the blur ones is ready??
Also, maybe you could use a UIVisualEffectView and see if performance are better?
Apple a while ago also released an example where they were using UIImageEffects to perform a blur. It is written in Obj-C but you could easily use it in Swift https://developer.apple.com/library/content/samplecode/UIImageEffects/Listings/UIImageEffects_UIImageEffects_h.html
Make use of dispatch queues. This one worked for me:
func applyBlurEffect(image: UIImage){
DispatchQueue.global(qos: DispatchQoS.QoSClass.userInitiated).async {
let imageToBlur = CIImage(image: image)!
let blurfilter = CIFilter(name: "CIGaussianBlur")!
blurfilter.setValue(10, forKey: kCIInputRadiusKey)
blurfilter.setValue(imageToBlur, forKey: "inputImage")
let resultImage = blurfilter.value(forKey: "outputImage") as! CIImage
let croppedImage: CIImage = resultImage.cropping(to: CGRect(x:0,y: 0,width: imageToBlur.extent.size.width,height: imageToBlur.extent.size.height))
let context = CIContext(options: nil)
let blurredImage = UIImage (cgImage: context.createCGImage(croppedImage, from: croppedImage.extent)!)
DispatchQueue.main.async {
self.backImage.image = blurredImage
}
}
}
But this method will create a delay of 3-4 seconds for image to become blur(but it won't block the loading of other UI contents). If you don't want that time delay too, then applying UIBlurEffect to imageView will produce a similar effect:
func applyBlurEffect(image: UIImage){
self.profileImageView.backgroundColor = UIColor.clear
let blurEffect = UIBlurEffect(style: .extraLight)
let blurEffectView = UIVisualEffectView(effect: blurEffect)
blurEffectView.frame = self.backImage.bounds
blurEffectView.alpha = 0.5
blurEffectView.autoresizingMask = [.flexibleWidth, .flexibleHeight] // for supporting device rotation
self.backImage.addSubview(blurEffectView)
}
By changing the blur effect style to .light or .dark and alpha value from 0 to 1, you can get your desired effect
So I am using AVCaptureSession to take pictures with front camera. I am also creating previewLayer from this session to display current image on screen.
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill
It all works like it should.
But now I have a problem because I need to implement a button which will flip / mirror (transform) this preview layer - so users have a choice to take normal selfie picture or take mirrored one.
I have already tried transforming previewLayer and it KINDA works. The problem is that if you rotate device, preview picture rotates in the other way since it is transformed. (in the default or any other camera app picture rotates with camera). Anyone has any idea how to achieve that?
Mirroring preview layer: (I tried transforming layer and even view later, same result).
#IBAction func mirrorCamera(_ sender: AnyObject) {
cameraMirrored = !cameraMirrored
if cameraMirrored {
// TRANSFORMING VIEW
self.videoPreviewView.transform = CGAffineTransform(scaleX: -1, y: 1);
// OR LAYER
self.previewLayer.transform = CATransform3DMakeScale(-1, 1, 1);
} else {
self.videoPreviewView.transform = CGAffineTransform(scaleX: 1, y: 1);
self.videoPreviewView.transform = CATransform3DMakeScale(1, 1, 1);
}
}
Nowadays, if you use the mirrored property of the preview layer directly, you will get a deprecation warning at runtime. The current way to do it is using directly the connection from the camera. You must do something like this (code below is not real code, property names probably will differ, but you get the idea)
if (cameraPreviewLayer.connection.SupportsVideoMirroring) {
cameraPreviewLayer.connection.automaticallyAdjustsVideoMirroring = false
cameraPreviewLayer.connection.videoMirrored = true
}
AVCaptureVideoPreviewLayer has a property mirrored. Set this true or false, as required.
I'm facing an issue which is driving my crazy rightnow.
I have an app which has a AVCaptureVideoPreviewLayer to show the camera, analog to snapchat.
When I take the still picture through AVCaptureStillImageOutput, it is way wider than the preview layer.
What I want to achieve is, that after taking the picture, the taken picture looks 1:1 same as in the preview, based on size.
func takePicture(){
if let videoConnection = stillImageOutput.connectionWithMediaType(AVMediaTypeVideo) {
// make sure we have the proper device orientation set
videoConnection.videoOrientation = previewLayer!.connection!.videoOrientation
stillImageOutput.captureStillImageAsynchronouslyFromConnection(videoConnection) {
(imageDataSampleBuffer, error) -> Void in
// take the picture from output
let imageData = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageDataSampleBuffer)
let imageOriginal:UIImage = UIImage(data: imageData)!
// imageOriginal must have the same aspect and size as the previewLayer!
}
}
}
I tried already with cropping based on calculated aspects, but this does not work.
The Previewlayer has the gravity allready set
previewLayer?.videoGravity = AVLayerVideoGravityResizeAspectFill
This doesn't change anything, and I think the fault lies within the stillImageOutput and not in the preview layer
If somebody knows help would be great, it looks like the whole AVFoundation-topic is very little covered across the www.
If somebody does not get what I mean, check those two screenshots from snapchat