Hey I an trying to constantly get the value of the devices camera / ARCamera. As far as I know there is only one function the allows me to access these ARCamera traits. That is this function here:
Code:
// Only gets called couple times when camera state changes
func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera)
print("\(camera.eulerAngles)")
}
I've been thinking about maybe using some trickery like putting a repeating timer in the function that would call that value. But I can't call a local selectors that get booted out. What I'm more looking for is something along the lines of how this function is:
func renderer(_ aRenderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
// This constantly gets called.
}
I wonder if there is a way to incorporate the ARCamera into the function.
If you want to continuously get updates on camera state, implement ARSessionDelegate.session(_:didUpdate:):
class MyDelegate: ARSessionDelegate {
func session(_ session: ARSession, didUpdate frame: ARFrame) {
print("\(frame.camera)")
}
/* ... */
}
The ARFrame object shall contain camera field with all the necessary information.
If you just want to know when tracking state changes, you might want to store the state from session(_:cameraDidChangeTrackingState:) in a field, and refer to it in your rendering loop:
class MyDelegate: SCNSceneRendererDelegate, ARSessionObserver {
var camera: ARCamera! = nil
func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera) {
self.camera = camera
}
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
print("\(self.camera.trackingState)")
}
/* ... more methods ... */
}
Related
in brief: How to pass every frame from AR Session to a funcion?
I'm making an app:
It displays the image of the rear camera in real time. (in AR View)（✅）
When it detects a QR code, it will read the text content in the QR code.（I don't know how to do it in real time, get data from AR session）
func getQRCodeContent(_ pixel: CVPixelBuffer) -> String {
let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixel, options: [:])
let request = VNDetectBarcodesRequest()
request.symbologies = [.qr]
try! requestHandler.perform([request])
let result = request.results?.first?.payloadStringValue
if let result = result {
return result
} else {
return "non"
}
}
And then do some logic with the content, and display the corresponding AR model in the AR View.
I know I have to feed images into the Vision Framework, I started with AVFoundation, but I found that when AR View is loaded, the AVCaptureSession is paused.
And I want to feed AR Session's frame into Vision Framework. However, all the tutorials I can find are based on story board and UI kit to complete this function. I don't know how to complete this function in Swift UI at all.
I tried to extent ARView:
extension ARView: ARSessionDelegate {
func renderer(_ renderer: SKRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {
let capturedImage = session.currentFrame?.capturedImage
print(capturedImage)
}
}
struct ARViewCustom: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
arView.session.delegate = arView
return arView
}
func updateUIView(_ uiView: ARView, context: Context) {}
}
No error, but it doesn't work.
I have a bit of a long winded process at the moment which retains an ARFrame from the ARSessionDelegate's func session(_ session: ARSession, didUpdate frame: ARFrame) callback.
I then do some processing which can take anywhere between 2-5 seconds in which the user of the app can move the camera and point it at a different location in AR then when I grabbed the ARFrame.
I noticed that the ARFrame object has it's own method to produce an ARRaycastQuery which I assume would be relative to that frame regardless of where the camera is currently pointed.
Here is a function where I use the ARRaycastQuery from an ARFrame and execute it against the ARSession
func getQuery(forPosition point: CGPoint, frame: ARFrame) -> ARRaycastResult? {
let estimatedPlane = ARRaycastQuery.Target.estimatedPlane
let alignment = ARRaycastQuery.TargetAlignment.any
let raycastQuery: ARRaycastQuery = frame.raycastQuery(from: point, allowing: estimatedPlane, alignment: alignment)
guard let raycastResult = arKitCoordinator.arView.session.raycast(raycastQuery).first else {
return nil
}
return raycastResult
}
However, If I have moved that camera (ARView view port) away from where I captured the ARFrame I always get nil. I would expect to be able to add objects into AR even if the user has moved the view port (iPhone) away from where I got the frame.
How can I add object into the "invisible" or "out of view" portions of the AR Space?
My app runs Vision on a CoreML model. The camera frames the machine learning model runs on are from an ARKit sceneView (basically, the camera). I have a method that's called loopCoreMLUpdate() that continuously runs CoreML so that we keep running the model on new camera frames. The code looks like this:
import UIKit
import SceneKit
import ARKit
class MyViewController: UIViewController {
var visionRequests = [VNRequest]()
let dispatchQueueML = DispatchQueue(label: "com.hw.dispatchqueueml") // A Serial Queue
override func viewDidLoad() {
super.viewDidLoad()
// Setup ARKit sceneview
// ...
// Begin Loop to Update CoreML
loopCoreMLUpdate()
}
// This is the problematic part.
// In fact - once it's run there's no way to stop it, is there?
func loopCoreMLUpdate() {
// Continuously run CoreML whenever it's ready. (Preventing 'hiccups' in Frame Rate)
dispatchQueueML.async {
// 1. Run Update.
self.updateCoreML()
// 2. Loop this function.
self.loopCoreMLUpdate()
}
}
func updateCoreML() {
///////////////////////////
// Get Camera Image as RGB
let pixbuff : CVPixelBuffer? = (sceneView.session.currentFrame?.capturedImage)
if pixbuff == nil { return }
let ciImage = CIImage(cvPixelBuffer: pixbuff!)
// Note: Not entirely sure if the ciImage is being interpreted as RGB, but for now it works with the Inception model.
// Note2: Also uncertain if the pixelBuffer should be rotated before handing off to Vision (VNImageRequestHandler) - regardless, for now, it still works well with the Inception model.
///////////////////////////
// Prepare CoreML/Vision Request
let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage, options: [:])
// let imageRequestHandler = VNImageRequestHandler(cgImage: cgImage!, orientation: myOrientation, options: [:]) // Alternatively; we can convert the above to an RGB CGImage and use that. Also UIInterfaceOrientation can inform orientation values.
///////////////////////////
// Run Image Request
do {
try imageRequestHandler.perform(self.visionRequests)
} catch {
print(error)
}
}
}
As you can see the loop effect is created by a DispatchQueue with the label com.hw.dispatchqueueml that keeps calling loopCoreMLUpdate(). Is there any way to stop the queue once CoreML is not needed anymore? Full code is here.
I suggest instead o running coreML model here in viewDidLoad, you can use ARSessionDelegate function for the same.
func session(_ session: ARSession, didUpdate frame: ARFrame) method to get the frame, you can set the flag, here to enable when you want the the model to work and when you dont.
Like this below:
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// This is where we will analyse our frame
// We return early if currentBuffer is not nil or the tracking state of camera is not normal
// TODO: - Core ML Functionality Commented
guard isMLFlow else { //
return
}
currentBuffer = frame.capturedImage
guard let buffer = currentBuffer, let image = UIImage(pixelBuffer: buffer) else { return }
<Code here to load model>
CoreMLManager.manager.updateClassifications(for: image)
}
This answer and others explain how to get notified when ARKit detects anchors or planes, but how do you get notifications when ARKit detects feature points?
Looking at the APIs it's somewhat similar to the answers that you have linked to.
Using ARSessionDelegate session(_ session: ARSession, didUpdate frame: ARFrame) you can access the rawFeaturePoints of the ARFrame that just got passed in.
So it would look something like:
// Not actually tested
class MyARSessionDelegate: ARSessionDelegate {
var previouslyDetectedPointCount = 0
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// Check if new points are detected
if previouslyDetectedPointCount != frame.rawFeaturePoints?.points.count {
// point count has changed
previouslyDetectedPointCount = frame.rawFeaturePoints!.points.count
}
}
}
Though as to why you would want to be looking for the specific points is curious. The documentation clearly states:
ARKit does not guarantee that the number and arrangement of raw
feature points will remain stable between software releases, or even
between subsequent frames in the same session.
This doesn't seem like the ideal solution, but it works. Implement the func session(_ session: ARSession, didUpdate frame: ARFrame) function from the ARSessionDelegate protocol, and check for feature points in each frame.
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// Show tap icon once we find feature points
if !detectedFeaturePoints, let points = frame.rawFeaturePoints?.points, let firstPoint = points.first {
detectedFeaturePoints = true
}
}
I understand how to extract feature points for a single ARFrame (ARFrame.rawFeaturePoints). Is there anyway to extract all feature points that have been detected in the session? Is this something I have to aggregate myself? If so, how should I handle point matching?
find the ways to capture the feature points.
You can implement the ARSCNViewDelegate, you will get a callback whenever AR finds plane..here you can capture all the feature points.
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
if let planeAnchor = anchor as? ARPlaneAnchor {
let cloudPoints = sceneView.session.currentFrame?.rawFeaturePoints
}
}
2.You can implement the SCNSceneRendererDelegate, implement the
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) { }.