Persist object orientations in ARKit ARWorldMap - ios

I'm trying to persist a model in ARKit using the ARWorldMap. I can save and load the models, but the orientation I apply to the objects before I save is not persisted with the object.
What I'm currently doing
Objects are saved and loaded:
/// - Tag: GetWorldMap
#objc func saveExperience(_ button: UIButton) {
sceneView.session.getCurrentWorldMap { worldMap, error in
guard let map = worldMap
else { self.showAlert(title: "Can't get current world map", message: error!.localizedDescription); return }
// Add a snapshot image indicating where the map was captured.
guard let snapshotAnchor = SnapshotAnchor(capturing: self.sceneView) else {
fatalError("Can't take snapshot")
}
map.anchors.append(snapshotAnchor)
do {
let data = try NSKeyedArchiver.archivedData(withRootObject: map, requiringSecureCoding: true)
try data.write(to: self.mapSaveURL, options: [.atomic])
DispatchQueue.main.async {
self.loadExperienceButton.isHidden = false
self.loadExperienceButton.isEnabled = true
}
} catch {
fatalError("Can't save map: \(error.localizedDescription)")
}
}
}
/// - Tag: RunWithWorldMap
#objc func loadExperience(_ button: UIButton) {
/// - Tag: ReadWorldMap
let worldMap: ARWorldMap = {
guard let data = mapDataFromFile
else { fatalError("Map data should already be verified to exist before Load button is enabled.") }
do {
guard let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: data)
else { fatalError("No ARWorldMap in archive.") }
return worldMap
} catch {
fatalError("Can't unarchive ARWorldMap from file data: \(error)")
}
}()
// Display the snapshot image stored in the world map to aid user in relocalizing.
if let snapshotData = worldMap.snapshotAnchor?.imageData,
let snapshot = UIImage(data: snapshotData) {
self.snapshotThumbnail.image = snapshot
} else {
print("No snapshot image in world map")
}
// Remove the snapshot anchor from the world map since we do not need it in the scene.
worldMap.anchors.removeAll(where: { $0 is SnapshotAnchor })
let configuration = self.defaultConfiguration // this app's standard world tracking settings
configuration.initialWorldMap = worldMap
sceneView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
isRelocalizingMap = true
virtualObjectAnchor = nil
}
Rotation:
#objc func didRotate(_ gesture: UIRotationGestureRecognizer) {
sceneView.scene.rootNode.eulerAngles.y = objectRotation
gesture.rotation = 0
}
And then it's rendered:
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard anchor.name == virtualObjectAnchorName else {
return
}
// save the reference to the virtual object anchor when the anchor is added from relocalizing
if virtualObjectAnchor == nil {
virtualObjectAnchor = anchor
}
node.addChildNode(virtualObject)
}
How can I do this?
How can I go about doing this? I have tried multiple solutions, but the orientation is never kept. It loads the object at the correct position, but rotation and scaling is never kept, even if I apply it to the rootnode. The only option I can see is to also store the transform as a seperate data object, and load that and apply it. But seems like it should be possible to store this data with the object.

Apple Documentation for ARWorldMap shows that the properties for an ARWorldMap class are:
When you archive a world map, these are the only information that get saved. Any information about the nodes added to the anchors during the session (e.g. changing node scale and orientation) are not saved along with the world map during the archiving.
I remember watching a WWDC session where they demoed a multiplayer AR game called SwiftShot where players hit different objects with balls. They provided the source code and I noticed they used a custom ARAnchor subclass called BoardAnchor which they used to store additional information in the anchor class such as the size of the game board.
See: SwiftShot: Creating a Game for Augmented Reality.
You can use the same approach to store, for example, the scale and orientation of a node, so that when you unarchive the world map and it get's relocalized, you can use ARSCNViewDelegate's renderer(_:didAdd:for:) to resize and scale the node based on the information stored in your custom ARAnchor.

Related

ARKit does not recognize reference images

I'm trying to place a 3D model on top of a recognized image with ARKit and RealityKit - all programmatically. Before I start the ARView I'm downloading the model I want to show when the reference image is detected.
This is my current setup:
override func viewDidLoad() {
super.viewDidLoad()
arView.session.delegate = self
// Check if the device supports the AR experience
if (!ARConfiguration.isSupported) {
TLogger.shared.error_objc("Device does not support Augmented Reality")
return
}
guard let qrCodeReferenceImage = UIImage(named: "QRCode") else { return }
let detectionImages: Set<ARReferenceImage> = convertToReferenceImages([qrCodeReferenceImage])
let configuration = ARWorldTrackingConfiguration()
configuration.detectionImages = detectionImages
arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
I use the ARSessionDelegate to get notified when a new image anchor was added which means the reference image got detected:
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
print("Hello")
for anchor in anchors {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let referenceImage = imageAnchor.referenceImage
addEntity(self.localModelPath!)
}
}
However, the delegate method never gets called while other delegate functions like func session(ARSession, didUpdate: ARFrame) are getting called so I assume that the session just doesn't detect the image. The image resolution is good and the printed image the big so it should definitely get recognized by the ARSession. I also checked that the image has been found before adding it to the configuration.
Can anyone lead me in the right direction here?
It looks like you have your configuration set up correctly. Your delegate-function should be called when the reference image is recognized. Make sure your configuration isn't overwritten at any point in your code.

Swift: Obtain and save the updated SCNNode over time using projectPoint in scenekit

I am trying to use projectPoint to get the 2D information of the updated SCNNode in scenekit and save them.
Based on ignotusverum's suggestion, I am able to save the SCNNode in to a path in a button.
var lastPosition: CGPoint?
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
guard anchor == currentFaceAnchor,
let contentNode = selectedContentController.contentNode,
contentNode.parent == node
else { return }
for (index, vertex) in vertices.enumerated() {
let vertex = sceneView.projectPoint(node.convertPosition(SCNVector3(vertex), to: nil))
let xVertex = CGFloat(vertex.x)
let yVertex = CGFloat(vertex.y)
Position = CGPoint(x: xVertex, y: yVertex)
}
selectedContentController.session = sceneView?.session
selectedContentController.sceneView = sceneView
selectedContentController.renderer(renderer, didUpdate: contentNode, for: anchor)
}
Started saving via a start button:
private var fpsTimer = Timer()
private var currentCaptureFrame = 0
#IBAction private func startPressed() {
currentCaptureFrame = 0 //inital capture frame
fpsTimer = Timer.scheduledTimer(withTimeInterval: 1/fps, repeats: true, block: {(timer) -> Void in self.recordData()})
}
Saved them via a stop button clicks:
#IBAction private func stopPressed() {
do {
fpsTimer.invalidate() //turn off the timer
let capturedData = captureData.map{$0.stringRepresentation}.joined(separator:"\(lastPosition)>")
let dir: URL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).last! as URL
let url = dir.appendingPathComponent("testing.txt")
try capturedData.appendLineToURL(fileURL: url as URL)
}
catch {
print("Could not write to file")
}
}
So far works fine with the points being saved. The problem is that in the saved data, I realized the data is only saving one frame of the x and y vertices. For example:
[(411.0618591308594, 534.4215087890625), (410.7286071777344, 544.9381713867188), (411.5425720214844, 522.1063232421875), (412.0340881347656, 512.1854248046875),...
[(411.0618591308594, 534.4215087890625), (410.7286071777344, 544.9381713867188), (411.5425720214844, 522.1063232421875), (412.0340881347656, 512.1854248046875)
The data is repeating with one frame rather than the period I want to save from the moment when I click start button to stop button.
My question is how to save the updated SCNNode over time from the moment when I click start button to stop button?
Thanks in advance!
If I understand your question correctly you want to save the vertex positions each time they are updated, keeping track of all previous updates as well as the most recent one. In order to do this you simply need to append the new vertex position array to the global array with saved data.
var savedPositions = [CGPoint]()
var beginSaving = false
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
guard anchor == currentFaceAnchor,
let contentNode = selectedContentController.contentNode,
contentNode.parent == node
else { return }
for vertex in vertices {
let projectedPoint = sceneView.projectPoint(node.convertPosition(SCNVector3(vertex), to: nil))
if beginSaving {
savedPositions.append(CGPoint(x: projectedPoint.x, y: projectedPoint.y))
}
}
selectedContentController.session = sceneView?.session
selectedContentController.sceneView = sceneView
selectedContentController.renderer(renderer, didUpdate: contentNode, for: anchor)
}
#IBAction private func startPressed() {
beginSaving = true
}
#IBAction private func stopPressed() {
beginSaving = false
....//do whatever with your array of saved positions
}
SceneKit calls this method exactly once per frame. If you do stuff inside the renderer, you gotta "get it done" and get out, otherwise it can have an impact on FPS.
If you need to move or check a lot of loop stuff, you can use timers to perform those efforts separately and that allows you some control over it. You can still keep your FPS up and break loops up or chunk work in the timers. If you do this, just make sure you put timers in the main thread.
You might also look at: node.presentation, ie: properties reflect the transitory values determined by any in-flight animations currently affecting the node.

How to stop CoreML from running when it's no longer needed in the app?

My app runs Vision on a CoreML model. The camera frames the machine learning model runs on are from an ARKit sceneView (basically, the camera). I have a method that's called loopCoreMLUpdate() that continuously runs CoreML so that we keep running the model on new camera frames. The code looks like this:
import UIKit
import SceneKit
import ARKit
class MyViewController: UIViewController {
var visionRequests = [VNRequest]()
let dispatchQueueML = DispatchQueue(label: "com.hw.dispatchqueueml") // A Serial Queue
override func viewDidLoad() {
super.viewDidLoad()
// Setup ARKit sceneview
// ...
// Begin Loop to Update CoreML
loopCoreMLUpdate()
}
// This is the problematic part.
// In fact - once it's run there's no way to stop it, is there?
func loopCoreMLUpdate() {
// Continuously run CoreML whenever it's ready. (Preventing 'hiccups' in Frame Rate)
dispatchQueueML.async {
// 1. Run Update.
self.updateCoreML()
// 2. Loop this function.
self.loopCoreMLUpdate()
}
}
func updateCoreML() {
///////////////////////////
// Get Camera Image as RGB
let pixbuff : CVPixelBuffer? = (sceneView.session.currentFrame?.capturedImage)
if pixbuff == nil { return }
let ciImage = CIImage(cvPixelBuffer: pixbuff!)
// Note: Not entirely sure if the ciImage is being interpreted as RGB, but for now it works with the Inception model.
// Note2: Also uncertain if the pixelBuffer should be rotated before handing off to Vision (VNImageRequestHandler) - regardless, for now, it still works well with the Inception model.
///////////////////////////
// Prepare CoreML/Vision Request
let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage, options: [:])
// let imageRequestHandler = VNImageRequestHandler(cgImage: cgImage!, orientation: myOrientation, options: [:]) // Alternatively; we can convert the above to an RGB CGImage and use that. Also UIInterfaceOrientation can inform orientation values.
///////////////////////////
// Run Image Request
do {
try imageRequestHandler.perform(self.visionRequests)
} catch {
print(error)
}
}
}
As you can see the loop effect is created by a DispatchQueue with the label com.hw.dispatchqueueml that keeps calling loopCoreMLUpdate(). Is there any way to stop the queue once CoreML is not needed anymore? Full code is here.
I suggest instead o running coreML model here in viewDidLoad, you can use ARSessionDelegate function for the same.
func session(_ session: ARSession, didUpdate frame: ARFrame) method to get the frame, you can set the flag, here to enable when you want the the model to work and when you dont.
Like this below:
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// This is where we will analyse our frame
// We return early if currentBuffer is not nil or the tracking state of camera is not normal
// TODO: - Core ML Functionality Commented
guard isMLFlow else { //
return
}
currentBuffer = frame.capturedImage
guard let buffer = currentBuffer, let image = UIImage(pixelBuffer: buffer) else { return }
<Code here to load model>
CoreMLManager.manager.updateClassifications(for: image)
}

ARKit removes a node when Reference Image disappeared

I'm building AR Scanner application where users are able to scan different images and receive rewards for this.
When they point camera at some specific image - I place SCNNode on top of that image and after they remove camera from that image - SCNNode get's dismissed.
But when image disappears and camera stays at the same position SCNNode didn't get dismissed.
How can I make it disappear together with Reference image disappearance?
I have studied lot's of other answers here, on SO, but they didn't help me
Here's my code for adding and removing SCNNode's:
extension ARScannerScreenViewController: ARSCNViewDelegate {
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
DispatchQueue.main.async { self.instructionLabel.isHidden = true }
if let imageAnchor = anchor as? ARImageAnchor {
handleFoundImage(imageAnchor, node)
imageAncors.append(imageAnchor)
trackedImages.append(node)
} else if let objectAnchor = anchor as? ARObjectAnchor {
handleFoundObject(objectAnchor, node)
}
}
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let pointOfView = sceneView.pointOfView else { return }
for (index, item) in trackedImages.enumerated() {
if !(sceneView.isNode(item, insideFrustumOf: pointOfView)) {
self.sceneView.session.remove(anchor: imageAncors[index])
}
}
}
private func handleFoundImage(_ imageAnchor: ARImageAnchor, _ node: SCNNode) {
let name = imageAnchor.referenceImage.name!
print("you found a \(name) image")
let size = imageAnchor.referenceImage.physicalSize
if let imageNode = showImage(size: size) {
node.addChildNode(imageNode)
node.opacity = 1
}
}
private func showImage(size: CGSize) -> SCNNode? {
let image = UIImage(named: "InfoImage")
let imageMaterial = SCNMaterial()
imageMaterial.diffuse.contents = image
let imagePlane = SCNPlane(width: size.width, height: size.height)
imagePlane.materials = [imageMaterial]
let imageNode = SCNNode(geometry: imagePlane)
imageNode.eulerAngles.x = -.pi / 2
return imageNode
}
private func handleFoundObject(_ objectAnchor: ARObjectAnchor, _ node: SCNNode) {
let name = objectAnchor.referenceObject.name!
print("You found a \(name) object")
}
}
I also tried to make it work using ARSession, but I couldn't even get to prints:
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
for anchor in anchors {
for myAnchor in imageAncors {
if let imageAnchor = anchor as? ARImageAnchor, imageAnchor == myAnchor {
if !imageAnchor.isTracked {
print("Not tracked")
} else {
print("tracked")
}
}
}
}
}
You have to use ARWorldTrackingConfiguration instead of ARImageTrackingConfiguration. It's quite bad idea to use both configurations in app because each time you switch between them – tracking state is reset and you have to track from scratch.
Let's see what Apple documentation says about ARImageTrackingConfiguration:
With ARImageTrackingConfiguration, ARKit establishes a 3D space not by tracking the motion of the device relative to the world, but solely by detecting and tracking the motion of known 2D images in view of the camera.
The basic differences between these two configs are about how ARAnchors behave:
ARImageTrackingConfiguration allows you get ARImageAnchors only if your reference images is in a Camera View. So if you can't see a reference image – there's no ARImageAnchor, thus there's no a 3D model (it's resetting each time you cannot-see-it-and-then-see-it-again). You can simultaneously detect up to 100 images.
ARWorldTrackingConfiguration allows you track a surrounding environment in 6DoF and get ARImageAnchor, ARObjectAnchor, or AREnvironmentProbeAnchor. If you can't see a reference image – there's no ARImageAnchor, but when you see it again ARImageAnchor is still there. So there's no reset.
Conclusion:
ARWorldTrackingConfiguration's cost of computation is much higher. However this configuration allows you perform not only image tracking but also hit-testing and ray-casting for detected planes, object detection, and a restoration of world maps.
Use nodeForAnchor to load your nodes, so when the anchors disappear, the nodes will go as well.

Show bounding box while detecting object using ARKit 2

I have scanned and trained multiple real world objects. I do have the ARReferenceObject and the app detects them fine.
The issue that I'm facing is when an object doest not have distinct, vibrant features it takes few seconds to return a detection result, which I can understand. Now, I want the app to show a bounding box and an activity indicator on top the object while it is trying to detect it.
I do not see any information regarding this. Also, if there is any way to get the time when detection starts or the confidence percentage of the object being detected.
Any help is appreciated.
It is possible to show a boundingBox in regard to the ARReferenceObject prior to it being detected; although I am not sure why you would want to do that (in advance anyway).
For example, assuming your referenceObject was on a horizontal surface you would first need to place your estimated bounding box on the plane (or use some other method to place it in advance), and in the time it took to detect the ARPlaneAnchor and place the boundingBox it is most likely that your model would already have been detected.
Possible Approach:
As you are no doubt aware an ARReferenceObject has a center, extent and scale property as well as a set of rawFeaturePoints associated with the object.
As such we can create our own boundingBox node based on some of the sample code from Apple in Scanning & Detecting 3D Objects and create our own SCNNode which will display a bounding box of the approximate size of the ARReferenceObject which is stored locally prior to it being detected.
Note you will need to locate the 'wireframe_shader' from the Apple Sample Code for the boundingBox to render transparent:
import Foundation
import ARKit
import SceneKit
class BlackMirrorzBoundingBox: SCNNode {
//-----------------------
// MARK: - Initialization
//-----------------------
/// Creates A WireFrame Bounding Box From The Data Retrieved From The ARReferenceObject
///
/// - Parameters:
/// - points: [float3]
/// - scale: CGFloat
/// - color: UIColor
init(points: [float3], scale: CGFloat, color: UIColor = .cyan) {
super.init()
var localMin = float3(Float.greatestFiniteMagnitude)
var localMax = float3(-Float.greatestFiniteMagnitude)
for point in points {
localMin = min(localMin, point)
localMax = max(localMax, point)
}
self.simdPosition += (localMax + localMin) / 2
let extent = localMax - localMin
let wireFrame = SCNNode()
let box = SCNBox(width: CGFloat(extent.x), height: CGFloat(extent.y), length: CGFloat(extent.z), chamferRadius: 0)
box.firstMaterial?.diffuse.contents = color
box.firstMaterial?.isDoubleSided = true
wireFrame.geometry = box
setupShaderOnGeometry(box)
self.addChildNode(wireFrame)
}
required init?(coder aDecoder: NSCoder) { fatalError("init(coder:) Has Not Been Implemented") }
//----------------
// MARK: - Shaders
//----------------
/// Sets A Shader To Render The Cube As A Wireframe
///
/// - Parameter geometry: SCNBox
func setupShaderOnGeometry(_ geometry: SCNBox) {
guard let path = Bundle.main.path(forResource: "wireframe_shader", ofType: "metal", inDirectory: "art.scnassets"),
let shader = try? String(contentsOfFile: path, encoding: .utf8) else {
return
}
geometry.firstMaterial?.shaderModifiers = [.surface: shader]
}
}
To display the bounding box you you would then do something like the following, noting that in my example I have the following variables:
#IBOutlet var augmentedRealityView: ARSCNView!
let configuration = ARWorldTrackingConfiguration()
let augmentedRealitySession = ARSession()
To display the boundingBox prior to detection of the actual object itself, you would call the func loadBoundigBox in viewDidLoad e.g:
/// Creates A Bounding Box From The Data Available From The ARObject In The Local Bundle
func loadBoundingBox(){
//1. Run Our Session
augmentedRealityView.session = augmentedRealitySession
augmentedRealityView.delegate = self
//2. Load A Single ARReferenceObject From The Main Bundle
if let objectURL = Bundle.main.url(forResource: "fox", withExtension: ".arobject"){
do{
var referenceObjects = [ARReferenceObject]()
let object = try ARReferenceObject(archiveURL: objectURL)
//3. Log it's Properties
print("""
Object Center = \(object.center)
Object Extent = \(object.extent)
Object Scale = \(object.scale)
""")
//4. Get It's Scale
let scale = CGFloat(object.scale.x)
//5. Create A Bounding Box
let boundingBoxNode = BlackMirrorzBoundingBox(points: object.rawFeaturePoints.points, scale: scale)
//6. Add It To The ARSCNView
self.augmentedRealityView.scene.rootNode.addChildNode(boundingBoxNode)
//7. Position It 0.5m Away From The Camera
boundingBoxNode.position = SCNVector3(0, -0.5, -0.5)
//8. Add It To The Configuration
referenceObjects.append(object)
configuration.detectionObjects = Set(referenceObjects)
}catch{
print(error)
}
}
//9. Run The Session
augmentedRealitySession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
augmentedRealityView.automaticallyUpdatesLighting = true
}
The above example simple creates a boundingBox from the non-detected ARReferenceObject and places it 0.5m down from and 0.5meter away from the Camera which yields something like this:
You would of course need to handle the position of the boundBox initially, as well as hoe to handle the removal of the boundingBox 'indicator'.
The method below simply shows a boundBox when the actual object is detected e.g:
//--------------------------
// MARK: - ARSCNViewDelegate
//--------------------------
extension ViewController: ARSCNViewDelegate{
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
//1. Check We Have A Valid ARObject Anchor
guard let objectAnchor = anchor as? ARObjectAnchor else { return }
//2. Create A Bounding Box Around Our Object
let scale = CGFloat(objectAnchor.referenceObject.scale.x)
let boundingBoxNode = BlackMirrorzBoundingBox(points: objectAnchor.referenceObject.rawFeaturePoints.points, scale: scale)
node.addChildNode(boundingBoxNode)
}
}
Which yields something like this:
In regard to the detection timer, there is an example in the Apple Sample Code, which displays how long it takes to detect the model.
In its crudest form (not accounting for milliseconds) you can do something like so:
Firstly create A Timer and a var to store the detection time e.g:
var detectionTimer = Timer()
var detectionTime: Int = 0
Then when you run your ARSessionConfiguration initialise the timer e.g:
/// Starts The Detection Timer
func startDetectionTimer(){
detectionTimer = Timer.scheduledTimer(timeInterval: 1.0, target: self, selector: #selector(logDetectionTime), userInfo: nil, repeats: true)
}
/// Increments The Total Detection Time Before The ARReference Object Is Detected
#objc func logDetectionTime(){
detectionTime += 1
}
Then when an ARReferenceObject has been detected invalidate the timer and log the time e.g:
//--------------------------
// MARK: - ARSCNViewDelegate
//--------------------------
extension ViewController: ARSCNViewDelegate{
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
//1. Check We Have A Valid ARObject Anchor
guard let _ = anchor as? ARObjectAnchor else { return }
//2. Stop The Timer
detectionTimer.invalidate()
//3. Log The Detection Time
print("Total Detection Time = \(detectionTime) Seconds")
//4. Reset The Detection Time
detectionTime = 0
}
}
This should be more than enough to get your started...
And please note, that this example doesn't provide a boundingBox when scanning an object (look at the Apple Sample Code for that), it provides one based on an existing ARReferenceObject which is implied in your question (assuming I interpreted it correctly).

Resources