ARKIT: Move Object with PanGesture (the right way) - ios

Question

I've been reading plenty of StackOverflow answers on how to move an object by dragging it across the screen. Some use hit tests against .featurePoints some use the gesture translation or just keeping track of the lastPosition of the object. But honestly.. none work the way everyone is expecting it to work.
Hit testing against .featurePoints just makes the object jump all around, because you dont always hit a featurepoint when dragging your finger. I dont understand why everyone keeps suggesting this.
Solutions like this one work: Dragging SCNNode in ARKit Using SceneKit
But the object doesnt really follow your finger, and the moment you take a few steps or change the angle of the object or the camera.. and try to move the object.. the x,z are all inverted.. and makes total sense to do that.
I really want to move objects as good as the Apple Demo, but I look at the code from Apple... and is insanely weird and overcomplicated I cant even understand a bit. Their technique to move the object so beautifly is not even close to what everyone propose online.
https://developer.apple.com/documentation/arkit/handling_3d_interaction_and_ui_controls_in_augmented_reality
There's gotta be a simpler way to do it.

Answer 1

Short answer:
To get this nice and fluent dragging effect like in the Apple demo project, you will have to do it like in the Apple demo project (Handling 3D Interaction). On the other side I agree with you, that the code might be confusing if you look at it for the first time. It is not easy at all to calculate the correct movement for an object placed on a floor plane - always and from every location or viewing angle. It’s a complex code construct, that is doing this superb dragging effect. Apple did a great job to achieve this, but didn’t make it too easy for us.
Full Answer:
Striping down the AR Interaction template for your needy results in a nightmare - but should work too if you invest enough time. If you prefer to begin from scratch, basically start using a common swift ARKit/SceneKit Xcode template (the one containing the space ship).
You will also require the entire AR Interaction Template Project from Apple. (The link is included in the SO question)
At the End you should be able to drag something called VirtualObject, which is in fact a special SCNNode. In Addition you will have a nice Focus Square, that can be useful for whatever purpose - like initially placing objects or adding a floor, or a wall. (Some code for the dragging effect and the focus square usage are kind of merged or linked together - doing it without the focus square will actually be more complicated)
Get started:
Copy the following files from the AR Interaction template to your empty project:
Utilities.swift (usually I name this file Extensions.swift, it contains some basic extensions that are required)
FocusSquare.swift
FocusSquareSegment.swift
ThresholdPanGesture.swift
VirtualObject.swift
VirtualObjectLoader.swift
VirtualObjectARView.swift
Add the UIGestureRecognizerDelegate to the ViewController class definition like so:
class ViewController: UIViewController, ARSCNViewDelegate, UIGestureRecognizerDelegate {
Add this code to your ViewController.swift, in the definitions section, right before viewDidLoad:
// MARK: for the Focus Square
// SUPER IMPORTANT: the screenCenter must be defined this way
var focusSquare = FocusSquare()
var screenCenter: CGPoint {
let bounds = sceneView.bounds
return CGPoint(x: bounds.midX, y: bounds.midY)
}
var isFocusSquareEnabled : Bool = true
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
/// The tracked screen position used to update the `trackedObject`'s position in `updateObjectToCurrentTrackingPosition()`.
private var currentTrackingPosition: CGPoint?
/**
The object that has been most recently intereacted with.
The `selectedObject` can be moved at any time with the tap gesture.
*/
var selectedObject: VirtualObject?
/// The object that is tracked for use by the pan and rotation gestures.
private var trackedObject: VirtualObject? {
didSet {
guard trackedObject != nil else { return }
selectedObject = trackedObject
}
}
/// Developer setting to translate assuming the detected plane extends infinitely.
let translateAssumingInfinitePlane = true
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
In viewDidLoad, before you setup the scene add this code:
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
let panGesture = ThresholdPanGesture(target: self, action: #selector(didPan(_:)))
panGesture.delegate = self
// Add gestures to the `sceneView`.
sceneView.addGestureRecognizer(panGesture)
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
At the very end of your ViewController.swift add this code:
// MARK: - Pan Gesture Block
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
#objc
func didPan(_ gesture: ThresholdPanGesture) {
switch gesture.state {
case .began:
// Check for interaction with a new object.
if let object = objectInteracting(with: gesture, in: sceneView) {
trackedObject = object // as? VirtualObject
}
case .changed where gesture.isThresholdExceeded:
guard let object = trackedObject else { return }
let translation = gesture.translation(in: sceneView)
let currentPosition = currentTrackingPosition ?? CGPoint(sceneView.projectPoint(object.position))
// The `currentTrackingPosition` is used to update the `selectedObject` in `updateObjectToCurrentTrackingPosition()`.
currentTrackingPosition = CGPoint(x: currentPosition.x + translation.x, y: currentPosition.y + translation.y)
gesture.setTranslation(.zero, in: sceneView)
case .changed:
// Ignore changes to the pan gesture until the threshold for displacment has been exceeded.
break
case .ended:
// Update the object's anchor when the gesture ended.
guard let existingTrackedObject = trackedObject else { break }
addOrUpdateAnchor(for: existingTrackedObject)
fallthrough
default:
// Clear the current position tracking.
currentTrackingPosition = nil
trackedObject = nil
}
}
// - MARK: Object anchors
/// - Tag: AddOrUpdateAnchor
func addOrUpdateAnchor(for object: VirtualObject) {
// If the anchor is not nil, remove it from the session.
if let anchor = object.anchor {
sceneView.session.remove(anchor: anchor)
}
// Create a new anchor with the object's current transform and add it to the session
let newAnchor = ARAnchor(transform: object.simdWorldTransform)
object.anchor = newAnchor
sceneView.session.add(anchor: newAnchor)
}
private func objectInteracting(with gesture: UIGestureRecognizer, in view: ARSCNView) -> VirtualObject? {
for index in 0..<gesture.numberOfTouches {
let touchLocation = gesture.location(ofTouch: index, in: view)
// Look for an object directly under the `touchLocation`.
if let object = virtualObject(at: touchLocation) {
return object
}
}
// As a last resort look for an object under the center of the touches.
// return virtualObject(at: gesture.center(in: view))
return virtualObject(at: (gesture.view?.center)!)
}
/// Hit tests against the `sceneView` to find an object at the provided point.
func virtualObject(at point: CGPoint) -> VirtualObject? {
// let hitTestOptions: [SCNHitTestOption: Any] = [.boundingBoxOnly: true]
let hitTestResults = sceneView.hitTest(point, options: [SCNHitTestOption.categoryBitMask: 0b00000010, SCNHitTestOption.searchMode: SCNHitTestSearchMode.any.rawValue as NSNumber])
// let hitTestOptions: [SCNHitTestOption: Any] = [.boundingBoxOnly: true]
// let hitTestResults = sceneView.hitTest(point, options: hitTestOptions)
return hitTestResults.lazy.compactMap { result in
return VirtualObject.existingObjectContainingNode(result.node)
}.first
}
/**
If a drag gesture is in progress, update the tracked object's position by
converting the 2D touch location on screen (`currentTrackingPosition`) to
3D world space.
This method is called per frame (via `SCNSceneRendererDelegate` callbacks),
allowing drag gestures to move virtual objects regardless of whether one
drags a finger across the screen or moves the device through space.
- Tag: updateObjectToCurrentTrackingPosition
*/
#objc
func updateObjectToCurrentTrackingPosition() {
guard let object = trackedObject, let position = currentTrackingPosition else { return }
translate(object, basedOn: position, infinitePlane: translateAssumingInfinitePlane, allowAnimation: true)
}
/// - Tag: DragVirtualObject
func translate(_ object: VirtualObject, basedOn screenPos: CGPoint, infinitePlane: Bool, allowAnimation: Bool) {
guard let cameraTransform = sceneView.session.currentFrame?.camera.transform,
let result = smartHitTest(screenPos,
infinitePlane: infinitePlane,
objectPosition: object.simdWorldPosition,
allowedAlignments: [ARPlaneAnchor.Alignment.horizontal]) else { return }
let planeAlignment: ARPlaneAnchor.Alignment
if let planeAnchor = result.anchor as? ARPlaneAnchor {
planeAlignment = planeAnchor.alignment
} else if result.type == .estimatedHorizontalPlane {
planeAlignment = .horizontal
} else if result.type == .estimatedVerticalPlane {
planeAlignment = .vertical
} else {
return
}
/*
Plane hit test results are generally smooth. If we did *not* hit a plane,
smooth the movement to prevent large jumps.
*/
let transform = result.worldTransform
let isOnPlane = result.anchor is ARPlaneAnchor
object.setTransform(transform,
relativeTo: cameraTransform,
smoothMovement: !isOnPlane,
alignment: planeAlignment,
allowAnimation: allowAnimation)
}
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
Add some Focus Square Code
// MARK: - Focus Square (code by Apple, some by me)
func updateFocusSquare(isObjectVisible: Bool) {
if isObjectVisible {
focusSquare.hide()
} else {
focusSquare.unhide()
}
// Perform hit testing only when ARKit tracking is in a good state.
if let camera = sceneView.session.currentFrame?.camera, case .normal = camera.trackingState,
let result = smartHitTest(screenCenter) {
DispatchQueue.main.async {
self.sceneView.scene.rootNode.addChildNode(self.focusSquare)
self.focusSquare.state = .detecting(hitTestResult: result, camera: camera)
}
} else {
DispatchQueue.main.async {
self.focusSquare.state = .initializing
self.sceneView.pointOfView?.addChildNode(self.focusSquare)
}
}
}
And add some control Functions:
func hideFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: true) } } // to hide the focus square
func showFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: false) } } // to show the focus square
From the VirtualObjectARView.swift COPY! the entire function smartHitTest to the ViewController.swift (so they exist twice)
func smartHitTest(_ point: CGPoint,
infinitePlane: Bool = false,
objectPosition: float3? = nil,
allowedAlignments: [ARPlaneAnchor.Alignment] = [.horizontal, .vertical]) -> ARHitTestResult? {
// Perform the hit test.
let results = sceneView.hitTest(point, types: [.existingPlaneUsingGeometry, .estimatedVerticalPlane, .estimatedHorizontalPlane])
// 1. Check for a result on an existing plane using geometry.
if let existingPlaneUsingGeometryResult = results.first(where: { $0.type == .existingPlaneUsingGeometry }),
let planeAnchor = existingPlaneUsingGeometryResult.anchor as? ARPlaneAnchor, allowedAlignments.contains(planeAnchor.alignment) {
return existingPlaneUsingGeometryResult
}
if infinitePlane {
// 2. Check for a result on an existing plane, assuming its dimensions are infinite.
// Loop through all hits against infinite existing planes and either return the
// nearest one (vertical planes) or return the nearest one which is within 5 cm
// of the object's position.
let infinitePlaneResults = sceneView.hitTest(point, types: .existingPlane)
for infinitePlaneResult in infinitePlaneResults {
if let planeAnchor = infinitePlaneResult.anchor as? ARPlaneAnchor, allowedAlignments.contains(planeAnchor.alignment) {
if planeAnchor.alignment == .vertical {
// Return the first vertical plane hit test result.
return infinitePlaneResult
} else {
// For horizontal planes we only want to return a hit test result
// if it is close to the current object's position.
if let objectY = objectPosition?.y {
let planeY = infinitePlaneResult.worldTransform.translation.y
if objectY > planeY - 0.05 && objectY < planeY + 0.05 {
return infinitePlaneResult
}
} else {
return infinitePlaneResult
}
}
}
}
}
// 3. As a final fallback, check for a result on estimated planes.
let vResult = results.first(where: { $0.type == .estimatedVerticalPlane })
let hResult = results.first(where: { $0.type == .estimatedHorizontalPlane })
switch (allowedAlignments.contains(.horizontal), allowedAlignments.contains(.vertical)) {
case (true, false):
return hResult
case (false, true):
// Allow fallback to horizontal because we assume that objects meant for vertical placement
// (like a picture) can always be placed on a horizontal surface, too.
return vResult ?? hResult
case (true, true):
if hResult != nil && vResult != nil {
return hResult!.distance < vResult!.distance ? hResult! : vResult!
} else {
return hResult ?? vResult
}
default:
return nil
}
}
You might see some errors in the copied function regarding the hitTest. Just correct it like so:
hitTest... // which gives an Error
sceneView.hitTest... // this should correct it
Implement the renderer updateAtTime function and add this lines:
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
// For the Focus Square
if isFocusSquareEnabled { showFocusSquare() }
self.updateObjectToCurrentTrackingPosition() // *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
}
And finally add some helper functions for the Focus Square
func hideFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: true) } } // to hide the focus square
func showFocusSquare() { DispatchQueue.main.async { self.updateFocusSquare(isObjectVisible: false) } } // to show the focus square
At this point you might still see about a dozen errors and warnings in the imported files, this might occur, when doing this in Swift 5 and you have some Swift 4 files. Just let Xcode correct the errors. (Its all about renaming some code statements, Xcode knows best)
Go in VirtualObject.swift and search for this code block:
if smoothMovement {
let hitTestResultDistance = simd_length(positionOffsetFromCamera)
// Add the latest position and keep up to 10 recent distances to smooth with.
recentVirtualObjectDistances.append(hitTestResultDistance)
recentVirtualObjectDistances = Array(recentVirtualObjectDistances.suffix(10))
let averageDistance = recentVirtualObjectDistances.average!
let averagedDistancePosition = simd_normalize(positionOffsetFromCamera) * averageDistance
simdPosition = cameraWorldPosition + averagedDistancePosition
} else {
simdPosition = cameraWorldPosition + positionOffsetFromCamera
}
Outcomment or replace this entire block by this single line of code:
simdPosition = cameraWorldPosition + positionOffsetFromCamera
At this point you should be able to compile the project and run it on a device. You should see the Spaceship and a yellow focus square that should already work.
To start placing an Object, that you can drag you need some function to create a so called VirtualObject as I said in the beginning.
Use this example function to test (add it somewhere in the view controller):
override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
if focusSquare.state != .initializing {
let position = SCNVector3(focusSquare.lastPosition!)
// *** FOR OBJECT DRAGGING PAN GESTURE - APPLE ***
let testObject = VirtualObject() // give it some name, when you dont have anything to load
testObject.geometry = SCNCone(topRadius: 0.0, bottomRadius: 0.2, height: 0.5)
testObject.geometry?.firstMaterial?.diffuse.contents = UIColor.red
testObject.categoryBitMask = 0b00000010
testObject.name = "test"
testObject.castsShadow = true
testObject.position = position
sceneView.scene.rootNode.addChildNode(testObject)
}
}
Note: everything you want to drag on a plane, must be setup using VirtualObject() instead of SCNNode(). Everything else regarding the VirtualObject stays the same as SCNNode
(You can also add some common SCNNode extensions as well, like the one to load scenes by its name - useful when referencing imported models)
Have fun!

Answer 2

I added some of my ideas to Claessons's answer. I noticed some lag when dragging the node around. I found that the node cannot follow the finger's movement.
To make the node move more smoothly, I added a variable that keeps track of the node that is currently being moved, and set the position to the location of the touch.
var selectedNode: SCNNode?
Also, I set a .categoryBitMask value to specify the category of nodes that I want to edit(move). The default bit mask value is 1.
The reason why we set the category bit mask is to distinguish between different kinds of nodes, and specify those that you wish to select (to move around, etc).
enum CategoryBitMask: Int {
case categoryToSelect = 2 // 010
case otherCategoryToSelect = 4 // 100
// you can add more bit masks below . . .
}
Then, I added a UILongPressGestureRecognizer in viewDidLoad().
let longPressRecognizer = UILongPressGestureRecognizer(target: self, action: #selector(longPressed))
self.sceneView.addGestureRecognizer(longPressRecognizer)
The following is the UILongPressGestureRecognizer I used to detect a long press, which initiates the dragging of the node.
First, obtain the touch location from the recognizerView
#objc func longPressed(recognizer: UILongPressGestureRecognizer) {
guard let recognizerView = recognizer.view as? ARSCNView else { return }
let touch = recognizer.location(in: recognizerView)
The following code runs once when a long press is detected.
Here, we perform a hitTest to select the node that has been touched. Note that here, we specify a .categoryBitMask option to select only nodes of the following category: CategoryBitMask.categoryToSelect
// Runs once when long press is detected.
if recognizer.state == .began {
// perform a hitTest
let hitTestResult = self.sceneView.hitTest(touch, options: [SCNHitTestOption.categoryBitMask: CategoryBitMask.categoryToSelect])
guard let hitNode = hitTestResult.first?.node else { return }
// Set hitNode as selected
self.selectedNode = hitNode
The following code will run periodically until the user releases the finger.
Here we perform another hitTest to obtain the plane you want the node to move along.
// Runs periodically after .began
} else if recognizer.state == .changed {
// make sure a node has been selected from .began
guard let hitNode = self.selectedNode else { return }
// perform a hitTest to obtain the plane
let hitTestPlane = self.sceneView.hitTest(touch, types: .existingPlane)
guard let hitPlane = hitTestPlane.first else { return }
hitNode.position = SCNVector3(hitPlane.worldTransform.columns.3.x,
hitNode.position.y,
hitPlane.worldTransform.columns.3.z)
Make sure you deselect the node when the finger is removed from the screen.
// Runs when finger is removed from screen. Only once.
} else if recognizer.state == .ended || recognizer.state == .cancelled || recognizer.state == .failed{
guard let hitNode = self.selectedNode else { return }
// Undo selection
self.selectedNode = nil
}
}

Answer 3

Kind of late answer but I know I had some problems solving this as well. Eventually I figured out a way to do it by performing two separate hit tests whenever my gesture recognizer is called.
First, I perform a hit test for my 3d-object to detect if I'm currently pressing an object or not (as you would get results for pressing featurePoints, planes etc. if you don't specify any options). I do this by using the .categoryBitMaskvalue of SCNHitTestOption.
Keep in mind you have to assign the correct .categoryBitMask value to your object node and all it's child nodes beforehand in order for the hit test to work. I declare an enum I can use for that:
enum BodyType : Int {
case ObjectModel = 2;
}
As becomes apparent by the answer to my question about .categoryBitMask values I posted here, it is important to consider what values you assign your bitmask.
Below is the code i use in in conjunction with UILongPressGestureRecognizer in
order to select the object I'm currently pressing:
guard let recognizerView = recognizer.view as? ARSCNView else { return }
let touch = recognizer.location(in: recognizerView)
let hitTestResult = self.sceneView.hitTest(touch, options: [SCNHitTestOption.categoryBitMask: BodyType.ObjectModel.rawValue])
guard let modelNodeHit = hitTestResult.first?.node else { return }
After that I perform a 2nd hit test in order to find a plane I'm pressing on.
You can use the type .existingPlaneUsingExtent if you don't want to move your object further than the edge of a plane, or .existingPlane if you want to move your object indefinitely along a detected plane surface.
var planeHit : ARHitTestResult!
if recognizer.state == .changed {
let hitTestPlane = self.sceneView.hitTest(touch, types: .existingPlane)
guard hitTestPlane.first != nil else { return }
planeHit = hitTestPlane.first!
modelNodeHit.position = SCNVector3(planeHit.worldTransform.columns.3.x,modelNodeHit.position.y,planeHit.worldTransform.columns.3.z)
}else if recognizer.state == .ended || recognizer.state == .cancelled || recognizer.state == .failed{
modelNodeHit.position = SCNVector3(planeHit.worldTransform.columns.3.x,modelNodeHit.position.y,planeHit.worldTransform.columns.3.z)
}
I made a GitHub repo when I tried this out while also experimenting with ARAnchors. You can check it out if you want to see my method in practice, but I did not make it with the intention of anyone else using it so it's quite unfinished. Also, the development branch should support some functionality for an object with more childNodes.
EDIT: ==================================
For clarification if you want to use a .scn object instead of a regular geometry, you need to iterate through all the child nodes of the object when creating it, setting the bitmask of each child like this:
let objectModelScene = SCNScene(named:
"art.scnassets/object/object.scn")!
let objectNode = objectModelScene.rootNode.childNode(
withName: "theNameOfTheParentNodeOfTheObject", recursively: true)
objectNode.categoryBitMask = BodyType.ObjectModel.rawValue
objectNode.enumerateChildNodes { (node, _) in
node.categoryBitMask = BodyType.ObjectModel.rawValue
}
Then, in the gesture recognizer after you get a hitTestResult
let hitTestResult = self.sceneView.hitTest(touch, options: [SCNHitTestOption.categoryBitMask: BodyType.ObjectModel.rawValue])
you need to find the parent node since otherwise you might be moving the individual child node you just pressed. Do this by searching recursively upwards through the node tree of the node you just found.
guard let objectNode = getParentNodeOf(hitTestResult.first?.node) else { return }
where you declare the getParentNode-method as follows
func getParentNodeOf(_ nodeFound: SCNNode?) -> SCNNode? {
if let node = nodeFound {
if node.name == "theNameOfTheParentNodeOfTheObject" {
return node
} else if let parent = node.parent {
return getParentNodeOf(parent)
}
}
return nil
}
Then you are free to perform any operation on the objectNode, as it will be the parent node of your .scn object, meaning that any transformation applied to it will also be applied to the child nodes.

Answer 4

As #ZAY mentioned out, Apple made it quite confusing in addition they used ARRaycastQuery which only works on iOS 13 and above. Therefore, I reached to a solution that works by using the current camera orientation to calculate the translation on a plane in the world coordinates.
First, by using this snippet we are able to get the current orientation the user is facing using quaternions.
private func getOrientationYRadians()-> Float {
guard let cameraNode = arSceneView.pointOfView else { return 0 }
//Get camera orientation expressed as a quaternion
let q = cameraNode.orientation
//Calculate rotation around y-axis (heading) from quaternion and convert angle so that
//0 is along -z-axis (forward in SceneKit) and positive angle is clockwise rotation.
let alpha = Float.pi - atan2f( (2*q.y*q.w)-(2*q.x*q.z), 1-(2*pow(q.y,2))-(2*pow(q.z,2)) )
// here I convert the angle to be 0 when the user is facing +z-axis
return alpha <= Float.pi ? abs(alpha - (Float.pi)) : (3*Float.pi) - alpha
}
Handle Pan Method
private var lastPanLocation2d: CGPoint!
#objc func handlePan(panGesture: UIPanGestureRecognizer) {
let state = panGesture.state
guard state != .failed && state != .cancelled else {
return
}
let touchLocation = panGesture.location(in: self)
if (state == .began) {
lastPanLocation2d = touchLocation
}
// 200 here is a random value that controls the smoothness of the dragging effect
let deltaX = Float(touchLocation.x - lastPanLocation2d!.x)/200
let deltaY = Float(touchLocation.y - lastPanLocation2d!.y)/200
let currentYOrientationRadians = getOrientationYRadians()
// convert delta in the 2D dimensions to the 3d world space using the current rotation
let deltaX3D = (deltaY*sin(currentYOrientationRadians))+(deltaX*cos(currentYOrientationRadians))
let deltaY3D = (deltaY*cos(currentYOrientationRadians))+(-deltaX*sin(currentYOrientationRadians))
// assuming that the node is currently positioned on a plane so the y-translation will be zero
let translation = SCNVector3Make(deltaX3D, 0.0, deltaY3D)
nodeToDrag.localTranslate(by: translation)
lastPanLocation2d = touchLocation
}