ARKit 3D Head tracking in scene - ios

Question

I am using ARKit to create an augmented camera app. When the ARSession initialises, a 3d character is shown in a ARSCNView. I am trying to get the character's
head to track the ARCamera's point of view so they are always looking at the camera as the user moves to take a photo.
I've used Apple's chameleon demo, which adds a focus node that tracks the cameras point of view using SCNLookAtConstraint but I am getting
strange behaviour. The head drops to the side and rotates as the ARCamera pans. If I add a SCNTransformConstraint to restrict the
head movement to up/down/side-to-side, it stays vertical but then looks away and doesn't track.
I've tried picking the chameleon demo apart to see why mine is not working but after a few days I am stuck.
The code I am using is:
class Daisy: SCNScene, ARCharacter, CAAnimationDelegate {
// Rig for animation
private var contentRootNode: SCNNode! = SCNNode()
private var geometryRoot: SCNNode!
private var head: SCNNode!
private var leftEye: SCNNode!
private var rightEye: SCNNode!
// Head tracking properties
private var focusOfTheHead = SCNNode()
private let focusNodeBasePosition = simd_float3(0, 0.1, 0.25)
// State properties
private var modelLoaded: Bool = false
private var headIsMoving: Bool = false
private var shouldTrackCamera: Bool = false
/*
* MARK: - Init methods
*/
override init() {
super.init()
loadModel()
setupSpecialNodes()
setupConstraints()
}
/*
* MARK: - Setup methods
*/
func loadModel() {
guard let virtualObjectScene = SCNScene(named: "daisy_3.dae", inDirectory: "art.scnassets") else {
print("virtualObjectScene not intialised")
return
}
let wrapper = SCNNode()
for child in virtualObjectScene.rootNode.childNodes {
wrapper.addChildNode(child)
}
self.rootNode.addChildNode(contentRootNode)
contentRootNode.addChildNode(wrapper)
hide()
modelLoaded = true
}
private func setupSpecialNodes() {
// Assign characters rig elements to nodes
geometryRoot = self.rootNode.childNode(withName: "D_Rig", recursively: true)
head = self.rootNode.childNode(withName: "D_RigFBXASC032Head", recursively: true)
leftEye = self.rootNode.childNode(withName: "D_Eye_L", recursively: true)
rightEye = self.rootNode.childNode(withName: "D_Eye_R", recursively: true)
// Set up looking position nodes
focusOfTheHead.simdPosition = focusNodeBasePosition
geometryRoot.addChildNode(focusOfTheHead)
}
/*
* MARK: - Head animations
*/
func updateForScene(_ scene: ARSCNView) {
guard shouldTrackCamera, let pointOfView = scene.pointOfView else {
print("Not going to updateForScene")
return
}
followUserWithHead(to: pointOfView)
}
private func followUserWithHead(to pov: SCNNode) {
guard !headIsMoving else { return }
// Update the focus node to the point of views position
let target = focusOfTheHead.simdConvertPosition(pov.simdWorldPosition, to: nil)
// Slightly delay the head movement and the animate it to the new focus position
DispatchQueue.main.asyncAfter(deadline: .now() + 0.2, execute: {
let moveToTarget = SCNAction.move(to: SCNVector3(target.x, target.y, target.z), duration: 1.5)
self.headIsMoving = true
self.focusOfTheHead.runAction(moveToTarget, completionHandler: {
self.headIsMoving = false
})
})
}
private func setupConstraints() {
let headConstraint = SCNLookAtConstraint(target: focusOfTheHead)
headConstraint.isGimbalLockEnabled = true
let headRotationConstraint = SCNTransformConstraint(inWorldSpace: false) { (node, transform) -> SCNMatrix4 in
// Only track the up/down and side to side movement
var eulerX = node.presentation.eulerAngles.x
var eulerZ = node.presentation.eulerAngles.z
// Restrict the head movement so it doesn't rotate too far
if eulerX < self.rad(-90) { eulerX = self.rad(-90) }
if eulerX > self.rad(90) { eulerX = self.rad(90) }
if eulerZ < self.rad(-30) { eulerZ = self.rad(-30) }
if eulerZ > self.rad(30) { eulerZ = self.rad(30) }
let tempNode = SCNNode()
tempNode.transform = node.presentation.transform
tempNode.eulerAngles = SCNVector3(eulerX, 0, eulerZ)
return tempNode.transform
}
head?.constraints = [headConstraint, headRotationConstraint]
}
// Helper to convert degrees to radians
private func rad(_ deg: Float) -> Float {
return deg * Float.pi / 180
}
}
The model in the Scene editor is:

Answer 1

I have solved the problem I was having. There were 2 issues:
The target in followUserWithHead should have converted the simdWorldPosition for it's parent and been convert from (not to)
focusOfTheHead.parent!.simdConvertPosition(pov.simdWorldPosition, from: nil)
The local coordinates for the head node are incorrect. The z-axis should be the x-axis so when I got the focus the head movement tracking, the ear was always following the camera.
I didn't realise that the Debug View Hierarchy in Xcode will show the details of an SCNScene. This helped me to debug the scene and find where the nodes were tracking. You can export the scene as a dae and then load into SceneKit editor
Edit:
I used localFront as mnuages suggested in the comments below, which got the tracking working in the correct direction. The head did occasionally moved about though. I have put this down to the animation that was running on the model trying to apply a transform that was then changed on the next update cycle. I decided to remove the tracking from the head and use the same approach to track the eyes only.