I'm new to SceneKit coming from 2D SpriteKit and was trying to figure out how to adjust the camera so that it's at the top of the world facing down. I have the location part right, however on the rotation I'm getting stuck. If I adjust the X,YorZaxis, nothing seems to happen, however on the W axis the slightest change (even0.1` higher or lower) seems to move the camera in an unknown direction. What am I doing wrong?
cameraNode.position = SCNVector3Make(0, 10, 0)
cameraNode.rotation = SCNVector4Make(0, 0, 0, 0.5)
the rotation vector is decomposed as (x_axis, y_axis, z_axis, angle)
Setting a rotation axis with a null angle is the identity (no effective rotation). Setting an angle with a null rotation axis does not actually define a rotation.
As for why a small change of the angle has a huge effect, it's because they are expressed in radians.
A rotation of 90º around the x axis can be achieved as follows
node.rotation = SCNVector4Make(1, 0, 0, M_PI_2)
But you can also use Euler angles (see SCNNode.eulerAngles) if you find it easier:
node.eulerAngles = SCNVector3Make(M_PI_2, 0, 0)
Related
I just started learning how to use SceneKit yesterday, so I may get some stuff wrong or incorrect. I am trying to make my cameraNode look at a SCNVector3 point in the scene.
I am trying to make my app available to people below iOS 11.0. However, the look(at:) function is only for iOS 11.0+.
Here is my function where I initialise the camera:
func initCamera() {
cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.position = SCNVector3(5, 12, 10)
if #available(iOS 11.0, *) {
cameraNode.look(at: SCNVector3(0, 5, 0)) // Calculate the look angle
} else {
// How can I calculate the orientation? <-----------
}
print(cameraNode.rotation) // Prints: SCNVector4(x: -0.7600127, y: 0.62465125, z: 0.17941462, w: 0.7226559)
gameScene.rootNode.addChildNode(cameraNode)
}
The orientation of SCNVector4(x: -0.7600127, y: 0.62465125, z: 0.17941462, w: 0.7226559) in degrees is x: -43.5, y: 35.8, z: 10.3, and I don't understand w. (Also, why isn't z = 0? I thought z was the roll...?)
Here is my workings out for recreating what I thought the Y-angle should be:
So I worked it out to be 63.4 degrees, but the returned rotation shows that it should be 35.8 degrees. Is there something wrong with my calculations, do I not fully understand SCNVector4, or is there another method to do this?
I looked at Explaining in Detail the ScnVector4 method for what SCNVector4 is, but I still don't really understand what w is for. It says that w is the 'angle of rotation' which I thought was what I thought X, Y & Z were for.
If you have any questions, please ask!
Although #rickster has given the explanations of the properties of the node, I have figured out a method to rotate the node to look at a point using maths (trigonometry).
Here is my code:
// Extension for Float
extension Float {
/// Convert degrees to radians
func asRadians() -> Float {
return self * Float.pi / 180
}
}
and also:
// Extension for SCNNode
extension SCNNode {
/// Look at a SCNVector3 point
func lookAt(_ point: SCNVector3) {
// Find change in positions
let changeX = self.position.x - point.x // Change in X position
let changeY = self.position.y - point.y // Change in Y position
let changeZ = self.position.z - point.z // Change in Z position
// Calculate the X and Y angles
let angleX = atan2(changeZ, changeY) * (changeZ > 0 ? -1 : 1)
let angleY = atan2(changeZ, changeX)
// Calculate the X and Y rotations
let xRot = Float(-90).asRadians() - angleX // X rotation
let yRot = Float(90).asRadians() - angleY // Y rotation
self.eulerAngles = SCNVector3(CGFloat(xRot), CGFloat(yRot), 0) // Rotate
}
}
And you call the function using:
cameraNode.lookAt(SCNVector3(0, 5, 0))
Hope this helps people in the future!
There are three ways to express a 3D rotation in SceneKit:
What you're doing on paper is calculating separate angles around the x, y, and z axes. These are called Euler angles, or pitch, yaw, and roll. You might get results that more resemble your hand-calculations if you use eulerAngles or simdEulerAngles instead of `rotation. (Or you might not, because one of the difficulties of an Euler-angle system is that you have to apply each of those three rotations in the correct order.)
simdRotation or rotation uses a four-component vector (float4 or SCNVector4) to express an axis-angle representation of the rotation. This relies on a bit of math that isn't obvious for many newcomers to 3D graphics: the result of any sequence of rotations around different axes can be minimally expressed as a single rotation around a new axis.
For example, a rotation of π/2 radians (90°) around the z-axis (0,0,1) followed by a rotation of π/2 around the y-axis (0,1,0) has the same result as a rotation of 2π/3 around the axis (-1/√3, 1/√3, 1/√3).
This is where you're getting confused about the x, y, z, and w components of a SceneKit rotation vector — the first three components are lengths, expressing a 3D vector, and the fourth is a rotation in radians around that vector.
Quaternions are another way to express 3D rotation (and one that's even further off the beaten path for those of us with the formal math education common to undergraduate computer science curricula, but not crazy advanced, either). These have lots of great features for 3D graphics, like being easy to compose and interpolate between. In SceneKit, the simdOrientation or orientation property lets you work with a node's rotation as a quaternion.
Explaining how quaternions work is too much for one SO answer, but the practical upshot is this: if you're working with a good vector math library (like the SIMD library built into iOS 9 and later), you can basically treat them as opaque — just convert from whichever other rotation representation is easiest for you, and reap the benefits.
I'm trying to estimate my device position related to a QR code in space. I'm using ARKit and the Vision framework, both introduced in iOS11, but the answer to this question probably doesn't depend on them.
With the Vision framework, I'm able to get the rectangle that bounds a QR code in the camera frame. I'd like to match this rectangle to the device translation and rotation necessary to transform the QR code from a standard position.
For instance if I observe the frame:
* *
B
C
A
D
* *
while if I was 1m away from the QR code, centered on it, and assuming the QR code has a side of 10cm I'd see:
* *
A0 B0
D0 C0
* *
what has been my device transformation between those two frames? I understand that an exact result might not be possible, because maybe the observed QR code is slightly non planar and we're trying to estimate an affine transform on something that is not one perfectly.
I guess the sceneView.pointOfView?.camera?.projectionTransform is more helpful than the sceneView.pointOfView?.camera?.projectionTransform?.camera.projectionMatrix since the later already takes into account transform inferred from the ARKit that I'm not interested into for this problem.
How would I fill
func get transform(
qrCodeRectangle: VNBarcodeObservation,
cameraTransform: SCNMatrix4) {
// qrCodeRectangle.topLeft etc is the position in [0, 1] * [0, 1] of A0
// expected real world position of the QR code in a referential coordinate system
let a0 = SCNVector3(x: -0.05, y: 0.05, z: 1)
let b0 = SCNVector3(x: 0.05, y: 0.05, z: 1)
let c0 = SCNVector3(x: 0.05, y: -0.05, z: 1)
let d0 = SCNVector3(x: -0.05, y: -0.05, z: 1)
let A0, B0, C0, D0 = ?? // CGPoints representing position in
// camera frame for camera in 0, 0, 0 facing Z+
// then get transform from 0, 0, 0 to current position/rotation that sees
// a0, b0, c0, d0 through the camera as qrCodeRectangle
}
====Edit====
After trying number of things, I ended up going for camera pose estimation using openCV projection and perspective solver, solvePnP This gives me a rotation and translation that should represent the camera pose in the QR code referential. However when using those values and placing objects corresponding to the inverse transformation, where the QR code should be in the camera space, I get inaccurate shifted values, and I'm not able to get the rotation to work:
// some flavor of pseudo code below
func renderer(_ sender: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let currentFrame = sceneView.session.currentFrame, let pov = sceneView.pointOfView else { return }
let intrisics = currentFrame.camera.intrinsics
let QRCornerCoordinatesInQRRef = [(-0.05, -0.05, 0), (0.05, -0.05, 0), (-0.05, 0.05, 0), (0.05, 0.05, 0)]
// uses VNDetectBarcodesRequest to find a QR code and returns a bounding rectangle
guard let qr = findQRCode(in: currentFrame) else { return }
let imageSize = CGSize(
width: CVPixelBufferGetWidth(currentFrame.capturedImage),
height: CVPixelBufferGetHeight(currentFrame.capturedImage)
)
let observations = [
qr.bottomLeft,
qr.bottomRight,
qr.topLeft,
qr.topRight,
].map({ (imageSize.height * (1 - $0.y), imageSize.width * $0.x) })
// image and SceneKit coordinated are not the same
// replacing this by:
// (imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
// weirdly fixes an issue, see below
let rotation, translation = openCV.solvePnP(QRCornerCoordinatesInQRRef, observations, intrisics)
// calls openCV solvePnP and get the results
let positionInCameraRef = -rotation.inverted * translation
let node = SCNNode(geometry: someGeometry)
pov.addChildNode(node)
node.position = translation
node.orientation = rotation.asQuaternion
}
Here is the output:
where A, B, C, D are the QR code corners in the order they are passed to the program.
The predicted origin stays in place when the phone rotates, but it's shifted from where it should be. Surprisingly, if I shift the observations values, I'm able to correct this:
// (imageSize.height * (1 - $0.y), imageSize.width * $0.x)
// replaced by:
(imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
and now the predicted origin stays robustly in place. However I don't understand where the shift values come from.
Finally, I've tried to get an orientation fixed relatively to the QR code referential:
var n = SCNNode(geometry: redGeometry)
node.addChildNode(n)
n.position = SCNVector3(0.1, 0, 0)
n = SCNNode(geometry: blueGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0.1, 0)
n = SCNNode(geometry: greenGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0, 0.1)
The orientation is fine when I look at the QR code straight, but then it shifts by something that seems to be related to the phone rotation:
Outstanding questions I have are:
How do I solve the rotation?
where do the position shift values come from?
What simple relationship do rotation, translation, QRCornerCoordinatesInQRRef, observations, intrisics verify? Is it O ~ K^-1 * (R_3x2 | T) Q ? Because if so that's off by a few order of magnitude.
If that's helpful, here are a few numerical values:
Intrisics matrix
Mat 3x3
1090.318, 0.000, 618.661
0.000, 1090.318, 359.616
0.000, 0.000, 1.000
imageSize
1280.0, 720.0
screenSize
414.0, 736.0
==== Edit2 ====
I've noticed that the rotation works fine when the phone stays horizontally parallel to the QR code (ie the rotation matrix is [[a, 0, b], [0, 1, 0], [c, 0, d]]), no matter what the actual QR code orientation is:
Other rotation don't work.
Coordinate systems' correspondence
Take into consideration that Vision/CoreML coordinate system doesn't correspond to ARKit/SceneKit coordinate system. For details look at this post.
Rotation's direction
I suppose the problem is not in matrix. It's in vertices placement. For tracking 2D images you need to place ABCD vertices counter-clockwise (the starting point is A vertex located in imaginary origin x:0, y:0). I think Apple Documentation on VNRectangleObservation class (info about projected rectangular regions detected by an image analysis request) is vague. You placed your vertices in the same order as is in official documentation:
var bottomLeft: CGPoint
var bottomRight: CGPoint
var topLeft: CGPoint
var topRight: CGPoint
But they need to be placed the same way like positive rotation direction (about Z axis) occurs in Cartesian coordinates system:
World Coordinate Space in ARKit (as well as in SceneKit and Vision) always follows a right-handed convention (the positive Y axis points upward, the positive Z axis points toward the viewer and the positive X axis points toward the viewer's right), but is oriented based on your session's configuration. Camera works in Local Coordinate Space.
Rotation direction about any axis is positive (Counter-Clockwise) and negative (Clockwise). For tracking in ARKit and Vision it's critically important.
The order of rotation also makes sense. ARKit, as well as SceneKit, applies rotation relative to the node’s pivot property in the reverse order of the components: first roll (about Z axis), then yaw (about Y axis), then pitch (about X axis). So the rotation order is ZYX.
Math (Trig.):
Notes: the bottom is l (the QR code length), the left angle is k, and the top angle is i (the camera)
I am moving from scene kit api to metal api. In have to do transforms on a node.
In scene kit i do this by simply setting
let camera = SCNCamera()
camera.zFar = 10000
camera.zNear = 0.1
camera.xFov = 93.299
camera.yFov = 77.65614
let cameraNode = SCNNode()
cameraNode.camera = camera
cameraNode.position = SCNVector3Make(0, 10, 0)
cameraNode.eulerAngles = SCNVector3Make(-2.degreesToRadians, 0, 0)
cameraNode.name = "CameraNode"
scene.rootNode.addChildNode(cameraNode)
let plane = SCNPlane(width:1, height:1)
plane.firstMaterial?.diffuse.contents = UIColor.blueColor()
plane.firstMaterial?.doubleSided = true
let node = SCNNode(geometry: plane)
node.position = SCNVector3Make(0, 0, 55)
node.eulerAngles = SCNVector3Make(90.degreesToRadians, -47.degreesToRadians, 0)
node.scale = SCNVector3Make(169, 169, 169)
scene.rootNode.addChildNode(node)
but in metal there is no way for setting eulerAngles and creating the projection transform based on xFieldOfView and yFieldOfView. As of now i am using GLKIT of all matrix calculations
Model Matrix - here i am doing the rotation in ZYX order (because thats how the scene kit does for the provided euler angle) (using node properties)
var modelMatrix = GLKMatrix4MakeZRotation(0)
modelMatrix = GLKMatrix4RotateY(modelMatrix, -47.degreesToRadians)
modelMatrix = GLKMatrix4RotateX(modelMatrix, 90.degreesToRadians)
modelMatrix = GLKMatrix4Translate(modelMatrix, 0, 0, 100)
modelMatrix = GLKMatrix4Scale(modelMatrix, 169, 169, 169)
View Matrix (using camera node properties)
var viewMatrix = GLKMatrix4MakeTranslation(0, 10, -0)
viewMatrix = GLKMatrix4RotateZ(viewMatrix, 0)
viewMatrix = GLKMatrix4RotateY(viewMatrix, 0)
viewMatrix = GLKMatrix4RotateX(viewMatrix,-2.degreesToRadians)
Perspective Transform (Using camera properties)
let projectionMatrix = GLKMatrix4MakePerspective(77.65614, Float(1024/682.66666666666674), 0.1, 10000)
here 1024 and 682.66666666666674 is my views's height and width.
The scene kit's output and metal output is not the same. for some reason the plane is always rendered on top half of the view in metal kit but in scene kit it is rendered on the bottom half (which is what i wanted) and adjusting the yFieldOfView in metal changes whether the plane is rendered in top half or bottom half and also the scaling and rotation of the output plane in metal view is different from the scene kit.
My Question is :
1. SceneKit coordinate system uses +z in front and -z at back. Whereas Metal uses -z in front and +z at back.
In metal, will simply multiplying all the z rotation and translation values by -1 fix it?
How to do euler angle in metal. I went through couple of links and stack overflow answers and my understanding is that euler angle is the rotation on each axis and multiplied in specific order. Here i am multiplying in zyx order because thats how scene kit uses it.
How do you generate perspectiveMatrix with xFieldOfView and yFieldOfView. I am using GLKMatrix for all matrix transformations. It doesn't have any function like scene kit does. I am passing viewWidth/viewHeight as a value to aspect. How does one generate aspect from xFieldOfView and yFieldOfView. Is that as simple as this
aspect = xFieldOfView/yFieldOfView or is it something different. There is this library https://github.com/nicklockwood/VectorMath/blob/master/VectorMath/VectorMath.swift
which calculates the aspect using xFieldOfView/yFieldOfView
when i am creating the plane in scene kit i am setting the size of plane as follows
let plane = SCNPlane(width:1, height:1)
Is the size matter in metal? In metal the clip space is from -1 to 1 along all axis. And In metal u can set the size of the plane by generating the plane vertices accordingly i have tried generating the vertices from -1 to 1 (here the width and height is 2.) and -0.5 to 0.5 (here the width and height is 1). But it doesn't give me right result.
In what order does the scene kit does the rotation, scaling and translation. I couldn't find any information related to this in apple documentation.
Basically i want to port my scene kit code to metal with same output. Any help greatly appreciated.
Thanks
What orthographicProjection does one have to use to be able to make a 2D application in SceneKit with 1:1 SceneKit points to screen points/pixels ratio?
Example:
I want to position something at (200, 200) on the screen and I want to use a SCNVector with (200, 200, 0) for it. What orthographicProjection do I need for this?
If you want an orthographic projection where a unit of scene space corresponds to a point of screen space, you need a projection where the left clipping plane is at zero and the right clipping plane is at whatever the screen's width in points is. (Ditto for top/bottom, and near/far doesn't matter so long as you keep objects within whatever near/far you set up.)
For this it's probably easiest to set up your own projection matrix, rather than working out what orthographicScale and camera position correspond to the dimensions you need:
GLKMatrix4 mat = GLKMatrix4MakeOrtho(0, self.view.bounds.size.width,
0, self.view.bounds.size.height,
1, 100); // z range arbitrary
cameraNode.camera.projectionTransform = SCNMatrix4FromGLKMatrix4(mat);
// still need to position the camera for its direction & z range
cameraNode.position = SCNVector3Make(0, 0, 50);
Trying to use CoreMotion to correctly rotate a SceneKit camera. The scene I've built is done rather simple ... all I do is create a bunch of boxes, distributed in an area, and the camera just points down the Z axis.
Unfortunately, the data coming back from device motion doesn't seem to relate to the device's physical position and orientation in any way. It just seems to meander randomly.
As suggested in this SO post, I'm passing the attitude's quaternion directly to the camera node's orientation property.
Am I misunderstanding what data core motion is giving me here? shouldn't the attitude reflect the device's physical orientation? or is it incremental movement and I should be building upon the prior orientation?
This snippet here might help you:
var motionManager = CMMotionManager()
motionManager?.deviceMotionUpdateInterval = 1.0 / 60.0
motionManager?.startDeviceMotionUpdatesToQueue(
NSOperationQueue.mainQueue(),
withHandler: { (motion: CMDeviceMotion!, error: NSError!) -> Void in
let currentAttitude = motion.attitude
var roll = Float(currentAttitude.roll) + (0.5*Float(M_PI))
var yaw = Float(currentAttitude.yaw)
var pitch = Float(currentAttitude.pitch)
self.cameraNode.eulerAngles = SCNVector3(
x: -roll,
y: yaw,
z: -pitch)
})
This setting is for the device in landscape right. You can play around with different orientations by changing the + and -
Import CoreMotion.
For anyone who stumbles on this, here's a more complete answer so you can understand the need for negations and pi/2 shifts. You first need to know your reference frame. Spherical coordinate systems define points as vectors angled away from the z- and x- axes. For the earth, let's define the z-axis as the line from the earth's center to the north pole and the x-axis as the line from the center through the equator at the prime meridian (mid-Africa in the Atlantic).
For (lat, lon, alt), we can then define roll and yaw around the z- and y- axes in radians:
let roll = lon * Float.pi / 180
let yaw = (90 - lat) * Float.pi / 180
I'm pairing roll, pitch, and yaw with z, x, and y, respectively, as defined for eulerAngles.
The extra 90 degrees accounts for the north pole being at 90 degrees latitude instead of zero.
To place my SCNCamera on the globe, I used two SCNNodes: an 'arm' node and the camera node:
let scnCamera = SCNNode()
scnCamera.camera = SCNCamera()
scnCamera.position = SCNVector3(x: 0.0, y: 0.0, z: alt + EARTH_RADIUS)
let scnCameraArm = SCNNode()
scnCameraArm?.position = SCNVector3(x: 0, y: 0, z: 0)
scnCameraArm?.addChildNode(scnCamera)
The arm is positioned at the center of the earth, and the camera is place at alt + EARTH_RADIUS away, i.e. the camera is now at the north pole. To move the camera on every location update, we can now just rotate the arm node with new roll and yaw values:
scnCameraArm.eulerAngles.z = roll
scnCameraArm.eulerAngles.y = yaw
Without changing the camera's orientation, it's virtual lens is always facing the ground and it's virtual 'up' direction is pointed westward.
To change the virtual camera's orientation, the CMMotion callback returns a CMAttitude with roll, pitch, and yaw values relative to a different z- and x- axis reference of your choosing. The magnetometer-based ones use a z-axis pointed away from gravity and an x-axis pointed at the north pole. So a phone with zero pitch, roll, and yaw, would have its screen facing away from gravity, it's back camera pointed at the ground, and its right side of portrait mode facing north. Notice that this orientation is relative to gravity, not to the phone's portrait/landscape mode (which is also relative to gravity). So portrait/landscape is irrelevant.
If you imagine the phone's camera in this orientation near the north pole on the prime meridian, you'll notice that the CMMotion reference is in a different orientation than the virtual camera (SCNCamera). Both cameras are facing the ground, but their respective y-axes (and x) are 180 degrees apart. To line them up, we need to spin one around its respective z-axis, i.e. add/subtract 180 degrees to the roll ...or, since they're expressed in radians, negate them for the same effect.
Also, as far as I can tell, CMAttitude doesn't explicitly document that its roll value means a rotation about the z-axis coming out of the phone's screen, and from experimenting, it seems that attitude.roll and attitude.yaw have opposite definitions than defined in eulerAngles, but maybe this is an artifact of the order that the rotational transformations are applied in virtual space with eulerAngles (?). Anyway, the callback:
motionManager?.startDeviceMotionUpdates(using: .xTrueNorthZVertical, to: OperationQueue.main, withHandler: { (motion: CMDeviceMotion?, err: Error?) in
guard let m = motion else { return }
scnCamera.eulerAngles.z = Float(m.attitude.yaw - Double.pi)
scnCamera.eulerAngles.x = Float(m.attitude.pitch)
scnCamera.eulerAngles.y = Float(m.attitude.roll)
})
You can also start with a different reference frame for your virtual camera, e.g. z-axis pointing through the prime meridian at the equator and x-axis pointing through the north pole (i.e. the CMMotion reference), but you'll still need to invert the longitude somewhere.
With this set up, you can build a scene heavily reliant on GPS locations pretty easily.