Calc reflection/re-bounce Vector in SceneKit Swift - ios

I try to calculate the bounce-vector or reflection-vector to a given direction at a specific intersection point/surface in 3D SceneKit space within a AR Session.
To do this, I send out a hittest from the exact center of the screen straight forward. There is i.Ex a cube positioned let’s say 2 meter in front of me. Now I’d like to continue this hittest in the logical re-bounce/reflection direction, just as a light-ray on a mirror would do. Of course the hittest is ended at its intersection point, but from there I would like to draw like a line or small and long SCNTube node to visualise the direction in which this hittest would continue if it was reflected by one of the faces of the cube. And this from any particular direction.
Lets say, I have the direction vector in which I send the hittest. I also have the intersection point given by the hittest result. And I have the normal of the surface at the intersection point.
Regarding to some Answers I found about this issue on Linear Algebra Forums:
https://math.stackexchange.com/questions/2235997/reflecting-ray-on-triangle-in-3d-space
https://math.stackexchange.com/questions/13261/how-to-get-a-reflection-vector
the following Formula should do, and this in 3D space:
(and it should give me the re-bounce/reflection vector)
r = d − 2(d⋅n)n
(where d⋅n is the dot product, and n must be normalised. r is the reflection vector.)
I tried to make a kind of Swift implementation of that resulting in nonsense. Here is my code:
let location: CGPoint = screenCenter
let hits = self.sceneView.hitTest(location, options: [SCNHitTestOption.categoryBitMask: NodeCategory.catCube.rawValue, SCNHitTestOption.searchMode: SCNHitTestSearchMode.any.rawValue as NSNumber])
if !hits.isEmpty {
print("we have a hittest")
let d = currentDirection
let p = hits.first?.worldCoordinates // Hit Location
let n = hits.first?.worldNormal // Normal of Hit Location
print("D = \(d)")
print("P = \(p)")
print("N = \(n)")
// r = d - 2*(d*n).normalized // the Formula
let r : SCNVector3 = d - (d.crossProduct(n!).crossProduct(d.crossProduct(n!))).normalized
// let r : SCNVector3 = d - (2 * d.crossProduct(n!).normalized) // I also tried this, but that gives me errors in Xcode
print("R = \(r)")
// This Function should setup then the node aligned to that new vector
setupRay(position: p!, euler: r)
}
All this results in nonsense. I get the following console output:
we are in gesture TAP recognizer
we have a hittest
D = SCNVector3(x: -0.29870644, y: 0.5494926, z: -0.7802771)
P = Optional(__C.SCNVector3(x: -0.111141175, y: 0.034069262, z: -0.62390435))
N = Optional(__C.SCNVector3(x: 2.672451e-08, y: 1.0, z: 5.3277716e-08))
R = SCNVector3(x: nan, y: nan, z: nan)
My Euler Angle: SCNVector3(x: nan, y: nan, z: nan)
(D is the direction of the hittest, P is the Point of intersetion, N is the Normal at the Point of intersection, R should be the reflection Vector but is always just nan, not a number)
I also tried the extension dotProduct instead of crossProduct, but dotProduct gives me a Float value, which I cannot calc with a SCNVector3
How can I calculate this re-bounce vector and align a SCNNode facing in that direction (with the Pivot at the start Point, the point of intersection from the hittest)
What am I doing wrong? Can anyone show me a working Swift implementation of that calculation?
Any Help would be so helpful. (Linear Algebra belongs not to my powers)
PS: I use standard SCNVector 3 math extensions as available from GitHub

Finally this Solution should work as far as I can say for iOS 12 (successfully tested). It gives you the reflection Vector from any surface and any Point of view.
let location: CGPoint = screenCenter
let hits = self.sceneView.hitTest(location, options: [SCNHitTestOption.categoryBitMask: NodeCategory.catCube.rawValue, SCNHitTestOption.searchMode: SCNHitTestSearchMode.any.rawValue as NSNumber])
if let hitResult = hits.first {
let direction = normalize(double3(sceneView.pointOfView!.worldFront))
// let reflectedDirection = reflect(direction, n: double3(hitResult.worldNormal))
let reflectedDirection = simd_reflect(direction, double3(hitResult.worldNormal))
print(reflectedDirection)
// Use the Result for whatever purpose
setupRay(position: hitResult.worldCoordinates, reflectDirection: SCNVector3(reflectedDirection))
}

the SIMD library is really useful for thing like that:
if let hitResult = hitResults.first {
let direction = normalize(scnView.pointOfView!.simdPosition - hitResult.simdWorldCoordinates)
let reflectedDirection = reflect(direction, n: hitResult.simdWorldNormal)
print(reflectedDirection)
}

Related

How tap by x and y coordinates iOS?

For Android world, we could tap any locations in by:
val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
device.click(x, y)
And the x and y above could be get from Pixelmator like:
For example, the coordinate of key a should be x=100 and y=1800 (Ruler from Pixelmator).
According to tapCoordinate , we might do something similar in iOS:
func tapCoordinate(at xCoordinate: Double, and yCoordinate: Double) {
let normalized = app.coordinate(withNormalizedOffset: CGVector(dx: 0, dy: 0))
let coordinate = normalized.withOffset(CGVector(dx: xCoordinate, dy: yCoordinate))
coordinate.tap()
}
But it didn't work as expected, I was wondering if maybe we could tap some points by the x and y of the global screen?
Your question doesn't make a lot of sense. The function debugDescription() generates a string from various data types, suitable for logging to the console. iOS doesn't "calculate the coordinates from debugDescription".
You don't tell us what you are logging. You should edit your question to tell us what it is that you are logging, show the code that captures the value, and also show the print statement that logs it.
Based on the format of the output, it looks like you are logging a rectangle (CGRect data type.)
A CGRect is in the form (origin, size), where origin is a CGPoint and size is a CGSize.
It sounds like tapCoordinate is giving your the center of the rectangle, which would be:
x = rect.origin.x + size.width/2
y = rect.origin.y + size.width/2
That gives
x = 23+41/2 = 43.5
y = 573+49/2 = 597.5
Assuming your x value is a typo and should be 33, that is quite close to the values you give in your question.

How to calculate distance from the centering point between two eyes to the camera using ARCore + SceneKit?

I'm using ARCore + SceneKit (Swift language) to calculate the distance from the centering point between two eyes to the camera.
I determine the coordinates of the camera:
let cameraPos = sceneView.pointOfView?.position
The coordinates of the left eye and right eye:
let buffer = face.mesh.vertices
let left = buffer[LF]
let right = buffer[RT]
NOTE:
LF and RT is defined base on: https://github.com/ManuelTS/augmentedFaceMeshIndices
LF = 159 is the index that contain the Vector3 condinate of the Left eye
RT = 386 is the index that contain the Vector3 condinate of the Right eye
Compute the centering point (in SCNVector3):
let center = SCNVector3(x: (left.x - right.x) * 0.5,
y: (left.y - right.y) * 0.5,
z: (left.z - right.z) * 0.5)
Finally, I calculate the distance:
let distance = distance(start: cameraPos!, end: center)
distance is defined as:
func distance(start: SCNVector3, end: SCNVector3) -> Float {
let dx = start.x - end.x
let dy = start.y - end.y
let dz = start.z - end.z
let distance = sqrt(dx * dx + dy * dy + dz * dz)
return round(distance * 100 * 10) / 10.0
}
Runtime result is incorrect.
Actual distance: ~20 cm
In-app distance: ~3 cm
Can someone tell me where the problem lies, even another solution?
Thanks.
Assuming center is the midpoint between the eyes, then shouldn't the formula be:
Midpoint:
(x1, y1, z1) and (x2, y2, z2) is (x1+x2 )/2,(y1+y2 )/2,(z1+z2 )/2.
Edit: Taking a guess here, but...
Example: So that a projectile will actually launch from a turret with a long barrel cannon exactly where the barrel is rotated to at the time of firing, you have to calculate that position at the end of the tube as it relates to the position of the node that the barrel is attached to, otherwise the shot will not look like it came from the right spot.
Requires a little imagination, but this is your face moving around = turret is moving around. I "think" that's what's happening to your math. I don't think you are getting the right LF/RF positions because you didn't mention converting the point. The link you sent [The face mesh consists of hundreds of vertices that make up the face, and is defined relative to the center pose.] Relative to the center pose - I'm pretty sure that means you have to convert LF with relation to the center to get the real position.
// Convert position something like this:
let REAL_LF = gNodes.gameNodes.convertPosition(LF.presentation.position, from: POSE_POSITION)
convertPosition(_:to:)
Converts a position from the node’s local coordinate space to that of another node

How to point the camera towards a SCNVector3 point below iOS 11

I just started learning how to use SceneKit yesterday, so I may get some stuff wrong or incorrect. I am trying to make my cameraNode look at a SCNVector3 point in the scene.
I am trying to make my app available to people below iOS 11.0. However, the look(at:) function is only for iOS 11.0+.
Here is my function where I initialise the camera:
func initCamera() {
cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.position = SCNVector3(5, 12, 10)
if #available(iOS 11.0, *) {
cameraNode.look(at: SCNVector3(0, 5, 0)) // Calculate the look angle
} else {
// How can I calculate the orientation? <-----------
}
print(cameraNode.rotation) // Prints: SCNVector4(x: -0.7600127, y: 0.62465125, z: 0.17941462, w: 0.7226559)
gameScene.rootNode.addChildNode(cameraNode)
}
The orientation of SCNVector4(x: -0.7600127, y: 0.62465125, z: 0.17941462, w: 0.7226559) in degrees is x: -43.5, y: 35.8, z: 10.3, and I don't understand w. (Also, why isn't z = 0? I thought z was the roll...?)
Here is my workings out for recreating what I thought the Y-angle should be:
So I worked it out to be 63.4 degrees, but the returned rotation shows that it should be 35.8 degrees. Is there something wrong with my calculations, do I not fully understand SCNVector4, or is there another method to do this?
I looked at Explaining in Detail the ScnVector4 method for what SCNVector4 is, but I still don't really understand what w is for. It says that w is the 'angle of rotation' which I thought was what I thought X, Y & Z were for.
If you have any questions, please ask!
Although #rickster has given the explanations of the properties of the node, I have figured out a method to rotate the node to look at a point using maths (trigonometry).
Here is my code:
// Extension for Float
extension Float {
/// Convert degrees to radians
func asRadians() -> Float {
return self * Float.pi / 180
}
}
and also:
// Extension for SCNNode
extension SCNNode {
/// Look at a SCNVector3 point
func lookAt(_ point: SCNVector3) {
// Find change in positions
let changeX = self.position.x - point.x // Change in X position
let changeY = self.position.y - point.y // Change in Y position
let changeZ = self.position.z - point.z // Change in Z position
// Calculate the X and Y angles
let angleX = atan2(changeZ, changeY) * (changeZ > 0 ? -1 : 1)
let angleY = atan2(changeZ, changeX)
// Calculate the X and Y rotations
let xRot = Float(-90).asRadians() - angleX // X rotation
let yRot = Float(90).asRadians() - angleY // Y rotation
self.eulerAngles = SCNVector3(CGFloat(xRot), CGFloat(yRot), 0) // Rotate
}
}
And you call the function using:
cameraNode.lookAt(SCNVector3(0, 5, 0))
Hope this helps people in the future!
There are three ways to express a 3D rotation in SceneKit:
What you're doing on paper is calculating separate angles around the x, y, and z axes. These are called Euler angles, or pitch, yaw, and roll. You might get results that more resemble your hand-calculations if you use eulerAngles or simdEulerAngles instead of `rotation. (Or you might not, because one of the difficulties of an Euler-angle system is that you have to apply each of those three rotations in the correct order.)
simdRotation or rotation uses a four-component vector (float4 or SCNVector4) to express an axis-angle representation of the rotation. This relies on a bit of math that isn't obvious for many newcomers to 3D graphics: the result of any sequence of rotations around different axes can be minimally expressed as a single rotation around a new axis.
For example, a rotation of π/2 radians (90°) around the z-axis (0,0,1) followed by a rotation of π/2 around the y-axis (0,1,0) has the same result as a rotation of 2π/3 around the axis (-1/√3, 1/√3, 1/√3).
This is where you're getting confused about the x, y, z, and w components of a SceneKit rotation vector — the first three components are lengths, expressing a 3D vector, and the fourth is a rotation in radians around that vector.
Quaternions are another way to express 3D rotation (and one that's even further off the beaten path for those of us with the formal math education common to undergraduate computer science curricula, but not crazy advanced, either). These have lots of great features for 3D graphics, like being easy to compose and interpolate between. In SceneKit, the simdOrientation or orientation property lets you work with a node's rotation as a quaternion.
Explaining how quaternions work is too much for one SO answer, but the practical upshot is this: if you're working with a good vector math library (like the SIMD library built into iOS 9 and later), you can basically treat them as opaque — just convert from whichever other rotation representation is easiest for you, and reap the benefits.

How do I rotate a 3d vector with a 4x4 matrix transform or euler angles?

I have a 3d vector I'm applying as a physics force:
let force = SCNVector3(x: 0, y: 0, z: -5)
node.physicsBody?.applyForce(force, asImpulse: true)
I need to rotate the force based on the mobile device's position which is available to me as a 4x4 matrix transform or euler angles.
var transform :matrix_float4x4 - The position and orientation of the camera in world coordinate space.
var eulerAngles :vector_float3 - The orientation of the camera, expressed as roll, pitch, and yaw values.
I think this is more of a fundamental 3d graphics question, but the application of this is a Swift based iOS app using SceneKit and ARKit.
There are some utilities available to me in the SceneKit and simd libraries. Unfortunately my naive attempts to do things like simd_mul(force, currentFrame.camera.transform) are failing me.
#orangenkopf provided a great answer that helped me come up with this:
let force = simd_make_float4(0, 0, -5, 0)
let rotatedForce = simd_mul(currentFrame.camera.transform, force)
let vectorForce = SCNVector3(x:rotatedForce.x, y:rotatedForce.y, z:rotatedForce.z)
node.physicsBody?.applyForce(vectorForce, asImpulse: true)
Your idea is right. You need to multiply the transform and the direction.
I can't find any documentation on simd_mul. But i suspect you have at least one of the following problems:
simd_mul applies both the rotation and the translation of the transform
The transform of the camera is in world coordinate space. Depending your node hierachy this can result in a direction that is way off.
SceneKit does not provide much linear algebra functions, so we have to build our own:
extension SCNMatrix4 {
static public func *(left: SCNMatrix4, right: SCNVector4) -> SCNVector4 {
let x = left.m11*right.x + left.m21*right.y + left.m31*right.z + left.m41*right.w
let y = left.m12*right.x + left.m22*right.y + left.m32*right.z + left.m42*right.w
let z = left.m13*right.x + left.m23*right.y + left.m33*right.z + left.m43*right.w
let w = left.m14*right.x + left.m24*right.y + left.m43*right.z + left.m44*right.w
return SCNVector4(x: x, y: y, z: z, w: w)
}
}
extension SCNVector4 {
public func to3() -> SCNVector3 {
return SCNVector3(self.x , self.y, self.z)
}
}
Now do the following:
Convert the camera transform to the nodes local coordinate system
Create the force as a 4d vector, set the fourth element to 0 to ignore the translation
Multiply the transform and the vector
// Convert the tranform to a SCNMatrix4
let transform = SCNMatrix4FromMat4(currentFrame.camera.transform)
// Convert the matrix to the nodes coordinate space
let localTransform = node.convertTransform(transform, from: nil)
let force = SCNVector4(0, 0, -5, 0)
let rotatedForce = (localTransform * force).to3()
node.physicsBody?.applyForce(rotatedForce, asImpulse: true)

iOS revert camera projection

I'm trying to estimate my device position related to a QR code in space. I'm using ARKit and the Vision framework, both introduced in iOS11, but the answer to this question probably doesn't depend on them.
With the Vision framework, I'm able to get the rectangle that bounds a QR code in the camera frame. I'd like to match this rectangle to the device translation and rotation necessary to transform the QR code from a standard position.
For instance if I observe the frame:
* *
B
C
A
D
* *
while if I was 1m away from the QR code, centered on it, and assuming the QR code has a side of 10cm I'd see:
* *
A0 B0
D0 C0
* *
what has been my device transformation between those two frames? I understand that an exact result might not be possible, because maybe the observed QR code is slightly non planar and we're trying to estimate an affine transform on something that is not one perfectly.
I guess the sceneView.pointOfView?.camera?.projectionTransform is more helpful than the sceneView.pointOfView?.camera?.projectionTransform?.camera.projectionMatrix since the later already takes into account transform inferred from the ARKit that I'm not interested into for this problem.
How would I fill
func get transform(
qrCodeRectangle: VNBarcodeObservation,
cameraTransform: SCNMatrix4) {
// qrCodeRectangle.topLeft etc is the position in [0, 1] * [0, 1] of A0
// expected real world position of the QR code in a referential coordinate system
let a0 = SCNVector3(x: -0.05, y: 0.05, z: 1)
let b0 = SCNVector3(x: 0.05, y: 0.05, z: 1)
let c0 = SCNVector3(x: 0.05, y: -0.05, z: 1)
let d0 = SCNVector3(x: -0.05, y: -0.05, z: 1)
let A0, B0, C0, D0 = ?? // CGPoints representing position in
// camera frame for camera in 0, 0, 0 facing Z+
// then get transform from 0, 0, 0 to current position/rotation that sees
// a0, b0, c0, d0 through the camera as qrCodeRectangle
}
====Edit====
After trying number of things, I ended up going for camera pose estimation using openCV projection and perspective solver, solvePnP This gives me a rotation and translation that should represent the camera pose in the QR code referential. However when using those values and placing objects corresponding to the inverse transformation, where the QR code should be in the camera space, I get inaccurate shifted values, and I'm not able to get the rotation to work:
// some flavor of pseudo code below
func renderer(_ sender: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let currentFrame = sceneView.session.currentFrame, let pov = sceneView.pointOfView else { return }
let intrisics = currentFrame.camera.intrinsics
let QRCornerCoordinatesInQRRef = [(-0.05, -0.05, 0), (0.05, -0.05, 0), (-0.05, 0.05, 0), (0.05, 0.05, 0)]
// uses VNDetectBarcodesRequest to find a QR code and returns a bounding rectangle
guard let qr = findQRCode(in: currentFrame) else { return }
let imageSize = CGSize(
width: CVPixelBufferGetWidth(currentFrame.capturedImage),
height: CVPixelBufferGetHeight(currentFrame.capturedImage)
)
let observations = [
qr.bottomLeft,
qr.bottomRight,
qr.topLeft,
qr.topRight,
].map({ (imageSize.height * (1 - $0.y), imageSize.width * $0.x) })
// image and SceneKit coordinated are not the same
// replacing this by:
// (imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
// weirdly fixes an issue, see below
let rotation, translation = openCV.solvePnP(QRCornerCoordinatesInQRRef, observations, intrisics)
// calls openCV solvePnP and get the results
let positionInCameraRef = -rotation.inverted * translation
let node = SCNNode(geometry: someGeometry)
pov.addChildNode(node)
node.position = translation
node.orientation = rotation.asQuaternion
}
Here is the output:
where A, B, C, D are the QR code corners in the order they are passed to the program.
The predicted origin stays in place when the phone rotates, but it's shifted from where it should be. Surprisingly, if I shift the observations values, I'm able to correct this:
// (imageSize.height * (1 - $0.y), imageSize.width * $0.x)
// replaced by:
(imageSize.height * (1.35 - $0.y), imageSize.width * ($0.x - 0.2))
and now the predicted origin stays robustly in place. However I don't understand where the shift values come from.
Finally, I've tried to get an orientation fixed relatively to the QR code referential:
var n = SCNNode(geometry: redGeometry)
node.addChildNode(n)
n.position = SCNVector3(0.1, 0, 0)
n = SCNNode(geometry: blueGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0.1, 0)
n = SCNNode(geometry: greenGeometry)
node.addChildNode(n)
n.position = SCNVector3(0, 0, 0.1)
The orientation is fine when I look at the QR code straight, but then it shifts by something that seems to be related to the phone rotation:
Outstanding questions I have are:
How do I solve the rotation?
where do the position shift values come from?
What simple relationship do rotation, translation, QRCornerCoordinatesInQRRef, observations, intrisics verify? Is it O ~ K^-1 * (R_3x2 | T) Q ? Because if so that's off by a few order of magnitude.
If that's helpful, here are a few numerical values:
Intrisics matrix
Mat 3x3
1090.318, 0.000, 618.661
0.000, 1090.318, 359.616
0.000, 0.000, 1.000
imageSize
1280.0, 720.0
screenSize
414.0, 736.0
==== Edit2 ====
I've noticed that the rotation works fine when the phone stays horizontally parallel to the QR code (ie the rotation matrix is [[a, 0, b], [0, 1, 0], [c, 0, d]]), no matter what the actual QR code orientation is:
Other rotation don't work.
Coordinate systems' correspondence
Take into consideration that Vision/CoreML coordinate system doesn't correspond to ARKit/SceneKit coordinate system. For details look at this post.
Rotation's direction
I suppose the problem is not in matrix. It's in vertices placement. For tracking 2D images you need to place ABCD vertices counter-clockwise (the starting point is A vertex located in imaginary origin x:0, y:0). I think Apple Documentation on VNRectangleObservation class (info about projected rectangular regions detected by an image analysis request) is vague. You placed your vertices in the same order as is in official documentation:
var bottomLeft: CGPoint
var bottomRight: CGPoint
var topLeft: CGPoint
var topRight: CGPoint
But they need to be placed the same way like positive rotation direction (about Z axis) occurs in Cartesian coordinates system:
World Coordinate Space in ARKit (as well as in SceneKit and Vision) always follows a right-handed convention (the positive Y axis points upward, the positive Z axis points toward the viewer and the positive X axis points toward the viewer's right), but is oriented based on your session's configuration. Camera works in Local Coordinate Space.
Rotation direction about any axis is positive (Counter-Clockwise) and negative (Clockwise). For tracking in ARKit and Vision it's critically important.
The order of rotation also makes sense. ARKit, as well as SceneKit, applies rotation relative to the node’s pivot property in the reverse order of the components: first roll (about Z axis), then yaw (about Y axis), then pitch (about X axis). So the rotation order is ZYX.
Math (Trig.):
Notes: the bottom is l (the QR code length), the left angle is k, and the top angle is i (the camera)

Resources