My application involves a user pointing their phone like a remote—as opposed to an 'AR window'—at virtual nodes. (See illustration.) These nodes are manually placed in the environment and aren't anchored to a physical plane. They will be floating.
To do this, I would need to cast along the Y axis from the camera/screen center and test for any hits along that ray. In Unity, I would do this by doing a Raycast hit test from the camera's transform.top, but I'm not sure how to do this in SceneKit.
I have had success using sceneView.hitTest on the Z axis, but this isn't my ultimate use case. I have also tried using scene.hitTestWithSegment(), using the camera world space as the from but I'm not sure how to get the to. I'm guessing I would need to cast a ray along the local Y axis and get a point along it, which just leads me back to my original problem.
Any tips on where to look? Is there a similar convenience to casting from a local transform.top in SceneKit?
Thank you, in advance!
Illustration:
** Update **
Per Josh Homann's answer below, I wound up solving it like so:
// Set local segment end and convert to world space
let segmentLocalEnd:SCNVector3 = SCNVector3(0,20,0)
let segmentWorldEnd:SCNVector3 = sceneView.pointOfView!.convertPosition(segmentLocalEnd, to: nil)
// Get hits
let hitResults:[SCNHitTestResult] = scene.hitTestWithSegment(
from: sceneView.pointOfView!.worldPosition,
to: segmentWorldEnd,
options: [SCNHitTestOption.firstFoundOnly.rawValue: true]
)
You already have most of the answer. In an AR view you would project a ray from the camera along the look at vector. In your case though you want to go perpendicular to the look at vector, so instead you just project a ray from the camera along the up vector (0,1,0) (assuming you haven't rotated your space and positive Y is up). In SceneKit you can't actually test along a infinite ray, so just pick a sufficiently large value and pass it into hitTestWithSegment(from:to:options:)
Related
I built a complete structure as a node (with its child nodes) and the user will walk through it using ARKit.
At some point, if the user cannot continue because of some real obstacle in the real world, I added a "pause" button which should freeze whatever the user currently sees in front of the camera, the user could then move freely to some other open space and when the user will release the pause button he/she will be able to resume where they left off (only someplace else in the real world).
A while ago I asked about it in the Apple Developer forum and an Apple Frameworks Engineer gave the following reply:
For "freezing" the scene, you could transform the anchor's position (in world coordinates) to camera coordinates, and then anchor your content to the camera. This will give you the effect that the scene is "frozen", i.e., does not move relative to the camera.
I'm currently not using an anchor because I don't necessarily need to find a flat surface. Rather, my node is placed at a certain position relative to where we start at (0,0,0).
My question is how do I exactly do what the Apple engineer told me to do?
I have the following code which I'm still stuck with. When I add the node to the camera (pointOfView, last line of the code below), it does freeze in place, but I can't get it to freeze in the same position and orientation as it was before it was frozen.
#IBAction func pauseButtonClicked(_ sender: UIButton) {
let currentPosition = sceneView.pointOfView?.position
let currentEulerAngles = sceneView.pointOfView?.eulerAngles
var internalNodeTraversal = lastNodeRootPosition - currentPosition! // for now, lastNodeRootPosition is (0,0,0)
internalNodeTraversal.y = lastNodeRootPosition.y + 20 // just so it’s positioned a little higher in front of the camera
myNode?.removeFromParentNode() // remove the node from the Real World view. Looks like this line has no effect and just adding the node as a child to the camera (pointOfView) is enough, but it feels more right to do this anyway.
myNode?.position = internalNodeTraversal // the whole node is moved respectively in the opposite direction from the root to where I’m standing to reposition the camera in my current position inside the node
// myNode?.eulerAngles = (currentEulerAngles! * -1) — this code put the whole node in weird positions so I removed it
myNode?.eulerAngles.y = currentEulerAngles!.y * -1 // opposite orientation of the node so the camera will be oriented in the same direction
myNode?.eulerAngles.x = 0.3 // just tilting it up a little bit to have a better view, more similar to the view as before it was locked to the camera
// I don’t think I need to change the eulerAngles.z
myNode!.convertPosition(internalNodeTraversal, to: sceneView.pointOfView) // I’m not sure I wrote this correctly. Also, this line doesn’t seem tp change anything
sceneView.pointOfView?.addChildNode(myNode!) // attaching the node to the camera so it will remain stuck while the user moves around until the button is released
}
So I first calculate where in the node I'm currently standing and then I change the position of the node in the opposite direction so that the camera will now be in that position. That seems to be correct.
Now I need to change the orientation of the node so that it will point in the right direction and here things get funky. I've been trying so many things for days now.
I use the eulerAngles for the orientation. If I set the whole vector multiplied by -1, it would show weird orientations. I ended up only using the eulerAngles.y which is the left/right orientation and I hardcoded the x orientation (up/down).
Ultimately what I have in the code above is the closest that I was able to get. If I'm pointing straight, the freeze will be correct. If I turn just a little bit, the freeze will be pretty close as well. Almost the same as what the user saw before the freeze. But the more I turn, the more the frozen image is off and more slanted. At some point (say I turn 50 or 60 degrees to the side) the whole node is off the camera and cannot be seen.
Somehow I have a feeling that there must be an easier and more correct way to achieve the above.
The Apple engineer wrote to "transform the anchor's position (in world coordinates) to camera coordinates". For that reason I added the "convertPosition" function in my code, but a) I'm not sure I used it correctly and b) it doesn't seem to change anything in my code if I have that line or not.
What am I doing wrong?
Any help would be very much appreciated.
Thanks!
I found the solution!
Actually, the problem I had was not even described as I didn't think it was relevant. I built the AR nodes 2 meters in front of the origin (-2 for the z-coordinate) while the center of my node was still at the origin. So when I changed the rotation or eulerAngles, it rotated around the origin so my nodes moved in a large curve and in fact also changed their position as a result.
The solution was to use a simdPivot. Instead of changing the position and rotation of the node itself, I created a translation matrix and a rotation matrix which was at the point of the camera (where the user is standing) and I then multiplied both matrices. Now when I added the node as a child of the camera (pointOfView) this would freeze the image and in effect show exactly what the user was seeing before it was frozen as the position is the same and the rotation is exactly around the user's standing position.
I am writing an application using ArCore. I am getting an object's pose as a Hitresult, therefore the object is placed correctly in the plane, but not correctly orientated/rotated towards the smartphone. For a more complete problem description:
The object is a 2d rectangle and is intended to stay inside the plane (= same normal vectors). The rectangle also needs to "point" to the camera (= nearest border needs to be aligned with the smartphone screen).
currentObstaclePose = hit.getHitPose().extractTranslation().compose(frame.getCamera().getPose().extractRotation())
Using this approach I do not get matching normal vectors for object and plane.
I have no idea how to construct the objects quaternions in a manner to achieve my goal. Thanks in advance for your help.
It sounds like you want to set the 'look direction' of the renderable.
Assuming you are using Scenefrom, which I see is tagged, you can use a TransformableNode and the method setLookDirection.
The example below will set the direction of the renderable, depending on the value you set for lookDirection:
var newAnchorNode:AnchorNode = AnchorNode(newAnchor)
var transNode = TransformableNode(arFragment.transformationSystem)
transNode.setLookDirection(Vector3(0f, xPoint.toFloat(), yPoint.toFloat()), lookDirectionValue)
transNode.renderable = yourRenderable
transNode.setParent(newAnchorNode)
newAnchorNode.setParent(arFragment.arSceneView.scene)
setAnchorNodes.add(newAnchorNode)
I'm working on developing iOS AR application using ARKit + Core location. And the points which are displayed on the map using coordinates move from place to place when I go. But I need they are displayed on the same place.
Here you can see the example of what I mean:
https://drive.google.com/file/d/1DQkTJFc9aChtGrgPJSziZVMgJYXyH9Da/view?usp=sharing
Could you help to handle with this issue? How can I have fixed places for points using coordinates? Any ideas?
Thanks.
Looks like you attach objects to planes. However, when you move the ARKit extends the existing planes. As a result if you put points, for example, at the center of the plane, then the center is always updated. You need to recalculate the coordinates of the point and place objects correctly.
The alternative is not to add objects to planes (or in relation to them). If you need to "put" object on a plane, then the best way is to wait, until the plane will be directed enough (it will not change his direction significantly if you will move), then select a point on the plane where you want to put your object, then convert this point coordinate to global coordinates (as a result if plane will change his size the coordinate you have will not be changed at all), and finally put object in root (or another object that it's not related to the plane).
It seems HitResult only gives us intersection with a surface (plane) or a point cloud. How can I get a point in the middle of air with my click, and thus put an object floating in the air?
It really depends on what you mean by "in the air". Two possibilities I see:
"Above a detected surface" Do a normal hit test against a plane, and offset the returned pose by some Y distance to get the hovering location. For example:
Pose.makeTranslation(0, 0.5f, 0).compose(hitResult.getHitPose())
returns a pose that is 50cm above the hit location. Create an anchor from this and you're good to go. You also could just create the anchor at the hit location and compose with the y translation each frame to allow for animating the hover height.
"Floating in front of the current device position" For this you probably want to compose a translation on the right hand side of the camera pose:
frame.getPose().compose(Pose.makeTranslation(0, 0, -1.0f)).extractTranslation()
gives you a translation-only pose that is 1m in front of the center of the display. If you want to be in front of a particular screen location, I put some code in this answer to do screen point to world ray conversion.
Apologies if you're in Unity/Unreal, your question didn't specify so I assumed Java.
The reason why you see so often a hit result being interpreted as the desired position by the user is that actually there is no closed-form solution for this user interaction. Which of the infinite possible positions along the ray starting from the camera pointing towards the scene was desired by the User? 2D coordinates from a click still leave the third dimension undefined.
As you said "middle of the air", why not take the centre between the camera position and the hitresult?
You can extract the current position using pose.getTranslation https://developers.google.com/ar/reference/java/com/google/ar/core/Pose.html#getTranslation(float[],%20int)
In GeoGebra, you can easily construct scenes with the GUI and the tools available in the Graphics view. I now have two functions and created some objects around them using that tools (Their intersection point, a circle tangent to both etc.). The whole depends on 5 parameters I defined as sliders for testing.
Now I want to know the coordinates of the point. It is defined as Intersect[l, h] which doesn't help me. I can access its coordinates too (0.8, 3.98) but I want to know how to calculate them depending on the parameters. (I'd expect it to be something like (3a, 7+b-2a)). I know GeoGebra can do this because it must have done it internally to be able to draw the whole image. But I don't know how to access this information.
If you want to get the current position of a Point P you can use the x and y commands. These will update whenever the position of P changes so that you don't have to recalculate where the point should be by hand.