I need to find the distance between two points inside a room using ARKit.
I will explain my scenario. Inside a room I have one predefined point. Say that point is (x1,y1,z1). I grabbed this value from ARKit ARCamera.transform's current position. I dynamically moved to another point whose ARCamera.transform's current position is (x2,y2,z2). My intention is to find real world distance between the two.
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let currentFrame = frame.camera.transform
let x = frame.camera.transform.columns.3.x
let y = frame.camera.transform.columns.3.y
let z = frame.camera.transform.columns.3.z
print( "camera transform :\(x),\(y),\(z)")
}
Please find the above code which am using to get camera current poistion.
Now here is the problem. (x2,y2,z2) is diifferent on each time I test at the same physical point (Please note that my starting point is kept same, so ARKit session starts on the same point each time),so the distance varies largerly. In short I can't rely on ARKit to give my camera's current point in a usefull manner , ARCamera.transform's current position seems a random value which varies depends on some unknown facts.
On googling I have seen that ARCamera.transform's current position gives the position of device camera.
Can anybody point a solution or correct me if am wrong or please tell me what exactly is ARCamera.transform's current position or how can we use ARCamera.transform's current position in real world positioning?
Related
I built a complete structure as a node (with its child nodes) and the user will walk through it using ARKit.
At some point, if the user cannot continue because of some real obstacle in the real world, I added a "pause" button which should freeze whatever the user currently sees in front of the camera, the user could then move freely to some other open space and when the user will release the pause button he/she will be able to resume where they left off (only someplace else in the real world).
A while ago I asked about it in the Apple Developer forum and an Apple Frameworks Engineer gave the following reply:
For "freezing" the scene, you could transform the anchor's position (in world coordinates) to camera coordinates, and then anchor your content to the camera. This will give you the effect that the scene is "frozen", i.e., does not move relative to the camera.
I'm currently not using an anchor because I don't necessarily need to find a flat surface. Rather, my node is placed at a certain position relative to where we start at (0,0,0).
My question is how do I exactly do what the Apple engineer told me to do?
I have the following code which I'm still stuck with. When I add the node to the camera (pointOfView, last line of the code below), it does freeze in place, but I can't get it to freeze in the same position and orientation as it was before it was frozen.
#IBAction func pauseButtonClicked(_ sender: UIButton) {
let currentPosition = sceneView.pointOfView?.position
let currentEulerAngles = sceneView.pointOfView?.eulerAngles
var internalNodeTraversal = lastNodeRootPosition - currentPosition! // for now, lastNodeRootPosition is (0,0,0)
internalNodeTraversal.y = lastNodeRootPosition.y + 20 // just so it’s positioned a little higher in front of the camera
myNode?.removeFromParentNode() // remove the node from the Real World view. Looks like this line has no effect and just adding the node as a child to the camera (pointOfView) is enough, but it feels more right to do this anyway.
myNode?.position = internalNodeTraversal // the whole node is moved respectively in the opposite direction from the root to where I’m standing to reposition the camera in my current position inside the node
// myNode?.eulerAngles = (currentEulerAngles! * -1) — this code put the whole node in weird positions so I removed it
myNode?.eulerAngles.y = currentEulerAngles!.y * -1 // opposite orientation of the node so the camera will be oriented in the same direction
myNode?.eulerAngles.x = 0.3 // just tilting it up a little bit to have a better view, more similar to the view as before it was locked to the camera
// I don’t think I need to change the eulerAngles.z
myNode!.convertPosition(internalNodeTraversal, to: sceneView.pointOfView) // I’m not sure I wrote this correctly. Also, this line doesn’t seem tp change anything
sceneView.pointOfView?.addChildNode(myNode!) // attaching the node to the camera so it will remain stuck while the user moves around until the button is released
}
So I first calculate where in the node I'm currently standing and then I change the position of the node in the opposite direction so that the camera will now be in that position. That seems to be correct.
Now I need to change the orientation of the node so that it will point in the right direction and here things get funky. I've been trying so many things for days now.
I use the eulerAngles for the orientation. If I set the whole vector multiplied by -1, it would show weird orientations. I ended up only using the eulerAngles.y which is the left/right orientation and I hardcoded the x orientation (up/down).
Ultimately what I have in the code above is the closest that I was able to get. If I'm pointing straight, the freeze will be correct. If I turn just a little bit, the freeze will be pretty close as well. Almost the same as what the user saw before the freeze. But the more I turn, the more the frozen image is off and more slanted. At some point (say I turn 50 or 60 degrees to the side) the whole node is off the camera and cannot be seen.
Somehow I have a feeling that there must be an easier and more correct way to achieve the above.
The Apple engineer wrote to "transform the anchor's position (in world coordinates) to camera coordinates". For that reason I added the "convertPosition" function in my code, but a) I'm not sure I used it correctly and b) it doesn't seem to change anything in my code if I have that line or not.
What am I doing wrong?
Any help would be very much appreciated.
Thanks!
I found the solution!
Actually, the problem I had was not even described as I didn't think it was relevant. I built the AR nodes 2 meters in front of the origin (-2 for the z-coordinate) while the center of my node was still at the origin. So when I changed the rotation or eulerAngles, it rotated around the origin so my nodes moved in a large curve and in fact also changed their position as a result.
The solution was to use a simdPivot. Instead of changing the position and rotation of the node itself, I created a translation matrix and a rotation matrix which was at the point of the camera (where the user is standing) and I then multiplied both matrices. Now when I added the node as a child of the camera (pointOfView) this would freeze the image and in effect show exactly what the user was seeing before it was frozen as the position is the same and the rotation is exactly around the user's standing position.
Hi I need to take a set of photos with my iphone and read the corresponding projection matrix, rotation and translation matrix for post-processing. I have never used ARKit or programmed in Swift/Object C before. What is the best way to get started?
If you create a default AR project in Xcode then you'll be able to study the basics of getting an AR app running in Swift.
To read the camera parameters you need to implement this function:
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let transform = frame.camera.transform else {return}
let position = SCNVector3Make(transform.columns.3.x, transform.columns.3.y, transform.columns.3.z)
let rotation = frame.camera.eulerAngles
let projection = frame.camera.projectionMatrix
print(position)
print(rotation)
print(projectionMatrix)
print("================="
}
(I.e copy / paste that code as another function in your ViewController class in the default app. The rotation is given in euler angles.
Given you've not used Swift before this may not be that helpful. But in theory it's all you need.
ARKit grabs frames at 60 frames per second (you can change this) - this function is called automatically for every new frame.
If you're using it to take photos you'll have to add more code to get all the timing right etc.
I am quite new and experimenting with Apple's ARKit and have a question regarding rotation information of the ARCamera. I am capturing photos and saving the current position, orientation and rotation of the camera with each image taken. The idea is to create 2d plane nodes with these images and have them appear in another view in the same position/orientation/rotation (with respect to the origin) as when when they were captured (as if the images were frozen in the air when they were captured). The position information seems to work fine, but the orientation/rotation comes up completely off as I’m having a difficulty in understanding when it’s relevant to use self.sceneView.session.currentFrame?.camera.eulerAngles vs self.sceneView.pointOfView?.orientation vs self.sceneView.pointOfView?.rotation.
This is how I set up my 2d image planes:
let imagePlane = SCNPlane(width: self.sceneView.bounds.width/6000, height: self.sceneView.bounds.height/6000)
imagePlane.firstMaterial?.diffuse.contents = self.image//<-- UIImage here
imagePlane.firstMaterial?.lightingModel = .constant
self.planeNode = SCNNode(geometry: imagePlane)
Then I set the self.planeNode.eulerAngles.x to the value I get from the view where the image is being captured using self.sceneView.session.currentFrame?.camera.eulerAngles.xfor x (and do the same for y and z as well).
I then set the rotation of the node as self.planeNode.rotation.x = self.rotX(where self.rotX is the information I get from self.sceneView.pointOfView?.rotation.x).
I have also tried to set it as follows:
let xAngle = SCNMatrix4MakeRotation(Float(self.rotX), 1, 0, 0);
let yAngle = SCNMatrix4MakeRotation(Float(self.rotY), 0, 1, 0);
let zAngle = SCNMatrix4MakeRotation(Float(self.rotZ), 0, 0, 1);
let rotationMatrix = SCNMatrix4Mult(SCNMatrix4Mult(xAngle, yAngle), zAngle);
self.planeNode.pivot = SCNMatrix4Mult(rotationMatrix, self.planeNode.transform);
The documentation states that eulerAngles is the “orientation” of the camera in roll, pitch and yaw values, but then what is self.sceneView.pointOfView?.orientation used for?
So when I specify the position, orientation and rotation of my plane nodes, is the information I get from eulerAngles enough to capture the correct orientation of the images?
Is my approach to this completely wrong or am I missing something obvious? Any help would be much appreciated!
If what you want to do is essentially create a billboard that is facing the camera at the time of capture then you can basically take the transform matrix of the camera (it already has the correct orientation) and just apply an inverse translation to it to move it to the objects location. They use that matric to position your billboard. This way you don't have to deal with any of the angles or worry about the correct order to composite the rotations. The translation is easy to do because all you need to do is subtract the object's location from the camera's location. One of the ARkit WWDC sessions actually has an example that sort of does this (it creates billboards at the camera's location). The only change you need to make is to translate the billboard away from the camer's position.
It seems HitResult only gives us intersection with a surface (plane) or a point cloud. How can I get a point in the middle of air with my click, and thus put an object floating in the air?
It really depends on what you mean by "in the air". Two possibilities I see:
"Above a detected surface" Do a normal hit test against a plane, and offset the returned pose by some Y distance to get the hovering location. For example:
Pose.makeTranslation(0, 0.5f, 0).compose(hitResult.getHitPose())
returns a pose that is 50cm above the hit location. Create an anchor from this and you're good to go. You also could just create the anchor at the hit location and compose with the y translation each frame to allow for animating the hover height.
"Floating in front of the current device position" For this you probably want to compose a translation on the right hand side of the camera pose:
frame.getPose().compose(Pose.makeTranslation(0, 0, -1.0f)).extractTranslation()
gives you a translation-only pose that is 1m in front of the center of the display. If you want to be in front of a particular screen location, I put some code in this answer to do screen point to world ray conversion.
Apologies if you're in Unity/Unreal, your question didn't specify so I assumed Java.
The reason why you see so often a hit result being interpreted as the desired position by the user is that actually there is no closed-form solution for this user interaction. Which of the infinite possible positions along the ray starting from the camera pointing towards the scene was desired by the User? 2D coordinates from a click still leave the third dimension undefined.
As you said "middle of the air", why not take the centre between the camera position and the hitresult?
You can extract the current position using pose.getTranslation https://developers.google.com/ar/reference/java/com/google/ar/core/Pose.html#getTranslation(float[],%20int)
So I have a bit of a project I am trying to do. I am trying to get the devices rotation relative to gravity, and translation from where it started. So basically getting "tracking" data for the device. I plan to basically apply this by making a 3d pt that will mimic the data I record from the device later on.
Anyway to attempt to achieve this I thought it would be best to work with scene kit that way I can see things in 3 dimensions just like the data I am trying to record. Right now I have been trying to get the ship to rotate so that it always looks like its following gravity (like its on the ground or something) no mater what the device rotation is. I figure once I have this down it will be a sinch to apply this to a point. So I made the following code:
if let attitude = motionManager.deviceMotion?.attitude {
print(attitude)
ship.eulerAngles.y = -Float(attitude.roll)
ship.eulerAngles.z = -Float(attitude.yaw)
ship.eulerAngles.x = -Float(attitude.pitch)
}
When you only run one of the rotation lines then everything is perfectly. It does behave properly on that axis. However when I do all three axis' at once it becomes chaotic and performs far from expected with jitter and everything.
I guess my question is:
Does anyone know how to fix my code above so that the ship properly stays "upright" no matter what the orientation.
J.Doe!
First there is a slight trick. If you want to use the iphone laying down as the default position you have to notice that the axis used on sceneKit are different then those used by the DeviceMotion. Check the axis:
(source: apple.com)
First thing you need to set is the camera position. When you start a SceneKit project it creates your camera in the position (0, 0, 15). There is a problem with that:
The values of eulerAngles = (0,0,0) would mean the object would be in the plane xz, but as long as you are looking from Z, you just see it from the side. For that to be equivalent to the iphone laying down, you would need to set the camera to look from above. So it would be like you were looking at it from the phone (like a camera, idk)
// create and add a camera to the scene
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
scene.rootNode.addChildNode(cameraNode)
// place the camera
cameraNode.position = SCNVector3(x: 0, y: 15, z: 0)
// but then you need to make the cameraNode face the ship (the origin of the axis), rotating it
cameraNode.eulerAngles.x = -Float(M_PI)*0.5 //or Float(M_PI)*1.5
With this we are going to see the ship from above, so the first part is done.
Now we gotta make the ship remain "still" (facing the ground) with the device rotation.
//First we need to use SCNRendererDelegate
class GameViewController : UIViewController SCNSceneRendererDelegate{
private let motion = CMMotionManager();
...
Then on viewDidLoad:
//important if you remove the sceneKit initial action from the ship.
//The scene would be static, and static scenes do not trigger the renderer update, setting the playing property to true forces that:
scnView.playing = true;
if(motion.deviceMotionAvailable){
motion.startDeviceMotionUpdates();
motion.deviceMotionUpdateInterval = 1.0/60.0;
}
Then we go to the update method
Look at the axis: the axis Y and Z are "switched" if you compare the sceneKit axis and the deviceMotion axis. Z is up on the phone, while is to the side on the scene, and Y is up on the scene, while to the side on the phone. So the pitch, roll and yaw, respectively associated to the X, Y and Z axis, will be applied as pitch, yaw and roll.
Notice I've put the roll value positive, that's because there is something else "switched". It's kinda hard to visualize. See the Y axis of device motion is correlated to the Z axis of the scene. Now imagine an object rotation along this axis, in the same direction (clock-wise for example), they would be going in opposite directions because of the disposition of the axis. (you can set the roll negative too see how it goes wrong)
func renderer(renderer: SCNSceneRenderer, updateAtTime time: NSTimeInterval) {
if let rot = motion.deviceMotion?.attitude{
print("\(rot.pitch) \(rot.roll) \(rot.yaw)")
ship.eulerAngles.x = -Float(rot.pitch);
ship.eulerAngles.y = -Float(rot.yaw);
ship.eulerAngles.z = Float(rot.roll);
}
Hope that helps! See ya!