How can I set text orientation in ARKit? - ios

I am creating a simple app with ARKit in which I add some text to the scene to the tapped position:
#objc func tapped(sender: UITapGestureRecognizer){
let sceneView = sender.view as! ARSCNView
let tapLocation = sender.location(in: sceneView)
let hitTest = sceneView.hitTest(tapLocation, types: .featurePoint)
if !hitTest.isEmpty{
self.addTag(tag: "A", hitTestResult: hitTest.first!)
}
else{
print("no match")
}
}
func addTag(tag: String, hitTestResult: ARHitTestResult){
let tag = SCNText(string:tag, extrusionDepth: 0.1)
tag.font = UIFont(name: "Optima", size: 1)
tag.firstMaterial?.diffuse.contents = UIColor.red
let tagNode = SCNNode(geometry: tag)
let transform = hitTestResult.worldTransform
let thirdColumn = transform.columns.3
tagNode.position = SCNVector3(thirdColumn.x,thirdColumn.y - tagNode.boundingBox.max.y / 2,thirdColumn.z)
print("\(thirdColumn.x) \(thirdColumn.y) \(thirdColumn.z)")
self.sceneView.scene.rootNode.addChildNode(tagNode)
}
It works, but I have problem with the orientation of the text. When I add it with the camera's original position, the text orientation is ok, I can see the text frontwise (Sample 1). But when I turn camera to the left / right, and add the text by tapping, I can see the added text from the side (Sample 2).
Sample 1:
Sample 2:
I know there should be some simple trick to solve it, but as a beginner in this topic I could not find it so far.

You want the text to always face the camera? SCNBillboardConstraint is your friend:
tagNode.constraints = [SCNBillboardConstraint()]

Am I correct in saying that you want the text to face the camera when you tap (wherever you happen to be facing), but then remain stationary?
There are a number of ways of adjusting the orientation of any node. For this case I would suggest simply setting the eulerAngles of the text node to be equal to those of the camera, at the point in which you instantiate the text.
In your addTag() function you add:
let eulerAngles = self.sceneView.session.currentFrame?.camera.eulerAngles
tagNode.eulerAngles = SCNVector3(eulerAngles.x, eulerAngles.y, eulerAngles.z + .pi / 2)
The additional .pi / 2 is there to ensure the text is in the correct orientation, as the default with ARKit is for a landscape orientation and therefore the text comes out funny. This applies a rotation around the local z axis.
It's also plausible (and some may argue it's better) to use .localRotate() of the node, or to access its transform property, however I like the approach of manipulating both the position and eulerAngles directly.
Hope this helps.
EDIT: replaced Float(1.57) with .pi / 2.

Related

Align 3D object parallel to vertical plane detected by estametedVerticalPlane

I have this book, but I'm currently remixing the furniture app from the video tutorial that was free on AR/VR week.
I would like to have a 3D wall canvas aligned with the wall/vertical plane detected.
This is proving to be harder than I thought. Positioning isn't an issue. Much like the furniture placement app you can just get the column3 of the hittest.worldtransform and provide the new geometry this vector3 for position.
But I do not know what I have to do to get my 3D object rotated to face forward on the aligned detected plane. As I have a canvas object, the photo is on one side of the canvas. On placement, the photo is ALWAYS facing away.
I thought about applying a arbitrary rotation to the canvas to face forward but that then was only correct if I was looking north and place a canvas on a wall to my right.
I'v tried quite a few solutions on line all but one always use .existingPlaneUsingExtent. for vertical plane detections. This allows for you to get the ARPlaneAnchor from the
hittest.anchor? as ARPlaneAnchor.
If you try this when using .estimatedVerticalPlane the anchor? is nil
I also didn't continue down this route as my horizontal 3D objects started getting placed in the air. This maybe down to a control flow logic but I am ignoring it until the vertical canvas placement is working.
My current train of thought is to get the front vector of the canvas and rotate it towards the front facing vector of the vertical plane detected UIImage or the hittest point.
How would I get a forward vector from a 3D point. OR get the front vector from the grid image, that is a UIImage that is placed as an overlay when ARKit detects a vertical wall?
Here is an example. The canvas is showing the back of the canvas and is not parallel with the detected vertical plane that is the column. But there is a "Place Poster Here" grid which is what I want the canvas to align with and I'm able to see the photo.
Things I have tried.
using .estimatedVerticalPlane
ARKit estimatedVerticalPlane hit test get plane rotation
I don't know how to correctly apply this matrix and eular angle results from the SO answer.
my add picture function.
func addPicture(hitTestResult: ARHitTestResult) {
// I would like to convert estimate hitTest to a anchorpoint
// it is easier to rotate a node to a anchorpoint over calculating eularAngles
// we have all detected anchors in the _Renderer SCNNode. however there are
// Get the current furniture item, correct its position if necessary,
// and add it to the scene.
let picture = pictureSettings.currentPicturePiece()
//look for the vertical node geometry in verticalAnchors
if let hitPlaneAnchor = hitTestResult.anchor as? ARPlaneAnchor {
if let anchoredNode = verticalAnchors[hitPlaneAnchor]{
//code removed as a .estimatedVerticalPlane hittestResult doesn't get here
}
}else{
// Transform hitresult to world coords
let worldTransform = hitTestResult.worldTransform
let anchoredNodeOrientation = worldTransform.eulerAngles
picture.rotation.y =
-.pi * anchoredNodeOrientation.y
//set the transform matirs
let positionMatris = worldTransform.columns.3
let position = SCNVector3 (
positionMatris.x,
positionMatris.y,
positionMatris.z
)
picture.position = position + pictureSettings.currentPictureOffset();
}
//parented to rootNode of the scene
sceneView.scene.rootNode.addChildNode(picture)
}
Thanks for any help available.
Edited:
I have notice the 'handness' or the 3D model isn't correct/ is opposite?
Positive Z is pointing to the Left and Positive X is facing the camera for what I would expects is the front of the model. Is this a issue?
You should try to avoid adding node directly into the scene using world coordinates. Rather you should notify the ARSession of an area of interest by adding an ARAnchor then use the session callback to vend an SCNNode for the added anchor.
For example your hit test might look something like:
#objc func tapped(_ sender: UITapGestureRecognizer) {
let location = sender.location(in: sender.view)
guard let hitTestResult = sceneView.hitTest(location, types: [.existingPlaneUsingGeometry, .estimatedVerticalPlane]).first,
let planeAnchor = hitTestResult.anchor as? ARPlaneAnchor,
planeAnchor.alignment == .vertical else { return }
let anchor = ARAnchor(transform: hitTestResult.worldTransform)
sceneView.session.add(anchor: anchor)
}
Here a tap gesture recognized is used to detect taps within an ARSCNView. When a tap is detected a hit test is performed looking for existing and estimated planes. If the plane is vertical, we add an ARAnchor is added with the worldTransform of the hit test result, and we add that anchor to the ARSession. This will register that point as an area of interest for the ARSession, so we'll receive better tracking and less drift after our content is added there.
Next, we need to vend our SCNNode for the newly added ARAnchor. For example
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
if anchor is ARPlaneAnchor {
let anchorNode = SCNNode()
anchorNode.name = "anchor"
return anchorNode
} else {
let plane = SCNPlane(width: 0.67, height: 1.0)
plane.firstMaterial?.diffuse.contents = UIImage(named: "monaLisa")
let planeNode = SCNNode(geometry: plane)
planeNode.eulerAngles = SCNVector3(CGFloat.pi * -0.5, 0.0, 0.0)
let node = SCNNode()
node.addChildNode(planeNode)
return node
}
}
Here we're first checking if the anchor is an ARPlaneAnchor. If it is, we vend an empty node for debugging purposes. If it is not, then it is an anchor that was added as the result of a hit test. So we create a geometry and node for the most recent tap. Because it is a vertical plane and our content is lying flat need to rotate it about the x axis. So we adjust it's eulerAngles to have it be upright. If we were to return planeNode directly adjustment to eulerAngles would be removed so we add it as a child node of an empty node and return it.
Should result in something like the following.

ARKit - Object stuck to camera after tap on screen

I started out with the template project which you get when you choose ARKit project. As you run the app you can see the ship and view it from any angle.
However, once I allow camera control and tap on the screen or zoom into the ship through panning the ship gets stuck to camera. Now wherever I go with the camera the ship is stuck to the screen.
I went through the Apple Guide and seems like the don't really consider this as unexpected behavior as there is nothing about this behavior.
How to keep the position of the ship fixed after I zoom it or touch the screen?
Well, looks like allowsCameraControl is not the answer at all. It's good for SceneKit but not for ARKit(maybe it's good for something in AR but I'm not aware of it yet).
In order to zoom into the view a UIPinchGestureRecognizer is required.
// 1. Find the touch location
// 2. Perform a hit test
// 3. From the results take the first result
// 4. Take the node from that first result and change the scale
#objc private func handlePan(recognizer: UIPinchGestureRecognizer) {
if recognizer.state == .changed {
// 1.
let location = recognizer.location(in: sceneView)
// 2.
let hitTestResults = sceneView.hitTest(location, options: nil)
// 3.
if let hitTest = hitTestResults.first {
let shipNode = hitTest.node
let newScaleX = Float(recognizer.scale) * shipNode.scale.x
let newScaleY = Float(recognizer.scale) * shipNode.scale.y
let newScaleZ = Float(recognizer.scale) * shipNode.scale.z
// 4.
shipNode.scale = SCNVector3(newScaleX, newScaleY, newScaleZ)
recognizer.scale = 1
}
}
Regarding #2. I got confused a little with another hitTest method called hitTest(_:types:)
Note from documentation
This method searches for AR anchors and real-world objects detected by
the AR session, not SceneKit content displayed in the view. To search
for SceneKit objects, use the view's hitTest(_:options:) method
instead.
So that method cannot be used if you want to scale a node which is a SceneKit content

making the arkit camera vertical rather than horizontal

I am trying to make an app where the user enters some text in a textfield and then the app displays this text in front of the ar camera to the user. I positioned the text correctly in front of the camera and I have changed the anchor of the text to be in the center of the text. but when I add the text into the scene the text is rotated 90 degrees around the z-axis. And I know why but I don't know how to solve it. The reason is that the camera of the arscene.session has a rotation of 0 for all x, y, z when the device is in landscape but since I want my app to be in portrait I rotate the device 90 degrees which rotates the camera as well and since the text has the same camera rotation, it's rotated as well. I tried correcting the rotation of the text by rotating it again around the z-axis but that doesn't solve the entire issue because when I change the direction of my phone, that affects the camera axis which will affect different axis of the text(not the same axis because I rotated the axis in the correction step). so I think the only way to solve the issue is to rotate the camera to be in consistent with the portrait mode from the beginning but I haven't found any way to set the rotation of the camera
here is the code of adding the text:
private func createTextNode(text:String?)
{
guard let text = text else {return}
let arText = SCNText(string: text, extrusionDepth: 1)
arText.font = UIFont(name: arText.font.fontName, size: 2)
arText.firstMaterial?.diffuse.contents = selectedColor
//making the node
let node = SCNNode()
node.geometry = arText
center(node: node)
guard let currentFrame = sceneView.session.currentFrame else {return}
let camera = currentFrame.camera
let cameraTransform = camera.transform
var newTransform = matrix_identity_float4x4
newTransform.columns.3.z = -0.2
let modifiedTransform = matrix_multiply(cameraTransform, newTransform)
node.transform = SCNMatrix4(modifiedTransform)
node.scale = SCNVector3(0.02, 0.02, 0.02)
self.sceneView.scene.rootNode.addChildNode(node)
node.eulerAngles.x = 90.degrees
}
and that's how the output looks like..
output
any help will be appreciated
You cannot use the matrix identity for any orientation, it has to be rotated depending on device orientation. I have a function in my apps that I call to update that before I perform the matrix multiplication :
var translation = matrix_identity_float4x4
func updateTranslationMatrix() {
switch UIDevice.current.orientation{
case .portrait, .portraitUpsideDown, .unknown, .faceDown, .faceUp:
print("portrait ")
translation.columns.0.x = -cos(.pi/2)
translation.columns.0.y = sin(.pi/2)
translation.columns.1.x = -sin(.pi/2)
translation.columns.1.y = -cos(.pi/2)
case .landscapeLeft :
print("landscape left")
translation.columns.0.x = 1
translation.columns.0.y = 0
translation.columns.1.x = 0
translation.columns.1.y = 1
case .landscapeRight :
print("landscape right")
translation.columns.0.x = cos(.pi)
translation.columns.0.y = -sin(.pi)
translation.columns.1.x = sin(.pi)
translation.columns.1.y = cos(.pi)
}
translation.columns.3.z = -0.6 //60cm in front of the camera
}
If you mean that you don't want the orientation of the device to change along with the rotation, then:
Go to Project > General > Deployment Info
Under device orientation, uncheck all boxes except 'Portrait'.
If this does not solve your problem and what you really want is to fix the euler angles of your text node, let me know, I'll be happy to help.

ARkit - Camera Position and 3D model Positions

I am trying to put several models in the scene.
for candidate in selectedCandidate {
sceneView.scene.rootNode.addChildNode(selectedObjects[candidate])
}
The candidate and selectedCandidate stands for the index of the model I want to use. Each model contains a rootNode and nodes attached to it. I use the API worldPosition and position of SCNNode to get and modify 3D model's position.
The thing I want to do is put those models right in front users' eyes. It means I need to get the camera's position and orientation vector to put the models in the right position I want. I also use these codes to get the camera's position according to this solution https://stackoverflow.com/a/47241952/7772038:
guard let pointOfView = sceneView.pointOfView else { return }
let transform = pointOfView.transform
let orientation = SCNVector3(-transform.m31, -transform.m32, transform.m33)
let location = SCNVector3(transform.m41, transform.m42, transform.m43)
The PROBLEM is that the camera's position and the model's position I printed out directly are severely different in order of magnitude. Camera's position is 10^-2 level like {0.038..., 0.047..., 0.024...} BUT the model's position is 10^2 level like {197.28, 100.29, -79.25}. From my point of view when I run the program, I am in the middle of those models and models are very near, but the positions are so different. So can you tell me how to modify the model's position to whatever I want? I really need to put the model right in front of user's eyes. If I simply do addChildNode() the models are behind me or somewhere else, while I need the model just be in front of users' eyes. Thank you in advance!
If you want to place an SCNNode infront of the camera you can do so like this:
/// Adds An SCNNode 3m Away From The Current Frame Of The Camera
func addNodeInFrontOfCamera(){
guard let currentTransform = augmentedRealitySession.currentFrame?.camera.transform else { return }
let nodeToAdd = SCNNode()
let boxGeometry = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0)
boxGeometry.firstMaterial?.diffuse.contents = UIColor.red
nodeToAdd.geometry = boxGeometry
var translation = matrix_identity_float4x4
//Change The X Value
translation.columns.3.x = 0
//Change The Y Value
translation.columns.3.y = 0
//Change The Z Value
translation.columns.3.z = -3
nodeToAdd.simdTransform = matrix_multiply(currentTransform, translation)
augmentedRealityView?.scene.rootNode.addChildNode(nodeToAdd)
}
And you can change any of the X,Y,Z values as you need.
Hope it points you in the right direction...
Update:
If you have multiple nodes e.g. in a scene, in order to use this function, it's probably best to create a 'holder' node, and then add all your content as a child.
Which means then you can simply call this function on the holder node.

ARKit Place a SCNNode facing the camera

I'm using ARKit to display 3D objects. I managed to place the nodes in the real world in front of the user (aka the camera). But I don't manage to make them to face the camera when I drop them.
let tap_point=CGPoint(x: x, y: y)
let results=arscn_view.hitTest(tap_point, types: .estimatedHorizontalPlane)
guard results.count>0 else{
return
}
guard let r=results.first else{
return
}
let hit_tf=SCNMatrix4(r.worldTransform)
let new_pos=SCNVector3Make(hit_tf.m41, hit_tf.m42+Float(0.2), hit_tf.m43)
guard let scene=SCNScene(named: file_name) else{
return
}
guard let node=scene.rootNode.childNode(withName: "Mesh", recursively: true) else{
return
}
node.position=new_pos
arscn_view.scene.rootNode.addChildNode(node)
The nodes are well positioned on the plane, in front of the camera. But they are all looking in the same direction. I guess I should rotate the SCNNode but I didn't manage to do this.
First, get the rotation matrix of the camera:
let rotate = simd_float4x4(SCNMatrix4MakeRotation(sceneView.session.currentFrame!.camera.eulerAngles.y, 0, 1, 0))
Then, combine the matrices:
let rotateTransform = simd_mul(r.worldTransform, rotate)
Lastly, apply a transform to your node, casting as SCNMatrix4:
node.transform = SCNMatrix4(rotateTransform)
Hope that helps
EDIT
here how you can create SCNMatrix4 from simd_float4x4
let rotateTransform = simd_mul(r.worldTransform, rotate)
node.transform = SCNMatrix4(m11: rotateTransform.columns.0.x, m12: rotateTransform.columns.0.y, m13: rotateTransform.columns.0.z, m14: rotateTransform.columns.0.w, m21: rotateTransform.columns.1.x, m22: rotateTransform.columns.1.y, m23: rotateTransform.columns.1.z, m24: rotateTransform.columns.1.w, m31: rotateTransform.columns.2.x, m32: rotateTransform.columns.2.y, m33: rotateTransform.columns.2.z, m34: rotateTransform.columns.2.w, m41: rotateTransform.columns.3.x, m42: rotateTransform.columns.3.y, m43: rotateTransform.columns.3.z, m44: rotateTransform.columns.3.w)
guard let frame = self.sceneView.session.currentFrame else {
return
}
node.eulerAngles.y = frame.camera.eulerAngles.y
here's my code for the SCNNode facing the camera..hope help for someone
let location = touches.first!.location(in: sceneView)
var hitTestOptions = [SCNHitTestOption: Any]()
hitTestOptions[SCNHitTestOption.boundingBoxOnly] = true
let hitResultsFeaturePoints: [ARHitTestResult] = sceneView.hitTest(location, types: .featurePoint)
let hitTestResults = sceneView.hitTest(location)
guard let node = hitTestResults.first?.node else {
if let hit = hitResultsFeaturePoints.first {
let rotate = simd_float4x4(SCNMatrix4MakeRotation(sceneView.session.currentFrame!.camera.eulerAngles.y, 0, 1, 0))
let finalTransform = simd_mul(hit.worldTransform, rotate)
sceneView.session.add(anchor: ARAnchor(transform: finalTransform))
}
return
}
Do you want the nodes to always face the camera, even as the camera moves? That's what SceneKit constraints are for. Either SCNLookAtConstraint or SCNBillboardConstraint can keep a node always pointing at the camera.
Do you want the node to face the camera when placed, but then hold still (so you can move the camera around and see the back of it)? There are a few ways to do that. Some involve fun math, but a simpler way to handle it might just be to design your 3D assets so that "front" is always in the positive Z-axis direction. Set a placed object's transform based on the camera transform, and its initial orientation will match the camera's.
Here's how I did it:
func faceCamera() {
guard constraints?.isEmpty ?? true else {
return
}
SCNTransaction.begin()
SCNTransaction.animationDuration = 5
SCNTransaction.completionBlock = { [weak self] in
self?.constraints = []
}
constraints = [billboardConstraint]
SCNTransaction.commit()
}
private lazy var billboardConstraint: SCNBillboardConstraint = {
let constraint = SCNBillboardConstraint()
constraint.freeAxes = [.Y]
return constraint
}()
As stated earlier a SCNBillboardConstraint will make the node always look at the camera. I am animating it so the node doesn't just immediately snap into place, this is optional. In the SCNTransaction.completionBlock I remove the constraint, also optional.
Also I set the SCNBillboardConstraint's freeAxes, which customizes on what axis the node follows the camera, again optional.
I want the node to face the camera when I place it then keep it here (and be able to move around). – Marie Dm
Blockquote
You can put object facing to camera, using this:
if let rotate = sceneView.session.currentFrame?.camera.transform {
node.simdTransform = rotate
}
This code will save you from gimbal lock and other troubles.
The four-component rotation vector specifies the direction of the rotation axis in the first three components and the angle of rotation (in radians) in the fourth. The default rotation is the zero vector, specifying no rotation. Rotation is applied relative to the node’s simdPivot property.
The simdRotation, simdEulerAngles, and simdOrientation properties all affect the rotational aspect of the node’s simdTransform property. Any change to one of these properties is reflected in the others.
https://developer.apple.com/documentation/scenekit/scnnode/2881845-simdrotation
https://developer.apple.com/documentation/scenekit/scnnode/2881843-simdtransform

Resources