First Person Experience in RealityKit - ios

I am fairly new to Swift and just started playing with RealityKit and ARKit. I am working on a personal project where I would like to have a 3D object stick to the camera in first person. Similar to the AR Angry birds or any FPS game. I have seen a few examples in SceneKit or SpriteKit, I am sure it is just a misunderstanding of how anchoring entities works.
My main question is:
How would I go about sticking a reality object I created in Reality Composer to the camera in first person? I want to create a Reality Scene, in this case an arm cannon and upon tapping it shots.
Below is the code for my ViewController
extension ViewController: ARSessionDelegate
{
func session(_ session: ARSession, didUpdate frame: ARFrame)
{
guard let arCamera = session.currentFrame?.camera else { return }
// Probably where I update the location of my reality experience
}
}
class ViewController: UIViewController
{
#IBOutlet var arView: ARView!
override func viewDidLoad()
{
super.viewDidLoad()
arView.session.delegate = self
// Load the "ArmCannon" scene from the "Experience" Reality File
let armCannonAnim = try! Experience.loadArmcannon()
// Create Anchor to anchor arm cannon to
let anchor = AnchorEntity(.camera)
anchor.transform = arView.cameraTransform
// Add the anchor to the scene
arView.scene.addAnchor(anchor)
// Setup tap gesture on arm cannon
let tapGesture = UITapGestureRecognizer(target: self, action:#selector(onTap))
arView.addGestureRecognizer(tapGesture)
// Add the the cannon animation to arView
arView.scene.anchors.append(armCannonAnim)
}
#IBAction func onTap(_ sender: UITapGestureRecognizer)
{
let tapLocation = sender.location(in: arView)
// Get the entity at the location we've tapped, if one exists
if let cannonFiring = arView.entity(at: tapLocation)
{
print(cannonFiring.name)
print("firing Cannon")
}
}
}
I have looked at and read Track camera position with RealityKit and Where is the .camera AnchorEntity located?

Instead of:
arView.scene.anchors.append(armCannonAnim)
put:
anchor.addChild(armCannonAnim)
You need this armCannonAnim to be a child of the camera, and the anchor object is an anchor at the camera transform. This is equivalent to in SceneKit adding a child to the cameraNode.

Related

view.setNeedsDisplay() and view.setNeedsDisplay(view.frame) is not equivalent

I was trying to solve this problem (TL;DR An overlaid SKScene using the overlaySKScene property in SCNView wasn't causing a redraw when children were added and removed from it) using view.setNeedsDisplay() to force a redraw since the SCNView wasn't doing it automatically.
The problem with using view.setNeedsDisplay() was that the CPU usage was spiking to 50% and I assumed it was because the entire SCNView was having to redraw its contents, which included a 3D SCNScene as well. My solution was to use view.setNeedsDisplay(_: CGRect) to minimise the region that needs to be redrawn. However, to my surprise, no matter what I put as the CGRect value the SCNView refused to render the SKScene contents that had been overlaid on it.
Steps to reproduce issue
Open SceneKit template
From the Main (Base) storyboard, set the "Scene" attribute on the SCNView to be "art.scnassets/ship.scn" or whatever the path is
Delete all boilerplate code and just leave
class CustomSKScene: SKScene {
override func didMove(to view: SKView) {
let tapGestureRecognizer = UITapGestureRecognizer(target: self, action: #selector(userTapped(_:)))
view.addGestureRecognizer(tapGestureRecognizer)
}
#objc func userTapped(_ sender: UITapGestureRecognizer) {
let finger = convertPoint(fromView: sender.location(in: view))
let circle = SKShapeNode(circleOfRadius: 25)
circle.position = finger
addChild(circle)
}
}
class GameViewController: UIViewController {
private var gameView: SCNView { view as! SCNView }
override func viewDidLoad() {
gameView.overlaySKScene = CustomSKScene(size: gameView.bounds.size)
}
}
(This should still allow the ship scene to render when you run the app)
When you tap the screen, circles shouldn't show up. Fix this issue by adding view!.setNeedsDisplay() below the addChild function. Notice how CPU usage goes up to around 40-50% if you tap repeatedly after adding this fix.
Replace view!.setNeedsDisplay() with view!.setNeedsDisplay(view!.frame) (which should be equivalent).
At this point we are now back to square one. The circles are not showing up on screen again and confusion ensues. view.setNeedsDisplay() and view.setNeedsDisplay(view.frame) should be equivalent, yet, nothing is redrawn.
Does anyone know how to fix this problem? I feel it only happens when using the overlaySKScene property so maybe there is some caveat with its implementation that I am unaware of.
Some observations:
When you debug the view hierarchy, the overlaid SKScene doesn't show up anywhere, which is strange
sender.view === view returns true
(sender.view as! SCNScene).overlaySKScene === self also returns true

How to place a 3D model on top of a reference image with RealityKit?

I want to place a 3D Model from the local device on top of the reference image when it's recognized. To achieve this I have tried the following:
Adding the reference image to the session configuration:
override func viewDidLoad() {
super.viewDidLoad()
arView.session.delegate = self
// Check if the device supports the AR experience
if (!ARConfiguration.isSupported) {
TLogger.shared.error_objc("Device does not support Augmented Reality")
return
}
guard let qrCodeReferenceImage = UIImage(named: "QRCode") else { return }
let detectionImages: Set<ARReferenceImage> = convertToReferenceImages([qrCodeReferenceImage])
let configuration = ARWorldTrackingConfiguration()
configuration.detectionImages = detectionImages
arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
Using the ARSessionDelegate to get notified when the reference image was detected and placing the 3D model at the same position as his ARImageAnchor:
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let position = imageAnchor.transform
addEntity(self.localModelPath!, position)
}
}
func addEntity(_ modelPath: URL, _ position: float4x4) {
// Load 3D Object as Entity
let entity = try! Entity.load(contentsOf: modelPath)
// Create the Anchor which gets added to the AR Scene
let anchor = AnchorEntity(world: position)
anchor.addChild(entity)
anchor.transform.matrix = position
arView.scene.addAnchor(anchor)
}
However, whenever I try to place the anchor including my 3D model (the entity) at a specific position, it doesn't appear in the arView. It seems like the model is getting loaded though since a few frames are getting lost when executing the addEntity function. When I don't specifically set the anchors position the model appears in front of the camera.
Can anyone lead me in the right direction here?
Solution I
To make your code work properly, remove this line:
anchor.transform.matrix = position
#TimLangner – "...But I want to make the model appear on top of the reference image and not at any other location..."
As you can see, my test sphere has appeared on top of the reference image. When changing its position, remember that the Y-axis of the image anchor is directed towards the camera if the QR is vertical, and is directed up if the QR code is on a horizontal surface.
Make sure, a pivot point is located where it should.
In my case (you can see that I'm using AppClipCode), to move sphere 30 cm up, I have to move along negative Z direction.
entity.position.z = anchor.position.z - 0.3
Solution II
In my opinion, the most simple and productive solution would be to use the RealityKit's native AnchorEntity(.image(...)), without the need of implementing ARImageAnchor in delegate's method.
AnchorEntity(.image(group: "GroupName", name: "forModel"))
Here's a code:
import UIKit
import RealityKit
class ViewController: UIViewController {
#IBOutlet var arView: ARView!
override func viewDidLoad() {
super.viewDidLoad()
let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
let anchor = AnchorEntity(.image(group: "AR Resources",
name: "appClipCode"))
anchor.addChild(entity)
entity.position.z = anchor.position.z - 0.3
arView.scene.anchors.append(anchor)
}
}

Send ARKit camera position and quaternion on OSC (SwiftOSC)

I'm currently starting to learn Swift to develop an app as a part of my research. I'm ok with python but I can't seem to wrap my head around using Swift and Xcode. I've been able to print accelerometer data on screen, and send values over OSC. However, I need help:
Find ARKit camera position (X,Y,Z),
Send these values over OSC.
I've set up the ARSession and set the ARDelegate etc, and set up a function to get the position values from the transform matrix, but I cannot seem to be able to display any data on screen and send over OSC. The ARKIT code is below. The first code break is just an example of using SwiftOSC, it should be easy to implement once I am able to get the ARKit data.
I know that this may be pretty simple but I can't get it working! If anyone is willing to help me doing that, it would be amazing. And if someone is willing to work with me on Zoom or Meets, my thesis grant will be able to give honorarium compensation for your time.
Thanks in advance!
override func viewDidAppear(_ animated: Bool) {
motionManager.accelerometerUpdateInterval = (1/60)
motionManager.startAccelerometerUpdates(to: OperationQueue.current!) {(data, error)in
if let myData = data{
let x = myData.acceleration.x
let y = myData.acceleration.y
let z = myData.acceleration.z
self.xAccel.text = "x \(Double(x))"
let message = OSCMessage(OSCAddressPattern("/accelx"), Double(x))
client.send(message)
let message_y = OSCMessage(OSCAddressPattern("/accely"), Double(y))
client.send(message_y)
let message_z = OSCMessage(OSCAddressPattern("/accelz"), Double(z))
client.send(message_z)
}
}
}
However, I can't seem to access camera position using ARKit:
import UIKit
import SceneKit
import ARKit
extension ViewController: ARSessionDelegate {
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let transform = frame.camera.transform
var position = transform.columns.3
print(position.x, position.y, position.z) //Get (x,y,z) and print
}
}
class ViewController: UIViewController, ARSCNViewDelegate {
#IBOutlet var sceneView: ARSCNView!
override func viewDidLoad() {
super.viewDidLoad()
// Set the view's delegate
sceneView.delegate = self
// Show statistics such as fps and timing information
sceneView.showsStatistics = true
// Create a new scene
let scene = SCNScene(named: "art.scnassets/ship.scn")!
// Set the scene to the view
sceneView.scene = scene
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
// Create a session configuration
let configuration = ARWorldTrackingConfiguration()
// Run the view's session
sceneView.session.run(configuration)
// How to call function and assign position.[x,y,z], to variable to read on screen and send on OSC? So far I can't see any data at all.
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
// Pause the view's session
sceneView.session.pause()
}
It always returns zero, or nothing at all.

SCNCamera made programmatically not in the correct position

I am trying to programmatically create a camera for my MainScene.scn file.
I need to create the camera in code, as I am wanting to make a camera orbit node, and this is the only way I can think of. I also would like to keep using my scene file.
This is my code (simplified) from my view controller:
import UIKit
import SceneKit
class GameViewController: UIViewController {
// MARK: Scene objects
private var gameView: SCNView!
private var gameScene: SCNScene!
private var gameCameraNode: SCNNode!
// MARK: View controller overrides
override func viewDidLoad() {
super.viewDidLoad()
// Setup game
initView()
initScene()
initCamera()
}
}
private extension GameViewController {
// Initialise the view and scene
private func initView() {
self.view = SCNView(frame: view.bounds) // Create an SCNView to play the game within
gameView = self.view as? SCNView // Assign the view
gameView.showsStatistics = true // Show game statistics
gameView.autoenablesDefaultLighting = true // Allow default lighting
gameView.antialiasingMode = .multisampling2X // Use anti-aliasing for a smoother look
}
private func initScene() {
gameScene = SCNScene(named: "art.scnassets/MainScene.scn")! // Assign the scene
gameView.scene = gameScene // Set the game view's scene
gameView.isPlaying = true // The scene is playing (not paused)
}
private func initCamera() {
gameCameraNode = SCNNode()
gameCameraNode.camera = SCNCamera()
gameCameraNode.position = SCNVector3(0, 3.5, 27)
gameCameraNode.eulerAngles = SCNVector3(-2, 0, 0)
gameScene.rootNode.addChildNode(gameCameraNode)
gameView.pointOfView = gameCameraNode
}
}
This code can easily be pasted to replace the default view controller code. All you need to do is add the MainScene.scn and drag in something like a box.
If you try the code, the camera is in the wrong position. If I use the same properties for the camera in the scene, it works, but that is not what I am looking for.
From what I have read, SceneKit may be creating a default camera as said here and here. However, I am setting the pointOfView property just as they said in those answers, but it still does not work.
How can I place my camera in the correct position in the scene programmatically?
After a while, I discovered that you can actually add empty nodes directly within the Scene Builder. I originally only wanted a programmatic answer, as I wanted to make a camera orbit node like the questions I linked to. Now I can add an empty node, I can make the child of the orbit the camera.
This requires no code, unless you want to access the nodes (e.g. changing position or rotation):
gameCameraNode = gameView.pointOfView // Use camera object from scene
gameCameraOrbitNode = gameCameraNode.parent // Use camera orbit object from scene
Here are the steps to create an orbit node:
1) Drag it in from the Objects Library:
2) Setup up your Scene Graph like so:

How to tap and move scene nodes in ARKit

I'm currently trying to build an AR Chess app and I'm having trouble getting the movement of the pieces working.
I would like to be able to tap on a chess piece, then the legal moves it can make on the chess board will be highlighted and it will move to whichever square the user tapped on.
Pic of the chess board design and nodes:
https://gyazo.com/2a88f9cda3f127301ed9b4a44f8be047
What I would like to implement:
https://imgur.com/a/IGhUDBW
Would greatly appreciate any suggestions on how to get this working.
Thanks!
ViewController Code:
import UIKit
import SceneKit
import ARKit
class ViewController: UIViewController, ARSCNViewDelegate {
#IBOutlet var sceneView: ARSCNView!
override func viewDidLoad() {
super.viewDidLoad()
// Set the view's delegate
sceneView.delegate = self
// Show statistics such as fps and timing information
sceneView.showsStatistics = true
// Add lighting to the scene
sceneView.autoenablesDefaultLighting = true
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
// Create a session configuration to track an external image
let configuration = ARImageTrackingConfiguration()
// Image detection
// Reference which group to find the image to detect in the Assets folder e.g. "Detection Card"
if let imageDetect = ARReferenceImage.referenceImages(inGroupNamed: "Detection Card", bundle: Bundle.main) {
// Sets image tracking properties to the image in the referenced group
configuration.trackingImages = imageDetect
// Amount of images to be tracked
configuration.maximumNumberOfTrackedImages = 1
}
// Run the view's session
sceneView.session.run(configuration)
}
// Run when horizontal surface is detected and display 3D object onto image
// ARAnchor - tells a certain point in world space is relevant to your app, makes virtual content appear "attached" to some real-world point of interest
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode {
// Creates 3D object
let obj = SCNNode()
// Check if image detected through camera is an ARImageAnchor - which contains position and orientation data about the image detected in the session
if let imageAnchor = anchor as? ARImageAnchor {
// Set dimensions of the horizontal plane to be displayed onto the image to be the same as the image uploaded
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
// Display mild transparent layer onto detected image
// This is to ensure image detection works by display a faint layer on the image
plane.firstMaterial?.diffuse.contents = UIColor(white: 1.0, alpha: 0.2)
// Set geometry shape of the plane
let planeNode = SCNNode(geometry: plane)
// Flip vertical plane to horizontal plane
planeNode.eulerAngles.x = -Float.pi / 2
obj.addChildNode(planeNode)
// Initialise chess scene
if let chessBoardSCN = SCNScene(named: "art.scnassets/chess.scn") {
// If there is a first in the scene file
if let chessNodes = chessBoardSCN.rootNode.childNodes.first {
// Displays chessboard upright
chessNodes.eulerAngles.x = Float.pi / 2
// Adds chessboard to the overall 3D scene
obj.addChildNode(chessNodes)
}
}
}
return obj
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
// Pause the view's session
sceneView.session.pause()
}
}
You will need to add gestures on to your view and use the ARSceneViews hitTest method to detect what the gesture is touching in your scene. You can then update the positions based on the movement from the gestures.
Here is a question that deals with roughly the same requirement of dragging nodes around.
Placing, Dragging and Removing SCNNodes in ARKit
First, you need to add a gesture recognizer for tap into your viewDidLoad, like this:
let tapGesture = UITapGestureRecognizer(target: self, action: #selector(handleTap(_:)))
myScnView.addGestureRecognizer(tapGesture)
Then realize the handler function:
#objc
func handleTap(_ gestureRecognize: UIGestureRecognizer) {
// HERE YOU NEED TO DETECT THE TAP
// check what nodes are tapped
let location = gestureRecognize.location(in: myScnView)
let hitResults = myScnView.hitTest(location, options: [:])
// check that we clicked on at least one object
if hitResults.count > 0 {
// retrieved the first clicked object
let tappedPiece = hitResults[0].node
// HERE YOU CAN SHOW POSSIBLE MOVES
//Ex. showPossibleMoves(for: tappedPiece)
}
}
Now, to show the possible moves, you need to identify all quadrants and your node position on the chessboard.
To do this, you can assign a name or a number, or a combination of letter and number, or moreover a combination of numbers. (I suggest combination of number, like row 1 column 1, like a matrix).
let's take my suggestion, so you can name each quadrant 1.1 1.2 ... 2.1 2.2 and so on.
Now, to detect where your piece is, you can check contact with the PhysicsContactDelegate.
Now you have the tappedPiece and the place where it is, so you have to define the rule for the pieces, for example:
let rules = ["tower":"cross"] //add the others
N.B You can choose what you want to define the rules.
Let's take my suggestion for good, now you should create the function to highlight:
func highlight(quadrant: SCNNode){
quadrant.geometry?.firstMaterial?.emission.contents = UIColor.yellow
}
Finally the showPossibleMoves(for: tappedPiece) could be something this:
func showPossibleMoves(for piece: SCNNode){
let pieceType = piece.name //You have to give the name as you did into your rules variable
//ex. if you have rules like ["tower":"cross"] you have to set all towers name to "tower"
let rule = rules[pieceType]
switch rule{
case "cross":
//you have to highlight all nodes on the right, left, above and bottom
// you can achieve this by selecting the start point and increase it
//assuming you named your quadrants like 1.1 1.2 or 11 12 13 ecc...
let startRow = Int(startQuadrant.name.first)
let startColumn = Int(startQuadrant.name.last)
//Now loop the highlight on right
for column in startColumn+1...MAX_COLUMN-1{
let quadrant = myScnView.scene.rootNode.childNode(withName:"\(startRow).\(column)" , recursively: true)
// call highlight function
highlight(quadrant: quadrant)
}
//Now loop for above quadrants
for row in startRow+1...MAX_ROW-1{
let quadrant = myScnView.scene.rootNode.childNode(withName:"\(row).\(startColumn)" , recursively: true)
// call highlight function
highlight(quadrant: quadrant)
}
//DO THE SAME FOR ALL DIRECTIONS
}
// ADD ALL CASES, like bishop movements "diagonals" and so on
}
NOTE: In the handlerTap function you have to check what you're tapping, for example, to check if you're tapping on a quadrant after selecting a piece (you want to move you're piece) you can check a boolean value and the name of the selected node
//assuming you have set the boolean value after selecting a piece
if pieceSelected && node.name != "tower"{
//HERE YOU CAN MOVE YOUR PIECE
}

Resources