2D Drawing On a Virtual 3D Plane in RealityKit - ios

Question

I'm trying to draw a 2D stroke using my finger on a 3D plane I generated and added to the scene:
let material = SimpleMaterial.init(color: .white,
roughness: 1,
isMetallic: false)
let plane = MeshResource.generatePlane(width: 1.5,
height: 1.0)
let planeEntity = ModelEntity(mesh: plane,
materials: [material])
planeEntity.collision = CollisionComponent(shapes: [.generateBox(size: [1.5, 1.0, 0.05])],
mode: .trigger,
filter: .sensor)
planeEntity.name = "TargetPlane"
planeEntity.position = SIMD3(x: 0.0,
y: 0.0,
z: 0.0)
self.scene.addChild(planeEntity)
I'm getting the points along the finger path using a pan recognizer on my ARView:
#objc func handlePan(_ sender: UIPanGestureRecognizer) {
let touchInView = sender.location(in: self.arView)
guard let hitEntity = self.arView.entity(at: touchInView) else { return } // No entity was hit.
if hitEntity.name == "TargetPlane" {
if sender.state == .began || sender.state == .changed {
let result = self.arView.hitTest(touchInView)
guard let collision = result.first else { return }
// Convert from world space to local plane space.
let position = planeEntity.convert(position: collision.position,
from: nil)
}
}
}
At this point I have points in the plane's local co-ordinates. I thought of using a UIBezierPath and drawing it onto a CAShapeLayer then rendering a UIImage out of it and using that as a texture on the plane.
The problem is I have no idea how to map the 3D points to 2D pixel co-ordinates on the plane.
I'm aware of the project(_:) method but that maps points to the whole viewport of the device. I need to map the points to the surface of the virtual plane.

Answer 1

I figured it out using the pinhole camera model.
if sender.state == .began || sender.state == .changed {
let result = self.arView.hitTest(touchInView)
guard let collision = result.first else { return }
// Convert the point from world space to the plane's local space.
let position = planeEntity.convert(position: collision.position,
from: nil)
// Get the focal length.
let intrinsics = (self.arView.session.currentFrame?.camera.intrinsics)!
let f_x = intrinsics.columns.0.x
let f_y = intrinsics.columns.1.y
// Co-ordinates of the principal point.
let c_x = intrinsics.columns.2.x
let c_y = intrinsics.columns.2.y
// (x, y, z) of the 3D point on the plane.
let x = position.x
let y = position.y
let z = collision.distance
// Map to 2D (u, v) co-ordinates.
let u = (x / z) * f_x + c_x
let v = (y / z) * f_y + c_y
// Shift the origin to the top left corner for Core Graphics.
let u_max = (planeEntity.model!.mesh.bounds.max.x / z) * f_x + c_x
let v_max = (planeEntity.model!.mesh.bounds.max.y / z) * f_y + c_y
let u_mapped = max(u, 0) // Clamp to avoid negative values.
let v_mapped = max(v_max - v, 0)
// Add this point to the UIBezierPath.
let mappedPoint = CGPoint(x: CGFloat(u_mapped),
y: CGFloat(v_mapped))
// Use this as the size of the image renderer.
let canvasSize = CGSize(width: u_max,
height: v_max)
}