right now my App can detect images and place some Models. I am usnig ARKit and RealityKit. This is my setup:
import ARKit
import RealityKit
class ViewController: UIViewController, ARSessionDelegate {
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let imageAnchor = anchors.first as? ARImageAnchor,
let _ = imageAnchor.referenceImage.name
else { return }
let anchor = AnchorEntity(anchor: imageAnchor)
// Add Model Entity to anchor
anchor.addChild(model)
arView.scene.anchors.append(anchor)
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
arView.session.delegate = self
resetTrackingConfig()
}
func resetTrackingConfig() {
guard let refImg = ARReferenceImage.referenceImages(inGroupNamed: "Sub",
bundle: nil)
else { return }
let config = ARWorldTrackingConfiguration()
config.detectionImages = refImg
config.maximumNumberOfTrackedImages = 1
let options = [ARSession.RunOptions.removeExistingAnchors,
ARSession.RunOptions.resetTracking]
arView.session.run(config, options: ARSession.RunOptions(options))
}
}
Now the problem is that I am not completely satisfied how the image detection works. It has troubles detecting images if for example the light is slightly different.
This is my image:
But it should also be able to detect these images:
(2nd one has no white background)
And for that I thought I could use Machine Leaning. But how can I combine that with my ARKit setup? Right now it just takes the images from my assets-folder. I tried searching for that topic but couldn't find anything.. Is this kind of project even possible the way I described it? Any help is appreciated! Let me know if you need any more information.
Related
in brief: How to pass every frame from AR Session to a funcion?
I'm making an app:
It displays the image of the rear camera in real time. (in AR View)（✅）
When it detects a QR code, it will read the text content in the QR code.（I don't know how to do it in real time, get data from AR session）
func getQRCodeContent(_ pixel: CVPixelBuffer) -> String {
let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixel, options: [:])
let request = VNDetectBarcodesRequest()
request.symbologies = [.qr]
try! requestHandler.perform([request])
let result = request.results?.first?.payloadStringValue
if let result = result {
return result
} else {
return "non"
}
}
And then do some logic with the content, and display the corresponding AR model in the AR View.
I know I have to feed images into the Vision Framework, I started with AVFoundation, but I found that when AR View is loaded, the AVCaptureSession is paused.
And I want to feed AR Session's frame into Vision Framework. However, all the tutorials I can find are based on story board and UI kit to complete this function. I don't know how to complete this function in Swift UI at all.
I tried to extent ARView:
extension ARView: ARSessionDelegate {
func renderer(_ renderer: SKRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {
let capturedImage = session.currentFrame?.capturedImage
print(capturedImage)
}
}
struct ARViewCustom: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
arView.session.delegate = arView
return arView
}
func updateUIView(_ uiView: ARView, context: Context) {}
}
No error, but it doesn't work.
I'm trying to build an ARKit based app which requires detection of roads and placing virtual content 30 feet away from the camera. However horizontal plane detection is stopping to add anchors after about 10 feet. Is there a workaround for this problem?
public func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let usdzEntity = usdzEntity else { return }
let camera = frame.camera
let transform = camera.transform
if let rayCast = arView.scene.raycast(from: transform.translation, to: usdzEntity.transform.translation, query: .nearest, mask: .default, relativeTo: nil).first {
print(rayCast.distance)
}
}
look at this, hope this can give you some help
I'm trying to place a 3D model on top of a recognized image with ARKit and RealityKit - all programmatically. Before I start the ARView I'm downloading the model I want to show when the reference image is detected.
This is my current setup:
override func viewDidLoad() {
super.viewDidLoad()
arView.session.delegate = self
// Check if the device supports the AR experience
if (!ARConfiguration.isSupported) {
TLogger.shared.error_objc("Device does not support Augmented Reality")
return
}
guard let qrCodeReferenceImage = UIImage(named: "QRCode") else { return }
let detectionImages: Set<ARReferenceImage> = convertToReferenceImages([qrCodeReferenceImage])
let configuration = ARWorldTrackingConfiguration()
configuration.detectionImages = detectionImages
arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
I use the ARSessionDelegate to get notified when a new image anchor was added which means the reference image got detected:
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
print("Hello")
for anchor in anchors {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let referenceImage = imageAnchor.referenceImage
addEntity(self.localModelPath!)
}
}
However, the delegate method never gets called while other delegate functions like func session(ARSession, didUpdate: ARFrame) are getting called so I assume that the session just doesn't detect the image. The image resolution is good and the printed image the big so it should definitely get recognized by the ARSession. I also checked that the image has been found before adding it to the configuration.
Can anyone lead me in the right direction here?
It looks like you have your configuration set up correctly. Your delegate-function should be called when the reference image is recognized. Make sure your configuration isn't overwritten at any point in your code.
I'm building AR Scanner application where users are able to scan different images and receive rewards for this.
When they point camera at some specific image - I place SCNNode on top of that image and after they remove camera from that image - SCNNode get's dismissed.
But when image disappears and camera stays at the same position SCNNode didn't get dismissed.
How can I make it disappear together with Reference image disappearance?
I have studied lot's of other answers here, on SO, but they didn't help me
Here's my code for adding and removing SCNNode's:
extension ARScannerScreenViewController: ARSCNViewDelegate {
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
DispatchQueue.main.async { self.instructionLabel.isHidden = true }
if let imageAnchor = anchor as? ARImageAnchor {
handleFoundImage(imageAnchor, node)
imageAncors.append(imageAnchor)
trackedImages.append(node)
} else if let objectAnchor = anchor as? ARObjectAnchor {
handleFoundObject(objectAnchor, node)
}
}
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let pointOfView = sceneView.pointOfView else { return }
for (index, item) in trackedImages.enumerated() {
if !(sceneView.isNode(item, insideFrustumOf: pointOfView)) {
self.sceneView.session.remove(anchor: imageAncors[index])
}
}
}
private func handleFoundImage(_ imageAnchor: ARImageAnchor, _ node: SCNNode) {
let name = imageAnchor.referenceImage.name!
print("you found a \(name) image")
let size = imageAnchor.referenceImage.physicalSize
if let imageNode = showImage(size: size) {
node.addChildNode(imageNode)
node.opacity = 1
}
}
private func showImage(size: CGSize) -> SCNNode? {
let image = UIImage(named: "InfoImage")
let imageMaterial = SCNMaterial()
imageMaterial.diffuse.contents = image
let imagePlane = SCNPlane(width: size.width, height: size.height)
imagePlane.materials = [imageMaterial]
let imageNode = SCNNode(geometry: imagePlane)
imageNode.eulerAngles.x = -.pi / 2
return imageNode
}
private func handleFoundObject(_ objectAnchor: ARObjectAnchor, _ node: SCNNode) {
let name = objectAnchor.referenceObject.name!
print("You found a \(name) object")
}
}
I also tried to make it work using ARSession, but I couldn't even get to prints:
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
for anchor in anchors {
for myAnchor in imageAncors {
if let imageAnchor = anchor as? ARImageAnchor, imageAnchor == myAnchor {
if !imageAnchor.isTracked {
print("Not tracked")
} else {
print("tracked")
}
}
}
}
}
You have to use ARWorldTrackingConfiguration instead of ARImageTrackingConfiguration. It's quite bad idea to use both configurations in app because each time you switch between them – tracking state is reset and you have to track from scratch.
Let's see what Apple documentation says about ARImageTrackingConfiguration:
With ARImageTrackingConfiguration, ARKit establishes a 3D space not by tracking the motion of the device relative to the world, but solely by detecting and tracking the motion of known 2D images in view of the camera.
The basic differences between these two configs are about how ARAnchors behave:
ARImageTrackingConfiguration allows you get ARImageAnchors only if your reference images is in a Camera View. So if you can't see a reference image – there's no ARImageAnchor, thus there's no a 3D model (it's resetting each time you cannot-see-it-and-then-see-it-again). You can simultaneously detect up to 100 images.
ARWorldTrackingConfiguration allows you track a surrounding environment in 6DoF and get ARImageAnchor, ARObjectAnchor, or AREnvironmentProbeAnchor. If you can't see a reference image – there's no ARImageAnchor, but when you see it again ARImageAnchor is still there. So there's no reset.
Conclusion:
ARWorldTrackingConfiguration's cost of computation is much higher. However this configuration allows you perform not only image tracking but also hit-testing and ray-casting for detected planes, object detection, and a restoration of world maps.
Use nodeForAnchor to load your nodes, so when the anchors disappear, the nodes will go as well.
I have a client that wants to recognize when an user smacks their screen with their whole hand, like a high-five. I suspect that Apple won't approve this, but let's look away from that.
I though of using a four-finger-tap recognizer, but that doesn't really cover it. The best approach would possibly be to check if the user is covering at least 70% of the screen with their hand, but I don't know how to do that.
Can someone help me out here?
You could use the accelerometer to detect the impact of a hand & examine the front camera feed to find a corresponding dark frame due to the hand covering the camera*
* N.B. a human hand might not be big enough to cover the front camera on an iPhone 6+
Sort of solved it. Proximity + accelerometer works good enough. Multitouch doesn't work, as it ignores stuff it doesn't think of as taps.
import UIKit
import CoreMotion
import AVFoundation
class ViewController: UIViewController {
var lastHighAccelerationEvent:NSDate? {
didSet {
checkForHighFive()
}
}
var lastProximityEvent:NSDate? {
didSet {
checkForHighFive()
}
}
var lastHighFive:NSDate?
var manager = CMMotionManager()
override func viewDidLoad() {
super.viewDidLoad()
//Start disabling the screen
UIDevice.currentDevice().proximityMonitoringEnabled = true
NSNotificationCenter.defaultCenter().addObserver(self, selector: #selector(proximityChanged), name: UIDeviceProximityStateDidChangeNotification, object: nil)
//Check for acceloremeter
manager.startAccelerometerUpdatesToQueue(NSOperationQueue.mainQueue()) { (data, error) in
let sum = abs(data!.acceleration.y + data!.acceleration.z + data!.acceleration.x)
if sum > 3 {
self.lastHighAccelerationEvent = NSDate()
}
}
//Enable multitouch
self.view.multipleTouchEnabled = true
}
func checkForHighFive() {
if let lastHighFive = lastHighFive where abs(lastHighFive.timeIntervalSinceDate(NSDate())) < 1 {
print("Time filter")
return
}
guard let lastProximityEvent = lastProximityEvent else {return}
guard let lastHighAccelerationEvent = lastHighAccelerationEvent else {return}
if abs(lastProximityEvent.timeIntervalSinceDate(lastHighAccelerationEvent)) < 0.1 {
lastHighFive = NSDate()
playBoratHighFive()
}
}
func playBoratHighFive() {
print("High Five")
let player = try! AudioPlayer(fileName: "borat.mp3")
player.play()
}
func proximityChanged() {
if UIDevice.currentDevice().proximityState {
self.lastProximityEvent = NSDate()
}
}
}
You can detect finger count with multi touch event handling. check this answer