Interacting with a SwiftUI View via ARKit blendshapes - ios

Question

I have a strange bug I can't resolve.
I'm trying to create an app where the user can interact with blend shapes instead of buttons.
I have my ContentView where I define an array of animals of type Card.
What the app do is that when the user blinks, a sheetView is shown. In this sheetView it's displayed a CardView with a random animal taken from the array. The user can play the sound of that specific animal with a button (for now).
The user can also close the sheetView opening his mouth.
I'll show my code
ContentView.swift
import SwiftUI
import AVKit
class ViewModel: ObservableObject {
#Published var changeAnimal = false
#Published var realMouthOpen = false
}
struct ContentView: View {
#ObservedObject var viewModel = ViewModel()
let animals: [Card] = [Card(image: "monkey", heading: "monkey", tag: 1, callSound: "Monkey", animal: .monkey), Card(image: "dog", heading: "dog", tag: 2, callSound: "dog", animal: .dog), Card(image: "chick", heading: "chick", tag: 3, callSound: "chick", animal: .chick)]
var body: some View {
ZStack {
SwiftUIViewController(viewModel: viewModel) // wrapper controller uikit
CardViewPreviewScroll(card: Card(image: "unknown", heading: "random animal", tag: 0, callSound: "", animal: .unknown))
.sheet(isPresented: $viewModel.changeAnimal) { // BUG: the view is changed 2 times
changeAnimal() // l'animale corrente, nel momento in cui cambia, la view viene aggiornata
}
}
}
//MARK: Functions
func changeAnimal() -> (CardView?) {
let animal = animals.randomElement()!
return (CardView(card: animal))
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
Card.swift
import Foundation
import SwiftUI
enum TypeAnimal: Identifiable {
case monkey, dog, chick, unknown
var id: Int {
hashValue
}
}
struct Card: Hashable{
var image: String
var heading: String
var id = UUID()
var tag: Int
var callSound: String
var animal: TypeAnimal
static let example = Card(image: "chick", heading: "chick", tag: 0, callSound: "chick", animal: .chick)
}
CardView.swift
import SwiftUI
import AVKit
struct CardView: View {
let card: Card
#State var audioPlayer: AVAudioPlayer!
var body: some View {
ZStack {
RoundedRectangle(cornerRadius: 30)
.fill(.green)
.shadow(radius: 3)
VStack {
Image(card.image)
.resizable()
.aspectRatio(contentMode: .fit)
VStack(alignment: .leading) {
Text(card.heading)
.font(.title)
.fontWeight(.black)
.foregroundColor(.primary)
.lineLimit(3)
HStack {
Spacer()
Button(action: {
self.audioPlayer.play()
}) {
Image(systemName: "play.circle.fill").resizable()
.frame(width: 50, height: 50)
.aspectRatio(contentMode: .fit)
}
Spacer()
Button(action: {
self.audioPlayer.pause()
}) {
Image(systemName: "pause.circle.fill").resizable()
.frame(width: 50, height: 50)
.aspectRatio(contentMode: .fit)
}
Spacer()
}
}
.layoutPriority(100)
.padding()
}
}
.onAppear {
let sound = Bundle.main.path(forResource: card.callSound, ofType: "mp3")
self.audioPlayer = try! AVAudioPlayer(contentsOf: URL(fileURLWithPath: sound!))
}
}
}
struct CardView_Previews: PreviewProvider {
static var previews: some View {
CardView(card: Card.example)
}
}
And this is my ViewController, wrapped in UIViewControllerRepresentable.
This ViewController conforms to the ARSessionDelegate protocol so that I can change the #Published property defined in my ObservableObject ViewModel and this means changing the $viewModel.changeAnimal binding in .sheet.
The fact is, as the CardView in the sheetView change whenever a blendshape of type browInnerUp is detected, I tried to catch this value only 1 time without finding a solution.
The best thing I managed to do is that, when the sheetView is displayed, an instance of CardView appears and this one is immediately replaced with another one.
I tried with DispatchQueue, timers and other things about treads in order to catch only a single value of the parameter of the blendshape in question but I failed
ViewController.swift
import UIKit
import SwiftUI
import ARKit
class ViewController: UIViewController, ARSessionDelegate {
var viewModel: ViewModel?
var session: ARSession!
override func viewDidLoad() {
super.viewDidLoad()
session = ARSession()
session.delegate = self
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
guard ARFaceTrackingConfiguration.isSupported else {print("IPhone X required"); return}
let configuration = ARFaceTrackingConfiguration()
session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
if let faceAnchor = anchors.first as? ARFaceAnchor {
update(withFaceAnchor: faceAnchor)
}
}
func update(withFaceAnchor faceAnchor: ARFaceAnchor) {
let bledShapes:[ARFaceAnchor.BlendShapeLocation:Any] = faceAnchor.blendShapes
guard let jawOpen = bledShapes[.jawOpen] as? Float else { return }
guard let browInnerUp = bledShapes[.browInnerUp] as? Float else { return }
if browInnerUp > 0.85 {
self.session.pause()
self.viewModel?.changeAnimal = true
// print("eyeBlinkLeft: \(eyeBlinkLeft)") // right eye
}
if jawOpen > 0.85 {
self.session.pause()
self.viewModel?.changeAnimal = false
// print("eyeBlinkRight: \(jawOpen)") // left eye
}
}
}
struct SwiftUIViewController: UIViewControllerRepresentable {
var viewModel: ViewModel
func makeUIViewController(context: Context) -> ViewController{
let controller = ViewController()
controller.viewModel = viewModel
return controller
}
func updateUIViewController(_ uiViewController: ViewController, context: Context) {
}
}
The question is: is there a way to fix this bug or I just messed up?