Action ML Classifier not giving expected results - ios

Question

I am creating an app which detect the exercises. i trained the model using create ML. i got 100% result in create ML application. But when i am integrating into the application using Vision framework it's always showing only one exercise. i followed the code exactly from Build an Action Classifier with Create ML for creating ml and requesting VNHumanBodyPoseObservation. Followed this for converting VNHumanBodyPoseObservation to MLMultiArray.
Here is the code what i do:
func didOutput(pixelBuffer: CVPixelBuffer) {
self.extractPoses(pixelBuffer)
}
func extractPoses(_ pixelBuffer: CVPixelBuffer) {
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
let request = VNDetectHumanBodyPoseRequest { (request, err) in
if err == nil {
if let observations =
request.results as? [VNRecognizedPointsObservation], observations.count > 0 {
if let prediction = try? self.makePrediction(observations) {
print("\(prediction.label), confidence: \(prediction.confidence)")
}
}
}
}
do {
// Perform the body pose-detection request.
try handler.perform([request])
} catch {
print("Unable to perform the request: \(error).\n")
}
}
func makePrediction(_ observations: [VNRecognizedPointsObservation]) throws -> (label: String, confidence: Double) {
let fitnessClassifier = try PlayerExcercise(configuration: MLModelConfiguration())
let numAvailableFrames = observations.count
let observationsNeeded = 60
var multiArrayBuffer = [MLMultiArray]()
for frameIndex in 0 ..< min(numAvailableFrames, observationsNeeded) {
let pose = observations[frameIndex]
do {
let oneFrameMultiArray = try pose.keypointsMultiArray()
multiArrayBuffer.append(oneFrameMultiArray)
} catch {
continue
}
}
// If poseWindow does not have enough frames (45) yet, we need to pad 0s
if numAvailableFrames < observationsNeeded {
for _ in 0 ..< (observationsNeeded - numAvailableFrames) {
do {
let oneFrameMultiArray = try MLMultiArray(shape: [1, 3, 18], dataType: .double)
try resetMultiArray(oneFrameMultiArray)
multiArrayBuffer.append(oneFrameMultiArray)
} catch {
continue
}
}
}
let modelInput = MLMultiArray(concatenating: [MLMultiArray](multiArrayBuffer), axis: 0, dataType: .float)
//
//
let predictions = try fitnessClassifier.prediction(poses: modelInput)
return (label: predictions.label, confidence: predictions.labelProbabilities[predictions.label]!)
}
func resetMultiArray(_ predictionWindow: MLMultiArray, with value: Double = 0.0) throws {
let pointer = try UnsafeMutableBufferPointer<Double>(predictionWindow)
pointer.initialize(repeating: value)
}
I suspect the issue happening while converting VNRecognizedPointsObservation to MLMultiArray Please help me, i am trying to achieve this so hard. Thanks in advance.

Answer 1

Are you running your app on a simulator? Because I had the same issue that the model predicted wrong results when I ran my image classifier app on a iPhone 12 simulator. But the issue was solved when I tried to run the app on a real device. So maybe there is nothing wrong with your model or code, try running it on a real device and see if you get your intended results.