Multiple Vision Requests per Frame - ios

Question

I am working with the Vision and CoreML frameworks. I have a real time video feed. For every frame, I first detect rectangles using VNDetectRectanglesRequest. For every rectangle I detect, I crop out that part of the image and perform a VNCoreMLRequest to classify that part of the image. After classifying the object, if it is the object type I am looking for, I draw the rectangle. It's like I built an object detector when I don't have data to train an actual neural network for detection.
Generally, I detect around 1 to 3 rectangles. Not that many. So for every VNDetectRectanglesRequest, I have 1 to 3 additional VNCoreMLRequest per frame to perform. However, performing all these requests make my video stream very laggy. It's quite noticeable when I point my camera at rectangularly shaped objects. I guess I should add that this video footage is coming from ARKit, so whatever background operations ARKit is performing might have made the lag worse.
I tried to optimize the code using DispatchQueue. Below is my pseudo-code. I'm happy with what the code is doing, but I need to get rid of the lag.
DispatchQueue.global(qos: .background).async {
let request = VNDetectRectanglesRequest(completionHandler: { (request, error) in
// ...
for observation in request.results {
let mlRequest = VNCoreMLRequest(model: model){ (request, error) in
// classify ... if is object I want then jump back to main queue and draw
DispatchQueue.main.async {
// draw rectangles
}
})
}
})
}

Answer 1

I don't think there's anything wrong with your code, just that making all of those requests for every frame is too high of a load for the device to handle.
Try reducing the frequency of the requests, and maybe add a few conditions to check before making the request in the first place. Simply adding a check to see if you're already waiting for other requests to finish might noticeably reduce the load. You could also check if the user is holding the device steady, the frame is in focus, the lighting is adequate, etc.

Answer 2

The lag is potentially caused by creation of new request objects (better check it in Instruments/Time Profiler, but object creation is often the first suspect).
In a similar situation (where I need to run a request on the entire image, then other requests on its parts) I am using https://github.com/maxvol/RxVision which does not re-create requests and hence does not introduce the lag.
let mlRequest: RxVNCoreMLRequest<CGImage> = VNCoreMLRequest.rx.request(model: model, imageCropAndScaleOption: .scaleFit)
mlRequest
.observable
.subscribe { [unowned self] (event) in
switch event {
case .next(let completion):
let cgImage = completion.value // NB you can easily pass the value along to the completion handler
if let result = completion.request.results?[0] as? VNClassificationObservation {
os_log("results: %#", type: .debug, result.identifier)
}
default:
break
}
}
.disposed(by: disposeBag)
let imageRequestHandler = VNImageRequestHandler(cgImage: cgImage, orientation: .up, options: requestOptions)
do {
try imageRequestHandler.rx.perform([mlRequest], with: cgImage) // NB you can easily pass the value along to the completion handler
} catch {
print(error)
}