Processing all frames in an AVAsset - ios

Question

I am trying to go through each frame in an AVAsset and process each frame as if it were an image. I have not been able to find anything from my searches.
The task I am trying to accomplish would look like this in pseudo-code
for each frame in asset
take the frame as an image and convert to a cvMat
Process and store data of center points
Store center points in array
The only part in that pseudo-code I do not know how to write is the going though each frame and capturing it in an image.
Can anyone help?

Answer 1

One answer is to use AVAssetImageGenerator.
1) Load the movie file into an AVAsset object.
2) Create an AVAssetImageGenerator object.
3) Pass in an estimated time of the frame where you want to get an image back from the movie.
Setting the 2 properties requestedTimeToleranceBefore and requestedTimeToleranceAfter on the AVAssetImageGenerator object to kCMTimeZero will increase the ability to get individual frames, but increases the processing time.
However this method is slow and I have not found a faster way.
//Load the Movie from a URL
self.movieAsset = [AVAsset assetWithURL:self.movieURL];
NSArray *movieTracks = [self.movieAsset tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *movieTrack = [movieTracks objectAtIndex:0];
//Make the image Generator
AVAssetImageGenerator *imageGenerator = [[AVAssetImageGenerator alloc] initWithAsset:self.movieAsset];
//Create a variables for the time estimation
Float64 durationSeconds = CMTimeGetSeconds(self.movieAsset.duration);
Float64 timePerFrame = 1.0 / (Float64)movieTrack.nominalFrameRate;
Float64 totalFrames = durationSeconds * movieTrack.nominalFrameRate;
//Step through the frames
for (int counter = 0; counter <= totalFrames; counter++){
CMTime actualTime;
Float64 secondsIn = ((float)counter/totalFrames)*durationSeconds;
CMTime imageTimeEstimate = CMTimeMakeWithSeconds(secondsIn, 600);
NSError *error;
CGImageRef image = [imageGenerator copyCGImageAtTime:imageTimeEstimate actualTime:&actualTime error:&error];
...Do some processing on the image
CGImageRelease(image);
}

Answer 2

You could simply gen each frame using AVAssetReaderTrackOutput:
let asset = AVAsset(url: inputUrl)
let reader = try! AVAssetReader(asset: asset)
let videoTrack = asset.tracks(withMediaType: .video).first!
let outputSettings = [String(kCVPixelBufferPixelFormatTypeKey): NSNumber(value: kCVPixelFormatType_32BGRA)]
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack,
outputSettings: outputSettings)
reader.add(trackReaderOutput)
reader.startReading()
while let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
// do what you want
}
}