My application involves getting video frames from the camera and running CoreML on it. This will then get displayed on the view. To do this, I have an AVCaptureSession() that is hooked to a video output. The main processing of CoreML is being done in captureOutput(didOutput Samplebuffer). Each processing of the frame takes about 0.15 seconds, meaning that I WILL have some dropped frames.After CoreML process is done, I have an AVAssetWriter that append all these frames together and saves it within the phone directory.
THE main problem is however, that my use-case also requires the original video to be saved, and this video should have HIGH FPS and since I am only able to get the image frames only in captureOutput(didOutput), the video quality will be choppy.
I have tried the following:
The reason that I'm using AVAssetWriter is because it is given here: https://forums.developer.apple.com/thread/98113#300885 that it is NOT POSSIBLE to have both AVCaptureVideoDataOutput and AVCaptureMovieFileOutput.
I have also tried extracting Image buffer from the captureOutput(didDrop) using guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } but it is giving me nil. This is because sampleBuffer only contains metadata but not imagebuffer as explained here: https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1388468-captureoutput.
Here is some of my code:
func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { /* HOW DO I possibly extract an image buffer here from the dropped frames here? */ }
func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let inputVideoImage = UIImage(pixelBuffer: pixelBuffer)
if self.isRecording{
let sourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
if let camera = self.videoWriterVideoInput, camera.isReadyForMoreMediaData {
videoWriterQueue.async() {
self.videoWriterInputPixelBufferAdaptor.append(pixelBuffer, withPresentationTime: sourceTime)
}
}else{
print("AVAssetInput is not ready for more media data ... ")
}
}
runCoreML()
}
Related
I'm working with Apple's own example here:
Applying Matte Effects to People in Images and Video
My goal is simple, I want to save the filtered video content on file as a video.
extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
//1. Grab the pixelbuffer frame from the camera output
guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
//2. create a new CIImage from the pixelBuffer that segments a person from the background
guard let image = processVideoFrame(pixelBuffer, sampleBuffer: sampleBuffer) else { return }
//3. #todo (need help here) - we want to somehow create a new CMSampleBuffer from the new image / frame
let updatedBuffer = image.convertToCMSampleBuffer()
//4. WORKS - Save the updated sample buffer to the current recording session
self.assetWriterHelper?.captureOutput(output, didOutput: sampleBuffer, from: connection)
}
}
Their example doesn't show this functionality (saving the actual video), and I've tried various solutions but nothing seems to work.
Anyone have an idea how you would go about saving the modified frame/image into the current recording session?
Thank ¥ou!
You can use the CIContext render(_:to:) function to render your image into a pixel buffer:
func render(
_ image: CIImage,
to buffer: CVPixelBuffer
)
Can anyone please advise why this solution works? I have same question using Swift, when I define my 'Scanner' in a standalone swift file, the AVCaptureVideoDataOutputSampleBufferDelegate.CaptureOutput not called but, when I move the Scanner in the same file when I call session.addOutput(videoOutput) and videoOutput.setSampleBufferDelegate it works.
I am binginer to Swift/iOS dev, this issue drives me crazy, finally this solution works. thank you for sharing this!
just clarify and add my code:
I have a CamService.swift which I have a method to setup the AVCaptureSession:
session.beginConfiguration()
// ...
//add AVCaptureVideoDataOutput
if session.canAddOutput(videoOutput) {
session.addOutput(videoOutput)
videoOutput.alwaysDiscardsLateVideoFrames = true
videoOutput.setSampleBufferDelegate(self, queue:DispatchQueue.global(qos: .userInteractive) )
videoOutput.connections.first?.videoOrientation = .landscapeLeft
} else {
//fatalError("could not add video output")
print("Couldn't add video output to the session.")
session.commitConfiguration()
return
}
// ...
session.commitConfiguration()
when I extension CamService as below, it works, the captureOutput can be called
extension CamService: AVCaptureVideoDataOutputSampleBufferDelegate {
public func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection:
AVCaptureConnection) {
print("sampleBuffer")
}
}
But, if I create a new swift file say,
MyDalegate: AVCaptureVideoDataOutputSampleBufferDelegate,
and config
videoOutput.setSampleBufferDelegate(MyDalegate(),
queue:DispatchQueue.global(qos: .userInteractive) )
this won't work captureOutput never get called.
I want to be able to track a users face from the camera feed. I have looked at this SO post. I used the code given in the answer but it did not seem to do anything. I have heard that
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!)
has been changed to something else in swift 4. Could this be the problem with the code?
While face tracking I want to also monitor face landmarks with CIFaceFeature. How would I do this?
I have found a starting point here: https://github.com/jeffreybergier/Blog-Getting-Started-with-Vision.
Basically you can instatiate a video capture session declaring a lazy variable like this:
private lazy var captureSession: AVCaptureSession = {
let session = AVCaptureSession()
session.sessionPreset = AVCaptureSession.Preset.photo
guard
let frontCamera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front),
let input = try? AVCaptureDeviceInput(device: frontCamera)
else { return session }
session.addInput(input)
return session
}()
Then inside viewDidLoad you start the session
self.captureSession.startRunning()
And finally you can perform your requests inside
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
}
for example:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer:
CMSampleBuffer, from connection: AVCaptureConnection) {
guard
// make sure the pixel buffer can be converted
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else { return }
let faceRequest = VNDetectFaceRectanglesRequest(completionHandler: self.faceDetectedRequestUpdate)
// perform the request
do {
try self.visionSequenceHandler.perform([faceRequest], on: pixelBuffer)
} catch {
print("Throws: \(error)")
}
}
And then you define your faceDetectedRequestUpdate function.
Anyway I have to say that I haven't been able to figure out how to create a working example from here. The best working example I have found is in Apple's documentation: https://developer.apple.com/documentation/vision/tracking_the_user_s_face_in_real_time
I am trying to make a simple camera application where the front camera can detect faces.
This should be simple enough:
Create a CameraView class that inherits from UIImage and place it in the UI. Make sure it implements AVCaptureVideoDataOutputSampleBufferDelegate in order to process frames from the camera in real time.
class CameraView: UIImageView, AVCaptureVideoDataOutputSampleBufferDelegate
Within a function handleCamera, called when the CameraView is instantiated, setup an AVCapture session. Add input from the camera.
override init(frame: CGRect) {
super.init(frame:frame)
handleCamera()
}
func handleCamera () {
camera = AVCaptureDevice.defaultDevice(withDeviceType: .builtInWideAngleCamera,
mediaType: AVMediaTypeVideo, position: .front)
session = AVCaptureSession()
// Set recovered camera as an input device for the capture session
do {
try input = AVCaptureDeviceInput(device: camera);
} catch _ as NSError {
print ("ERROR: Front camera can't be used as input")
input = nil
}
// Add the input from the camera to the capture session
if (session?.canAddInput(input) == true) {
session?.addInput(input)
}
Create output. Create a serial output queue to pass the data to which will then be processed by the AVCaptureVideoDataOutputSampleBufferDelegate (the class itself in this case). Add output to session.
output = AVCaptureVideoDataOutput()
output?.alwaysDiscardsLateVideoFrames = true
outputQueue = DispatchQueue(label: "outputQueue")
output?.setSampleBufferDelegate(self, queue: outputQueue)
// add front camera output to the session for use and modification
if(session?.canAddOutput(output) == true){
session?.addOutput(output)
} // front camera can't be used as output, not working: handle error
else {
print("ERROR: Output not viable")
}
Setup the camera preview view and run the session
// Setup camera preview with the session input
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer?.videoGravity = AVLayerVideoGravityResizeAspectFill
previewLayer?.connection.videoOrientation = AVCaptureVideoOrientation.portrait
previewLayer?.frame = self.bounds
self.layer.addSublayer(previewLayer!)
// Process the camera and run it onto the preview
session?.startRunning()
in the captureOutput function run by the delegate, convert the recieved sample buffer to CIImage in order to detect faces. Give feedback if a face is found.
func captureOutput(_ captureOutput: AVCaptureOutput!, didDrop sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(cvPixelBuffer: pixelBuffer!)
let accuracy = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: accuracy)
let faces = faceDetector?.features(in: cameraImage)
for face in faces as! [CIFaceFeature] {
print("Found bounds are \(face.bounds)")
let faceBox = UIView(frame: face.bounds)
faceBox.layer.borderWidth = 3
faceBox.layer.borderColor = UIColor.red.cgColor
faceBox.backgroundColor = UIColor.clear
self.addSubview(faceBox)
if face.hasLeftEyePosition {
print("Left eye bounds are \(face.leftEyePosition)")
}
if face.hasRightEyePosition {
print("Right eye bounds are \(face.rightEyePosition)")
}
}
}
My problem: I can get the camera running but with the multitude of different codes I have tried from all over the internet, I have never been able to get captureOutput to detect a face. Either the application doesn't enter the function or it crashes because of a varible that doesn't work, the most often being that the sampleBuffer variable is nul.
What am I doing wrong?
You need to change your captureOutput function arguments to the following: func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!)
Your captureOutput function calls when buffer drops, not when it gets from camera.
I'am developing camera application in Swift, that edit every frame realtime.
So what I need:
Edit every frame using CIFilter - probably using AVCaptureVideoDataOutput
Write data buffer to AVAssetWriter
Add audio
Add some labels to the video
Realtime view of what camera "see", but already edited by CIFilter
What I have:
I'am able to edit video output and show it to screen using imageView. But I think that this is not best way to do that and with iPhone 4S I have really low FPS:
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!)
{
guard let filter = Filters["Saturation"]
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(CVPixelBuffer: pixelBuffer!)
filter!.setValue(cameraImage, forKey: kCIInputImageKey)
let filteredImage = UIImage(CIImage: filter!.valueForKey(kCIOutputImageKey) as! CIImage!)
dispatch_async(dispatch_get_main_queue())
{
self.imageView.image = filteredImage
}
}
So how to change my code to edit frame more efficiently, show it to screen and when its needed write it to a file (I know, that I must convert filtred image to CMSampleBuffer, but I don't know how to use NSTime)?