I have recently been onboarded onto an existing project at work which is written in Swift.
The metadataOutput(_:didOutput:from:) delegate function looks like this:
func metadataOutput(_ output: AVCaptureMetadataOutput,
didOutput metadataObjects: [AVMetadataObject],
from connection: AVCaptureConnection) {
captureSession.stopRunning()
if let metadataObject = metadataObjects.first,
let readableObject = metadataObject as? AVMetadataMachineReadableCodeObject {
print("\(readableObject.stringValue)")
}
... // other code
}
Theoretically speaking, the print function should only print once; however, in the console, I can see that the output is printed several times.
Is there something that I am missing out here? I have created a barebones standalone scanning app, and it prints only once there.
Related
Can anyone please advise why this solution works? I have same question using Swift, when I define my 'Scanner' in a standalone swift file, the AVCaptureVideoDataOutputSampleBufferDelegate.CaptureOutput not called but, when I move the Scanner in the same file when I call session.addOutput(videoOutput) and videoOutput.setSampleBufferDelegate it works.
I am binginer to Swift/iOS dev, this issue drives me crazy, finally this solution works. thank you for sharing this!
just clarify and add my code:
I have a CamService.swift which I have a method to setup the AVCaptureSession:
session.beginConfiguration()
// ...
//add AVCaptureVideoDataOutput
if session.canAddOutput(videoOutput) {
session.addOutput(videoOutput)
videoOutput.alwaysDiscardsLateVideoFrames = true
videoOutput.setSampleBufferDelegate(self, queue:DispatchQueue.global(qos: .userInteractive) )
videoOutput.connections.first?.videoOrientation = .landscapeLeft
} else {
//fatalError("could not add video output")
print("Couldn't add video output to the session.")
session.commitConfiguration()
return
}
// ...
session.commitConfiguration()
when I extension CamService as below, it works, the captureOutput can be called
extension CamService: AVCaptureVideoDataOutputSampleBufferDelegate {
public func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection:
AVCaptureConnection) {
print("sampleBuffer")
}
}
But, if I create a new swift file say,
MyDalegate: AVCaptureVideoDataOutputSampleBufferDelegate,
and config
videoOutput.setSampleBufferDelegate(MyDalegate(),
queue:DispatchQueue.global(qos: .userInteractive) )
this won't work captureOutput never get called.
My application involves getting video frames from the camera and running CoreML on it. This will then get displayed on the view. To do this, I have an AVCaptureSession() that is hooked to a video output. The main processing of CoreML is being done in captureOutput(didOutput Samplebuffer). Each processing of the frame takes about 0.15 seconds, meaning that I WILL have some dropped frames.After CoreML process is done, I have an AVAssetWriter that append all these frames together and saves it within the phone directory.
THE main problem is however, that my use-case also requires the original video to be saved, and this video should have HIGH FPS and since I am only able to get the image frames only in captureOutput(didOutput), the video quality will be choppy.
I have tried the following:
The reason that I'm using AVAssetWriter is because it is given here: https://forums.developer.apple.com/thread/98113#300885 that it is NOT POSSIBLE to have both AVCaptureVideoDataOutput and AVCaptureMovieFileOutput.
I have also tried extracting Image buffer from the captureOutput(didDrop) using guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } but it is giving me nil. This is because sampleBuffer only contains metadata but not imagebuffer as explained here: https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1388468-captureoutput.
Here is some of my code:
func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { /* HOW DO I possibly extract an image buffer here from the dropped frames here? */ }
func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let inputVideoImage = UIImage(pixelBuffer: pixelBuffer)
if self.isRecording{
let sourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
if let camera = self.videoWriterVideoInput, camera.isReadyForMoreMediaData {
videoWriterQueue.async() {
self.videoWriterInputPixelBufferAdaptor.append(pixelBuffer, withPresentationTime: sourceTime)
}
}else{
print("AVAssetInput is not ready for more media data ... ")
}
}
runCoreML()
}
I have an app that records video, but I need it to show pitch levels of the sounds captured on the microphone in real-time to the user. I have been able to successfully record audio and video to MP4 using AVCaptureSession. However, when I add AVCaptureAudioDataOutput to the session and assign the AVCaptureAudioDataOutputSampleBufferDelegate I receive no errors, and yet the captureOutput function is never called once the session starts.
Here is the code:
import UIKit
import AVFoundation
import CoreLocation
class ViewController: UIViewController,
AVCaptureVideoDataOutputSampleBufferDelegate,
AVCaptureFileOutputRecordingDelegate, CLLocationManagerDelegate ,
AVCaptureAudioDataOutputSampleBufferDelegate {
var videoFileOutput: AVCaptureMovieFileOutput!
let session = AVCaptureSession()
var outputURL: URL!
var timer:Timer!
var locationManager:CLLocationManager!
var currentMagnitudeValue:CGFloat!
var defaultMagnitudeValue:CGFloat!
var visualMagnitudeValue:CGFloat!
var soundLiveOutput: AVCaptureAudioDataOutput!
override func viewDidLoad() {
super.viewDidLoad()
self.setupAVCapture()
}
func setupAVCapture(){
session.beginConfiguration()
//Add the camera INPUT to the session
let videoDevice = AVCaptureDevice.default(.builtInWideAngleCamera,
for: .video, position: .front)
guard
let videoDeviceInput = try? AVCaptureDeviceInput(device: videoDevice!),
session.canAddInput(videoDeviceInput)
else { return }
session.addInput(videoDeviceInput)
//Add the microphone INPUT to the session
let microphoneDevice = AVCaptureDevice.default(.builtInMicrophone, for: .audio, position: .unspecified)
guard
let audioDeviceInput = try? AVCaptureDeviceInput(device: microphoneDevice!),
session.canAddInput(audioDeviceInput)
else { return }
session.addInput(audioDeviceInput)
//Add the video file OUTPUT to the session
videoFileOutput = AVCaptureMovieFileOutput()
guard session.canAddOutput(videoFileOutput) else {return}
if (session.canAddOutput(videoFileOutput)) {
session.addOutput(videoFileOutput)
}
//Add the audio output so we can get PITCH of the sounds
//AND assign the SampleBufferDelegate
soundLiveOutput = AVCaptureAudioDataOutput()
soundLiveOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "test"))
if (session.canAddOutput(soundLiveOutput)) {
session.addOutput(soundLiveOutput)
print ("Live AudioDataOutput added")
} else
{
print("Could not add AudioDataOutput")
}
//Preview Layer
let previewLayer = AVCaptureVideoPreviewLayer(session: session)
let rootLayer :CALayer = self.cameraView.layer
rootLayer.masksToBounds=true
previewLayer.frame = rootLayer.bounds
rootLayer.addSublayer(previewLayer)
previewLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill;
//Finalize the session
session.commitConfiguration()
//Begin the session
session.startRunning()
}
func captureOutput(_: AVCaptureOutput, didOutput: CMSampleBuffer, from:
AVCaptureConnection) {
print("Bingo")
}
}
Expected output:
Bingo
Bingo
Bingo
...
I have read:
StackOverflow: captureOutput not being called - The user was not declaring the captureOutput method correctly.
StackOverflow: AVCaptureVideoDataOutput captureOutput not being called - The user was not declaring the captureOutput method at all.
Apple - AVCaptureAudioDataOutputSampleBufferDelegate - Apple's documentation on the delegate and it's method - the method matches the method I have declared.
Other common errors I have encountered online:
Using the declaration for older versions of Swift (I am using v4.1)
Apparently on one article after Swift 4.0, AVCaptureMetadataOutput replaces AVCaptureAudioDataOutput - Although I couldn't find this in Apple's documentation, I tried this also, but similarly, the metadataOutput function is never called.
I am fresh out of ideas. Am I missing something obvious?
Ok, nobody got back to me but after playing around with it I worked out the correct way to declare the captureOutput method for Swift4 is as follows:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
//Do your stuff here
}
Unfortunately, the documentation for this online is very poor. I guess you just need to get it exactly right - there are no errors thrown if you mispell or misname the variables as it is an optional function.
The method you're using is been updated with this one, Which will get called for both AVCaptureAudioDataOutput & AVCaptureVideoDataOutput. you make sure you check the output before writing the sample buffers to asset writer.
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
//Make sure you check the output before using sample buffer
if output == audioDataOutput {
//Use sample buffer for audio
}
}
The problem for me turned out to be this, and that the AVAudioSession and AVCaptureSession where declared as local variables and when I started the session, it just went away. Once I moved them to class level variables, everything worked great!
I want to be able to track a users face from the camera feed. I have looked at this SO post. I used the code given in the answer but it did not seem to do anything. I have heard that
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!)
has been changed to something else in swift 4. Could this be the problem with the code?
While face tracking I want to also monitor face landmarks with CIFaceFeature. How would I do this?
I have found a starting point here: https://github.com/jeffreybergier/Blog-Getting-Started-with-Vision.
Basically you can instatiate a video capture session declaring a lazy variable like this:
private lazy var captureSession: AVCaptureSession = {
let session = AVCaptureSession()
session.sessionPreset = AVCaptureSession.Preset.photo
guard
let frontCamera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front),
let input = try? AVCaptureDeviceInput(device: frontCamera)
else { return session }
session.addInput(input)
return session
}()
Then inside viewDidLoad you start the session
self.captureSession.startRunning()
And finally you can perform your requests inside
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
}
for example:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer:
CMSampleBuffer, from connection: AVCaptureConnection) {
guard
// make sure the pixel buffer can be converted
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else { return }
let faceRequest = VNDetectFaceRectanglesRequest(completionHandler: self.faceDetectedRequestUpdate)
// perform the request
do {
try self.visionSequenceHandler.perform([faceRequest], on: pixelBuffer)
} catch {
print("Throws: \(error)")
}
}
And then you define your faceDetectedRequestUpdate function.
Anyway I have to say that I haven't been able to figure out how to create a working example from here. The best working example I have found is in Apple's documentation: https://developer.apple.com/documentation/vision/tracking_the_user_s_face_in_real_time
I'm trying to understand why my AVCaptureOutput is dropping frames. In the captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) delegate method, I get a CMSampleBuffer that should contains an attachement explaining the reason the frame was dropped (doc)
The reason is expected to be one of those CFString:
kCMSampleBufferDroppedFrameReason_FrameWasLate // "FrameWasLate"
kCMSampleBufferDroppedFrameReason_OutOfBuffers // "OutOfBuffers"
kCMSampleBufferDroppedFrameReason_Discontinuity // "Discontinuity"
From the docs it's really not clear how to get this value. I've tried using CMGetAttachment but this returns a CMAttachmentMode aka UInt32:
func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
var reason: CMAttachmentMode = 0
CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_DroppedFrameReason, &reason)
print("reason \(reason)") // 1
}
and I don't really know how to match this UInt32 to the CFString constant
I was stupidly not looking at the right output:
var mode: CMAttachmentMode = 0
let reason = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_DroppedFrameReason, &mode)
print("reason \(String(describing: reason))") // Optional(OutOfBuffers)