I'm trying to process CMSampleBuffers on the go, but I quickly stop receiving samples, even though I'm copying the samples and processing the copies.
I get the samples with the AVCaptureAudioDataOutputSampleBufferDelegate and process them like that:
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {
if (bufferCopy != nil) {
bufferCopy == nil
}
if CMSampleBufferCreateCopy(kCFAllocatorDefault, sampleBuffer, &bufferCopy) == noErr {
let encodedSamples = self.encoder.encodeSample(bufferCopy?.takeRetainedValue())
println(encodedSamples.count)
CMSampleBufferInvalidate(sampleBuffer)
// println("Copy: \(bufferCopy!.takeRetainedValue())")
}
else {
println("Failed to copy sampleBuffer")
}
}
Objective-C solution would work too.
I also created a GitHub repo with this example: https://github.com/Devxhkl/SwiftAudioStreamingOggOpus
Related
Can anyone please advise why this solution works? I have same question using Swift, when I define my 'Scanner' in a standalone swift file, the AVCaptureVideoDataOutputSampleBufferDelegate.CaptureOutput not called but, when I move the Scanner in the same file when I call session.addOutput(videoOutput) and videoOutput.setSampleBufferDelegate it works.
I am binginer to Swift/iOS dev, this issue drives me crazy, finally this solution works. thank you for sharing this!
just clarify and add my code:
I have a CamService.swift which I have a method to setup the AVCaptureSession:
session.beginConfiguration()
// ...
//add AVCaptureVideoDataOutput
if session.canAddOutput(videoOutput) {
session.addOutput(videoOutput)
videoOutput.alwaysDiscardsLateVideoFrames = true
videoOutput.setSampleBufferDelegate(self, queue:DispatchQueue.global(qos: .userInteractive) )
videoOutput.connections.first?.videoOrientation = .landscapeLeft
} else {
//fatalError("could not add video output")
print("Couldn't add video output to the session.")
session.commitConfiguration()
return
}
// ...
session.commitConfiguration()
when I extension CamService as below, it works, the captureOutput can be called
extension CamService: AVCaptureVideoDataOutputSampleBufferDelegate {
public func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection:
AVCaptureConnection) {
print("sampleBuffer")
}
}
But, if I create a new swift file say,
MyDalegate: AVCaptureVideoDataOutputSampleBufferDelegate,
and config
videoOutput.setSampleBufferDelegate(MyDalegate(),
queue:DispatchQueue.global(qos: .userInteractive) )
this won't work captureOutput never get called.
My application involves getting video frames from the camera and running CoreML on it. This will then get displayed on the view. To do this, I have an AVCaptureSession() that is hooked to a video output. The main processing of CoreML is being done in captureOutput(didOutput Samplebuffer). Each processing of the frame takes about 0.15 seconds, meaning that I WILL have some dropped frames.After CoreML process is done, I have an AVAssetWriter that append all these frames together and saves it within the phone directory.
THE main problem is however, that my use-case also requires the original video to be saved, and this video should have HIGH FPS and since I am only able to get the image frames only in captureOutput(didOutput), the video quality will be choppy.
I have tried the following:
The reason that I'm using AVAssetWriter is because it is given here: https://forums.developer.apple.com/thread/98113#300885 that it is NOT POSSIBLE to have both AVCaptureVideoDataOutput and AVCaptureMovieFileOutput.
I have also tried extracting Image buffer from the captureOutput(didDrop) using guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } but it is giving me nil. This is because sampleBuffer only contains metadata but not imagebuffer as explained here: https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1388468-captureoutput.
Here is some of my code:
func captureOutput(_ output: AVCaptureOutput, didDrop sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { /* HOW DO I possibly extract an image buffer here from the dropped frames here? */ }
func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let inputVideoImage = UIImage(pixelBuffer: pixelBuffer)
if self.isRecording{
let sourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
if let camera = self.videoWriterVideoInput, camera.isReadyForMoreMediaData {
videoWriterQueue.async() {
self.videoWriterInputPixelBufferAdaptor.append(pixelBuffer, withPresentationTime: sourceTime)
}
}else{
print("AVAssetInput is not ready for more media data ... ")
}
}
runCoreML()
}
I want to be able to track a users face from the camera feed. I have looked at this SO post. I used the code given in the answer but it did not seem to do anything. I have heard that
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!)
has been changed to something else in swift 4. Could this be the problem with the code?
While face tracking I want to also monitor face landmarks with CIFaceFeature. How would I do this?
I have found a starting point here: https://github.com/jeffreybergier/Blog-Getting-Started-with-Vision.
Basically you can instatiate a video capture session declaring a lazy variable like this:
private lazy var captureSession: AVCaptureSession = {
let session = AVCaptureSession()
session.sessionPreset = AVCaptureSession.Preset.photo
guard
let frontCamera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front),
let input = try? AVCaptureDeviceInput(device: frontCamera)
else { return session }
session.addInput(input)
return session
}()
Then inside viewDidLoad you start the session
self.captureSession.startRunning()
And finally you can perform your requests inside
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
}
for example:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer:
CMSampleBuffer, from connection: AVCaptureConnection) {
guard
// make sure the pixel buffer can be converted
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else { return }
let faceRequest = VNDetectFaceRectanglesRequest(completionHandler: self.faceDetectedRequestUpdate)
// perform the request
do {
try self.visionSequenceHandler.perform([faceRequest], on: pixelBuffer)
} catch {
print("Throws: \(error)")
}
}
And then you define your faceDetectedRequestUpdate function.
Anyway I have to say that I haven't been able to figure out how to create a working example from here. The best working example I have found is in Apple's documentation: https://developer.apple.com/documentation/vision/tracking_the_user_s_face_in_real_time
I'm using Google Mobile Vision to process camera frames and detect faces. After detecting a face I want to save the frame to a file but it has a wrong orientation.
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
guard let image = GMVUtility.sampleBufferTo32RGBA(sampleBuffer) else {
print("No Image")
return
}
// wrong orientation when saving image here
}
I've found code that I want to use in a swift ios app but I'm having trouble writing it the swift way.
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer( sampleBuffer );
CGSize imageSize = CVImageBufferGetEncodedSize( imageBuffer );
// also in the 'mediaSpecific' dict of the sampleBuffer
NSLog( #"frame captured at %.fx%.f", imageSize.width, imageSize.height );
}
It's just the function declaration I'm having trouble with, the contents of the function should be easier. I'm not used to the syntax of:
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
My attempt:
I started trying to write it like this but when nothing would autocomplete within the () I realised it was definitely wrong:
func captureOutput(AVCaptureOutput(didOutputSampleBuffer:)) {
}
You can find the correct declaration in the AVCaptureVideoDataOutputSampleBufferDelegate Protocol Reference, where it is declared like this:
optional func captureOutput(_ captureOutput: AVCaptureOutput!,
didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
fromConnection connection: AVCaptureConnection!)
You wouldn't write optional in your method definition, though:
class MyDelegate: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(captureOutput:AVCaptureOutput,
didOutputSampleBuffer sampleBuffer:CMSampleBuffer,
fromConnection connection:AVCaptureConnection) {
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let imageSize = CVImageBufferGetEncodedSize(imageBuffer)
NSLog("frame captured at %.fx%.f", imageSize.width, imageSize.height)
}
}