I am working on iOS application which uses camera. I am using AVCaptureVideoDataOutput delegate method to get video frame. I always getting video frame with 1920 * 1080 regardless of device I am using which is iPhone X.
I am using AVCaptureSession.Preset.high
Here is my code snipped
func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
let ciImage = CIImage(cvPixelBuffer: CMSampleBufferGetImageBuffer(sampleBuffer))
let image = UIImage(ciImage: ciImage)
}
When I do
let device = AVCaptureDevice.devices(for: AVMediaType.video).first {
($0 as AVCaptureDevice).position == AVCaptureDevice.Position.back
}
print("resolutions supported:: \(String(describing: device?.activeFormat.highResolutionStillImageDimensions)))")
This always gives me 3840 * 2160 for iPhone x which is having 12 megapixel
I am expecting same kind of highest possible resolution video frame through AVCaptureVideoDataOutput.
I tried using AVCaptureSession.Preset.photo it also doesn't give me high resolution.
I did try AVCaptureSession.Preset.hd4K3840x2160 which gives me expected resolution for frame but it may not work with older iPhone???
I know AVCapturePhotoOutput can give me higher resolution image. But for my use case I want to create image from video frame.
What I am doing wrong here?
I agree with #adamfowlerphoto. And the answer Why you need to check and then apply 4K video preset is the function is according to hardware specification. Like, if your old phone doesn't have a high resolution sensor or lens which is good enough, you cannot use it;hd4K3840x2160
Related
I am trying to modify some sample code from Apple Developer codes for my own purpose (I am very new to programming for iOS). I am trying to get images from a camera and apply some detection and just show the detections.
Currently, I am using the AVCaptureVideoPreviewLayer and basically the camera feed gets displayed on the screen. I actually want to zero out the camera feed and draw some detections only. So, I am basically trying to handle this in the captureOutputfunction. So something like:
extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
// Grab the pixelbuffer frame from the camera output
guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
// Here now I should be able to set it 0s (all black)
}
}
I am trying to do something basic like setting this CVImageBuffer to a black background but have not been able to figure that out in the last hours!
EDIT
So, I discovered that I can do something like:
var image: CGImage?
// Create a Core Graphics bitmap image from the buffer.
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &image)
This copies the buffer data to the CGImage, which I can then use for my purposes. Now, is there an API that can basically make an all black image with the same size as one represented by the input image buffer?
Im instanciating the AVPlayerItemVideoOutput like so:
let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: [String(kCVPixelBufferPixelFormatTypeKey): NSNumber(value: kCVPixelFormatType_32BGRA)])
And retrieving the pixelBuffers like this:
#objc func displayLinkDidRefresh(link: CADisplayLink) {
let itemTime = videoOutput.itemTime(forHostTime: CACurrentMediaTime())
if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: nil) {
}
}
}
But for some reason CVPixelBufferGetHeight(pixelBuffer) or Width. always return 1280x720 when the video was taken when the iPhone's camera (landscape or portrait) always height=1280 width=720. EVEN if the video is 4k. If I load a square video from instagram or any other video downloaded from the internet (not created directly with the camera app) the width and height are printed correctly when the resolution is less than 720p. But a different resolution, for ex. a 1008x1792 will throw CVPixelBufferGetHeight(pixelBuffer) = 1280
Videos taken with the camera... it always throws a lower res. I tried 4k and 1080 settings (you can change that in iOS Settings > Camera). still.. even in 1080, I get 1280x720 pixel buffers.
I figured out that th UIPickerController I was using was set to default transcode the selected video from library to a Medium setting. in this case it was 1280x720
I ended up changing this properties of the picker
picker.videoQuality = .typeHigh
picker.videoExportPreset = AVAssetExportPresetHighestQuality
Altho the property that actually makes the change is the videoExportPreset the other one I dont know what it does, even tho the Documentation specifies it is for when you record a video OR you pick a video.
I'm capturing video via my device's camera, and feeding it to the Vision framework to perform rectangle detection. The code looks something like this (compressed for brevity ... hidden lines not relevant to this question):
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer:
CMSampleBuffer, from connection: AVCaptureConnection) {
// Get a CIImage from the buffer
guard let buffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
let image = CIImage(cvImageBuffer: buffer)
// Set up corner detector
let handler = VNImageRequestHandler(ciImage: image, orientation: .up options: [:])
let request = VNDetectRectanglesRequest()
// Perform corner detection
do {
try handler.perform([request])
guard let observation = request.results?.first as? VNRectangleObservation else {
print("error at \(#line)")
return
}
handleCorners(observation)
} catch {
print("Error: \(error)")
return
}
}
This works just fine on an iPad Air 2, and I can use the corners in the observation object to draw a nice overlay. But on an iPhone X the corners in the x-axis are "compressed".
For example, if I capture an image with a business card that occupies almost the entire width of the screen, I would expect observation.topLeft to have an x value close to zero. Instead it's nearly 0.15. This is true for the righthand corners too (expected: ~1.0, actual: ~0.85).
Any idea why this might be the case? The CIImage extent property is the same on both devices. It's just that Vision's corners are compressed in the x-axis.
I had a pretty similar problem with detecting rectangles in realtime using ARKit. And after some investigation I saw this answer and figure out that: "The problem is that ARKit provides the image buffer (frame.capturedImage) with a camera resolution of 1920 x 1440. The screen of the iPhone X is 375 x 812 points. It seems like ARKit can see more than it can display on the phone screen." So I just corrected capturedImage size using screen proportion, and this "solution" fix my problem.
I have made a face detection app that is "functional". However, the problem is that .featuresInImage() in CIdetector is not detecting faces every time func captureOutput() is called. I have already tried setting CIDetectorAccuracyLow too which didn't improve anything significantly.
I have tried both my application and the native iPhone camera which the latter can detect faces in an instance, even with faces that are slightly blocked (e.g. glasses). Why is this? Is Apple using a different face detecting algorithm from this one? Or is there some optimizing I should do before sending to CIDetector?
Code as below for further reference:
private var faceDetector: CIDetector?
dynamic var faceFeature: CGRect
...
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {
// This function is called whenever a new frame is available
let pixelBuffer: CVPixelBufferRef = CMSampleBufferGetImageBuffer(sampleBuffer)!
let inputImage = CIImage(CVPixelBuffer: pixelBuffer)
let features = self.faceDetector!.featuresInImage(inputImage)
for feature in features as! [CIFaceFeature] {
faceFeature = feature.bounds
print("\(faceFeature)")
}
}
.
UPDATE:
I have further tested my "functional" code, and it seems that there are certain times when my face is at a certain angle and size, .featuresInImage() will detect my face with the video frame rate.
Does this mean the CIDetector is working correctly but I'll have to do some adjustments to the input sample? Make it more easy for the algorithm to run?
I am trying to process the video frames and extracting the concentrated color out of it. I was using the AVCaptureStillImageOutput but it was making the shutter sound everytime I take a frame for the processing so I switched to AVCaptureVideoDataOutput and now processing each frame as it comes by.
Here is the code I am using:
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {
currentFrame = self.convertImageFromCMSampleBufferRef(sampleBuffer);
if let image = UIImage(CIImage: currentFrame){
if let color = self.extractColor(image) {
// print the color code
}
}
}
func convertImageFromCMSampleBufferRef(sampleBuffer:CMSampleBuffer) -> CIImage{
let pixelBuffer:CVPixelBufferRef = CMSampleBufferGetImageBuffer(sampleBuffer);
var ciImage:CIImage = CIImage(CVPixelBuffer: pixelBuffer)
return ciImage;
}
With the AVCaptureStillImageOutput I was getting almost correct output but with the AVCaptureVideoDataOutput the values are always near to black even if the camera view is into the bright light. I am guessing the problem is around the framerate or something but not able to figure it out.
In the last few test run this is the only color code I am getting #1b1f01
I would love to use the original AVCaptureStillImageOutput code but it should not make the Shutter sound and I am not able to disable it.
Had this same issue myself. It was just that it was very early; for whatever reason the camera sensor starts at 0 and is willing to give you frames before what you'd think of as the first frame is fully exposed.
Solution: just wait a second before you expect any real images.