How to obtain detailed zoom specs for iPhone camera(-s)? [closed] - ios

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 months ago.
Improve this question
I'm making a zoom control for my app, and I'd like to make it advanced, like in default camera app from Apple: sample
I did some research, and still have some questions about it.
Is it possible to get focal length value programmaticaly? There are labels like 13mm, 26mm for different back cameras in the default app, but there is no such property on AVCaptureDevice. (It is probably needed to determine zoom values, see the next question)
How can we determine zoom values to display in UI? The thing is that AVCaptureDevice's minZoomFactor always starts from 1x, but in the camera app we can see that on devices with ultrawide camera the scale starts at 0.5x, so there should be some way to map this values onto each other. As I understand, Apple considers "usual" back camera as default (that is, 1x), and all other values are relative to it: 13mm is 0.5 * 26mm, so the first value on iphone 13 pro zoom control will be 0.5x, the second value is the "default" and is 1x (26mm), and telephoto camera is 77mm, so the third value is 3x (26mm * 3 = 78mm ~= 77mm). Please clarify how it is actually calculated and correct me if my assumption is wrong.
What is the correct way to get max zoom value? If I try AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInTripleCamera], mediaType: .video, position: .back).devices.first!.maxAvailableVideoZoomFactor, it says 123.75 (iphone 13 pro), but in the default camera app max zoom value is 15x. Why is it exactly 15x and where does it come from? (My assumption is that max digital zoom for all iPhones equals to 5x, so on 13 Pro telephoto camera zooms 3x as "usual" camera, thus we get 3x * 5x = 15x max zoom)
Is there any universal way to get "the best" (i.e with all features) camera? For example, now I can specify [.builtInTripleCamera, .builtInDualWideCamera, .builtInDualCamera, .builtInWideAngleCamera] for discovery session and pick the first item in devices array, but if Apple will release, lets say, some ".builtInQuadrupleCamera" in a couple of years, this code will have to be modified, because it won't include it automatically.
To sum up (TL;DR version):
As I suppose, final code should look something like this:
let deviceTypes: [AVCaptureDevice.DeviceType]
if #available(iOS 13, *) {
deviceTypes = [.builtInTripleCamera, .builtInDualWideCamera, .builtInDualCamera, .builtInWideAngleCamera]
} else {
deviceTypes = [.builtInDualCamera, .builtInWideAngleCamera]
}
let session: AVCaptureDevice.DiscoverySession(
deviceTypes: deviceTypes,
mediaType: .video,
position: .back
)
if let device = session.devices.first {
device.getUIZoomValues()
}
extension AVCaptureDevice {
func getUIZoomValues() -> [Float] {
// Hardcode. Seems like all iPhones limit digital zoom to 5x
let maxDigitalZoom: Float = 5
// fallback for old iOS versions
guard #available(iOS 13, *) else { return [1, maxDigitalZoom] }
let uiZoomValues: [Float]
let factors = virtualDeviceSwitchOverVideoZoomFactors
switch deviceType {
case .builtInTripleCamera, .builtInDualWideCamera:
// ultrawide camera is available - starting zoom from 0.5x
let firstZoom: Float = 1.0 / factors.first!.floatValue
uiZoomValues = [firstZoom] + factors.map { $0.floatValue * firstZoom } + [firstZoom * factors.last!.floatValue * maxDigitalZoom]
case .builtInDualCamera:
// no ultrawide. Starting from 1x
uiZoomValues = [1.0] + factors.map { $0.floatValue } + [factors.last!.floatValue * maxDigitalZoom]
case .builtInWideAngleCamera:
// just a single "usual" camera.
uiZoomValues = [1, maxDigitalZoom]
default:
fatalError("this should not happen on a real device")
}
return uiZoomValues
}
}
2 main concerns about this code:
1 - We have to hardcode maxDigitalZoom. Is there any way to get it programmaticaly? Apple states 5x in iPhone specs, and there is AVCaptureDevice.maxAvailableVideoZoomFactor, but those values are different (for example, iPhone 13 pro has 15x in specs vs 123.75x in maxAvailableVideoZoomFactor).
2 - Case builtInDualCamera (iPhone XS Max, for example). All the code above relies on virtualDeviceSwitchOverVideoZoomFactors var, which is available only from iOS 13, but builtInDualCamera is available from iOS 10.2, so what will happen if user has XS Max? Will it work on iOS >= 13 but break on earlier versions? Or it will not work at all?

Questions in order:
I think not
Works for me:
Created dictionary var zoomFactors: [String: CGFloat] = ["1": 1]
Than manage AVCaptureDevice?
I think u can play with getApproximation() to achieve the goal
No
Code Questions:
No. but i think one of the easiest method it's to play with getApproximation() idea
I believe it will be crash

Related

Choosing suitable camera for barcode scanning when using AVCaptureDeviceTypeBuiltInTripleCamera

I've had some barcode scanning code in my iOS app for many years now. Recently, users have begun complaining that it doesn't work with an iPhone 13 Pro.
During investigation, it seemed that I should be using the built in triple camera if available. Doing that did fix it for iPhone 13 Pro but subsequently broke it for iPhone 12 Pro, which seemed to be working fine with the previous code.
How are you supposed to choose a suitable camera for all devices? It seems bizarre to me that Apple has suddenly made it so difficult to use this previously working code.
Here is my current code. The "fallback" section is what the code has used for years.
_session = [[AVCaptureSession alloc] init];
// Must use macro camera for barcode scanning on newer devices, otherwise the image is blurry
if (#available(iOS 13.0, *)) {
AVCaptureDeviceDiscoverySession * discoverySession =
[AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:#[AVCaptureDeviceTypeBuiltInTripleCamera]
mediaType:AVMediaTypeVideo
position:AVCaptureDevicePositionBack];
if (discoverySession.devices.count == 0) {
// no BuiltInTripleCamera
_device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
} else {
_device = discoverySession.devices.firstObject;
}
} else {
// Fallback on earlier versions
_device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
}
The accepted answer works but not all the time. Because lenses have different minimum focus distance it is harder for the device to focus on small barcodes because you have to put you device too close (before the minimum focus distance). This way it will never autofocus on small barcodes. It used to work on older lenses where autofocus was 10-12 cm but newer lenses especially those on iPhone 14 Pros that have the distance 20cm will be problematic.
The solution is to use ideally AVCaptureDeviceTypeBuiltInWideAngleCamera and setting videoZoomFactor on the AVCaptureDevice to zoom in little bit so the barcode will be nicely focused. The value should be calculated based on the input video properties and minimum size of barcode.
For details please refer to this WWDC 2019 video where they address exactly this issue https://developer.apple.com/videos/play/wwdc2021/10047/?time=133.
Here is implementation of class that sets zoom factor on a device that works for me. You can instantiate this class providing your device instance and call applyAutomaticZoomFactorIfNeeded() just before you are about to commit your capture session configuration.
///
/// Calling this method will automatically zoom the device to increase minimum focus distance. This distance appears to be problematic
/// when scanning barcodes too small or if a device's minimum focus distance is too large (like on iPhone 14 Pro and Max - 20cm, iPhone 13 Pro - 15 cm, older iPhones 12 or less.). By zooming
/// the input the device will be able to focus on a preview and complete the scan more easily.
///
/// - See https://developer.apple.com/videos/play/wwdc2021/10047/?time=133 for more detailed explanation and
/// - See https://developer.apple.com/documentation/avfoundation/capture_setup/avcambarcode_detecting_barcodes_and_faces
/// for implementation instructions.
///
#available(iOS 15.0, *)
final class DeviceAutomaticVideoZoomFactor {
enum Errors : Error {
case minimumFocusDistanceUnknown
case deviceLockFailed
}
private let device: AVCaptureDevice
private let minimumCodeSize: Float
init(device: AVCaptureDevice, minimumCodeSize: Float) {
self.device = device
self.minimumCodeSize = minimumCodeSize
}
///
/// Optimize the user experience for scanning QR codes down to smaller sizes (determined by `minimumCodeSize`, for example 2x2 cm).
/// When scanning a QR code of that size, the user may need to get closer than the camera's minimum focus distance to fill the rect of interest.
/// To have the QR code both fill the rect and still be in focus, we may need to apply some zoom.
///
func applyAutomaticZoomFactorIfNeeded() throws {
let deviceMinimumFocusDistance = Float(self.device.minimumFocusDistance)
guard deviceMinimumFocusDistance != -1 else {
throw Errors.minimumFocusDistanceUnknown
}
Logger.logIfStaging("Video Zoom Factor", "using device: \(self.device)")
Logger.logIfStaging("Video Zoom Factor", "device minimum focus distance: \(deviceMinimumFocusDistance)")
/*
Set an inital square rect of interest that is 100% of the view's shortest side.
This means that the region of interest will appear in the same spot regardless
of whether the app starts in portrait or landscape.
*/
let formatDimensions = CMVideoFormatDescriptionGetDimensions(self.device.activeFormat.formatDescription)
let rectOfInterestWidth = Double(formatDimensions.height) / Double(formatDimensions.width)
let deviceFieldOfView = self.device.activeFormat.videoFieldOfView
let minimumSubjectDistanceForCode = self.minimumSubjectDistanceForCode(fieldOfView: deviceFieldOfView,
minimumCodeSize: self.minimumCodeSize,
previewFillPercentage: Float(rectOfInterestWidth))
Logger.logIfStaging("Video Zoom Factor", "minimum subject distance: \(minimumSubjectDistanceForCode)")
guard minimumSubjectDistanceForCode < deviceMinimumFocusDistance else {
return
}
let zoomFactor = deviceMinimumFocusDistance / minimumSubjectDistanceForCode
Logger.logIfStaging("Video Zoom Factor", "computed zoom factor: \(zoomFactor)")
try self.device.lockForConfiguration()
self.device.videoZoomFactor = CGFloat(zoomFactor)
self.device.unlockForConfiguration()
Logger.logIfStaging("Video Zoom Factor", "applied zoom factor: \(self.device.videoZoomFactor)")
}
private func minimumSubjectDistanceForCode(fieldOfView: Float,
minimumCodeSize: Float,
previewFillPercentage: Float) -> Float {
/*
Given the camera horizontal field of view, we can compute the distance (mm) to make a code
of minimumCodeSize (mm) fill the previewFillPercentage.
*/
let radians = self.degreesToRadians(fieldOfView / 2)
let filledCodeSize = minimumCodeSize / previewFillPercentage
return filledCodeSize / tan(radians)
}
private func degreesToRadians(_ degrees: Float) -> Float {
return degrees * Float.pi / 180
}
}
Thankfully with the help of reddit I was able to figure out that the solution is simply to replace
AVCaptureDeviceTypeBuiltInTripleCamera
with
AVCaptureDeviceTypeBuiltInWideAngleCamera

why is my app's display on iPhone12 mini incorrectly sized?

I read the native height of the phone to size my display output, and determine top and bottom keep out areas. This is a tab based app. My display is purely programmatic, it is graphically layed out based on screen dimension only. On the iPhone 12 mini, the display is smaller than it should be. The other iPhone 12 types display correctly.
While testing my new app on various iPhone types in the xcode simulator, they
looked correct. But when I put the app on my iPhone 12 mini, the display was smaller than it should be. I learned the simulator returns the value 2436 for native height, but the phone returns the value 2340.
This code examines the display and tells me how much to reserve on the top and bottom of the screen by type of phone:
nativeHeight=UIScreen.main.nativeBounds.height
if nativeHeight==1136 || nativeHeight==1334 // SE,6S,7,8
{reserveTopPixels=40; reserveBottomPixels=98}
else if nativeHeight==1792 // 11, XR
{reserveTopPixels=88; reserveBottomPixels=198}
else if nativeHeight==2208 // 6s+ 7+ 8+
{reserveTopPixels=54; reserveBottomPixels=146}
else if nativeHeight==2436 || nativeHeight==2688 // X, XS, 11Pro, 11, Xr
{reserveTopPixels=132; reserveBottomPixels=260}
else if nativeHeight==2340 //iPhone 12 mini "downsampled screen"
{reserveTopPixels=132; reserveBottomPixels=260; nativeHeight = 2436}
else if nativeHeight==2532 // 12, 12 pro
{reserveTopPixels=132; reserveBottomPixels=260}
else if nativeHeight==2778 // 12 pro max
{reserveTopPixels=132; reserveBottomPixels=260}
else {reserveTopPixels=40; reserveBottomPixels=98}
My correction is to change the value received from the phone, 2340, to a larger value, 2436.
The app displays correctly now on both the simulator and the phone. My question is, is this a reasonable solution, or should I try something else.
nativeBounds: CGRect. : provides the bounding rectangle of the physical screen, measured in pixels. And the physical screen for iPhone 12 mini is 1080×2340 pixels
The iPhone 12 mini has a physical screen resolution of 1080 x 2340 pixels (476 ppi). This is lower than the native resolution reported by the operating system (using nativeBounds on UIScreen). I assume this is to maintain compatibility with the resolutions of the 5.8" models (iPhone X/XS/11 Pro).
Reference :
https://hacknicity.medium.com/how-ios-apps-adapt-to-the-various-iphone-12-screen-sizes-e45c021e1b8b
https://useyourloaf.com/blog/iphone-12-screen-sizes/
It depends on where you are using this nativeHeight to layout.
As you indicated you have problem just for on the iPhone 12 mini simulator. Then Why do you need all checking logic ? Just be sure you working on simulator and catch the iPhone 12 mini and make your adjustment. I think that extension would make sense.
static var isSimulator: Bool {
return TARGET_OS_SIMULATOR != 0
}
extension UIDevice {
var modelIdentifier = ""
modelIdentifier = ProcessInfo.processInfo.environment["SIMULATOR_MODEL_IDENTIFIER"] ?? ""
let nativeHeight = UIScreen.main.nativeBounds.height
if nativeHeight==2340 // your specific case
{
reserveTopPixels=132; reserveBottomPixels=260; nativeHeight = 2436
}
}
Edit
Then If you want certain code to be run only in the iOS simulator either device, you should use this:
func adjust() {
#if targetEnvironment(simulator)
// we're on the simulator
}
#else
// we're on a device
}
#endif
}

iPhone X/Xs Max AVCaptureVideoPreviewLayer scale factor and coordinates in resizeAspectFill mode

I am working on an application with standard AVCaptureDevice flow for getting and displaying frames from iPhone camera, but then it processes them through OpenCV algorithm and puts markers on the display.
The flow is:
Setting up a AVCaptureVideoPreviewLayer.
Getting frames from AVCaptureVideoDataOutputSampleBufferDelegate's function captureOutput
Converting frame to OpenCV mat and processing it.
Getting results from the algorithm as rectangles.
Scaling them back to iPhone's screen and showing in UI.
My current problem is that everything works accurately on rectangle screened devices (iPhone 7, 8, 7 Plus, 8 Plus), but I've got a lot of problems with top-notch devices like iPhone X, iPhone Xs Max and later.
The fact is that due to usage of previewLayer?.videoGravity = .resizeAspectFill on iPhone X family devices the image on the screen (e.g. in AVCaptureVideoPreviewLayer) gets scaled and cropped if to compare with the original from from the camera , but I can't calculate the exact difference to perform correct back scaling.
If I try to render results in OpenCV straight on the device and save them into the memory, the output image is correct. If I do all the scaling and rendering on rectangle-screened devices, the result is also correct. The only problem are top notch devices, as filling their screen with camera frames make them look differently.
I tried getting such methods as metadataOutputRectConverted, but couldn't understand the right usage of the results I get.
let metaRect = self.camera.previewLayer?.metadataOutputRectConverted(fromLayerRect: self.camera.previewLayer?.bounds ?? CGRect.zero) ?? CGRect.zero
// on iPhone 8 I get: (-3.442597823690085e-17, 0.0, 1.0, 1.0)
// so it means that width and height coefficients are 1 and it's nearly not skewed on both x and y,
//so it gives me the right result on the screen
// on iPhone X I get (-3.136083667459394e-17, 0.08949096880131369, 1.0, 0.8210180623973728)
// I see that there's a skew on Y axis and in height, but I don't know how to correctly use it
The code that I use to initialise the layer:
session.sessionPreset = AVCaptureSession.Preset.hd1920x1080
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer?.videoGravity = .resizeAspectFill
DispatchQueue.main.async {
layer.connection?.videoOrientation = orientation
layer.frame = UIScreen.main.bounds
view.layer.insertSublayer(layer, at: 0)
}
The code that I use to put objects at screen, resultRect I get from my C++ OpenCV module:
let aspectRatioWidth = CGFloat(1080)/UIScreen.main.bounds.size.width
let aspectRatioHeight = CGFloat(1920)/UIScreen.main.bounds.size.height
let width = CGFloat(resultRect.width) / aspectRatioWidth
let height = CGFloat(resultRect.height) / aspectRatioHeight
let rectx = CGFloat(resultRect.x) / aspectRatioWidth - width / 2.0
let recty = CGFloat(resultRect.y) / aspectRatioHeight - height / 2.0
I would appreciate any help, thank you very much in advance.

iOS 11.3 ARKit Autofocus and higher resolution

As mentioned here Apple is allowing us to use higher resolution and Autofocus in the apps based on ArKit, have anybody already tried implementing those features in own apps ?
Where apple is usually sharing more technical details about such updates ?
Regards !
You haven’t been able to set the ARCamera image resolution... which is why you may not find anything relating to adjusting the camera image resolution. Changing ARCamera Resolution
If you want to check the ARCamera image resolution you can do so access via the currentFrame.
let currentFrame = sceneView.session.currentFrame
print(currentFrame?.camera.imageResolution)
to date it was set to 1280.0, 720.0
if you want more information about focal-length, which i believe auto-focus may now be able to adjust automatically. You can just check the camera property of currentFrame.
print(currentFrame?.camera)
Ok, autofocus can be added with:
var isAutoFocusEnabled: Bool { get set }
var configuration = ARWorldTrackingConfiguration()
configuration.isAutoFocusEnabled = true // or false
Any clues what about higher res ?
I am able to select desired resolution for AR camera using following code -
ARWorldTrackingConfiguration* configuration = [ARWorldTrackingConfiguration new];
NSArray<ARVideoFormat*>* supportedVideoFormats = [ARWorldTrackingConfiguration supportedVideoFormats];
int bestFormatIndex = 0;
for (int i = 0; i < [supportedVideoFormats count]; i++) {
float width = supportedVideoFormats[i].imageResolution.width;
float height = supportedVideoFormats[i].imageResolution.height;
NSLog(#"AR Video Format %f x %f", width, height);
if (width * 9 == height * 16) { // <-- Use your own condition for selecting video format
bestFormatIndex = i;
break;
}
}
[configuration setVideoFormat:supportedVideoFormats[bestFormatIndex]];
// Run the view's session
[_mSceneView.session runWithConfiguration:configuration];
For my requirement I wanted biggest 16:9 ratio. On iPhone 8+, following image resolutions are getting listed -
1920 x 1440
1920 x 1080
1280 x 720
Notice that they are sorted in the supportedVideoFormats array, with biggest resolution at 0 index. And 0th index is the default selected video format.

on device face recognition - Swift [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have an app that takes all the images of the user (all the Assets from the Photo app). After that the app should run throw all the images and detect faces and return their facial landmarks, and then look in the database to see if there is any friend with the same landmark (recognizing friend faces), similar to what Facebook do on moments app and on the web. The app will then show all the photos that the friend appear in them. Important part of my app is the user privacy, so I would like to keep the entire process on the device, and not sending it to online service. Another benefit of keeping it on the device is that every user in my app can have thousands of images, and working with external service will be expansive and might low down the performance (if every image need to be sent to the server).
From the research that I done there are many online services (but they don't fit my requirements - keeping the process offline). There is also the CIDector that detect the faces, and then you can return few features such as eye location and mouth location (which I don't believe that is good enough for reliable recognition). I also heard about Luxand, openCV, and openFace, which are all on device recognition, but are C++ class, which make it difficult to integrate with swift project (the documentation are not very good, and don't explain how do integrate it to your project and how to perform face recognition on swift).
So my question is if there is any way to perform face detection that return the facial landmarks on the device?
If not is there any other way or service I could use.
Also if there is any efficient and fast way to perform face detection and recognition, if a user could have thousand of images.
By the way, I am in early stage of development and I am looking for free services that I could use for the development stage.
iOS have native face detection in CoreImage framework that works pretty cool. You can as well detect eyes etc. Just check out this code, there's show how you can work with it.
func detect() {
guard let personciImage = CIImage(image: personPic.image!) else {
return
}
let accuracy = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: accuracy)
let faces = faceDetector.featuresInImage(personciImage)
// converting to other coordinate system
let ciImageSize = personciImage.extent.size
var transform = CGAffineTransformMakeScale(1, -1)
transform = CGAffineTransformTranslate(transform, 0, -ciImageSize.height)
for face in faces as! [CIFaceFeature] {
print("Found bounds are \(face.bounds)")
// calculating place for faceBox
var faceViewBounds = CGRectApplyAffineTransform(face.bounds, transform)
let viewSize = personPic.bounds.size
let scale = min(viewSize.width / ciImageSize.width,
viewSize.height / ciImageSize.height)
let offsetX = (viewSize.width - ciImageSize.width * scale) / 2
let offsetY = (viewSize.height - ciImageSize.height * scale) / 2
faceViewBounds = CGRectApplyAffineTransform(faceViewBounds, CGAffineTransformMakeScale(scale, scale))
faceViewBounds.origin.x += offsetX
faceViewBounds.origin.y += offsetY
let faceBox = UIView(frame: faceViewBounds)
faceBox.layer.borderWidth = 3
faceBox.layer.borderColor = UIColor.redColor().CGColor
faceBox.backgroundColor = UIColor.clearColor()
personPic.addSubview(faceBox)
if face.hasLeftEyePosition {
print("Left eye bounds are \(face.leftEyePosition)")
}
if face.hasRightEyePosition {
print("Right eye bounds are \(face.rightEyePosition)")
}
}
}

Resources