I am working on a video editing app where each video gets squared in such a way that no portion of the video gets cropped.For this, in case of portrait video, it contains black portion on left & right and for landscape video, it contains black portion on top & bottom side of the video. Black portions are part of the video, they are not for AVPlayerViewController. Here is the sample,
I need to cover these black portions with some CALayers.
What will be the frame(CGRect) of the CALayer?
I am getting the video dimension with naturalSize property which includes the black portions.
Is there any way to get the video dimension without the black portions?(I mean the dimension of actual video content) or
is there any way to get the CGRect of black area of the video?
func initAspectRatioOfVideo(with fileURL: URL) -> Double {
let resolution = resolutionForLocalVideo(url: fileURL)
guard let width = resolution?.width, let height = resolution?.height else {
return 0
}
return Double(height / width)
}
private func resolutionForLocalVideo(url: URL) -> CGSize? {
guard let track = AVURLAsset(url: url).tracks(withMediaType: AVMediaType.video).first else { return nil }
let size = track.naturalSize.applying(track.preferredTransform)
return CGSize(width: abs(size.width), height: abs(size.height))
}
This is a more concise version of Vlad Pulichev's answer.
var aspectRatio: CGFloat! // use the function to assign your variable
func getVideoResolution(url: String) -> CGFloat? {
guard let track = AVURLAsset(url: URL(string: url)!).tracks(withMediaType: AVMediaType.video).first else { return nil }
let size = track.naturalSize.applying(track.preferredTransform)
return abs(size.height) / abs(size.width)
}
Related
I have an app which does real time filtering on camera feed, i'm getting each frame from camera and then do some filtering using CIFilter and then pass the final frame(CIImage) to MTKView to be shown on my swiftUI view, it works fine, but when i want to do face/body detection, real time, on camera feed, frame rate goes down to 8 frames per second and super laggy.
i tried anything i could find on the internet, using vision, CIDetector, CoreML, everything is the same result, well, i would do this on global thread, which makes the UI responsive but the feed which i'm showing into the main view is still laggy, but things like scrollview are working fine.
so i tried to change the view from MTKView to UIImageView, Xcode shows its rendering at 120FPS (which i dont understand why, its 30FPS when not using any face detection) but the feed is still laggy, cannot keep up somehow to the output frame rate, i'm new to this, i dont understand why is it like that.
i also tried just to pass the coming image to MTKView (without any filtering in between, with face detection) also the same laggy result, without face detection, it goes to 30FPS (why not 120?).
this is the code i'm using for converting sampleBuffer to ciImage
extension CICameraCapture: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
var ciImage = CIImage(cvImageBuffer: imageBuffer)
if self.cameraPosition == AVCaptureDevice.Position.front {
ciImage = ciImage.oriented(.downMirrored)
}
ciImage = ciImage.transformed(by: CGAffineTransform(rotationAngle: 3 * .pi / 2))
ciImage = ciImage.transformToOrigin(withSize: ciImage.extent.size)
detectFace(image: ciImage) // this is for detecting face realtime, i have done it in vision
//and also cidetector - cidetector is a little bit faster when setted to low accuracy
//but still not desired result(frame rate)
DispatchQueue.main.async {
self.callback(ciImage)
}
}
}
and this is the MTKView code, which is very simple and basic implementation of it:
import MetalKit
import CoreImage
class MetalRenderView: MTKView {
//var textureCache: CVMetalTextureCache?
override init(frame frameRect: CGRect, device: MTLDevice?) {
super.init(frame: frameRect, device: device)
if super.device == nil {
fatalError("No support for Metal. Sorry")
}
framebufferOnly = false
preferredFramesPerSecond = 120
sampleCount = 2
}
required init(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
private lazy var commandQueue: MTLCommandQueue? = {
[unowned self] in
return self.device!.makeCommandQueue()
}()
private lazy var ciContext: CIContext = {
[unowned self] in
return CIContext(mtlDevice: self.device!)
}()
var image: CIImage? {
didSet {
renderImage()
}
}
private func renderImage() {
guard var image = image else { return }
image = image.transformToOrigin(withSize: drawableSize) // this is an extension to resize
//the image to the render size so i dont get the render error while rendering a frame
let commandBuffer = commandQueue?.makeCommandBuffer()
let destination = CIRenderDestination(width: Int(drawableSize.width),
height: Int(drawableSize.height),
pixelFormat: .bgra8Unorm,
commandBuffer: commandBuffer) { () -> MTLTexture in
return self.currentDrawable!.texture
}
try! ciContext.startTask(toRender: image, to: destination)
commandBuffer?.present(currentDrawable!)
commandBuffer?.commit()
draw()
}
}
and here is the code for face detection using CIDetector:
func detectFace (image: CIImage){
//DispatchQueue.global().async {
let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh,
CIDetectorSmile: true, CIDetectorTypeFace: true] as [String : Any]
let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil,
options: options)!
let faces = faceDetector.features(in: image)
if let face = faces.first as? CIFaceFeature {
AppState.shared.mouth = face.mouthPosition
AppState.shared.leftEye = face.leftEyePosition
AppState.shared.rightEye = face.rightEyePosition
}
//}
}
what I have tried
1) different face detection methods, using Vision, CIDetector and also CoreML(this one not very deeply as i dont have experience in it)
I would get the detection info, but frame rate is 8 or at the best case its 15 (which would be a delayed detection)
2) I've read somewhere that it might be result of the image colorsapce so i have tried different video setting and different rendering colorspace, still no change in the frame rate.
3) I'm somehow sure that it might be regarding to pixelbuffer release time, so i deep copied the imageBuffer and pass it to the detection, beside some memory issues it went up to 15 FPS, but still not minimum 30FPS. in here i also tried to convert imageBuffer to ciimage and then render ciimage to cgimage and the back to ciimage to just release the buffer, but also could not get more than 15FPS (well on average, sometimes goes to 17 or 19, but still laggy)
i'm new in this and still trying to figure it out, i would appreciate any suggestions, samples or tips that could direct me to a better path of solving this.
update
this is the camera capture setup code:
class CICameraCapture: NSObject {
typealias Callback = (CIImage?) -> ()
private var cameraPosition = AVCaptureDevice.Position.front
var ciContext: CIContext?
let callback: Callback
private let session = AVCaptureSession()
private let sampleBufferQueue = DispatchQueue(label: "buffer", qos: .userInitiated)//, attributes: [], autoreleaseFrequency: .workItem)
// face detection
//private var sequenceHandler = VNSequenceRequestHandler()
//var request: VNCoreMLRequest!
//var visionModel: VNCoreMLModel!
//let detectionQ = DispatchQueue(label: "detectionQ", qos: .background)//, attributes: [], autoreleaseFrequency: .workItem)
init(callback: #escaping Callback) {
self.callback = callback
super.init()
prepareSession()
ciContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!)
}
func start() {
session.startRunning()
}
func stop() {
session.stopRunning()
}
private func prepareSession() {
session.sessionPreset = .high //.hd1920x1080
let cameraDiscovery = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInDualCamera, .builtInWideAngleCamera], mediaType: .video, position: cameraPosition)
guard let camera = cameraDiscovery.devices.first else { fatalError("Can't get hold of the camera") }
//try! camera.lockForConfiguration()
//camera.activeVideoMinFrameDuration = camera.formats[0].videoSupportedFrameRateRanges[0].minFrameDuration
//camera.activeVideoMaxFrameDuration = camera.formats[0].videoSupportedFrameRateRanges[0].maxFrameDuration
//camera.unlockForConfiguration()
guard let input = try? AVCaptureDeviceInput(device: camera) else { fatalError("Can't get hold of the camera") }
session.addInput(input)
let output = AVCaptureVideoDataOutput()
output.videoSettings = [:]
//print(output.videoSettings.description)
//[875704438, 875704422, 1111970369]
//output.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String : Int(kCVPixelFormatType_32BGRA)]
output.setSampleBufferDelegate(self, queue: sampleBufferQueue)
session.addOutput(output)
session.commitConfiguration()
}
}
I've created several UIViews inside of a UIScrollView that resize dynamically based on values I type into Height and Width text fields. Once the UIView resizes I save the contents of the UIScrollView as PDF data.
I find that the dimensions of the UIView within the PDF (when measured in Adobe Illustrator) are always rounded to a third.
For example:
1.5 -> 1.333
1.75 -> 1.666
I check the constant values each time before the constraints are updated and they are accurate. Can anyone explain why the UIViews have incorrect dimensions once rendered as a PDF?
#IBAction func updateDimensions(_ sender: Any) {
guard let length = NumberFormatter().number(from:
lengthTextField.text ?? "") else { return }
guard let width = NumberFormatter().number(from:
widthTextField.text ?? "") else { return }
guard let height = NumberFormatter().number(from:
heightTextField.text ?? "") else { return }
let flapHeight = CGFloat(truncating: width)/2
let lengthFloat = CGFloat(truncating: length)
let widthFloat = CGFloat(truncating: width)
let heightFloat = CGFloat(truncating: height)
UIView.animate(withDuration: 0.3) {
self.faceAWidthConstraint.constant = lengthFloat
self.faceAHeightConstraint.constant = heightFloat
self.faceBWidthConstraint.constant = widthFloat
self.faceA1HeightConstraint.constant = flapHeight
self.view.layoutIfNeeded()
}
}
func createPDFfrom(aView: UIView, saveToDocumentsWithFileName fileName: String)
{
let pdfData = NSMutableData()
UIGraphicsBeginPDFContextToData(pdfData, aView.bounds, nil)
UIGraphicsBeginPDFPage()
guard let pdfContext = UIGraphicsGetCurrentContext() else { return }
aView.layer.render(in: pdfContext)
UIGraphicsEndPDFContext()
if let documentDirectories = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true).first {
let documentsFileName = documentDirectories + "/" + fileName
debugPrint(documentsFileName)
pdfData.write(toFile: documentsFileName, atomically: true)
}
}
You should not be using layer.render(in:) to render your pdf. The reason its always a third is because you must be on a 3x device (it would be 1/2 on a 2x device and simply 1 on a 1x device), so there are 3 pixels per point. When iOS converts your constraints to pixels, then best it can do is round to the nearest third because its picking an integer pixel. The pdf can have much higher pixel density (or use vector art with infinite) resolution, so instead of using layer.render(in:), which is dumping pixels in the rasterized vector layer into your PDF, you should actually draw the contents into the PDF context manually (ie use UIBezier curve, UIImage.draw, etc). This will allow the pdf to capture the full resolution of any rasterized images you have and will allow it to capture any vectors you use without degrading them into rasterized pixels that are constrained by the device screen that you happen to be on.
I'm trying to add highlight annotations to documents using PDFKit on iOS.
let highlight = PDFAnnotation(bounds: selection.bounds(for: page),
forType: PDFAnnotationSubtype.highlight,
withProperties: nil)
highlight.color = color
page.addAnnotation(highlight)
page.displaysAnnotations = true
When adding them using the code above, they appear as two differently shaped layers. When saving them to the PDF file and reopening it they display correctly.
In this screen capture
The top and bottom highlights have been added the same way using the snippet provided here. The top one has been saved to the pdf document and appears as expected when reopening it, the bottom one has just been added.
Does anyone know how to have them display correctly (i.e. like the top one) without resorting to saving and reopening the file?
So this is a know bug in 10.13. There is a workaround by scrolling the page away and then back to the highlight
You can create the highlight using this code:
let page = self.pdfDocument?.page(at: 10)
let bounds = CGRect(x: 85.8660965, y: 786.8891167, width: 298.41, height: 12.1485)
let annotation = PDFAnnotation(bounds: bounds, forType: .highlight, withProperties: nil)
annotation.color = NSColor.blue
page?.addAnnotation(annotation)
Then you need to scroll away from the page and back to the highlight
func annotationScrollHack(page: PDFPage) {
guard let pdfDocument = self.pdfDocument else { return }
//When adding highlights to macOS 10.13 it seems like 2 highlights are added.
//If you scroll to a different page and back the "extra" highlight is removed
//This function scrolls to the first/last page in the book and then back to the current page
//rdar://34784917
let bookScrollView = self.pdfView.documentView?.enclosingScrollView
let currentVisibleRect = bookScrollView?.contentView.documentVisibleRect
if (0 ... 3).contains(pdfDocument.index(for: page)) {
if self.pdfView.canGoToLastPage {
self.pdfView.goToLastPage(self)
}
} else {
if self.pdfView.canGoToFirstPage {
self.pdfView.goToFirstPage(self)
}
}
if let currentVisibleRect = currentVisibleRect {
bookScrollView?.contentView.scroll(to: CGPoint(x: currentVisibleRect.origin.x, y: currentVisibleRect.origin.y))
}
}
Response from Apple:
There is no workaround other than the “hack" they described: scrolling
away then back is the best option. Changing zoom factor and reverting
it might fix it in the same way, but not guaranteed.
I've ended up with this hack method which takes into consideration all cases in my opinion. Hope that will help.
private func hackScroll(to page: PDFPage) {
switch (pdfView.canGoToFirstPage(), pdfView.canGoToLastPage()) {
case (true, false):
pdfView.goToFirstPage(nil)
pdfView.go(to: page)
case (false, true):
pdfView.goToLastPage(nil)
pdfView.go(to: page)
case (false, false):
guard let dir = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first else { return }
let file = "temp.pdf"
/// Save to temp file
let fileURL = dir.appendingPathComponent(file)
pdfView.document?.write(to: fileURL)
/// Read from temp file
guard let document = PDFDocument(url: fileURL) else { return }
pdfView.document = document
pdfView.autoScales = true
default:
pdfView.goToFirstPage(nil)
pdfView.go(to: page)
}
}
I've been generating QR Codes using the CIQRCodeGenerator CIFilter and it works very well:
But when I resize the UIImageView and generate again
#IBAction func sizeSliderValueChanged(_ sender: UISlider) {
qrImageView.transform = CGAffineTransform(scaleX: CGFloat(sender.value), y: CGFloat(sender.value))
}
I get a weird Border/DropShadow around the image sometimes:
How can I prevent it from appearing at all times or remove it altogether?
I have no idea what it is exactly, a border, a dropShadow or a Mask, as I'm new to Swift/iOS.
Thanks in advance!
PS. I didn't post any of the QR-Code generating code as it's pretty boilerplate and can be found in many tutorials out there, but let me know if you need it
EDIT:
code to generate the QR Code Image
private func generateQRCode(from string: String) -> UIImage? {
let data = string.data(using: String.Encoding.ascii)
guard let filter = CIFilter(name: "CIQRCodeGenerator") else {
return nil
}
filter.setValue(data, forKey: "inputMessage")
guard let qrEncodedImage = filter.outputImage else {
return nil
}
let scaleX = qrImageView.frame.size.width / qrEncodedImage.extent.size.width
let scaleY = qrImageView.frame.size.height / qrEncodedImage.extent.size.height
let transform = CGAffineTransform(scaleX: scaleX, y: scaleY )
if let outputImage = filter.outputImage?.applying(transform) {
return UIImage(ciImage: outputImage)
}
return nil
}
Code for button pressed
#IBAction func generateCodeButtonPressed(_ sender: CustomButton) {
if codeTextField.text == "" {
return
}
let newEncodedMessage = codeTextField.text!
let encodedImage: UIImage = generateQRCode(from: newEncodedMessage)!
qrImageView.image = encodedImage
qrImageView.transform = CGAffineTransform(scaleX: CGFloat(sizeSlider.value), y: CGFloat(sizeSlider.value))
qrImageView.layer.minificationFilter = kCAFilterNearest
qrImageView.layer.magnificationFilter = kCAFilterNearest
}
It’s a little hard to be sure without the code you’re using to generate the image for the image view, but that looks like a resizing artifact—the CIImage may be black or transparent outside the edges of the QR code, and when the image view size doesn’t match the image’s intended size, the edges get fuzzy and either the image-outside-its-boundaries or the image view’s background color start bleeding in. Might be able to fix it by setting the image view layer’s minification/magnification filters to “nearest neighbor”, like so:
imageView.layer.minificationFilter = kCAFilterNearest
imageView.layer.magnificationFilter = kCAFilterNearest
Update from seeing the code you added—you’re currently resizing the image twice, first with the call to applying(transform) and then by setting a transform on the image view itself. I suspect the first resize is adding the blurriness, which the minification / magnification filter I suggested earlier then can’t fix. Try shortening generateQRCode to this:
private func generateQRCode(from string: String) -> UIImage? {
let data = string.data(using: String.Encoding.ascii)
guard let filter = CIFilter(name: "CIQRCodeGenerator") else {
return nil
}
filter.setValue(data, forKey: "inputMessage")
if let qrEncodedImage = filter.outputImage {
return UIImage(cgImage: qrEncodedImage)
}
return nil
}
I think the problem here is that you try to resize it to "non-square" (as your scaleX isn't always the same as scaleY), while the QR code is always square so both side should have the same scale factor to get a non-blurred image.
Something like:
let scaleX = qrImageView.frame.size.width / qrEncodedImage.extent.size.width
let scaleY = qrImageView.frame.size.height / qrEncodedImage.extent.size.height
let scale = max(scaleX, scaleY)
let transform = CGAffineTransform(scaleX: scale, y: scale)
will make sure you have "non-bordered/non-blurred/squared" UIImage.
I guess the issue is with the image(png) file not with your UIImageView. Try to use another image and I hope it will work!
Is there a way to check the image dimensions (i.e. height and width) before downloading (or partially downloading) the image from a URL? I have found ways to get the image size, but that doesn't help.
Basically I want to calculate the correct height of a UITableView row before the image is downloaded. Is this possible?
You can do a partial download of the image data and then extract the image size from that. You will have to get the data structure of the image format you are using and parse it to some extent. It is possible and not that hard if you are capable of lower level coding.
You can do it by accessing its header details
In Swift 3.0 below code will help you,
if let imageSource = CGImageSourceCreateWithURL(url! as CFURL, nil) {
if let imageProperties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, nil) as Dictionary? {
let pixelWidth = imageProperties[kCGImagePropertyPixelWidth] as! Int
let pixelHeight = imageProperties[kCGImagePropertyPixelHeight] as! Int
print("the image width is: \(pixelWidth)")
print("the image height is: \(pixelHeight)")
}
}
Create a IBOutlet of HeightConstraint
e.g Here image is in 16:9 ratio from server
This will automatically adjust height for all screen. ImageView ContentMode is AspectFit
override func viewDidLoad() {
cnstHeight.constant = (self.view.frame.width/16)*9
}
Swift 4 Method:
func getImageDimensions(from url: URL) -> (width: Int, height: Int) {
if let imageSource = CGImageSourceCreateWithURL(url as CFURL, nil) {
if let imageProperties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, nil) as Dictionary? {
let pixelWidth = imageProperties[kCGImagePropertyPixelWidth] as! Int
let pixelHeight = imageProperties[kCGImagePropertyPixelHeight] as! Int
return (pixelWidth, pixelHeight)
}
}
return (0, 0)
}