I created two applications: one for mac and one for iPhone. iPhone sends the video frames it captured to mac using MultipeerConnectivity framework. I have managed to find code for converting an UIimage to grayscale using this code:
func convertToGrayScale(image: UIImage) -> UIImage {
let imageRect:CGRect = CGRectMake(0, 0, image.size.width, image.size.height)
let colorSpace = CGColorSpaceCreateDeviceGray()
let width = image.size.width
let height = image.size.height
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.None.rawValue)
let context = CGBitmapContextCreate(nil, Int(width), Int(height), 8, 0, colorSpace, bitmapInfo.rawValue)
CGContextDrawImage(context, imageRect, image.CGImage)
let imageRef = CGBitmapContextCreateImage(context)
let newImage = UIImage(CGImage: imageRef!)
return newImage
}
In the code below, it sends the video frame to Mac:
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
CVPixelBufferLockBaseAddress(imageBuffer!, kCVPixelBufferLock_ReadOnly)
let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer!)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer!)
let width = CVPixelBufferGetWidth(imageBuffer!)
let height = CVPixelBufferGetHeight(imageBuffer!)
CVPixelBufferUnlockBaseAddress(imageBuffer!, 0)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedLast.rawValue)
let context = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, bitmapInfo.rawValue)
let quarzImage = CGBitmapContextCreateImage(context)
let image = UIImage(CGImage: quarzImage!)
let grayImage = convertToGrayScale(image)
let data: NSData = UIImagePNGRepresentation(grayImage)!
delegate?.recievedOutput(data)
}
The delegate method is just sending the data using session.sendData()
So, here comes to the Mac side. When mac received NSData, I created an NSImage from the data and created a .png image file using this code:
func session(session: MCSession, didReceiveData data: NSData, fromPeer peerID: MCPeerID) {
let image: NSImage = NSImage(data: data)!.imageRotatedByDegreess(270)
let cgRef = image.CGImageForProposedRect(nil, context: nil, hints: nil)
let representation = NSBitmapImageRep(CGImage: cgRef!)
let pngData = representation.representationUsingType(NSBitmapImageFileType.NSPNGFileType, properties: [NSImageCompressionFactor: 1.0])
pngData?.writeToFile("/Users/JunhongXu/Desktop/image/\(result.description).png", atomically: true)
result[4]++
self.delegate?.presentRecievedImage(image)
}
Although the image is like the picture below, when I checked my image file property, it is in RGB format. How can I change the ColorSpace of my NSImage to grayscale instead of RGB?
enter image description here
I have found a simple solution to my problem. Since it is already in grayscale when it transimitted to my Mac, I am able to use the code below to convert the image representation's ColorSpace to grayscale and save it as a .png file:
let newRep = representation.bitmapImageRepByConvertingToColorSpace(NSColorSpace.genericGrayColorSpace(), renderingIntent: NSColorRenderingIntent.Default)
let pngData = newRep!.representationUsingType(NSBitmapImageFileType.NSPNGFileType, properties: [NSImageCompressionFactor: 1.0])
pngData?.writeToFile("/Users/JunhongXu/Desktop/image/\(result.description).png", atomically: true)
Related
I am using the below code to get and append a pixel buffer from a metal layer. On some non specific devices the output looks like below and the drawable textures pixelFormat is .invalid
static func make(with currentDrawable: CAMetalDrawable, usingBuffer pool: CVPixelBufferPool) -> (CVPixelBuffer?, UIImage) {
let destinationTexture = currentDrawable.texture
var pixelBuffer: CVPixelBuffer?
_ = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &pixelBuffer)
if let pixelBuffer = pixelBuffer {
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.init(rawValue: 0))
let region = MTLRegionMake2D(0, 0, Int(currentDrawable.layer.drawableSize.width), Int(currentDrawable.layer.drawableSize.height))
let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
let tempBuffer = CVPixelBufferGetBaseAddress(pixelBuffer)
destinationTexture.getBytes(tempBuffer!, bytesPerRow: Int(bytesPerRow), from: region, mipmapLevel: 0)
let image = imageFromCVPixelBuffer(buffer: pixelBuffer)
CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.init(rawValue: 0))
return (pixelBuffer, image)
}
return (nil, UIImage())
}
static func imageFromCVPixelBuffer(buffer: CVPixelBuffer) -> UIImage {
let ciimage = CIImage(cvPixelBuffer: buffer)
let cgimgage = context.createCGImage(ciimage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(buffer), height: CVPixelBufferGetHeight(buffer)))
let uiimage = UIImage(cgImage: cgimgage!)
return uiimage
}
Does anybody have any idea why this happens and how to prevent it?
There are several more feedback from people experiencing this can be found here: https://github.com/svtek/SceneKitVideoRecorder/issues/3
I'm trying to convert a CMSampleBuffer to an UIImage with Swift 3.0. A popular solution is to write an extension for the CMSampleBuffer class and add a getter to convert the buffer to an image. This is what it looks like:
import Foundation
import AVFoundation
extension CMSampleBuffer {
#available(iOS 9.0, *)
var uiImage: UIImage? {
guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil }
let ciimage: CIImage = CIImage(cvImageBuffer: imageBuffer)
let image:UIImage = UIImage(ciImage: ciimage)
return image
}
}
It works fine but it's taking up a lot of memory, 40% of the total app memory. Is there a more memory efficient solution?
EDIT:
I have changed my code and it looks like this:
var uiImage: UIImage? {
guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil }
CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags(rawValue: 0))
let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.noneSkipFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)
var image: UIImage?
autoreleasepool(invoking: {() -> () in
guard let context = CGContext(data: baseAddress,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bytesPerRow,
space: colorSpace,
bitmapInfo: bitmapInfo.rawValue) else { return }
guard let cgImage = context.makeImage() else { return }
image = UIImage(cgImage: cgImage)
})
CVPixelBufferUnlockBaseAddress(imageBuffer,CVPixelBufferLockFlags(rawValue: 0));
return image
}
The memory leak has something to do with the CGContext. Is there any other way I can free/release/deallocate it besides using an autoreleasepool?
I have a set of user-generated PNG images that I need to embed into print-ready PDFs. One of the constraints set by the printing company is that all images be in the CMYK colorspace. How do I convert a PNG (or UIImage containing that PNG) to CMYK? By default the colorspace is sRGB.
You should consider CGColorSpace class and CGColorSpaceCreateDeviceCMYK function.
You can create a CGContex using this initializer and pass CMYK color space as a parameter to it. Then just draw a CGImage from that context and create UIImage from it.
Example:
func createCMYK(fromRGB image: UIImage) -> UIImage? {
let size = image.size
let colorSpace:CGColorSpace = CGColorSpaceCreateDeviceCMYK()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue)
guard let context = CGContext(data: nil,
width: Int(size.width),
height: Int(size.height),
bitsPerComponent: 8,
bytesPerRow: 4 * Int(size.width),
space: colorSpace,
bitmapInfo: bitmapInfo.rawValue) else { return nil }
context.draw(image.cgImage!, in: CGRect(origin: .zero, size: size))
guard let cgImage = context.makeImage() else { return nil }
return UIImage(cgImage: cgImage)
}
According to https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/drawingwithquartz2d/dq_context/dq_context.html#//apple_ref/doc/uid/TP30001066-CH203 "Supported Pixel Formats" table you must use
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue)
to correctly create a context
I am displaying a video feed of CMSampleBuffers converted to UIImages inside a UIImageView. In the photo below, the background layer is an AVCapturePreviewLayer and the center is the buffer feed. My goal is to remove the blue tint.
Here is the CMSampleBuffer to UIImage code
extension CMSampleBuffer {
func imageRepresentation() -> UIImage? {
let imageBuffer: CVImageBufferRef = CMSampleBufferGetImageBuffer(self)!
CVPixelBufferLockBaseAddress(imageBuffer, 0)
let address = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGBitmapContextCreate(address, width, height, 8, bytesPerRow, colorSpace, CGImageAlphaInfo.NoneSkipFirst.rawValue)
let imageRef = CGBitmapContextCreateImage(context)
CVPixelBufferUnlockBaseAddress(imageBuffer, 0)
let resultImage: UIImage = UIImage(CGImage: imageRef!)
return resultImage
}
}
AVCaptureVideoDataOutput setup:
class MovieRecorder: NSObject {
// vars
private let captureVideoDataOutput = AVCaptureVideoDataOutput()
// capture session boilerplate setup...
captureVideoDataOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey: Int(kCVPixelFormatType_32BGRA)]
captureVideoDataOutput.alwaysDiscardsLateVideoFrames = true
captureVideoDataOutput.setSampleBufferDelegate(self, queue: captureDataOutputQueue)
}
The problem was with the bitmapInfo. This bitmap info fixed it.
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.NoneSkipFirst.rawValue | CGBitmapInfo.ByteOrder32Little.rawValue)
let context = CGBitmapContextCreate(address, width, height, 8, bytesPerRow, colorSpace, bitmapInfo.rawValue)
I'm trying to implement uncompressed photos in a little practice app of mine, but I can't get it to work for the life of me.
This is how I capture a photo:
print("Shutter!")
if let videoConnection = stillImageOutput.connectionWithMediaType(AVMediaTypeVideo) {
stillImageOutput.captureStillImageAsynchronouslyFromConnection(videoConnection) {
(imageDataSampleBuffer, error) -> Void in
if let cameraFrame = CMSampleBufferGetImageBuffer(imageDataSampleBuffer) {
CVPixelBufferLockBaseAddress(cameraFrame, 0)
let rawImageBytes = CVPixelBufferGetBaseAddress(cameraFrame)
let bytesPerRow = CVPixelBufferGetBytesPerRow(cameraFrame)
let width = CVPixelBufferGetWidth(cameraFrame)
let height = CVPixelBufferGetHeight(cameraFrame)
CVPixelBufferUnlockBaseAddress(cameraFrame, 0)
let provider = CGDataProviderCreateWithData(nil, rawImageBytes, width * height * 4, nil)
let bitsPerComponent = 8
let bitsPerPixel = 32
let colorSpaceRef = CGColorSpaceCreateDeviceRGB()
let bitmapInfo: CGBitmapInfo = .ByteOrderDefault
let renderingIntent: CGColorRenderingIntent = .RenderingIntentDefault
if let imageRef = CGImageCreate(width, height, bitsPerComponent, bitsPerPixel, bytesPerRow * 4, colorSpaceRef, bitmapInfo, provider, nil, false, renderingIntent) {
print("Attempting to save image")
let tiffImage = UIImage(CGImage: imageRef)
UIImageWriteToSavedPhotosAlbum(tiffImage, nil, "image:didFinishSavingWithError:contextInfo:", nil)
} else {
print("CGImageCreate failed")
}
} else {
print("CMSampleBufferGetImageBuffer failed")
}
}
}
Executing that chunk of code prints "Shutter!" and "Attempting to save image" to Xcode's output panel, then crashes the app, causing Xcode to show me a bunch of what I think is Assembly code. Needless to say, this doesn't help me at all.
I have tried displaying the UIImage in a UIImageView, which also caused the app to crash. What's going on here?