Compute the histogram of an image using vImageHistogramCalculation in swift - ios

I'm trying to compute the histogram of an image using Accelerate vImageHistogramCalculation_ARGBFFFF function, but I'm getting a vImage_Error of type kvImageNullPointerArgument (error code is -21772).
This is the exact same question, but I'm working in Swift: Compute the histogram of an image using vImageHistogramCalculation
// Get CGImage from UIImage
var image:UIImage = UIImage(named: "happiness1")!
var img:CGImageRef = image.CGImage
// Create vImage_Buffer with data from CGImageRef
var inProvider:CGDataProviderRef = CGImageGetDataProvider(img)
var inBitmapData:CFDataRef = CGDataProviderCopyData(inProvider)
// The next three lines set up the inBuffer object
var height:vImagePixelCount = CGImageGetHeight(img)
var width:vImagePixelCount = CGImageGetWidth(img)
var rowBytes:UInt = CGImageGetBytesPerRow(img)
var data:UnsafePointer<Void> = UnsafePointer<Void>(CFDataGetBytePtr(inBitmapData))
// Setup inBuffer
var inBuffer = vImage_Buffer(data: &data, height: height, width: width, rowBytes: rowBytes)
var histogram_entries:UInt32 = 4
var minVal:Pixel_F = 0
var maxVal:Pixel_F = 255
//let flags:vImage_Flags = kvImageNoFlags = 0
var histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>()
var error:vImage_Error = vImageHistogramCalculation_ARGBFFFF(&inBuffer, histogram, histogram_entries, minVal, maxVal, 0)
println(error)
The problem is in the histogram variable, I need to recreate something like this:
// create an array of four histograms with eight entries each.
vImagePixelCount histogram[4][8] = {{0}};
// vImageHistogramCalculation requires an array of pointers to the histograms.
vImagePixelCount *histogramPointers[4] = { &histogram[0][0], &histogram[1][0], &histogram[2][0], &histogram[3][0] };
vImage_Error error = vImageHistogramCalculation_ARGBFFFF(&inBuffer, histogramPointers, 8, 0, 255, kvImageNoFlags);
// You can now access bin j of the histogram for channel i as histogram[i][j].
// The storage for the histogram will be cleaned up when execution leaves the
// current lexical block.
Suggestion?

I've implemented vImageHistogramCalculation_ARGB8888 as an extension to UIImage in Swift with the following:
func SIHistogramCalculation() -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) {
let imageRef = self.CGImage
let inProvider = CGImageGetDataProvider(imageRef)
let inBitmapData = CGDataProviderCopyData(inProvider)
var inBuffer = vImage_Buffer(data: UnsafeMutablePointer(CFDataGetBytePtr(inBitmapData)), height: UInt(CGImageGetHeight(imageRef)), width: UInt(CGImageGetWidth(imageRef)), rowBytes: CGImageGetBytesPerRow(imageRef))
var alpha = [UInt](count: 256, repeatedValue: 0)
var red = [UInt](count: 256, repeatedValue: 0)
var green = [UInt](count: 256, repeatedValue: 0)
var blue = [UInt](count: 256, repeatedValue: 0)
var alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha)
var redPtr = UnsafeMutablePointer<vImagePixelCount>(red)
var greenPtr = UnsafeMutablePointer<vImagePixelCount>(green)
var bluePtr = UnsafeMutablePointer<vImagePixelCount> (blue)
var rgba = [redPtr, greenPtr, bluePtr, alphaPtr]
var histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba)
var error = vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags))
return (alpha, red, green, blue)
}
(Taken from https://github.com/FlexMonkey/ShinpuruImage)

For Swift 5, you need to explicitly let the compiler know that your pointers are optional. Change your UnsafeMutablePointer declarations to the following:
Swift 5 version:
let redPtr = red.withUnsafeMutableBufferPointer { $0.baseAddress }
let greenPtr = green.withUnsafeMutableBufferPointer { $0.baseAddress }
let bluePtr = blue.withUnsafeMutableBufferPointer { $0.baseAddress }
let alphaPtr = alphaChannel.withUnsafeMutableBufferPointer { $0.baseAddress }
let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>?>.allocate(capacity: 4)
histogram[0] = redPtr
histogram[1] = greenPtr
histogram[2] = bluePtr
histogram[3] = alphaPtr
let error:vImage_Error = vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags))
Swift 4 version:
let redPtr: UnsafeMutablePointer<vImagePixelCount>? = UnsafeMutablePointer(mutating: red)
let greenPtr: UnsafeMutablePointer<vImagePixelCount>? = UnsafeMutablePointer(mutating:green)
let bluePtr: UnsafeMutablePointer<vImagePixelCount>? = UnsafeMutablePointer(mutating:blue)
let alphaPtr: UnsafeMutablePointer<vImagePixelCount>? = UnsafeMutablePointer(mutating:alpha)

Today, I write the code to analyze photo's RGB histogram. it's working now.
func getHistogram(_ image: UIImage) -> (alpha: [UInt], red: [UInt], green: [UInt], blue: [UInt]) {
guard
let cgImage = image.cgImage,
var imageBuffer = try? vImage_Buffer(cgImage: cgImage)
else {
return nil
}
defer {
imageBuffer.free()
}
var redArray: [vImagePixelCount] = Array(repeating: 0, count: 256)
var greenArray: [vImagePixelCount] = Array(repeating: 0, count: 256)
var blueArray: [vImagePixelCount] = Array(repeating: 0, count: 256)
var alphaArray: [vImagePixelCount] = Array(repeating: 0, count: 256)
var error: vImage_Error = kvImageNoError
redArray.withUnsafeMutableBufferPointer { rPointer in
greenArray.withUnsafeMutableBufferPointer { gPointer in
blueArray.withUnsafeMutableBufferPointer { bPointer in
alphaArray.withUnsafeMutableBufferPointer { aPointer in
var histogram = [ rPointer.baseAddress, gPointer.baseAddress, bPointer.baseAddress, aPointer.baseAddress ]
histogram.withUnsafeMutableBufferPointer { hPointer in
if let hBaseAddress = hPointer.baseAddress {
error = vImageHistogramCalculation_ARGB8888(&imageBuffer, hBaseAddress, vNoFlags)
}
}
}
}
}
}
guard error == kvImageNoError else {
printVImageError(error: error)
return nil
}
return (alphaArray, redArray, greenArray, blueArrat)
}

Related

Issue applying a Shader after a MPSImageLanczosScale on Apple Metal

I'm having weird result when I apply a shader on a MTLTexture after applying a MPSImageLanczosScale.
Even if the transform as scale = 1 and translationX = 0 and translationY = 0.
It's working well if I don't apply the MPSImageLanczosScale. Below you can see the result without and with applying the MPSImageLanczosScale.
My render method look like this:
func filter(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
guard let commandQueue = commandQueue, var commandBuffer = commandQueue.makeCommandBuffer() else {
print("Failed to create Metal command queue")
CVMetalTextureCacheFlush(textureCache!, 0)
return nil
}
var newPixelBuffer: CVPixelBuffer?
CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, outputPixelBufferPool!, &newPixelBuffer)
guard var outputPixelBuffer = newPixelBuffer else {
print("Allocation failure: Could not get pixel buffer from pool (\(self.description))")
return nil
}
guard let inputTexture = makeTextureFromCVPixelBuffer(pixelBuffer: pixelBuffer, textureFormat: .bgra8Unorm) else {
return nil
}
guard var intermediateTexture = makeTextureFromCVPixelBuffer(pixelBuffer: outputPixelBuffer, textureFormat: .bgra8Unorm) else {
return nil
}
let imageLanczosScale = MPSImageLanczosScale(device: metalDevice)
let transform = MPSScaleTransform(scaleX: Double(scale), scaleY: Double(scale), translateX: Double(translationX), translateY: Double(translationY))
withUnsafePointer(to: &transform) { (transformPtr: UnsafePointer<MPSScaleTransform>) -> () in
imageLanczosScale.scaleTransform = transformPtr
}
imageLanczosScale.encode(commandBuffer: commandBuffer, sourceTexture: inputTexture, destinationTexture: outputTexture)
guard let commandEncoder = commandBuffer.makeComputeCommandEncoder(),
let outputTexture = makeTextureFromCVPixelBuffer(pixelBuffer: outputPixelBuffer, textureFormat: .bgra8Unorm) else { return nil }
commandEncoder.label = "Shader"
commandEncoder.setComputePipelineState(shaderPipline)
commandEncoder.setTexture(intermediateTexture, index: 1)
commandEncoder.setTexture(outputTexture, index: 0)
let w = shaderPipline.threadExecutionWidth
let h = shaderPipline.maxTotalThreadsPerThreadgroup / w
let threadsPerThreadgroup = MTLSizeMake(w, h, 1)
let threadgroupsPerGrid = MTLSize(width: (intermediateTexture.width + w - 1) / w, height: (intermediateTexture.height + h - 1) / h, depth: 1)
commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup)
commandEncoder.endEncoding()
commandBuffer.commit()
return outputPixelBuffer
}
No idea what im doing wrong. any ideas?

error: use of undeclared type 'Accelerate'

Converting YUV420 to RGB using Accelerate Framework in swift.
call to vImageConvert_420Yp8_Cb8_Cr8ToARGB8888is throwing
Thread 1: EXC_BAD_ACCESS (code=1, address=0x108bc9000)
While debugging source and destination pointer then showing this message.
Printing description of yPlaneBuffer: expression produced error:
swift:1:65: error: use of undeclared type 'Accelerate'
Swift._DebuggerSupport.stringForPrintObject(Swift.UnsafePointer(bitPattern:
0x108788240)!.pointee)
Same error is coming for all source yPlaneBuffer, uPlaneBuffer and vPlaneBuffer.
Following is my code.
import Foundation
import Accelerate.vImage
import UIKit
import OpenTok
class I420Converter{
var infoYpCbCrToARGB = vImage_YpCbCrToARGB()
init() {
configureYpCbCrToARGBInfo()
}
func configureYpCbCrToARGBInfo() -> vImage_Error {
print("Configuring")
var pixelRange = vImage_YpCbCrPixelRange(Yp_bias: 0,
CbCr_bias: 128,
YpRangeMax: 255,
CbCrRangeMax: 255,
YpMax: 255,
YpMin: 1,
CbCrMax: 255,
CbCrMin: 0)
let error = vImageConvert_YpCbCrToARGB_GenerateConversion(
kvImage_YpCbCrToARGBMatrix_ITU_R_601_4!,
&pixelRange,
&infoYpCbCrToARGB,
kvImage420Yp8_Cb8_Cr8,
kvImageARGB8888,
vImage_Flags(kvImagePrintDiagnosticsToConsole))
print("Configration done \(error)")
return error
}
public func convertFrameVImageYUV(toUIImage frame: OTVideoFrame) -> UIImage {
var result: UIImage? = nil
let width = frame.format?.imageWidth ?? 0
let height = frame.format?.imageHeight ?? 0
var pixelBuffer: CVPixelBuffer? = nil
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(width), Int(height), kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, nil, &pixelBuffer)
convertFrameVImageYUV(frame, to: pixelBuffer)
var ciImage: CIImage? = nil
if let pixelBuffer = pixelBuffer {
ciImage = CIImage(cvPixelBuffer: pixelBuffer)
}
let temporaryContext = CIContext(options: nil)
var uiImage: CGImage? = nil
if let ciImage = ciImage {
uiImage = temporaryContext.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(pixelBuffer!), height: CVPixelBufferGetHeight(pixelBuffer!)))
}
if let uiImage = uiImage {
result = UIImage(cgImage: uiImage)
}
//CGImageRelease(uiImage!)
print("done")
return result!
}
func convertFrameVImageYUV(_ frame: OTVideoFrame, to pixelBufferRef: CVPixelBuffer?) -> vImage_Error{
if pixelBufferRef == nil {
print("No PixelBuffer refrance found")
return vImage_Error(kvImageInvalidParameter)
}
let width = frame.format?.imageWidth ?? 0
let height = frame.format?.imageHeight ?? 0
let subsampledWidth = frame.format!.imageWidth/2
let subsampledHeight = frame.format!.imageHeight/2
print("subsample height \(subsampledHeight) \(subsampledWidth)")
let planeSize = calculatePlaneSize(forFrame: frame)
let yPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.ySize)
let uPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.uSize)
let vPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.vSize)
memcpy(yPlane, frame.planes?.pointer(at: 0), planeSize.ySize)
memcpy(uPlane, frame.planes?.pointer(at: 1), planeSize.uSize)
memcpy(vPlane, frame.planes?.pointer(at: 2), planeSize.vSize)
print("192")
var yPlaneBuffer = vImage_Buffer(data: yPlane, height: vImagePixelCount(height), width: vImagePixelCount(width), rowBytes: planeSize.ySize)
var uPlaneBuffer = vImage_Buffer(data: uPlane, height: vImagePixelCount(subsampledHeight), width: vImagePixelCount(subsampledHeight), rowBytes: planeSize.uSize)
var vPlaneBuffer = vImage_Buffer(data: vPlane, height: vImagePixelCount(subsampledHeight), width: vImagePixelCount(subsampledWidth), rowBytes: planeSize.vSize)
CVPixelBufferLockBaseAddress(pixelBufferRef!, .readOnly)
let pixelBufferData = CVPixelBufferGetBaseAddress(pixelBufferRef!)
let rowBytes = CVPixelBufferGetBytesPerRow(pixelBufferRef!)
var destinationImageBuffer = vImage_Buffer()
destinationImageBuffer.data = pixelBufferData
destinationImageBuffer.height = vImagePixelCount(height)
destinationImageBuffer.width = vImagePixelCount(width)
destinationImageBuffer.rowBytes = rowBytes
var permuteMap: [UInt8] = [3, 2, 1, 0] // BGRA
let convertError = vImageConvert_420Yp8_Cb8_Cr8ToARGB8888(&yPlaneBuffer, &uPlaneBuffer, &vPlaneBuffer, &destinationImageBuffer, &infoYpCbCrToARGB, &permuteMap, 255, vImage_Flags(kvImagePrintDiagnosticsToConsole))
CVPixelBufferUnlockBaseAddress(pixelBufferRef!, [])
print("is error \(convertError)")
return convertError
}
fileprivate func calculatePlaneSize(forFrame frame: OTVideoFrame)
-> (ySize: Int, uSize: Int, vSize: Int)
{
guard let frameFormat = frame.format
else {
return (0, 0 ,0)
}
let baseSize = Int(frameFormat.imageWidth * frameFormat.imageHeight) * MemoryLayout<GLubyte>.size
return (baseSize, baseSize / 4, baseSize / 4)
}
}

Convert colours of every pixel in video preview - Swift

I have the following code which displays a camera preview, retrieves a single pixel's colour from the UIImage and converts this value to a 'filtered' colour.
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
connection.videoOrientation = orientation
let videoOutput = AVCaptureVideoDataOutput()
videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue.main)
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(cvImageBuffer: pixelBuffer!)
let typeOfColourBlindness = ColourBlindType(rawValue: "deuteranomaly")
/* Gets colour from a single pixel - currently 0,0 and converts it into the 'colour blind' version */
let captureImage = convert(cmage: cameraImage)
let colour = captureImage.getPixelColour(pos: CGPoint(x: 0, y: 0))
var redval: CGFloat = 0
var greenval: CGFloat = 0
var blueval: CGFloat = 0
var alphaval: CGFloat = 0
_ = colour.getRed(&redval, green: &greenval, blue: &blueval, alpha: &alphaval)
print("Colours are r: \(redval) g: \(greenval) b: \(blueval) a: \(alphaval)")
let filteredColour = CBColourBlindTypes.getModifiedColour(.deuteranomaly, red: Float(redval), green: Float(greenval), blue: Float(blueval))
print(filteredColour)
/* #################################################################################### */
DispatchQueue.main.async {
// placeholder for now
self.filteredImage.image = self.applyFilter(cameraImage: cameraImage, colourBlindness: typeOfColourBlindness!)
}
}
Here is where the x: 0, y: 0 pixel value is converted:
import Foundation
enum ColourBlindType: String {
case deuteranomaly = "deuteranomaly"
case protanopia = "protanopia"
case deuteranopia = "deuteranopia"
case protanomaly = "protanomaly"
}
class CBColourBlindTypes: NSObject {
class func getModifiedColour(_ type: ColourBlindType, red: Float, green: Float, blue: Float) -> Array<Float> {
switch type {
case .deuteranomaly:
return [(red*0.80)+(green*0.20)+(blue*0),
(red*0.25833)+(green*0.74167)+(blue*0),
(red*0)+(green*0.14167)+(blue*0.85833)]
case .protanopia:
return [(red*0.56667)+(green*0.43333)+(blue*0),
(red*0.55833)+(green*0.44167)+(blue*0),
(red*0)+(green*0.24167)+(blue*0.75833)]
case .deuteranopia:
return [(red*0.625)+(green*0.375)+(blue*0),
(red*0.7)+(green*0.3)+(blue*0),
(red*0)+(green*0.3)+(blue*0.7)]
case .protanomaly:
return [(red*0.81667)+(green*0.18333)+(blue*0.0),
(red*0.33333)+(green*0.66667)+(blue*0.0),
(red*0.0)+(green*0.125)+(blue*0.875)]
}
}
}
The placeholder for now comment refers to the following function:
func applyFilter(cameraImage: CIImage, colourBlindness: ColourBlindType) -> UIImage {
//do stuff with pixels to render new image
/* Placeholder code for shifting the hue */
// Create a place to render the filtered image
let context = CIContext(options: nil)
// Create filter angle
let filterAngle = 207 * Double.pi / 180
// Create a random color to pass to a filter
let randomColor = [kCIInputAngleKey: filterAngle]
// Apply a filter to the image
let filteredImage = cameraImage.applyingFilter("CIHueAdjust", parameters: randomColor)
// Render the filtered image
let renderedImage = context.createCGImage(filteredImage, from: filteredImage.extent)
// Return a UIImage
return UIImage(cgImage: renderedImage!)
}
And here is my extension for retrieving a pixel colour:
extension UIImage {
func getPixelColour(pos: CGPoint) -> UIColor {
let pixelData = self.cgImage!.dataProvider!.data
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(self.size.width) * Int(pos.y)) + Int(pos.x)) * 4
let r = CGFloat(data[pixelInfo]) / CGFloat(255.0)
let g = CGFloat(data[pixelInfo+1]) / CGFloat(255.0)
let b = CGFloat(data[pixelInfo+2]) / CGFloat(255.0)
let a = CGFloat(data[pixelInfo+3]) / CGFloat(255.0)
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
How can I create a filter for the following colour range for example?
I want to take in the camera input, replace the colours to be of the Deuteranopia range and display this on the screen, in real time, using Swift.
I am using a UIImageView for the image display.
To learn how to perform filtering of video capture and real-time display of the filtered image, you may want to study the AVCamPhotoFilter sample code from Apple, and other sources such as this objc.io tutorial
In short, using a UIImage for real-time rendering is not a good idea - it's too slow. Use a OpenGL (e.g. GLKView) of Metal (e.g. MTKView). The AVCamPhotoFilter code uses MTKView and renders to intermediate buffers, but you can also render a CIImage directly using the appropriate CIContext methods, e.g. for metal https://developer.apple.com/documentation/coreimage/cicontext/1437835-render
In addition, regarding your color filter - you may want to look at the CIColorCube core image filter as shown here.
let filterName = "CIColorCrossPolynomial"
//deuteronomaly
let param = ["inputRedCoefficients" : CIVector(values: [0.8, 0.2, 0, 0, 0, 0, 0, 0, 0, 0], count: 10),
"inputGreenCoefficients" : CIVector(values: [0.25833, 0.74167, 0, 0, 0, 0, 0, 0, 0, 0], count: 10),
"inputBlueCoefficients" : CIVector(values: [0, 0.14167, 0.85833, 0, 0, 0, 0, 0, 0, 0], count: 10)]
let filter = CIFilter(name: filterName, parameters: param)
let startImage = CIImage(image: image!)
filter?.setValue(startImage, forKey: kCIInputImageKey)
let newImage = UIImage(ciImage: ((filter?.outputImage)!))
filter result:
filter result 2:

How to convert YUV frames (from OTVideoFrame) to CVPixelBuffer

I need to convert YUV Frames to CVPixelBuffer that I get from OTVideoFrame Class
This class provides an array of planes in the video frame which contains three elements for y,u,v frame each at index 0,1,2.
#property (nonatomic, retain) NSPointerArray *planes
and format of the video frame
#property (nonatomic, retain) OTVideoFormat *format
That contains Properties like width, height, bytesPerRow etc. of the frame
I need to add filter to the image I receive in the form of OTVideoFrame, I have already tried these answers :
How to convert from YUV to CIImage for iOS
Create CVPixelBuffer from YUV with IOSurface backed
These two links have the solutions in Objective-C but I want to do it in swift. One of the answers in second link was in swift but it lacks some information about the YUVFrame struct that the answer has reference to.
The Format that I receive is NV12
Here is what I have been trying to do till now but I don't know how to proceed next :-
/**
* Calcualte the size of each plane from OTVideoFrame.
*
* #param frame The frame to render.
* #return tuple containing three elements for size of each plane
*/
fileprivate func calculatePlaneSize(forFrame frame: OTVideoFrame)
-> (ySize: Int, uSize: Int, vSize: Int){
guard let frameFormat = frame.format
else {
return (0, 0 ,0)
}
let baseSize = Int(frameFormat.imageWidth * frameFormat.imageHeight) * MemoryLayout<GLubyte>.size
return (baseSize, baseSize / 4, baseSize / 4)
}
/**
* Renders a frame to the video renderer.
*
* #param frame The frame to render.
*/
func renderVideoFrame(_ frame: OTVideoFrame) {
let planeSize = calculatePlaneSize(forFrame: frame)
let yPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.ySize)
let uPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.uSize)
let vPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.vSize)
memcpy(yPlane, frame.planes?.pointer(at: 0), planeSize.ySize)
memcpy(uPlane, frame.planes?.pointer(at: 1), planeSize.uSize)
memcpy(vPlane, frame.planes?.pointer(at: 2), planeSize.vSize)
let yStride = frame.format!.bytesPerRow.object(at: 0) as! Int
// multiply chroma strides by 2 as bytesPerRow represents 2x2 subsample
let uStride = frame.format!.bytesPerRow.object(at: 1) as! Int
let vStride = frame.format!.bytesPerRow.object(at: 2) as! Int
let width = frame.format!.imageWidth
let height = frame.format!.imageHeight
var pixelBuffer: CVPixelBuffer? = nil
var err: CVReturn;
err = CVPixelBufferCreate(kCFAllocatorDefault, Int(width), Int(height), kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, nil, &pixelBuffer)
if (err != 0) {
NSLog("Error at CVPixelBufferCreate %d", err)
fatalError()
}
}
Taking Guidance from those two links I tried to create Pixel buffer but I got stuck every time at this point because the conversion of the Objective-C code after this is not similar to what we have in Swift 3.
For those who are looking for a fast solution, I did with swift Accelerate
using vImageConvert_AnyToAny(_:_:_:_:_:) function.
import Foundation
import Accelerate
import UIKit
import OpenTok
class Accelerater{
var infoYpCbCrToARGB = vImage_YpCbCrToARGB()
init() {
_ = configureYpCbCrToARGBInfo()
}
func configureYpCbCrToARGBInfo() -> vImage_Error {
print("Configuring")
var pixelRange = vImage_YpCbCrPixelRange(Yp_bias: 0,
CbCr_bias: 128,
YpRangeMax: 255,
CbCrRangeMax: 255,
YpMax: 255,
YpMin: 1,
CbCrMax: 255,
CbCrMin: 0)
let error = vImageConvert_YpCbCrToARGB_GenerateConversion(
kvImage_YpCbCrToARGBMatrix_ITU_R_601_4!,
&pixelRange,
&infoYpCbCrToARGB,
kvImage420Yp8_Cb8_Cr8,
kvImageARGB8888,
vImage_Flags(kvImagePrintDiagnosticsToConsole))
print("Configration done \(error)")
return error
}
public func convertFrameVImageYUV(toUIImage frame: OTVideoFrame, flag: Bool) -> UIImage {
var result: UIImage? = nil
let width = frame.format?.imageWidth ?? 0
let height = frame.format?.imageHeight ?? 0
var pixelBuffer: CVPixelBuffer? = nil
_ = CVPixelBufferCreate(kCFAllocatorDefault, Int(width), Int(height), kCVPixelFormatType_32BGRA, nil, &pixelBuffer)
_ = convertFrameVImageYUV(frame, to: pixelBuffer)
var ciImage: CIImage? = nil
if let pixelBuffer = pixelBuffer {
ciImage = CIImage(cvPixelBuffer: pixelBuffer)
}
let temporaryContext = CIContext(options: nil)
var uiImage: CGImage? = nil
if let ciImage = ciImage {
uiImage = temporaryContext.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(pixelBuffer!), height: CVPixelBufferGetHeight(pixelBuffer!)))
}
if let uiImage = uiImage {
result = UIImage(cgImage: uiImage)
}
CVPixelBufferUnlockBaseAddress(pixelBuffer!, [])
return result!
}
func convertFrameVImageYUV(_ frame: OTVideoFrame, to pixelBufferRef: CVPixelBuffer?) -> vImage_Error{
let start = CFAbsoluteTimeGetCurrent()
if pixelBufferRef == nil {
print("No PixelBuffer refrance found")
return vImage_Error(kvImageInvalidParameter)
}
let width = frame.format?.imageWidth ?? 0
let height = frame.format?.imageHeight ?? 0
let subsampledWidth = frame.format!.imageWidth/2
let subsampledHeight = frame.format!.imageHeight/2
print("subsample height \(subsampledHeight) \(subsampledWidth)")
let planeSize = calculatePlaneSize(forFrame: frame)
print("ysize : \(planeSize.ySize) \(planeSize.uSize) \(planeSize.vSize)")
let yPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.ySize)
let uPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.uSize)
let vPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.vSize)
memcpy(yPlane, frame.planes?.pointer(at: 0), planeSize.ySize)
memcpy(uPlane, frame.planes?.pointer(at: 1), planeSize.uSize)
memcpy(vPlane, frame.planes?.pointer(at: 2), planeSize.vSize)
let yStride = frame.format!.bytesPerRow.object(at: 0) as! Int
// multiply chroma strides by 2 as bytesPerRow represents 2x2 subsample
let uStride = frame.format!.bytesPerRow.object(at: 1) as! Int
let vStride = frame.format!.bytesPerRow.object(at: 2) as! Int
var yPlaneBuffer = vImage_Buffer(data: yPlane, height: vImagePixelCount(height), width: vImagePixelCount(width), rowBytes: yStride)
var uPlaneBuffer = vImage_Buffer(data: uPlane, height: vImagePixelCount(subsampledHeight), width: vImagePixelCount(subsampledWidth), rowBytes: uStride)
var vPlaneBuffer = vImage_Buffer(data: vPlane, height: vImagePixelCount(subsampledHeight), width: vImagePixelCount(subsampledWidth), rowBytes: vStride)
CVPixelBufferLockBaseAddress(pixelBufferRef!, .readOnly)
let pixelBufferData = CVPixelBufferGetBaseAddress(pixelBufferRef!)
let rowBytes = CVPixelBufferGetBytesPerRow(pixelBufferRef!)
var destinationImageBuffer = vImage_Buffer()
destinationImageBuffer.data = pixelBufferData
destinationImageBuffer.height = vImagePixelCount(height)
destinationImageBuffer.width = vImagePixelCount(width)
destinationImageBuffer.rowBytes = rowBytes
var permuteMap: [UInt8] = [3, 2, 1, 0] // BGRA
let convertError = vImageConvert_420Yp8_Cb8_Cr8ToARGB8888(&yPlaneBuffer, &uPlaneBuffer, &vPlaneBuffer, &destinationImageBuffer, &infoYpCbCrToARGB, &permuteMap, 255, vImage_Flags(kvImagePrintDiagnosticsToConsole))
CVPixelBufferUnlockBaseAddress(pixelBufferRef!, [])
yPlane.deallocate()
uPlane.deallocate()
vPlane.deallocate()
let end = CFAbsoluteTimeGetCurrent()
print("Decoding time \((end-start)*1000)")
return convertError
}
fileprivate func calculatePlaneSize(forFrame frame: OTVideoFrame)
-> (ySize: Int, uSize: Int, vSize: Int)
{
guard let frameFormat = frame.format
else {
return (0, 0 ,0)
}
let baseSize = Int(frameFormat.imageWidth * frameFormat.imageHeight) * MemoryLayout<GLubyte>.size
return (baseSize, baseSize / 4, baseSize / 4)
}
}
Performance tested on iPhone7, one frame conversion is less than a millisecond.
Here's what worked for me (I've taken your function and changed it a bit):
func createPixelBufferWithVideoFrame(_ frame: OTVideoFrame) -> CVPixelBuffer? {
if let fLock = frameLock {
fLock.lock()
let planeSize = calculatePlaneSize(forFrame: frame)
let yPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.ySize)
let uPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.uSize)
let vPlane = UnsafeMutablePointer<GLubyte>.allocate(capacity: planeSize.vSize)
memcpy(yPlane, frame.planes?.pointer(at: 0), planeSize.ySize)
memcpy(uPlane, frame.planes?.pointer(at: 1), planeSize.uSize)
memcpy(vPlane, frame.planes?.pointer(at: 2), planeSize.vSize)
let width = frame.format!.imageWidth
let height = frame.format!.imageHeight
var pixelBuffer: CVPixelBuffer? = nil
var err: CVReturn;
err = CVPixelBufferCreate(kCFAllocatorDefault, Int(width), Int(height), kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, nil, &pixelBuffer)
if (err != 0) {
NSLog("Error at CVPixelBufferCreate %d", err)
return nil
}
if let pixelBuffer = pixelBuffer {
CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
let yPlaneTo = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
memcpy(yPlaneTo, yPlane, planeSize.ySize)
let uvRow: Int = planeSize.uSize*2/Int(width)
let halfWidth: Int = Int(width)/2
if let uPlaneTo = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1) {
let uvPlaneTo = uPlaneTo.bindMemory(to: GLubyte.self, capacity: Int(uvRow*halfWidth*2))
for i in 0..<uvRow {
for j in 0..<halfWidth {
let dataIndex: Int = Int(i) * Int(halfWidth) + Int(j)
let uIndex: Int = (i * Int(width)) + Int(j) * 2
let vIndex: Int = uIndex + 1
uvPlaneTo[uIndex] = uPlane[dataIndex]
uvPlaneTo[vIndex] = vPlane[dataIndex]
}
}
}
}
fLock.unlock()
return pixelBuffer
}
return nil
}

vImageBuffer_InitWithCGImage Memory Leak in Swift 3

I am trying to get histogram calculation. Everything works fine, except following method shows an immense memory leak when profiled in Instruments.
Every time following method is called, it uses 200-300 MB of memory and never releases:
func histogramCalculation(_ imageRef: CGImage) -> (red: [UInt], green: [UInt], blue: [UInt]) {
var inBuffer = vImage_Buffer()
vImageBuffer_InitWithCGImage(
&inBuffer,
&format,
nil,
imageRef,
UInt32(kvImageNoFlags))
let alpha = [UInt](repeating: 0, count: 256)
let red = [UInt](repeating: 0, count: 256)
let green = [UInt](repeating: 0, count: 256)
let blue = [UInt](repeating: 0, count: 256)
let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(mutating: alpha) as UnsafeMutablePointer<vImagePixelCount>?
let redPtr = UnsafeMutablePointer<vImagePixelCount>(mutating: red) as UnsafeMutablePointer<vImagePixelCount>?
let greenPtr = UnsafeMutablePointer<vImagePixelCount>(mutating: green) as UnsafeMutablePointer<vImagePixelCount>?
let bluePtr = UnsafeMutablePointer<vImagePixelCount>(mutating: blue) as UnsafeMutablePointer<vImagePixelCount>?
let rgba = [redPtr, greenPtr, bluePtr, alphaPtr]
let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>?>(mutating: rgba)
let error : vImage_Error = vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags))
if (error == kvImageNoError) {
return (red, green, blue)
}
return (red, green, blue)
}
What could be wrong here.....
The docs for vImageBuffer_InitWithCGImage explain:
You are responsible for returning the memory referenced by buf->data to the system using free() when you are done with it.
So I would expect something along these lines to clean up the memory:
inBuffer.data.deallocate(bytes: inBuffer.rowBytes * Int(inBuffer.height),
alignedTo: MemoryLayout<vImage_Buffer>.alignment)
As a side note, your use of UnsafeMutablePointer here is not safe. There's no promise, for instance, that alpha will exist by the time you use reference it. Swift is allowed to destroy alpha immediately after you create alphaPtr (because it's never referenced again). It is rare that you want to use UnsafeMutablePointer.init. Instead, you want to use withUnsafe... methods to establish guaranteed lifetimes. For example (untested, but compiles):
var alpha = [vImagePixelCount](repeating: 0, count: 256)
var red = [vImagePixelCount](repeating: 0, count: 256)
var green = [vImagePixelCount](repeating: 0, count: 256)
var blue = [vImagePixelCount](repeating: 0, count: 256)
let error = alpha.withUnsafeMutableBufferPointer { alphaPtr -> vImage_Error in
return red.withUnsafeMutableBufferPointer { redPtr in
return green.withUnsafeMutableBufferPointer { greenPtr in
return blue.withUnsafeMutableBufferPointer { bluePtr in
var rgba = [redPtr.baseAddress, greenPtr.baseAddress, bluePtr.baseAddress, alphaPtr.baseAddress]
return rgba.withUnsafeMutableBufferPointer { buffer in
return vImageHistogramCalculation_ARGB8888(&inBuffer, buffer.baseAddress!, UInt32(kvImageNoFlags))
}
}
}
}
}

Resources