I am using SWIFT language and trying to take snapshot images from the camera viewfinder buffer. So far everything works well except for the image color. It seems incorrect or being swapped. Below is the code snippets where I set the video settings and capturing the image frames
func addVideoOutput() {
videoDeviceOutput = AVCaptureVideoDataOutput()
videoDeviceOutput.videoSettings = NSDictionary(objectsAndKeys: Int(kCVPixelFormatType_32BGRA), kCVPixelBufferPixelFormatTypeKey) as[NSObject: AnyObject]
// kCVPixelFormatType_32ARGB tested and found not supported
videoDeviceOutput.alwaysDiscardsLateVideoFrames = true
videoDeviceOutput.setSampleBufferDelegate(self, queue: sessionQueue)
if captureSession!.canAddOutput(videoDeviceOutput) {
captureSession!.addOutput(videoDeviceOutput)
}
}
/* AVCaptureVideoDataOutput Delegate
------------------------------------------*/
func captureOutput(captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, fromConnection connection: AVCaptureConnection!) {
sessionDelegate ? .cameraSessionDidOutputSampleBuffer ? (sampleBuffer)
// Extract a UImage
//var pixel_buffer : CVPixelBufferRef?
let pixel_buffer = CMSampleBufferGetImageBuffer(sampleBuffer)
CVPixelBufferLockBaseAddress(pixel_buffer, 0);
// Get the number of bytes per row for the pixel buffer
var baseAddress = CVPixelBufferGetBaseAddress(pixel_buffer);
// Get the number of bytes per row for the pixel buffer
var bytesPerRow = CVPixelBufferGetBytesPerRow(pixel_buffer);
// Get the pixel buffer width and height
let width : Int = CVPixelBufferGetWidth(pixel_buffer);
let height : Int = CVPixelBufferGetHeight(pixel_buffer);
/*Create a CGImageRef from the CVImageBufferRef*/
let colorSpace: CGColorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue)
var newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, bitmapInfo)
CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);
// get image frame and save to local storage
var refImage: CGImageRef = CGBitmapContextCreateImage(newContext)
var pixelData = CGDataProviderCopyData(CGImageGetDataProvider(refImage))
var image: UIImage = UIImage(CGImage: refImage)!;
self.SaveImageToDocumentStorage(image)
}
As you can see one of the comment line in the addVideoOutput function, I tried the kCVPixelFormatType_32ARGB format but it says not supported in iOS???
I kinda suspect the video format is 32BGRA but the color space for the image frame is set with CGColorSpaceCreateDeviceRGB(), but I could not find any other suitable RGB format for the video setting.
Any solutions or hints are much appreciated.
Thanks
I found the cause and a solution.
Just in case anyone experiences the same problem. Just change the bitmapInfo as follow:
// let bitmapInfo = CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue)
var bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedFirst.rawValue) | CGBitmapInfo.ByteOrder32Little
Related
I am trying to use a MTKTextureLoader to load a CGImage as a texture. Here is the original image
However after I convert that CGImage into a MTLTexture and that texture back to a CGImage it looks horrible, like this:
Here is sorta what is going on in code.
The image is loaded in as a CGImage (I have checked and that image does appear to have the full visual quality)
I have a function view() that allows me to view a NSImage by using it in a CALayer like so:
func view() {
.....
imageView!.layer = CALayer()
imageView!.layer!.contentsGravity = kCAGravityResizeAspectFill
imageView!.layer!.contents = img
imageView!.wantsLayer = true
So I did the following
let cg = CoolImage()
let ns = NSImage(cgImage: cg, size: Size(width: cg.width, height: cg.height))
view(image: ns)
And checked sure enough it had the full visual fidelity.
So then I loaded the cg image into a MTLTexture like so
let textureLoader = MTKTextureLoader(device: metalState.sharedDevice!)
let options = [
MTKTextureLoader.Option.textureUsage: NSNumber(value: MTLTextureUsage.shaderRead.rawValue | MTLTextureUsage.shaderWrite.rawValue | MTLTextureUsage.renderTarget.rawValue),
MTKTextureLoader.Option.SRGB: false
]
return ensure(try textureLoader.newTexture(cgImage: cg, options: options))
I then converted the MTLTexture back to a UIImage like so:
let texture = self
let width = texture.width
let height = texture.height
let bytesPerRow = width * 4
let data = UnsafeMutableRawPointer.allocate(bytes: bytesPerRow * height, alignedTo: 4)
defer {
data.deallocate(bytes: bytesPerRow * height, alignedTo: 4)
}
let region = MTLRegionMake2D(0, 0, width, height)
texture.getBytes(data, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
var buffer = vImage_Buffer(data: data, height: UInt(height), width: UInt(width), rowBytes: bytesPerRow)
var map: [UInt8] = [0, 1, 2, 3]
if (pixelFormat == .bgra8Unorm) {
map = [2, 1, 0, 3]
}
vImagePermuteChannels_ARGB8888(&buffer, &buffer, map, 0)
guard let colorSpace = CGColorSpace(name: CGColorSpace.genericRGBLinear) else { return nil }
guard let context = CGContext(data: data, width: width, height: height, bitsPerComponent: 8, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue) else { return nil }
guard let cgImage = context.makeImage() else { return nil }
return NSImage(cgImage: cgImage, size: Size(width: width, height: height))
And viewed it.
The resulting image was quite saturated and I believe it was because of the CGImage to MTLTexture conversion which I have been fairly successful with in the past.
Please note that this texture was never rendered only converted.
You are probably wondering why I am using all of these conversions and that is a great point. My actual pipeline does not work anything like this HOWEVER it does require each of these conversion components to be working smoothly. This is not my actual use case just something to show the problem.
The problem here isn't the conversion from CGImage to MTLTexture. The problem is that you're assuming that the color space of the source image is linear. More likely than not, the image data is actually sRGB-encoded, so by creating a bitmap context with a generic linear color space, you're incorrectly telling CG that it should gamma-encode the image data before display, which leads to the desaturation you're seeing.
You can fix this by using the native color space of the original CGImage, or by otherwise accounting for the fact that your image data is sRGB-encoded.
I'm trying to save a 16-bit depth PNG image with P3 color space from a Metal texture on iOS. The texture has pixelformat = .rgba16Unorm, and I extract the data with this code
func dataProviderRef() -> CGDataProvider? {
let pixelCount = width * height
var imageBytes = [UInt8](repeating: 0, count: pixelCount * bytesPerPixel)
let region = MTLRegionMake2D(0, 0, width, height)
getBytes(&imageBytes, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
return CGDataProvider(data: NSData(bytes: &imageBytes, length: pixelCount * bytesPerPixel * MemoryLayout<UInt8>.size))
}
I figured out that the way to save a PNG image on iOS would be to create a UIImage first, and to initialize it, I need to create a CGImage. The problem is I don't know what to pass to CGIBitmapInfo. In the documentation I can see you can specify the byteOrder for 32-bit formats, but not for 64-bit.
The function I use to convert the texture to an UIImage is this,
extension UIImage {
public convenience init?(texture: MTLTexture) {
guard let rgbColorSpace = texture.defaultColorSpace else {
return nil
}
let bitmapInfo:CGBitmapInfo = [CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue)]
guard let provider = texture.dataProviderRef() else {
return nil
}
guard let cgim = CGImage(
width: texture.width,
height: texture.height,
bitsPerComponent: texture.bitsPerComponent,
bitsPerPixel: texture.bitsPerPixel,
bytesPerRow: texture.bytesPerRow,
space: rgbColorSpace,
bitmapInfo: bitmapInfo,
provider: provider,
decode: nil,
shouldInterpolate: false,
intent: .defaultIntent
)
else {
return nil
}
self.init(cgImage: cgim)
}
}
Note that "texture" is using a series of attributes that do not exist in MTLTexture. I created a simple extension for convenience. The only interesting bit I guess it's the color space, that at the moment is simply,
public extension MTLTexture {
var defaultColorSpace: CGColorSpace? {
get {
switch pixelFormat {
case .rgba16Unorm:
return CGColorSpace(name: CGColorSpace.displayP3)
default:
return CGColorSpaceCreateDeviceRGB()
}
}
}
}
It looks like the image I'm creating with that code above is sampling 4 bytes per pixel, instead of 8. So I obviously end up with a funny looking image...
How do I create the appropriate CGBitmapInfo? Is it even possible?
P.S. If you want to see the full code with an example, it's all in github: https://github.com/endavid/VidEngine/tree/master/SampleColorPalette
The answer was using byteOrder16. For instance, I've replace bitmapInfo in the code above for this,
let isFloat = texture.bitsPerComponent == 16
let bitmapInfo:CGBitmapInfo = [isFloat ? .byteOrder16Little : .byteOrder32Big, CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue)]
(The alpha can be premultiplied as well).
The SDK documentation does not provide many hints of why this is, but the book Programming with Quartz has a nice explanation of the meaning of these 16 bits:
The value byteOrder16Little specifies to Quartz that each 16-bit chunk of data supplied by your data provider should be treated in little endian order [...] For example, when using a value of byteOrder16Little for an image that specifies RGB format with 16 bits per component and 48 bits per pixel, your data provider supplies the data for each pixel where the components are ordered R, G, B, but each color component value is in little-endian order [...] For best performance when using byteOrder16Little, either the pixel size or the component size of the image must be 16 bits.
So for a 64-bit image in rgba16, the pixel size is 64 bits, but the component size is 16 bits. It works nicely :)
(Thanks #warrenm !)
I am converting the CMSampleBuffer argument in the captureOutput function of my AVCaptureVideoDataOuput delegate into a MTLTexture like so (side note, I have set the pixel format of the video output to kCVPixelFormatType_32BGRA):
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
var outTexture: CVMetalTexture? = nil
var textCache : CVMetalTextureCache?
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textCache)
var textureRef : CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textCache!, imageBuffer, nil, MTLPixelFormat.bgra8Unorm, width, height, 0, &textureRef)
let texture = CVMetalTextureGetTexture(textureRef!)!
print(texture.bufferBytesPerRow)
}
The issue is when I print the bytes per row of the texture, it always prints 0, which is problematic because I later try to convert the texture back into a UIImage using the methodology in this article: https://www.invasivecode.com/weblog/metal-image-processing. Why is the texture I receive seemingly empty? I know the CMSampleBuffer property is fine because I can convert it into a UIIMage and draw it like so:
let myPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let myCIimage = CIImage(cvPixelBuffer: myPixelBuffer!)
let image = UIImage(ciImage: myCIimage)
self.imageView.image = image
The bufferBytesPerRow property is only meaningful for a texture that was created using the makeTexture(descriptor:offset:bytesPerRow:) method of a MTLBuffer. As you can see, the bytes-per-row is an input to that method to tell Metal how to interpret the data in the buffer. (The texture descriptor provides additional information, too, of course.) This method is only a means to get that back out.
Note that textures created from buffers can also report which buffer they were created from and the offset supplied to the above method.
Textures created in other ways don't have that information. These textures have no intrinsic bytes-per-row. Their data is not necessarily organized internally in a simple raster buffer.
If/when you want to get the data from a texture to either a Metal buffer or a plain old byte array, you have the freedom to choose a bytes-per-row value that's useful for your purposes, so long as it's at least the bytes-per-pixel of the texture pixel format times the texture's width. (It's more complicated for compressed formats.) The docs for getBytes(_:bytesPerRow:from:mipmapLevel:) and copy(from:sourceSlice:sourceLevel:sourceOrigin:sourceSize:to:destinationOffset:destinationBytesPerRow:destinationBytesPerImage:) explain further.
What is the best way to convert CGImage to OTVideoFrame?
I tried to get the underlying CGImage pixel buffer and feed it into an OTVideoBuffer, but got a distorted image.
Here is what I have done:
created a new OTVideoFormat object with ARGB pixel format
Set the bytesPerRow of the OTVideoFormat to height*width*4. Taking the value of CGImageGetBytesPerRow(...) did not work, got no error messages but also no frames on the other end of the line.
Copied the rows truncating them to convert from CGImageGetBytesPerRow(...) to height*width*4 bytes per row.
Got a distorted image with rows slightly shifted
Here is the code:
func toOTVideoFrame() throws -> OTVideoFrame {
let width : UInt32 = UInt32(CGImageGetWidth(self)) // self is a CGImage
let height : UInt32 = UInt32(CGImageGetHeight(self))
assert(CGImageGetBitsPerPixel(self) == 32)
assert(CGImageGetBitsPerComponent(self) == 8)
let bitmapInfo = CGImageGetBitmapInfo(self)
assert(bitmapInfo.contains(CGBitmapInfo.FloatComponents) == false)
assert(bitmapInfo.contains(CGBitmapInfo.ByteOrderDefault))
assert(CGImageGetAlphaInfo(self) == .NoneSkipFirst)
let bytesPerPixel : UInt32 = 4
let cgImageBytesPerRow : UInt32 = UInt32(CGImageGetBytesPerRow(self))
let otFrameBytesPerRow : UInt32 = bytesPerPixel * width
let videoFormat = OTVideoFormat()
videoFormat.pixelFormat = .ARGB
videoFormat.bytesPerRow.addObject(NSNumber(unsignedInt: otFrameBytesPerRow))
videoFormat.imageWidth = width
videoFormat.imageHeight = height
videoFormat.estimatedFramesPerSecond = 15
videoFormat.estimatedCaptureDelay = 100
let videoFrame = OTVideoFrame(format: videoFormat)
videoFrame.timestamp = CMTimeMake(0, 1) // This is temporary
videoFrame.orientation = OTVideoOrientation.Up // This is temporary
let dataProvider = CGImageGetDataProvider(self)
let imageData : NSData = CGDataProviderCopyData(dataProvider)!
let buffer = UnsafeMutablePointer<UInt8>.alloc(Int(otFrameBytesPerRow * height))
for currentRow in 0..<height {
let currentRowStartOffsetCGImage = currentRow * cgImageBytesPerRow
let currentRowStartOffsetOTVideoFrame = currentRow * otFrameBytesPerRow
let cgImageRange = NSRange(location: Int(currentRowStartOffsetCGImage), length: Int(otFrameBytesPerRow))
imageData.getBytes(buffer.advancedBy(Int(currentRowStartOffsetOTVideoFrame)),
range: cgImageRange)
}
do {
let planes = UnsafeMutablePointer<UnsafeMutablePointer<UInt8>>.alloc(1)
planes.initialize(buffer)
videoFrame.setPlanesWithPointers(planes, numPlanes: 1)
planes.dealloc(1)
}
return videoFrame
}
The result image:
Solved this issue by my own.
It appears to be a bug in the OpenTok SDK. The SDK does not seem to be able to handle images whose size is not a multiple of 16. When I changed all image sizes to be multiple of 16, everything started to work fine.
TokBox did not bother to state this limitation in the API documentation, nor throw an exception when the input image size is not a multiple of 16.
This is a second critical bug I have found in OpenTok SDK. I strongly suggest you do not use this product. It is of very low quality.
I am trying to create a video with an input of images using AVAssetWriterInput for which I need to create a pixel buffer of my images.
For that I call the function below which works, but after creating a few videos the app receives a memory warning and crashes. I have debugged it using Instruments and it appears I have a memory leak here.
I have tried to put the variables pixelBufferPointer and pxData as class variables and destroy/dealloc once the video is created but that didn't appear to make any difference. Is there something that I should be doing to release this memory?
func createPixelBufferFromCGImage(image: CGImageRef) -> CVPixelBufferRef {
let options = [
"kCVPixelBufferCGImageCompatibilityKey": true,
"kCVPixelBufferCGBitmapContextCompatibilityKey": true
]
var videoWidth = 496
var videoHeight = 668
let frameSize = CGSizeMake(CGFloat(videoWidth), CGFloat(videoHeight))
var pixelBufferPointer = UnsafeMutablePointer<Unmanaged<CVPixelBuffer>?>.alloc(1)
var status:CVReturn = CVPixelBufferCreate(
kCFAllocatorDefault,
Int(frameSize.width),
Int(frameSize.height),
OSType(kCVPixelFormatType_32ARGB),
options,
pixelBufferPointer
)
var lockStatus:CVReturn = CVPixelBufferLockBaseAddress(pixelBufferPointer.memory?.takeUnretainedValue(), 0)
var pxData:UnsafeMutablePointer<(Void)> = CVPixelBufferGetBaseAddress(pixelBufferPointer.memory?.takeUnretainedValue())
let bitmapinfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.NoneSkipFirst.rawValue)
let rgbColorSpace:CGColorSpace = CGColorSpaceCreateDeviceRGB()
var context:CGContextRef = CGBitmapContextCreate(
pxData,
Int(frameSize.width),
Int(frameSize.height),
8,
//4 * CGImageGetWidth(image),
4 * Int(frameSize.width),
rgbColorSpace,
bitmapinfo
)
CGContextDrawImage(context, CGRectMake(0, 0, frameSize.width, frameSize.height), image)
CVPixelBufferUnlockBaseAddress(pixelBufferPointer.memory?.takeUnretainedValue(), 0)
UIGraphicsEndImageContext()
return pixelBufferPointer.memory!.takeUnretainedValue()
}
The Unmanaged<CVPixelBuffer> is leaking.
I would prefer Russell Austin's answer, but I couldn't figure out how to pass the pixelBufferPointer to CVPixelBufferCreate without syntax error (swift noob). Failing that, changing the line
return pixelBufferPointer.memory!.takeUnretainedValue()
to
return pixelBufferPointer.memory!.takeRetainedValue()
fixes the leak
Take a look at Memory leak on CIContext createCGImage at iOS 9?
I'm having a similar issue, the leak is due to the following code
CGImageRef processedCGImage = [_context createCGImage:ciImage
fromRect:[ciImage extent]];
Here for more contextual info: Memory Leak in CMSampleBufferGetImageBuffer
You don't have to use the UnsafeMutablePointer types. You were probably trying to convert from some Objective-C example. I was doing the same thing but eventually found how to do it without. Try
var pixelBufferPointer: CVPixelBuffer?
...
var status:CVReturn = CVPixelBufferCreate(
kCFAllocatorDefault,
Int(frameSize.width),
Int(frameSize.height),
OSType(kCVPixelFormatType_32ARGB),
options,
&pixelBufferPointer
)
and
var pxData = CVPixelBufferGetBaseAddress(pixelBufferPointer)
Then you don't have to do the takeUnRetainedValue, etc.
You might also try using AVAssetWriterInputPixelBufferAdaptor and a pixel buffer pool. That is supposed to be more efficient. See this example
When using CVPixelBufferCreate the UnsafeMutablePointer has to be destroyed after retrieving the memory of it.
When I create a CVPixelBuffer, I do it like this:
func allocPixelBuffer() -> CVPixelBuffer {
let pixelBufferAttributes : CFDictionary = [...]
let pixelBufferOut = UnsafeMutablePointer<CVPixelBuffer?>.alloc(1)
_ = CVPixelBufferCreate(kCFAllocatorDefault,
Int(Width),
Int(Height),
OSType(kCVPixelFormatType_32ARGB),
pixelBufferAttributes,
pixelBufferOut)
let pixelBuffer = pixelBufferOut.memory!
pixelBufferOut.destroy()
return pixelBuffer
}