how to create a bitmap image from the byte array in swift - ios

I have a byte array which comes from a finger print sensor device. I wanted to create a bitmap out of it. I have tried few examples but all I am getting is a nil UIImage.
If there are any steps to do that, pls tell me.
Thanks.
This is what my func does:
func didFingerGrabDataReceived(data: [UInt8]) {
if data[0] == 0 {
let width1 = Int16(data[0]) << 8
let finalWidth = Int(Int16(data[1]) | width1)
let height1 = Int16(data[2]) << 8
let finalHeight = Int(Int16(data[3]) | height1)
var finalData:[UInt8] = [UInt8]()
// i dont want the first 8 bytes, so am removing it
for i in 8 ..< data.count {
finalData.append(data[i])
}
dispatch_async(dispatch_get_main_queue()) { () -> Void in
let msgData = NSMutableData(bytes: finalData, length: finalData.count)
let ptr = UnsafeMutablePointer<UInt8>(msgData.mutableBytes)
let colorSpace = CGColorSpaceCreateDeviceGray()
if colorSpace == nil {
self.showToast("color space is nil")
return
}
let bitmapContext = CGBitmapContextCreate(ptr, finalWidth, finalHeight, 8, 4 * finalWidth, colorSpace, CGImageAlphaInfo.Only.rawValue);
if bitmapContext == nil {
self.showToast("context is nil")
return
}
let cgImage=CGBitmapContextCreateImage(bitmapContext);
if cgImage == nil {
self.showToast("image is nil")
return
}
let newimage = UIImage(CGImage: cgImage!)
self.imageViewFinger.image = newimage
}
}
I am getting a distorted image. someone please help

The significant issue here is that when you called CGBitmapContextCreate, you specified that you're building an alpha channel alone, your data buffer is clearly using one byte per pixel, but for the "bytes per row" parameter, you've specified 4 * width. It should just be width. You generally use 4x when you're capturing four bytes per pixel (e.g. RGBA), but since your buffer is using one byte per pixel, you should remove that 4x factor.
Personally, I'd also advise a range of other improvements, namely:
The only thing that should be dispatched to the main queue is the updating of the UIKit control
You can retire finalData, as you don't need to copy from one buffer to another, but rather you can build msgData directly.
You should probably bypass the creation of your own buffer completely, though, and call CGBitmapContextCreate with nil for the data parameter, in which case, it will create its own buffer which you can retrieve via CGBitmapContextGetData. If you pass it a buffer, it assumes you'll manage this buffer yourself, which we're not doing here.
If you create your own buffer and don't manage that memory properly, you'll experience difficult-to-reproduce errors where it looks like it works, but suddenly you'll see the buffer corrupted for no reason in seemingly similar situations. By letting Core Graphics manage the memory, these sorts of problems are prevented.
I might separate the conversion of this byte buffer to a UIImage from the updating of the UIImageView.
So that yields something like:
func mask(from data: [UInt8]) -> UIImage? {
guard data.count >= 8 else {
print("data too small")
return nil
}
let width = Int(data[1]) | Int(data[0]) << 8
let height = Int(data[3]) | Int(data[2]) << 8
let colorSpace = CGColorSpaceCreateDeviceGray()
guard
data.count >= width * height + 8,
let context = CGContext(data: nil, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width, space: colorSpace, bitmapInfo: CGImageAlphaInfo.alphaOnly.rawValue),
let buffer = context.data?.bindMemory(to: UInt8.self, capacity: width * height)
else {
return nil
}
for index in 0 ..< width * height {
buffer[index] = data[index + 8]
}
return context.makeImage().flatMap { UIImage(cgImage: $0) }
}
And then
if let image = mask(from: data) {
DispatchQueue.main.async {
self.imageViewFinger.image = image
}
}
For Swift 2 rendition, see previous revision of this answer.

Related

What's the easiest way to get an Image from RGB raw data in swift?

I have an UnsafeMutablePointer<UInt8> which contains the raw RGB data to construct the image.
But I cannot find a API that can render the image from raw RGB data.
var content = UnsafeMutablePointer<UInt8>.allocate(capacity: 6)
apply_raw_data(content) // set content to [255,0,0,255,0,0]
let data = Data(bytes: content, count: 6)
let ui_image = UIImage(data: data) // we get nil
One way to do it is by creating an intermediate CGImage from a CGContext. However, this solution will require you to use RGBA (or ARGB) and not RGB as you actually asked about.
let width = 2
let height = 1
let bytesPerPixel = 4 // RGBA
let content = UnsafeMutablePointer<UInt8>.allocate(capacity: width * height * bytesPerPixel)
apply_raw_data(content) // set content to [255,0,0,0,255,0,0,0]
let colorSpace = CGColorSpaceCreateDeviceRGB()
guard let context = CGContext(data: content, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width * 4, space: colorSpace, bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)
else { return }
if let cgImage = context.makeImage() {
let image = UIImage(cgImage)
// use image here
}

Tensorflow lite image output different between python and iOS/Android

I convert Keras model to TF lite the output dimension is (1, 256, 256, 1).
the result on python is correct, but when I try to construct the image on ios swift the result is wrong.
Here is the code, that I use to construct an UIImage from a list of output.
// helper function
---------------------------------------
// MARK: - Extensions
extension Data {
init<T>(copyingBufferOf array: [T]) {
self = array.withUnsafeBufferPointer(Data.init)
}
/// Convert a Data instance to Array representation.
func toArray<T>(type: T.Type) -> [T] where T: ExpressibleByIntegerLiteral {
var array = [T](repeating: 0, count: self.count/MemoryLayout<T>.stride)
_ = array.withUnsafeMutableBytes { copyBytes(to: $0) }
return array
}
}
func imageFromSRGBColorArray(pixels: [UInt32], width: Int, height: Int) -> UIImage?
{
guard width > 0 && height > 0 else { return nil }
guard pixels.count == width * height else { return nil }
// Make a mutable copy
var data = pixels
// Convert array of pixels to a CGImage instance.
let cgImage = data.withUnsafeMutableBytes { (ptr) -> CGImage in
let ctx = CGContext(
data: ptr.baseAddress,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: MemoryLayout<UInt32>.size * width,
space: CGColorSpace(name: CGColorSpace.sRGB)!,
bitmapInfo: CGBitmapInfo.byteOrder32Little.rawValue
+ CGImageAlphaInfo.premultipliedFirst.rawValue
)!
return ctx.makeImage()!
}
// Convert the CGImage instance to an UIImage instance.
return UIImage(cgImage: cgImage)
}
let results = outputTensor.data.toArray(type: UInt32.self)
let maskImage = imageFromSRGBColorArray(pixels: results, width: 256, height: 256)
the result I get is completely wrong compared to python.
I think the function imageFromSRGBColorArray is not correct.
can anyone help me to figure out the problem?

How to create depth data and add it to an image?

Sorry, I duplicated this question How to build AVDepthData manually, because it doesn't have answers I want and I don't have enough rep to comment there. If you don't mind, I could remove my question in the future and ask somebody to move future answers to that topic.
So, my goal is to create depth data and attach it to an arbitrary image. There is an article on how to do it https://developer.apple.com/documentation/avfoundation/avdepthdata/creating_auxiliary_depth_data_manually, but I don't know how to implement any step of it. I won't post all questions at once and start with the first one.
As a first step a depth image must be converted per-pixel from grayscale to depth or disparity values. I took this snippet from the aforementioned topic:
func buildDepth(image: UIImage) -> AVDepthData? {
let width = Int(image.size.width)
let height = Int(image.size.height)
var maybeDepthMapPixelBuffer: CVPixelBuffer?
let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_DisparityFloat32, nil, &maybeDepthMapPixelBuffer)
guard status == kCVReturnSuccess, let depthMapPixelBuffer = maybeDepthMapPixelBuffer else {
return nil
}
CVPixelBufferLockBaseAddress(depthMapPixelBuffer, .init(rawValue: 0))
guard let baseAddress = CVPixelBufferGetBaseAddress(depthMapPixelBuffer) else {
return nil
}
let buffer = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Float32>.self)
for i in 0..<width * height {
buffer[i] = 0 // disparity must be calculated somehow, but set to 0 for testing purposes
}
CVPixelBufferUnlockBaseAddress(depthMapPixelBuffer, .init(rawValue: 0))
let info: [AnyHashable: Any] = [kCGImagePropertyPixelFormat: kCVPixelFormatType_DisparityFloat32,
kCGImagePropertyWidth: image.size.width,
kCGImagePropertyHeight: image.size.height,
kCGImagePropertyBytesPerRow: CVPixelBufferGetBytesPerRow(depthMapPixelBuffer)]
let metadata = generateMetadata(image: image)
let dic: [AnyHashable: Any] = [kCGImageAuxiliaryDataInfoDataDescription: info,
// I get an error when converting baseAddress to CFData
kCGImageAuxiliaryDataInfoData: baseAddress as! CFData,
kCGImageAuxiliaryDataInfoMetadata: metadata]
guard let depthData = try? AVDepthData(fromDictionaryRepresentation: dic) else {
return nil
}
return depthData
}
Then the article says to load a base address of a pixel buffer (in which is the disparity map) as CFData and pass it as kCGImageAuxiliaryDataInfoData value into a CFDictionary. But I get an error when converting baseAddress to CFData. I tried to convert the pixel buffer itself too, but without luck. What do I have to pass as kCGImageAuxiliaryDataInfoData? Did I create the disparity buffer correctly in the first place?
Aside from this problem it would be cool if somebody could direct me to some sample code on how to do the whole thing.
Your question really helped me get from cvPixelBuffer to AVDepthData so thank you. It was about 95% of the way there.
To fix your (and mine) issue I added the following:
let bytesPerRow = CVPixelBufferGetBytesPerRow(depthMapPixelBuffer)
let size = bytesPerRow * height;
... code code code ...
CVPixelBufferLockBaseAddress(depthMapPixelBuffer!, .init(rawValue: 0))
let baseAddress = CVPixelBufferGetBaseAddressOfPlane(depthMapPixelBuffer!, 0)
let data = NSData(bytes: baseAddress, length: size);
... code code code ...
let dic: [AnyHashable: Any] = [kCGImageAuxiliaryDataInfoDataDescription: info,
kCGImageAuxiliaryDataInfoData: data,
kCGImageAuxiliaryDataInfoMetadata: metadata]

Thread 1: EXC_BAD_ACCESS (code=EXC_I386_GPFLT)

In my swift project, I have two classes that work together to hold Pixel values of an image to be able to modify red, green, blue and alpha values. An UnsafeMutableBufferPointer holds lots of bites that are comprised of the Pixel class objects.
I can interact with that the class that holds the UnsafeMutableBufferPointer<Pixel> property. I can access all of the properties on that object and that all works fine. The only problem I'm having with the UnsafeMutableBufferPoint<Pixel> is trying to loop through it with my Pixel object and it keeps crashing with the Thread 1: EXC_BAD_ACCESS (code=EXC_I386_GPFLT) exception.
init!(image: UIImage)
{
_width = Int(image.size.width)
_height = Int(image.size.height)
guard let cgImage = image.cgImage else { return nil }
_width = Int(image.size.width)
_height = Int(image.size.height)
let bitsPerComponent = 8
let bytesPerPixel = 4
let bytesPerRow = _width * bytesPerPixel
let imageData = UnsafeMutablePointer<Pixel>.allocate(capacity: _width * _height)
let colorSpace = CGColorSpaceCreateDeviceRGB()
var bitmapInfo: UInt32 = CGBitmapInfo.byteOrder32Big.rawValue
bitmapInfo |= CGImageAlphaInfo.premultipliedLast.rawValue & CGBitmapInfo.alphaInfoMask.rawValue
guard let imageContext = CGContext(data: imageData, width: _width, height: _height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo) else { return nil }
imageContext.draw(cgImage, in: CGRect(origin: CGPoint.zero, size: image.size))
_pixels = UnsafeMutableBufferPointer<Pixel>(start: imageData, count: _width * _height)
}
This function is the part that is crashing the program. The exact part that is crashing is the for loop that is looping through the rgba.pixels. rgba.pixels is the UnsafeMutableBufferPointer.
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any])
{
let image: UIImage = info[UIImagePickerControllerEditedImage] as! UIImage!
let rgba = RGBA(image: image)!
for pixel in rgba.pixels
{
print(pixel.red)
}
self.dismiss(animated: true, completion: nil);
}
This is the constructor where I create the UnsafeMutableBufferPointer<Pixel>. Is there an easier way to do this and still be able to get the RBGA values and change them easily.
The Pixel class is a UInt32 value this is split into four UInt 8 values.
Am I using the wrong construct to hold those values and if so, is there a safer or easier construct to use? Or am I doing something wrong when accessing the Pixel values?
This is how I got the pixels of an image -
// Grab and set up variables for the original image
let inputCGImage = inputImage.CGImage
let inputWidth: Int = CGImageGetWidth(inputCGImage)
let inputHeight: Int = CGImageGetHeight(inputCGImage)
// Get the colorspace that will be used for image processing (RGB/HSV)
let colorSpace: CGColorSpaceRef = CGColorSpaceCreateDeviceRGB()!
// Hardcode memory variables
let bytesPerPixel = 4 // 32 bits = 4 bytes
let bitsPerComponent = 8 // 32 bits div. by 4 components (RGBA) = 8 bits per component
let inputBytesPerRow = bytesPerPixel * inputWidth
// Get a pointer pointing to an allocated array to hold all the pixel data of the image
let inputPixels = UnsafeMutablePointer<UInt32>(calloc(inputHeight * inputWidth, sizeof(UInt32)))
// Create a context to draw the original image in (aka put the pixel data into the above array)
let context: CGContextRef = CGBitmapContextCreate(inputPixels, inputWidth, inputHeight, bitsPerComponent, inputBytesPerRow, colorSpace, CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Big.rawValue)!
CGContextDrawImage(context, CGRect(x: 0, y: 0, width: inputWidth, height: inputHeight), inputCGImage)
Keep in mind this is not Swift 3 syntax incase that's what you're using, but that's the basic algorithm. Now to grab the individual color values of each pixel, you will have to implement these functions -
func Mask8(x: UInt32) -> UInt32
{
return x & 0xFF
}
func R(x: UInt32) -> UInt32
{
return Mask8(x)
}
func G(x: UInt32) -> UInt32
{
return Mask8(x >> 8)
}
func B(x: UInt32) -> UInt32
{
return Mask8(x >> 16)
}
func A(x: UInt32) -> UInt32
{
return Mask8(x >> 24)
}
To create a completely new color after processing the RGBA values, you use this function -
func RGBAMake(r: UInt32, g: UInt32, b: UInt32, a: UInt32) -> UInt32
{
return (Mask8(r) | Mask8(g) << 8 | Mask8(b) << 16 | Mask8(a) << 24)
}
To iterate through the pixels array, you do it as so -
var currentPixel = inputPixels
for _ in 0..<height
{
for i in 0..<width
{
let color: UInt32 = currentPixel.memory
if i < width - 1
{
print(NSString(format: "%3.0f", R(x: color), terminator: " "))
}
else
{
print(NSString(format: "%3.0f", R(x: color)))
}
currentPixel += 1
}
}

Scaling Images: how can the accelerate be the slowest method?

I am testing several methods to rescale a UIImage.
I have tested all these methods posted here and measured the time they take to resize an image.
1) UIGraphicsBeginImageContextWithOptions & UIImage -drawInRect:
let image = UIImage(contentsOfFile: self.URL.path!)
let size = CGSizeApplyAffineTransform(image.size, CGAffineTransformMakeScale(0.5, 0.5))
let hasAlpha = false
let scale: CGFloat = 0.0 // Automatically use scale factor of main screen
UIGraphicsBeginImageContextWithOptions(size, !hasAlpha, scale)
image.drawInRect(CGRect(origin: CGPointZero, size: size))
let scaledImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
2) CGBitmapContextCreate & CGContextDrawImage
let cgImage = UIImage(contentsOfFile: self.URL.path!).CGImage
let width = CGImageGetWidth(cgImage) / 2
let height = CGImageGetHeight(cgImage) / 2
let bitsPerComponent = CGImageGetBitsPerComponent(cgImage)
let bytesPerRow = CGImageGetBytesPerRow(cgImage)
let colorSpace = CGImageGetColorSpace(cgImage)
let bitmapInfo = CGImageGetBitmapInfo(cgImage)
let context = CGBitmapContextCreate(nil, width, height, bitsPerComponent, bytesPerRow, colorSpace, bitmapInfo.rawValue)
CGContextSetInterpolationQuality(context, kCGInterpolationHigh)
CGContextDrawImage(context, CGRect(origin: CGPointZero, size: CGSize(width: CGFloat(width), height: CGFloat(height))), cgImage)
let scaledImage = CGBitmapContextCreateImage(context).flatMap { UIImage(CGImage: $0) }
3) CGImageSourceCreateThumbnailAtIndex
import ImageIO
if let imageSource = CGImageSourceCreateWithURL(self.URL, nil) {
let options: [NSString: NSObject] = [
kCGImageSourceThumbnailMaxPixelSize: max(size.width, size.height) / 2.0,
kCGImageSourceCreateThumbnailFromImageAlways: true
]
let scaledImage = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, options).flatMap { UIImage(CGImage: $0) }
}
4) Lanczos Resampling with Core Image
let image = CIImage(contentsOfURL: self.URL)
let filter = CIFilter(name: "CILanczosScaleTransform")!
filter.setValue(image, forKey: "inputImage")
filter.setValue(0.5, forKey: "inputScale")
filter.setValue(1.0, forKey: "inputAspectRatio")
let outputImage = filter.valueForKey("outputImage") as! CIImage
let context = CIContext(options: [kCIContextUseSoftwareRenderer: false])
let scaledImage = UIImage(CGImage: self.context.createCGImage(outputImage, fromRect: outputImage.extent()))
5) vImage in Accelerate
let cgImage = UIImage(contentsOfFile: self.URL.path!).CGImage
// create a source buffer
var format = vImage_CGImageFormat(bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil,
bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.First.rawValue),
version: 0, decode: nil, renderingIntent: CGColorRenderingIntent.RenderingIntentDefault)
var sourceBuffer = vImage_Buffer()
defer {
sourceBuffer.data.dealloc(Int(sourceBuffer.height) * Int(sourceBuffer.height) * 4)
}
var error = vImageBuffer_InitWithCGImage(&sourceBuffer, &format, nil, cgImage, numericCast(kvImageNoFlags))
guard error == kvImageNoError else { return nil }
// create a destination buffer
let scale = UIScreen.mainScreen().scale
let destWidth = Int(image.size.width * 0.5 * scale)
let destHeight = Int(image.size.height * 0.5 * scale)
let bytesPerPixel = CGImageGetBitsPerPixel(image.CGImage) / 8
let destBytesPerRow = destWidth * bytesPerPixel
let destData = UnsafeMutablePointer<UInt8>.alloc(destHeight * destBytesPerRow)
defer {
destData.dealloc(destHeight * destBytesPerRow)
}
var destBuffer = vImage_Buffer(data: destData, height: vImagePixelCount(destHeight), width: vImagePixelCount(destWidth), rowBytes: destBytesPerRow)
// scale the image
error = vImageScale_ARGB8888(&sourceBuffer, &destBuffer, nil, numericCast(kvImageHighQualityResampling))
guard error == kvImageNoError else { return nil }
// create a CGImage from vImage_Buffer
let destCGImage = vImageCreateCGImageFromBuffer(&destBuffer, &format, nil, nil, numericCast(kvImageNoFlags), &error)?.takeRetainedValue()
guard error == kvImageNoError else { return nil }
// create a UIImage
let scaledImage = destCGImage.flatMap { UIImage(CGImage: $0, scale: 0.0, orientation: image.imageOrientation) }
After testing this for hours and measure the time every method took for rescaling the images to 100x100, my conclusions are completely different from NSHipster. First of all the vImage in accelerate is 200 times slower than the first method, that in my opinion is the poor cousin of the other ones. The core image method is also slow. But I am intrigued how method #1 can smash methods 3, 4 and 5, some of them in theory process stuff on the GPU.
Method #3 for example, took 2 seconds to resize a 1024x1024 image to 100x100. On the other hand #1 took 0.01 seconds!
Am I missing something?
Something must be wrong or Apple would not take time to write accelerate and CIImage stuff.
NOTE: I am measuring the time from the time the image is already loaded on a variable to the time a scaled version is saved to another variable. I am not considering the time it takes to read from the file.
Accelerate can be the slowest method for a variety of reasons:
The code you show may spend a lot of time just extracting the data
from the CGImage and making a new image. You didn't, for example,
use any features that would allow the CGImage to use your vImage result
directly rather than make a copy. Possibly a colorspace conversion was also required as part of some of those extract / create CGImage operations. Hard to tell from here.
Some of the other methods may not have done anything, deferring the
work until later when absolutely forced to do it. If that was after your end time, then the work wasn't measured.
Some of the other methods have the advantage of being able to
directly use the contents of the image without having to make a copy
first.
Different resampling methods (e.g. Bilinear vs. Lanczos) have
different cost
The GPU can actually be faster at some stuff, and resampling is one
of the tasks it is specially optimized to do. On the flip side, random data access (such as occurs in resampling) is not a nice thing to do to the vector unit.
Timing methods can impact the result. Accelerate is multithreaded.
If you use wall clock time, you will get one answer. If you use
getrusage or a sampler, you'll get another.
If you really think Accelerate is way off the mark here, file a bug. I certainly would check with Instruments Time Profile that you are spending the majority of your time in vImageScale in your benchmark loop before doing so, though.

Resources