Thread 1: EXC_BAD_ACCESS (code=EXC_I386_GPFLT) - ios

In my swift project, I have two classes that work together to hold Pixel values of an image to be able to modify red, green, blue and alpha values. An UnsafeMutableBufferPointer holds lots of bites that are comprised of the Pixel class objects.
I can interact with that the class that holds the UnsafeMutableBufferPointer<Pixel> property. I can access all of the properties on that object and that all works fine. The only problem I'm having with the UnsafeMutableBufferPoint<Pixel> is trying to loop through it with my Pixel object and it keeps crashing with the Thread 1: EXC_BAD_ACCESS (code=EXC_I386_GPFLT) exception.
init!(image: UIImage)
{
_width = Int(image.size.width)
_height = Int(image.size.height)
guard let cgImage = image.cgImage else { return nil }
_width = Int(image.size.width)
_height = Int(image.size.height)
let bitsPerComponent = 8
let bytesPerPixel = 4
let bytesPerRow = _width * bytesPerPixel
let imageData = UnsafeMutablePointer<Pixel>.allocate(capacity: _width * _height)
let colorSpace = CGColorSpaceCreateDeviceRGB()
var bitmapInfo: UInt32 = CGBitmapInfo.byteOrder32Big.rawValue
bitmapInfo |= CGImageAlphaInfo.premultipliedLast.rawValue & CGBitmapInfo.alphaInfoMask.rawValue
guard let imageContext = CGContext(data: imageData, width: _width, height: _height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo) else { return nil }
imageContext.draw(cgImage, in: CGRect(origin: CGPoint.zero, size: image.size))
_pixels = UnsafeMutableBufferPointer<Pixel>(start: imageData, count: _width * _height)
}
This function is the part that is crashing the program. The exact part that is crashing is the for loop that is looping through the rgba.pixels. rgba.pixels is the UnsafeMutableBufferPointer.
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any])
{
let image: UIImage = info[UIImagePickerControllerEditedImage] as! UIImage!
let rgba = RGBA(image: image)!
for pixel in rgba.pixels
{
print(pixel.red)
}
self.dismiss(animated: true, completion: nil);
}
This is the constructor where I create the UnsafeMutableBufferPointer<Pixel>. Is there an easier way to do this and still be able to get the RBGA values and change them easily.
The Pixel class is a UInt32 value this is split into four UInt 8 values.
Am I using the wrong construct to hold those values and if so, is there a safer or easier construct to use? Or am I doing something wrong when accessing the Pixel values?

This is how I got the pixels of an image -
// Grab and set up variables for the original image
let inputCGImage = inputImage.CGImage
let inputWidth: Int = CGImageGetWidth(inputCGImage)
let inputHeight: Int = CGImageGetHeight(inputCGImage)
// Get the colorspace that will be used for image processing (RGB/HSV)
let colorSpace: CGColorSpaceRef = CGColorSpaceCreateDeviceRGB()!
// Hardcode memory variables
let bytesPerPixel = 4 // 32 bits = 4 bytes
let bitsPerComponent = 8 // 32 bits div. by 4 components (RGBA) = 8 bits per component
let inputBytesPerRow = bytesPerPixel * inputWidth
// Get a pointer pointing to an allocated array to hold all the pixel data of the image
let inputPixels = UnsafeMutablePointer<UInt32>(calloc(inputHeight * inputWidth, sizeof(UInt32)))
// Create a context to draw the original image in (aka put the pixel data into the above array)
let context: CGContextRef = CGBitmapContextCreate(inputPixels, inputWidth, inputHeight, bitsPerComponent, inputBytesPerRow, colorSpace, CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Big.rawValue)!
CGContextDrawImage(context, CGRect(x: 0, y: 0, width: inputWidth, height: inputHeight), inputCGImage)
Keep in mind this is not Swift 3 syntax incase that's what you're using, but that's the basic algorithm. Now to grab the individual color values of each pixel, you will have to implement these functions -
func Mask8(x: UInt32) -> UInt32
{
return x & 0xFF
}
func R(x: UInt32) -> UInt32
{
return Mask8(x)
}
func G(x: UInt32) -> UInt32
{
return Mask8(x >> 8)
}
func B(x: UInt32) -> UInt32
{
return Mask8(x >> 16)
}
func A(x: UInt32) -> UInt32
{
return Mask8(x >> 24)
}
To create a completely new color after processing the RGBA values, you use this function -
func RGBAMake(r: UInt32, g: UInt32, b: UInt32, a: UInt32) -> UInt32
{
return (Mask8(r) | Mask8(g) << 8 | Mask8(b) << 16 | Mask8(a) << 24)
}
To iterate through the pixels array, you do it as so -
var currentPixel = inputPixels
for _ in 0..<height
{
for i in 0..<width
{
let color: UInt32 = currentPixel.memory
if i < width - 1
{
print(NSString(format: "%3.0f", R(x: color), terminator: " "))
}
else
{
print(NSString(format: "%3.0f", R(x: color)))
}
currentPixel += 1
}
}

Related

TensorFlowLite.Tensor to UUImage

I am new with swift, TFlite and IOS. I succeed to convert, run my model. However at the end, I need to reconstruct an image. My TFlite model return a TFLite.tensor Float32 4d - shape (1, height, width, 3).
let outputTensor: Tensor
outputTensor = try myInterpreter.output(at: 0)
I am looking to make a RGB picture without alpha. In python, it will like this:
Image.fromarray((np.array(outputTensor.data) * 255).astype(np.uint8))
From my understanding the best way will be to make a CVPixelBuffer, apply a CoreOS transformation (for the x255) and finally make the UUImage. I am deeply lost in the IOS doc, it exists many possibilities, does the community has a suggestion ?
++t
From Google example, an extension of UIImage can be coded:
extension UIImage {
convenience init?(data: Data, size: CGSize) {
let width = Int(size.width)
let height = Int(size.height)
let floats = data.toArray(type: Float32.self)
let bufferCapacity = width * height * 4
let unsafePointer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferCapacity)
let unsafeBuffer = UnsafeMutableBufferPointer<UInt8>(
start: unsafePointer,
count: bufferCapacity)
defer {
unsafePointer.deallocate()
}
for x in 0..<width {
for y in 0..<height {
let floatIndex = (y * width + x) * 3
let index = (y * width + x) * 4
let red = UInt8(floats[floatIndex] * 255)
let green = UInt8(floats[floatIndex + 1] * 255)
let blue = UInt8(floats[floatIndex + 2] * 255)
unsafeBuffer[index] = red
unsafeBuffer[index + 1] = green
unsafeBuffer[index + 2] = blue
unsafeBuffer[index + 3] = 0
}
}
let outData = Data(buffer: unsafeBuffer)
// Construct image from output tensor data
let alphaInfo = CGImageAlphaInfo.noneSkipLast
let bitmapInfo = CGBitmapInfo(rawValue: alphaInfo.rawValue)
.union(.byteOrder32Big)
let colorSpace = CGColorSpaceCreateDeviceRGB()
guard
let imageDataProvider = CGDataProvider(data: outData as CFData),
let cgImage = CGImage(
width: width,
height: height,
bitsPerComponent: 8,
bitsPerPixel: 32,
bytesPerRow: MemoryLayout<UInt8>.size * 4 * Int(size.width),
space: colorSpace,
bitmapInfo: bitmapInfo,
provider: imageDataProvider,
decode: nil,
shouldInterpolate: false,
intent: .defaultIntent
)
else {
return nil
}
self.init(cgImage: cgImage)
}
}
Then the image can be easily constructed from the inference of TFLite.
let outputTensor: Tensor
outputTensor = try decoder.output(at: 0)
image = UIImage(data: outputTensor.data, size: size) ?? UIImage()

EXC_BAD_ACCESS (code=1) when updating UIImage of UIImageView

I am new with iOS and CoreML. I have a very simple UI with two UIImageViews (one should be the input, and the second should be the output). When tapping the first one, the image should be processed by a neural network and the output should be displayed in the second one.
However, when I try to download the image from the MLMultiArray output object and create an UIImage from it which I can then upload to the second UIImageView I get an EXC_BAD_ACCESS (code=1) .
I have reduced the problem to not calling the neural network processing at all, just trying to create a new image from a MLMultiArray. The outcome was the same.
After that I tried generating an UIImage from an empty buffer. The image is created correctly, but if I attempt to update the UIImageView to use it, I get the same error.
If I try to update the second UIImageView to a different image (e.g.: the input image) everything works fine.
I assume this is a memory management issue about the UIImage object I am creating but I am not able to figure out what I am doing wrong
class ViewController: UIViewController {
#IBOutlet weak var out: UIImageView!
override func viewDidLoad() {
super.viewDidLoad()
}
#IBAction func imageTapped(_ sender: UITapGestureRecognizer) {
let imageView = sender.view as? UIImageView
if let imageToAnalyse = imageView?.image {
if let outImg = process(forImage: imageToAnalyse) {
out.image = outImg
}
}
}
func process (forImage inImage:UIImage) -> UIImage? {
let size = CGSize(width: 512, height: 512)
let mlOut = try? MLMultiArray(shape: [1, size.height, size.width] as [NSNumber], dataType: .float32)
let newImage = getSinglePlaneImage(inBuffer: mlOut!, width: Int(size.width), height: Int(size.height))
return newImage
}
func getSinglePlaneImage(inBuffer: MLMultiArray, width: Int, height: Int) -> UIImage
{
var newImage: UIImage
// let floatPtr = inBuffer.dataPointer.bindMemory(to: Float32.self, capacity: inBuffer.count)
// let floatBuffer = UnsafeBufferPointer(start: floatPtr, count: inBuffer.count)
// let pixelValues : [UInt8]? = Array(floatBuffer).map({UInt8( ImageProcessor.clamp( (($0) + 1.0) * 128.0, minValue: 0.0, maxValue: 255.0) ) })
//simulating pixels from MLMultiArray
let pixels : [UInt8]? = Array(repeating: 0, count: 512*512)
var imageRef: CGImage?
if var pixelValues = pixels {
let bitsPerComponent = 8
let bytesPerPixel = 1
let bitsPerPixel = bytesPerPixel * bitsPerComponent
let bytesPerRow = bytesPerPixel * width
let totalBytes = height * bytesPerRow
imageRef = withUnsafePointer(to: &pixelValues, {
ptr -> CGImage? in
var imageRef: CGImage?
let colorSpaceRef = CGColorSpaceCreateDeviceGray()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue).union(CGBitmapInfo())
let data = UnsafeRawPointer(ptr.pointee).assumingMemoryBound(to: UInt8.self)
let releaseData: CGDataProviderReleaseDataCallback = {
(info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
}
if let providerRef = CGDataProvider(dataInfo: nil, data: data, size: totalBytes, releaseData: releaseData) {
imageRef = CGImage(width: width,
height: height,
bitsPerComponent: bitsPerComponent,
bitsPerPixel: bitsPerPixel,
bytesPerRow: bytesPerRow,
space: colorSpaceRef,
bitmapInfo: bitmapInfo,
provider: providerRef,
decode: nil,
shouldInterpolate: false,
intent: CGColorRenderingIntent.defaultIntent)
}
return imageRef
})
}
newImage = UIImage(cgImage: imageRef!)
return newImage
}
}
Seems your code would convert 512x512-float32 array into 512x512-UInt8 Array successfully, so I write this answer based on the uncommented version of your code. (Though, the conversion is not efficient enough and has some room to improve.)
UPDATE
The following description is not the right solution for the OP's issue. Just kept for record. Please skip to UPDATED CODE at the bottom of this answer.
OLD CODE (NOT the right solution)
First of all, the worst flaw in the code are the following two lines:
imageRef = withUnsafePointer(to: &pixelValues, {
let data = UnsafeRawPointer(ptr.pointee).assumingMemoryBound(to: UInt8.self)
The first line above passes a pointer to [UInt8]?, in Swift, [UInt8]? (aka Optional<Array<UInt8>>) is an 8-byte struct, not a contiguous region like C-arrays.
The second is more dangerous. ptr.pointee is [UInt8]?, but accessing Swift Arrays through pointer is not guaranteed. And passing an Array to UnsafeRawPointer.init(_:) may create a temporal region which would be deallocated just after the call to the initializer.
As you know, accessing a dangling pointer does not make harm in some limited condition occasionally, but may generate unexpected result at any time.
I would write something like this:
func getSinglePlaneImage(inBuffer: MLMultiArray, width: Int, height: Int) -> UIImage {
//simulating pixels from MLMultiArray
//...
let pixelValues: [UInt8] = Array(repeating: 0, count: 1*512*512)
let bitsPerComponent = 8
let bytesPerPixel = 1
let bitsPerPixel = bytesPerPixel * 8
let bytesPerRow = bytesPerPixel * width
let totalBytes = height * bytesPerRow
let imageRef = pixelValues.withUnsafeBytes({bytes -> CGImage? in
var imageRef: CGImage?
let colorSpaceRef = CGColorSpaceCreateDeviceGray()
let bitmapInfo: CGBitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue)
let data = bytes.baseAddress!.assumingMemoryBound(to: UInt8.self)
let releaseData: CGDataProviderReleaseDataCallback = {_,_,_ in }
if let providerRef = CGDataProvider(dataInfo: nil, data: data, size: totalBytes, releaseData: releaseData) {
imageRef = CGImage(width: width,
height: height,
bitsPerComponent: bitsPerComponent,
bitsPerPixel: bitsPerPixel,
bytesPerRow: bytesPerRow,
space: colorSpaceRef,
bitmapInfo: bitmapInfo,
provider: providerRef,
decode: nil,
shouldInterpolate: false,
intent: .defaultIntent)
}
return imageRef
})
let newImage = UIImage(cgImage: imageRef!)
return newImage
}
When you want a pointer pointing to the starting element of an Array, use withUnsafeBytes and use the pointer (in fact, it is an UnsafeRawBufferPointer) inside the closure argument.
One more, your pixels or pixelValues have no need to be an Optional.
Or else, you can create a grey-scale image with Float32 for each pixel.
func getSinglePlaneImage(inBuffer: MLMultiArray, width: Int, height: Int) -> UIImage {
//simulating pixels from MLMultiArray
//...
let pixelValues: [Float32] = Array(repeating: 0, count: 1*512*512)
let bitsPerComponent = 32 //<-
let bytesPerPixel = 4 //<-
let bitsPerPixel = bytesPerPixel * 8
let bytesPerRow = bytesPerPixel * width
let totalBytes = height * bytesPerRow
let imageRef = pixelValues.withUnsafeBytes({bytes -> CGImage? in
var imageRef: CGImage?
let colorSpaceRef = CGColorSpaceCreateDeviceGray()
let bitmapInfo: CGBitmapInfo = [CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue),
.byteOrder32Little, .floatComponents] //<-
let data = bytes.baseAddress!.assumingMemoryBound(to: Float32.self)
let releaseData: CGDataProviderReleaseDataCallback = {_,_,_ in }
if let providerRef = CGDataProvider(dataInfo: nil, data: data, size: totalBytes, releaseData: releaseData) {
imageRef = CGImage(width: width,
height: height,
bitsPerComponent: bitsPerComponent,
bitsPerPixel: bitsPerPixel,
bytesPerRow: bytesPerRow,
space: colorSpaceRef,
bitmapInfo: bitmapInfo,
provider: providerRef,
decode: nil,
shouldInterpolate: false,
intent: CGColorRenderingIntent.defaultIntent)
}
return imageRef
})
let newImage = UIImage(cgImage: imageRef!)
return newImage
}
Both work as expected in my testing project, but if you find something wrong, please inform me.
UPDATED CODE (Hope this is the right solution)
I was missing the fact that CGDataProvider keeps the pointer when created with init(dataInfo:data:size:releaseData:) even after a CGImage is created. So, it may be referenced after the closure to withUnsafeBytes is finished, when it is not valid.
You should better use CGDataProvider.init(data:) in such cases.
func getSinglePlaneImage(inBuffer: MLMultiArray, width: Int, height: Int) -> UIImage {
var newImage: UIImage
//let floatPtr = inBuffer.dataPointer.bindMemory(to: Float32.self, capacity: inBuffer.count)
//let floatBuffer = UnsafeBufferPointer(start: floatPtr, count: inBuffer.count)
//let pixelValues: Data = Data((floatBuffer.lazy.map{
// UInt8(ImageProcessor.clamp((($0) + 1.0) * 128.0, minValue: 0.0, maxValue: 255.0))
//})
//simulating pixels from MLMultiArray
//...
let pixelValues = Data(count: 1*512*512) // <- ###
var imageRef: CGImage?
let bitsPerComponent = 8
let bytesPerPixel = 1
let bitsPerPixel = bytesPerPixel * bitsPerComponent
let bytesPerRow = bytesPerPixel * width
let colorSpaceRef = CGColorSpaceCreateDeviceGray()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue)
if let providerRef = CGDataProvider(data: pixelValues as CFData) { // <-###
imageRef = CGImage(width: width,
height: height,
bitsPerComponent: bitsPerComponent,
bitsPerPixel: bitsPerPixel,
bytesPerRow: bytesPerRow,
space: colorSpaceRef,
bitmapInfo: bitmapInfo,
provider: providerRef,
decode: nil,
shouldInterpolate: false,
intent: CGColorRenderingIntent.defaultIntent)
}
newImage = UIImage(cgImage: imageRef!)
return newImage
}
As far as I tried, this does not crash even in actual device with number of repeated touches. Please try. Thanks for your patience.

EXC_BAD_INSTRUCTION Swift Dithering Function

I've been trying to create a dithering function in Swift but I keep running into issues. I've noticed the code loops just fine for the first 9000 or so pixels of a random image I selected. But then it gives me a runtime error, I've looked everywhere and I can't seem to solve the issue. Please help.
file:///Users/jeffn/Desktop/MyPlayground34.playground/: error: Playground execution aborted: error: Execution was interrupted, reason: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0).
The process has been left at the point where it was interrupted, use "thread return -x" to return to the state before expression evaluation.
import UIKit
struct Pixel {
var value: UInt32
var red: UInt8 {
get { return UInt8(value & 0xFF) }
set { value = UInt32(newValue) | (value & 0xFFFFFF00) }
}
var green: UInt8 {
get { return UInt8((value >> 8) & 0xFF) }
set { value = (UInt32(newValue) << 8) | (value & 0xFFFF00FF) }
}
var blue: UInt8 {
get { return UInt8((value >> 16) & 0xFF) }
set { value = (UInt32(newValue) << 16) | (value & 0xFF00FFFF) }
}
var alpha: UInt8 {
get { return UInt8((value >> 24) & 0xFF) }
set { value = (UInt32(newValue) << 24) | (value & 0x00FFFFFF) }
}}
public struct RGBA {
var pixels: UnsafeMutableBufferPointer<Pixel>
var width: Int
var height: Int
init?(image: UIImage) {
guard let cgImage = image.cgImage else { return nil }
width = Int(image.size.width)
height = Int(image.size.height)
let bitsPerComponent = 8
let bytesPerPixel = 4
let bytesPerRow = width * bytesPerPixel
let imageData = UnsafeMutablePointer<Pixel>.allocate(capacity: width * height)
let colorSpace = CGColorSpaceCreateDeviceRGB()
var bitmapInfo: UInt32 = CGBitmapInfo.byteOrder32Big.rawValue
bitmapInfo |= CGImageAlphaInfo.premultipliedLast.rawValue & CGBitmapInfo.alphaInfoMask.rawValue
guard let imageContext = CGContext(data: imageData, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo) else { return nil }
imageContext.draw(cgImage, in: CGRect(origin: CGPoint(x: 0,y :0), size: image.size))
pixels = UnsafeMutableBufferPointer<Pixel>(start: imageData, count: width * height)
}
public func toUIImage() -> UIImage? {
let bitsPerComponent = 8
let bytesPerPixel = 4
let bytesPerRow = width * bytesPerPixel
let colorSpace = CGColorSpaceCreateDeviceRGB()
var bitmapInfo: UInt32 = CGBitmapInfo.byteOrder32Big.rawValue
bitmapInfo |= CGImageAlphaInfo.premultipliedLast.rawValue & CGBitmapInfo.alphaInfoMask.rawValue
let imageContext = CGContext(data: pixels.baseAddress, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo, releaseCallback: nil, releaseInfo: nil)
guard let cgImage = imageContext!.makeImage() else {return nil}
let image = UIImage(cgImage: cgImage)
return image
}
}
public func contrast(image: UIImage) -> RGBA {
let rgba = RGBA(image: image)!
var new_red: UInt8 = 0
var new_green = 0
var new_blue = 0
var new_alpha = 0
var error_red: UInt8 = 0
var error_green = 0
var error_blue = 0
var error_alpha = 0
var pixel_1_red: UInt8 = 0
var output_red: UInt8 = 0
let w1: Double = 7.00/16.00
let w2: Double = 3.00/16.00
let w3: Double = 5.00/16.00
let w4: Double = 1.00/16.00
for y in 0..<rgba.height-1{
for x in 0..<rgba.width-1{
var index = y * rgba.width + x
var index_1 = y*rgba.width + x + 1
var pixel = rgba.pixels[index]
var pixel_1 = rgba.pixels[index_1]
pixel_1_red = pixel_1.red
if(pixel.red < 128){new_red = 0} else {new_red = 255}
error_red = new_red - pixel.red
var double_error_red = Double(error_red)*w1
var int_error_red = UInt8(double_error_red)
output_red = pixel_1_red + int_error_red
pixel_1.red = output_red
rgba.pixels[index_1] = pixel_1
}
}
return rgba
}
let image = UIImage(named: "newlowpassfilter.jpg")!
let rgba = contrast(image: image)
let newImage = rgba.toUIImage()
image
newImage
Maybe integer overflow is your issue.
With testing with one of my sample jpg-image, this line caused overflow:
output_red = pixel_1_red + int_error_red
You may need to change the line to something like this:
let temp_red = Int(pixel_1_red) + Int(int_error_red)
output_red = temp_red > 255 ? 255 : UInt8(temp_red)
In Swift, + operator detects overflow and causes app crash, and overflow ignoring operator &+ does not suit for image processing. You may need to care about the result range of each operation in your code.

How to read and log the raw pixels of image in swift iOS

I need to read pixel values of an image and iterate to print in swift output, I have written this so far and used a RGBAImage class to read out pixels. I'm getting lost from CGContextRef to Iteration. I tried to write from CGImage, getting pixel data from objective C language to swift since I wanted to work in swift.
func createRGBAPixel(inImage: CGImageRef) -> CGContextRef {
//Image width, height
let pixelWidth = CGImageGetWidth(inImage)
let pixelHeight = CGImageGetHeight(inImage)
//Declaring number of bytes
let bytesPerRow = Int(pixelWidth) * 4
let byteCount = bytesPerRow * Int(pixelHeight)
//RGB color space
let colorSpace = CGColorSpaceCreateDeviceRGB()
//Allocating image data
let mapData = malloc(byteCount)
let mapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedFirst.rawValue)
//Create bitmap context
let context = CGBitmapContextCreate(mapData, pixelWidth, pixelHeight, Int(8), Int(bytesPerRow), colorSpace, mapInfo.rawValue)
let pixelImage = CGBitmapContextCreate(pixels, pixelWidth, pixelHeight, bitsPerComponent, bytesPerRow, colorSpace, mapInfo)
let CGContextRef = pixelImage
let CGContextDrawImage(context, CGRectMake(0, 0, pixelWidth, pixelHeight), inImage)
//Iterating and logging
print("Logging pixel counts")
let pixels = calloc(pixelHeight * pixelWidth, sizeof(UInt32))
let myImage = CGImageRef: inImage
let myRGBA = RGBAImage(image: myImage)! //RGBAImage class to read pixels.
var number = 0
var currentPixel:Int32 = 0
currentPixel = pixels * UInt32
for number in 0..<pixelHeight {
for number in 0..<pixelWidth {
var color = color * currentPixel
print((pixel.red + pixel.green + pixel.blue) / 3.0)
currentPixel++
}
}
return context!
}
I created small class for this:
class ImagePixelReader {
enum Component:Int {
case r = 0
case g = 1
case b = 2
case alpha = 3
}
struct Color {
var r:UInt8
var g:UInt8
var b:UInt8
var a:UInt8
var uiColor:UIColor {
return UIColor(red:CGFloat(r)/255.0,green:CGFloat(g)/255.0,blue:CGFloat(b)/255.0,alpha:CGFloat(alpha)/255.0)
}
}
let image:UIImage
private var data:CFData
private let pointer:UnsafePointer<UInt8>
private let scale:Int
init?(image:UIImage){
self.image = image
guard let cfdata = self.image.cgImage?.dataProvider?.data,
let pointer = CFDataGetBytePtr(cfdata) else {
return nil
}
self.scale = Int(image.scale)
self.data = cfdata
self.pointer = pointer
}
func componentAt(_ component:Component,x:Int,y:Int)->UInt8{
assert(CGFloat(x) < image.size.width)
assert(CGFloat(y) < image.size.height)
let pixelPosition = (Int(image.size.width) * y * scale + x) * 4 * scale
return pointer[pixelPosition + component.rawValue]
}
func colorAt(x:Int,y:Int)->Color{
assert(CGFloat(x) < image.size.width)
assert(CGFloat(y) < image.size.height)
let pixelPosition = (Int(image.size.width) * y * scale + x) * 4 * scale
return Color(r: pointer[pixelPosition + Component.r.rawValue],
g: pointer[pixelPosition + Component.g.rawValue],
b: pointer[pixelPosition + Component.b.rawValue],
a: pointer[pixelPosition + Component.alpha.rawValue])
}
}
How to use:
if let reader = ImagePixelReader(image: yourImage) {
//get alpha or color
let alpha = reader.componentAt(.alpha, x: 10, y:10)
let color = reader.colorAt(x:10, y: 10).uiColor
//getting all the pixels you need
var values = ""
//iterate over all pixels
for x in 0 ..< Int(image.size.width){
for y in 0 ..< Int(image.size.height){
let color = reader.colorAt(x: x, y: y)
values += "[\(x):\(y):\(color)] "
}
//add new line for every new row
values += "\n"
}
print(values)
}

Render off-screen SCNScene into UIImage

How can I render render an off-screen SCNScene into a UIImage?
I know that SCNView provides a -snapshot method, but unfortunately that doesn't work for off-screen views. A similar question have been asked before where one of the answers suggest reading the bitmap data from OpenGL using glReadPixels, but that approach doesn't work for me with an off-screen scene.
I tried rendering into the context of an GLKView using SCNRenderer without success.
Swift 4 with SCNRenderer:
You can use SCNRenderer's snapshot method to render the off-screen SCNScene to a UIImage pretty easily.
Some caveats here, this uses Metal. I don't know where the device/iOS version cutoff is, but you'll need a newer device. You also won't be able to run it on the simulator.
Step 1 - Set up your scene like you normally would:
// Set up your scene which won't be displayed
let hiddenScene = SCNScene()
[insert code to set up your nodes, cameras, and lights here]
Step 2 - Set up the SCNRenderer -- renderer will be nil on simulator:
// Set up the renderer -- this returns nil on simulator
let renderer = SCNRenderer(device: MTLCreateSystemDefaultDevice(), options: nil)
renderer!.scene = hiddenScene
Step 3 - Render scene to UIImage:
// You can use zero for renderTime unless you are using animations,
// in which case, renderTime should be the current scene time.
let renderTime = TimeInterval(0)
// Output size
let size = CGSize(width:300, height: 150)
// Render the image
let image = renderer!.snapshot(atTime: renderTime, with: size,
antialiasingMode: SCNAntialiasingMode.multisampling4X)
If you are running animations, you'll need to increment renderTime or set it to the time index you want to render. For example, if you want to render the frame 4 seconds into the scene, you would set it to 4. This only affects animations -- it won't go back in time and show you a historical view of your scene.
For example, if you run are running animations with SCNNode.runAction, you may want to keep incrementing renderTime every 60th of a second (0.16667 seconds), so that whenever you decide to render, you've got an updated renderTime:
var timer : Timer
var renderTime = TimeInterval(0)
timer = Timer.scheduledTimer(withTimeInterval: 0.016667, repeats: true, block: { (t) in
self?.renderTime += 0.016667
}
})
Using CADisplayLink is probably a better solution for the timing though.
Here's a very quick and dirty implementation example.
Here’s a little code I wrote. You can use it note for note.
public extension SCNRenderer {
public func renderToImageSize(size: CGSize, floatComponents: Bool, atTime time: NSTimeInterval) -> CGImage? {
var thumbnailCGImage: CGImage?
let width = GLsizei(size.width), height = GLsizei(size.height)
let samplesPerPixel = 4
#if os(iOS)
let oldGLContext = EAGLContext.currentContext()
let glContext = unsafeBitCast(context, EAGLContext.self)
EAGLContext.setCurrentContext(glContext)
objc_sync_enter(glContext)
#elseif os(OSX)
let oldGLContext = CGLGetCurrentContext()
let glContext = unsafeBitCast(context, CGLContextObj.self)
CGLSetCurrentContext(glContext)
CGLLockContext(glContext)
#endif
// set up the OpenGL buffers
var thumbnailFramebuffer: GLuint = 0
glGenFramebuffers(1, &thumbnailFramebuffer)
glBindFramebuffer(GLenum(GL_FRAMEBUFFER), thumbnailFramebuffer); checkGLErrors()
var colorRenderbuffer: GLuint = 0
glGenRenderbuffers(1, &colorRenderbuffer)
glBindRenderbuffer(GLenum(GL_RENDERBUFFER), colorRenderbuffer)
if floatComponents {
glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_RGBA16F), width, height)
} else {
glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_RGBA8), width, height)
}
glFramebufferRenderbuffer(GLenum(GL_FRAMEBUFFER), GLenum(GL_COLOR_ATTACHMENT0), GLenum(GL_RENDERBUFFER), colorRenderbuffer); checkGLErrors()
var depthRenderbuffer: GLuint = 0
glGenRenderbuffers(1, &depthRenderbuffer)
glBindRenderbuffer(GLenum(GL_RENDERBUFFER), depthRenderbuffer)
glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_DEPTH_COMPONENT24), width, height)
glFramebufferRenderbuffer(GLenum(GL_FRAMEBUFFER), GLenum(GL_DEPTH_ATTACHMENT), GLenum(GL_RENDERBUFFER), depthRenderbuffer); checkGLErrors()
let framebufferStatus = Int32(glCheckFramebufferStatus(GLenum(GL_FRAMEBUFFER)))
assert(framebufferStatus == GL_FRAMEBUFFER_COMPLETE)
if framebufferStatus != GL_FRAMEBUFFER_COMPLETE {
return nil
}
// clear buffer
glViewport(0, 0, width, height)
glClear(GLbitfield(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)); checkGLErrors()
// render
renderAtTime(time); checkGLErrors()
// create the image
if floatComponents { // float components (16-bits of actual precision)
// slurp bytes out of OpenGL
typealias ComponentType = Float
var imageRawBuffer = [ComponentType](count: Int(width * height) * samplesPerPixel * sizeof(ComponentType), repeatedValue: 0)
glReadPixels(GLint(0), GLint(0), width, height, GLenum(GL_RGBA), GLenum(GL_FLOAT), &imageRawBuffer)
// flip image vertically — OpenGL has a different 'up' than CoreGraphics
let rowLength = Int(width) * samplesPerPixel
for rowIndex in 0..<(Int(height) / 2) {
let baseIndex = rowIndex * rowLength
let destinationIndex = (Int(height) - 1 - rowIndex) * rowLength
swap(&imageRawBuffer[baseIndex..<(baseIndex + rowLength)], &imageRawBuffer[destinationIndex..<(destinationIndex + rowLength)])
}
// make the CGImage
var imageBuffer = vImage_Buffer(
data: UnsafeMutablePointer<Float>(imageRawBuffer),
height: vImagePixelCount(height),
width: vImagePixelCount(width),
rowBytes: Int(width) * sizeof(ComponentType) * samplesPerPixel)
var format = vImage_CGImageFormat(
bitsPerComponent: UInt32(sizeof(ComponentType) * 8),
bitsPerPixel: UInt32(sizeof(ComponentType) * samplesPerPixel * 8),
colorSpace: nil, // defaults to sRGB
bitmapInfo: CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Little.rawValue | CGBitmapInfo.FloatComponents.rawValue),
version: UInt32(0),
decode: nil,
renderingIntent: kCGRenderingIntentDefault)
var error: vImage_Error = 0
thumbnailCGImage = vImageCreateCGImageFromBuffer(&imageBuffer, &format, nil, nil, vImage_Flags(kvImagePrintDiagnosticsToConsole), &error)!.takeRetainedValue()
} else { // byte components
// slurp bytes out of OpenGL
typealias ComponentType = UInt8
var imageRawBuffer = [ComponentType](count: Int(width * height) * samplesPerPixel * sizeof(ComponentType), repeatedValue: 0)
glReadPixels(GLint(0), GLint(0), width, height, GLenum(GL_RGBA), GLenum(GL_UNSIGNED_BYTE), &imageRawBuffer)
// flip image vertically — OpenGL has a different 'up' than CoreGraphics
let rowLength = Int(width) * samplesPerPixel
for rowIndex in 0..<(Int(height) / 2) {
let baseIndex = rowIndex * rowLength
let destinationIndex = (Int(height) - 1 - rowIndex) * rowLength
swap(&imageRawBuffer[baseIndex..<(baseIndex + rowLength)], &imageRawBuffer[destinationIndex..<(destinationIndex + rowLength)])
}
// make the CGImage
var imageBuffer = vImage_Buffer(
data: UnsafeMutablePointer<Float>(imageRawBuffer),
height: vImagePixelCount(height),
width: vImagePixelCount(width),
rowBytes: Int(width) * sizeof(ComponentType) * samplesPerPixel)
var format = vImage_CGImageFormat(
bitsPerComponent: UInt32(sizeof(ComponentType) * 8),
bitsPerPixel: UInt32(sizeof(ComponentType) * samplesPerPixel * 8),
colorSpace: nil, // defaults to sRGB
bitmapInfo: CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Big.rawValue),
version: UInt32(0),
decode: nil,
renderingIntent: kCGRenderingIntentDefault)
var error: vImage_Error = 0
thumbnailCGImage = vImageCreateCGImageFromBuffer(&imageBuffer, &format, nil, nil, vImage_Flags(kvImagePrintDiagnosticsToConsole), &error)!.takeRetainedValue()
}
#if os(iOS)
objc_sync_exit(glContext)
if oldGLContext != nil {
EAGLContext.setCurrentContext(oldGLContext)
}
#elseif os(OSX)
CGLUnlockContext(glContext)
if oldGLContext != nil {
CGLSetCurrentContext(oldGLContext)
}
#endif
return thumbnailCGImage
}
}
func checkGLErrors() {
var glError: GLenum
var hadError = false
do {
glError = glGetError()
if glError != 0 {
println(String(format: "OpenGL error %#x", glError))
hadError = true
}
} while glError != 0
assert(!hadError)
}

Resources