I need to read a colour of a pixel located at a point in an image and the code I have works in simulator for all iPhones (including iPhone 6 Plus) except iPhone 6.
I do not know why, but my guess is the index of the pixel is not correct since it detects a colour in a wrong location.
I appreciate any help.
This is the code that I have.
UIGraphicsBeginImageContext(upperCaseView.frame.size)
upperCaseView.layer.renderInContext(UIGraphicsGetCurrentContext()!)
snapshotImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(snapshotImage.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(snapshotImage.size.width) * Int(point.y)) + Int(point.x)) * 4
print(data[pixelInfo])
print(data[pixelInfo+1])
print(data[pixelInfo+2])
Thank you so much Scott Thompson.
I used CGImageGetBytesPerRow instead of the image width and now it works.
The correct code is below:
UIGraphicsBeginImageContext(upperCaseView.frame.size)
upperCaseView.layer.renderInContext(UIGraphicsGetCurrentContext()!)
snapshotImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(snapshotImage.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let bytesPerPixel = (CGImageGetBitsPerPixel(snapshotImage.CGImage) / 8)
let bytesPerRow = CGImageGetBytesPerRow(snapshotImage.CGImage)
let pixelInfo: Int = (bytesPerRow * Int(point.y)) + (Int(point.x) * bytesPerPixel)
print(data[pixelInfo])
print(data[pixelInfo+1])
print(data[pixelInfo+2])
Related
Maybe this is a very stupid question.
I am using AVFoundation in my app and I am able to get the frames(32BGRA Format).
The width of the frame is 1504, Height is 1128 and the bytes-Per-Row value is 6016.
When I create a UInt8 pixel array from this samplebuffer the length (array.count) of this array is 1696512 which happens to be equal to width * height.
What I am not getting is why the array length is width * height. Should it not be width * height * 4.
What am I missing here?
Edit - 1: Code
func BufferToArray(sampleBuffer: CMSampleBuffer) -> ([UInt8], Int, Int, Int) {
var rgbBufferArray = [UInt8]()
//Get pixel Buffer from CMSSampleBUffer
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
//Lock the base Address
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly)
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
//get pixel count
let pixelCount = CVPixelBufferGetWidth(pixelBuffer) * CVPixelBufferGetHeight(pixelBuffer)
//Get base address
let baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer)
//Get bytes per row of the image
let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
//Cast the base address to UInt8. This is like an array now
let frameBuffer = baseAddress?.assumingMemoryBound(to: UInt8.self)
rgbBufferArray = Array(UnsafeMutableBufferPointer(start: frameBuffer, count: pixelCount))
//Unlock and release memory
CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
return (rgbBufferArray, bytesPerRow, width, height)
}
The culprit is the data type (UInt8) in combination with the count:
You are assuming the memory contains UInt8 values (assumingMemoryBound(to: UInt8.self)) of pixelCount count. But as you concluded correctly it should be four times that number.
I'd recommend you import simd and use simd_uchar4 as data type. That's a struct type containing 4 UInt8. Then your array will contain pixelCount values of 4-tuple pixel values. You can access the channels with array[index].x , .y, .z, and .w respectively.
I'm currently making a game with Spritekit and struggling to figure out why my normalized Textures are not being applied properly when rendering on device, while they seem to be fine in the simulator.
Here is the code that adds the normal textures to the tile definitions, among other things:
self.wallTileMap = self.scene?.childNode(withName: "Walls") as? SKTileMapNode
let textureAtlas = SKTextureAtlas(named: "Wall Normal Maps")
if let tileMap = self.wallTileMap {
let startingLocation:CGPoint = tileMap.position
let tileSize = tileMap.tileSize
let halfWidth = CGFloat(tileMap.numberOfColumns) / 2.0 * tileSize.width
let halfHeight = CGFloat(tileMap.numberOfRows) / 2.0 * tileSize.height
let rows = tileMap.numberOfRows
let columns = tileMap.numberOfColumns
for column in 0..<columns {
for row in 0..<rows {
let x = CGFloat(column) * tileSize.width - halfWidth + (tileSize.width / 2)
let y = CGFloat(row) * tileSize.height - halfHeight + (tileSize.height / 2)
if let tileDefinition = tileMap.tileDefinition(atColumn: column, row: row) {
if let name = tileDefinition.name {
let normalTexture = textureAtlas.textureNamed("\(name)_n")
tileDefinition.normalTextures = [normalTexture]
}
if (tileDefinition.userData?["shouldKill"] as? Bool ?? false) {
let newNode = SKShapeNode(rectOf: tileDefinition.size)
newNode.position = CGPoint(x: x, y: y)
newNode.isHidden = true
newNode.physicsBody = SKPhysicsBody(texture: tileDefinition.textures[0], size: tileDefinition.size)
newNode.physicsBody?.isDynamic = false
newNode.physicsBody?.affectedByGravity = false
newNode.physicsBody?.categoryBitMask = CollisionTypes.wall.rawValue
newNode.physicsBody?.collisionBitMask = CollisionTypes.dynamicComponents.rawValue
newNode.physicsBody?.contactTestBitMask = CollisionTypes.dynamicComponents.rawValue
self.addChild(newNode)
newNode.position = CGPoint(x: newNode.position.x + startingLocation.x, y: newNode.position.y + startingLocation.y)
}
}
}
}
}
The result for simulator--which is expected:
The result on device--which is incorrect:
I tried multiple simulators, it worked on them all. I've also tried multiple physical devices and it was broken on all of them.
The only thing that I could find while debugging is that the normal images on device seemed to be off by one pixel in size occasionally. So the normal size is 128 x 128 and occasionally the size on device would be 128 x 127 or 127 x 127. No clue what could cause this, nor if that is the actual problem.
Does anyone have any ideas as to why the normal maps would be rendered properly in the simulator, but not on device?
In WWDC session "Image Editing with Depth" they mentioned few times normalizedDisparity and normalizedDisparityImage:
"The basic idea is that we're going to map our normalized disparity
values into values between 0 and 1"
"So once you know the min and max you can normalize the depth or disparity between 0 and 1."
I tried to first get the disparit image like this:
let disparityImage = depthImage.applyingFilter(
"CIDepthToDisparity", withInputParameters: nil)
Then I tried to get depthDataMap and do normalization but it didn't work. I'm I on the right track? would be appreciate some hint on what to do.
Edit:
This is my test code, sorry for the quality. I get the min and max then I try to loop over the data to normalize it (let normalizedPoint = (point - min) / (max - min))
let depthDataMap = depthData!.depthDataMap
let width = CVPixelBufferGetWidth(depthDataMap) //768 on an iPhone 7+
let height = CVPixelBufferGetHeight(depthDataMap) //576 on an iPhone 7+
CVPixelBufferLockBaseAddress(depthDataMap, CVPixelBufferLockFlags(rawValue: 0))
// Convert the base address to a safe pointer of the appropriate type
let floatBuffer = unsafeBitCast(CVPixelBufferGetBaseAddress(depthDataMap),
to: UnsafeMutablePointer<Float32>.self)
var min = floatBuffer[0]
var max = floatBuffer[0]
for x in 0..<width{
for y in 0..<height{
let distanceAtXYPoint = floatBuffer[Int(x * y)]
if(distanceAtXYPoint < min){
min = distanceAtXYPoint
}
if(distanceAtXYPoint > max){
max = distanceAtXYPoint
}
}
}
What I expected is the the data will reflect the disparity where the user clicked on the image but it didn't match. The code to find the disparity where the user clicked is here:
// Apply the filter with the sampleRect from the user’s tap. Don’t forget to clamp!
let minMaxImage = normalized?.clampingToExtent().applyingFilter(
"CIAreaMinMaxRed", withInputParameters:
[kCIInputExtentKey : CIVector(cgRect:rect2)])
// A four-byte buffer to store a single pixel value
var pixel = [UInt8](repeating: 0, count: 4)
// Render the image to a 1x1 rect. Be sure to use a nil color space.
context.render(minMaxImage!, toBitmap: &pixel, rowBytes: 4,
bounds: CGRect(x:0, y:0, width:1, height:1),
format: kCIFormatRGBA8, colorSpace: nil)
// The max is stored in the green channel. Min is in the red.
let disparity = Float(pixel[1]) / 255.0
There's a new blog post on raywenderlich.com called "Image Depth Maps Tutorial for iOS" contains sample app and details related to working with depth. The sample code shows how to normalize the depth data using a CVPixelBuffer extension:
extension CVPixelBuffer {
func normalize() {
let width = CVPixelBufferGetWidth(self)
let height = CVPixelBufferGetHeight(self)
CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
let floatBuffer = unsafeBitCast(CVPixelBufferGetBaseAddress(self), to: UnsafeMutablePointer<Float>.self)
var minPixel: Float = 1.0
var maxPixel: Float = 0.0
for y in 0 ..< height {
for x in 0 ..< width {
let pixel = floatBuffer[y * width + x]
minPixel = min(pixel, minPixel)
maxPixel = max(pixel, maxPixel)
}
}
let range = maxPixel - minPixel
for y in 0 ..< height {
for x in 0 ..< width {
let pixel = floatBuffer[y * width + x]
floatBuffer[y * width + x] = (pixel - minPixel) / range
}
}
CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
}
}
Something to keep in mind when working with depth data that they are lower resolution than the actual image so you need to scale up (more info in the blog and in the WWDC video)
Will's answer above is very good, but it can be improved as follows. I'm using it with depth data from a photo, it's possible that if the depth data doesn't follow 16-bits, as mentioned above, it won't work. Haven't found such a photo yet. I'm surprised there isn't a filter to handle this in Core Image.
extension CVPixelBuffer {
func normalize() {
CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
let width = CVPixelBufferGetWidthOfPlane(self, 0)
let height = CVPixelBufferGetHeightOfPlane(self, 0)
let count = width * height
let pixelBufferBase = unsafeBitCast(CVPixelBufferGetBaseAddressOfPlane(self, 0), to: UnsafeMutablePointer<Float>.self)
let depthCopyBuffer = UnsafeMutableBufferPointer<Float>(start: pixelBufferBase, count: count)
let maxValue = vDSP.maximum(depthCopyBuffer)
let minValue = vDSP.minimum(depthCopyBuffer)
let range = maxValue - minValue
let negMinValue = -minValue
let subtractVector = vDSP.add(negMinValue, depthCopyBuffer)
let normalizedDisparity = vDSP.divide(subtractVector, range)
pixelBufferBase.initialize(from: normalizedDisparity, count: count)
CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
}
}
Try using the Accelerate Framework vDSP vector functions.. here is a normalize in two functions.
to change the cvPixel buffer to a 0..1 normalized range
myCVPixelBuffer.setUpNormalize()
import Accelerate
extension CVPixelBuffer {
func vectorNormalize( targetVector: UnsafeMutableBufferPointer<Float>) -> [Float] {
// range = max - min
// normalized to 0..1 is (pixel - minPixel) / range
// see Documentation "Using vDSP for Vector-based Arithmetic" in vDSP under system "Accelerate" documentation
// see also the Accelerate documentation section 'Vector extrema calculation'
// Maximium static func maximum<U>(U) -> Float
// Returns the maximum element of a single-precision vector.
//static func minimum<U>(U) -> Float
// Returns the minimum element of a single-precision vector.
let maxValue = vDSP.maximum(targetVector)
let minValue = vDSP.minimum(targetVector)
let range = maxValue - minValue
let negMinValue = -minValue
let subtractVector = vDSP.add(negMinValue, targetVector)
// adding negative value is subtracting
let result = vDSP.divide(subtractVector, range)
return result
}
func setUpNormalize() -> CVPixelBuffer {
// grayscale buffer float32 ie Float
// return normalized CVPixelBuffer
CVPixelBufferLockBaseAddress(self,
CVPixelBufferLockFlags(rawValue: 0))
let width = CVPixelBufferGetWidthOfPlane(self, 0)
let height = CVPixelBufferGetHeightOfPlane(self, 0)
let count = width * height
let bufferBaseAddress = CVPixelBufferGetBaseAddressOfPlane(self, 0)
// UnsafeMutableRawPointer
let pixelBufferBase = unsafeBitCast(bufferBaseAddress, to: UnsafeMutablePointer<Float>.self)
let depthCopy = UnsafeMutablePointer<Float>.allocate(capacity: count)
depthCopy.initialize(from: pixelBufferBase, count: count)
let depthCopyBuffer = UnsafeMutableBufferPointer<Float>(start: depthCopy, count: count)
let normalizedDisparity = vectorNormalize(targetVector: depthCopyBuffer)
pixelBufferBase.initialize(from: normalizedDisparity, count: count)
// copy back the normalized map into the CVPixelBuffer
depthCopy.deallocate()
// depthCopyBuffer.deallocate()
CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
return self
}
}
You can see it in action in a modified version of the Apple sample 'PhotoBrowse' app at
https://github.com/racewalkWill/PhotoBrowseModified
I'm trying to get the per-pixel RGBA values for a CIImage in floating point.
I expect the following to work, using CIContext and rendering as kCIFormatRGBAh, but the output is all zeroes. Otherwise my next step would be converting from half floats to full.
What am I doing wrong? I've also tried this in Objective-C and get the same result.
let image = UIImage(named: "test")!
let sourceImage = CIImage(CGImage: image.CGImage)
let context = CIContext(options: [kCIContextWorkingColorSpace: NSNull()])
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bounds = sourceImage.extent()
let bytesPerPixel: UInt = 8
let format = kCIFormatRGBAh
let rowBytes = Int(bytesPerPixel * UInt(bounds.size.width))
let totalBytes = UInt(rowBytes * Int(bounds.size.height))
var bitmap = calloc(totalBytes, UInt(sizeof(UInt8)))
context.render(sourceImage, toBitmap: bitmap, rowBytes: rowBytes, bounds: bounds, format: format, colorSpace: colorSpace)
let bytes = UnsafeBufferPointer<UInt8>(start: UnsafePointer<UInt8>(bitmap), count: Int(totalBytes))
for (var i = 0; i < Int(totalBytes); i += 2) {
println("half float :: left: \(bytes[i]) / right: \(bytes[i + 1])")
// prints all zeroes!
}
free(bitmap)
Here's a related question about getting the output of CIAreaHistogram, which is why I want floating point values rather than integer, but I can't seem to make kCIFormatRGBAh work on any CIImage regardless of its origin, filter output or otherwise.
There are two constraints on using RGBAh with [CIContext render:toBitmap:rowBytes:bounds:format:colorSpace:] on iOS
the rowBytes must be a multiple of 8 bytes
calling it under simulator is not supported
These constraints come from the behavior of OpenGLES with RGBAh on iOS.
I want to manipulate an image at the pixel level in Swift. This question was answered for objective c: How do I access and manipulate JPEG image pixels?, but I would like to see the equivalent source for Swift.
This is how I am getting a color from an image at a touch location.
I translated this answer: https://stackoverflow.com/a/12579413/359578
(This sample does no error checking for nil)
func createARGBBitmapContext(inImage: CGImage) -> CGContext {
var bitmapByteCount = 0
var bitmapBytesPerRow = 0
//Get image width, height
let pixelsWide = CGImageGetWidth(inImage)
let pixelsHigh = CGImageGetHeight(inImage)
// Declare the number of bytes per row. Each pixel in the bitmap in this
// example is represented by 4 bytes; 8 bits each of red, green, blue, and
// alpha.
bitmapBytesPerRow = Int(pixelsWide) * 4
bitmapByteCount = bitmapBytesPerRow * Int(pixelsHigh)
// Use the generic RGB color space.
let colorSpace = CGColorSpaceCreateDeviceRGB()
// Allocate memory for image data. This is the destination in memory
// where any drawing to the bitmap context will be rendered.
let bitmapData = malloc(bitmapByteCount)
// Create the bitmap context. We want pre-multiplied ARGB, 8-bits
// per component. Regardless of what the source image format is
// (CMYK, Grayscale, and so on) it will be converted over to the format
// specified here by CGBitmapContextCreate.
let context = CGBitmapContextCreate(bitmapData, pixelsWide, pixelsHigh, 8, bitmapBytesPerRow, colorSpace, CGImageAlphaInfo.PremultipliedFirst.rawValue)
// Make sure and release colorspace before returning
return context!
}
func getPixelColorAtLocation(point:CGPoint, inImage:CGImageRef) -> NSColor {
// Create off screen bitmap context to draw the image into. Format ARGB is 4 bytes for each pixel: Alpa, Red, Green, Blue
let context = self.createARGBBitmapContext(inImage)
let pixelsWide = CGImageGetWidth(inImage)
let pixelsHigh = CGImageGetHeight(inImage)
let rect = CGRect(x:0, y:0, width:Int(pixelsWide), height:Int(pixelsHigh))
//Clear the context
CGContextClearRect(context, rect)
// Draw the image to the bitmap context. Once we draw, the memory
// allocated for the context for rendering will then contain the
// raw image data in the specified color space.
CGContextDrawImage(context, rect, inImage)
// Now we can get a pointer to the image data associated with the bitmap
// context.
let data = CGBitmapContextGetData(context)
let dataType = UnsafePointer<UInt8>(data)
let offset = 4*((Int(pixelsWide) * Int(point.y)) + Int(point.x))
let alpha = dataType[offset]
let red = dataType[offset+1]
let green = dataType[offset+2]
let blue = dataType[offset+3]
let color = NSColor(red: CGFloat(red)/255.0, green: CGFloat(green)/255.0, blue: CGFloat(blue)/255.0, alpha: CGFloat(alpha)/255.0)
// Free image data memory for the context
free(data)
return color;
}