I know there have been lots of posts for how to get the colour of a pixel in a UIImage given a CGPoint but they are all outdated as far as I can tell. Most of them contain CGImageGetDataProvider and CGDataProviderCopyData which in Swift 4 is an error:
'CGImageGetDataProvider' has been replaced by property 'CGImage.dataProvider'
'CGDataProviderCopyData' has been replaced by property 'CGDataProvider.data'
Xcode suggests these substitutes, but they do not exist so I have been having trouble trying to recreate a Swift 4 function to get the colour of a pixel in a UIImage.
Here is the typical Swift 3 function:
extension UIImage {
subscript (x: Int, y: Int) -> UIColor? {
if x < 0 || x > Int(size.width) || y < 0 || y > Int(size.height) {
return nil
}
let provider = CGImageGetDataProvider(self.cgImage!)
let providerData = CGDataProviderCopyData(provider!)
let data = CFDataGetBytePtr(providerData)
let numberOfComponents = 4
let pixelData = ((Int(size.width) * y) + x) * numberOfComponents
let r = CGFloat(data![pixelData]) / 255.0
let g = CGFloat(data![pixelData + 1]) / 255.0
let b = CGFloat(data![pixelData + 2]) / 255.0
let a = CGFloat(data![pixelData + 3]) / 255.0
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
Any suggestions or comments are greatly appreciated. Thank you!
EDIT
I tried #Mukesh 's solution but even though all the build errors are fixed, the program crashes. It says:
The image I give to the function is not nil, I have checked.
P.S. The image I am using for the function is a snapshot of the camera. I have a live camera and after every frame, this function (below) is called where I turn the current frame into a UIImage. This UIImage is what I want to find the pixel colour of:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let ciImg = CIImage(cvPixelBuffer: pixelBuffer)
let cameraImage = UIImage(ciImage: ciImg)
let col = cameraImage.getPixelColor(pos: CGPoint(x: 100, y: 100))
}
I used CGPoint(x: 100, y: 100) to see if it crashed as an example, and it did. This point is also in the image because if it wasn't, it would have returned nil here:
if x < 0 || x > Int(size.width) || y < 0 || y > Int(size.height) {
return nil
}
Is there a way to find out why it gets a nil value? Or maybe a different solution? Thank you :)
Here is the code I get after removing the errors:
extension UIImage {
subscript (x: Int, y: Int) -> UIColor? {
if x < 0 || x > Int(size.width) || y < 0 || y > Int(size.height) {
return nil
}
let provider = self.cgImage!.dataProvider
let providerData = provider!.data
let data = CFDataGetBytePtr(providerData)
let numberOfComponents = 4
let pixelData = ((Int(size.width) * y) + x) * numberOfComponents
let r = CGFloat(data![pixelData]) / 255.0
let g = CGFloat(data![pixelData + 1]) / 255.0
let b = CGFloat(data![pixelData + 2]) / 255.0
let a = CGFloat(data![pixelData + 3]) / 255.0
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
Related
I need to print an image from my iOS app to bluetooth thermal printer which is not AirPrint enabled. As we can not use UIKit print for non AirPrint printer, I chose to use 3rd party SDK.
Used this SDK and tried to print image where i can print small sized images, but when it comes to large one, it got crash on appending bytes saying index out of range, this is the function
`private func eachLinePixToCmd(src: [UInt8], nWidth: Int, nHeight: Int, nMode: Int) -> [UInt8] {
var data = [UInt8]
let p0 = [0, 0x80]
let p1 = [0, 0x40]
let p2 = [0, 0x20]
let p3 = [0, 0x10]
let p4 = [0, 0x08]
let p5 = [0, 0x04]
let p6 = [0, 0x02]
let nBytesPerLine: Int = (nWidth + 7) / 8
var k: Int = 0
for _ in 0..<nHeight {
data.append(ESC_POSCommand.beginPrintImage(xl: UInt8(nBytesPerLine % 0xff), xH: UInt8(nBytesPerLine / 0xff), yl: UInt8(1), yH: UInt8(0)).rawValue)
var bytes = [UInt8]()
for _ in 0..<nBytesPerLine {
bytes.append(UInt8(p0[Int(src[k])] + p1[Int(src[k + 1])] + p2[Int(src[k + 2])] + p3[Int(src[k + 3])] + p4[Int(src[k + 4])] + p5[Int(src[k + 5])] + p6[Int(src[k + 6])] + Int(src[k + 7])))
k = k + 8
}
data.append(bytes)
}
let rdata: [UInt8] = data.flatMap { $0 }
return rdata
}
}`
Please let me know if any other SDK available or where to make changes on appending bytes?
Hope for the reply!!
I faced the same issue with the print images, this is due to thermal printer limitations. Here I have compressed the image before sending it to print image.
func resizeWithWidth(width: CGFloat) - > UIImage ? {
let imageView = UIImageView(frame: CGRect(origin: .zero, size: CGSize(width: width, height: CGFloat(ceil(width / size.width * size.height)))))
imageView.contentMode = .scaleAspectFit
imageView.image = self
UIGraphicsBeginImageContextWithOptions(imageView.bounds.size, false, scale)
guard
let context = UIGraphicsGetCurrentContext()
else {
return nil
}
imageView.layer.render( in: context)
guard
let result = UIGraphicsGetImageFromCurrentImageContext()
else {
return nil
}
UIGraphicsEndImageContext()
return result
}
//Set the width to 256 for example
let myImage = image.resizeWithWidth(width: 256) !
let compressData = myImage.jpegData(compressionQuality: 0.6)
//max value is 1.0 and minimum is 0.0
let compressedImage = UIImage(data: compressData!)
var ticImage = Ticket(.image(image, attributes: .alignment(.center)))
if bluetoothPrinterManager.canPrint {
bluetoothPrinterManager.print(ticImage)
}
I want to create sample app for DJI Osmo Mobile2 but when I tried to fetch camera from DJIHandheld it was always nil. How can I use native camera? I tried to map CMSampleBuffer of AVCaptureVideoDataOutputSampleBufferDelegate to UnsafeMutablePointer<UInt8> in captureOutput delegate method but the preview was always black.
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
let lumaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
let chromaBaseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
let lumaBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0)
let chromaBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1)
let lumaBuffer = lumaBaseAddress?.assumingMemoryBound(to: UInt8.self)
let chromaBuffer = chromaBaseAddress?.assumingMemoryBound(to: UInt8.self)
var rgbaImage = [UInt8](repeating: 0, count: 4*width*height)
for x in 0 ..< width {
for y in 0 ..< height {
let lumaIndex = x+y*lumaBytesPerRow
let chromaIndex = (y/2)*chromaBytesPerRow+(x/2)*2
let yp = lumaBuffer?[lumaIndex]
let cb = chromaBuffer?[chromaIndex]
let cr = chromaBuffer?[chromaIndex+1]
let ri = Double(yp!) + 1.402 * (Double(cr!) - 128)
let gi = Double(yp!) - 0.34414 * (Double(cb!) - 128) - 0.71414 * (Double(cr!) - 128)
let bi = Double(yp!) + 1.772 * (Double(cb!) - 128)
let r = UInt8(min(max(ri,0), 255))
let g = UInt8(min(max(gi,0), 255))
let b = UInt8(min(max(bi,0), 255))
rgbaImage[(x + y * width) * 4] = b
rgbaImage[(x + y * width) * 4 + 1] = g
rgbaImage[(x + y * width) * 4 + 2] = r
rgbaImage[(x + y * width) * 4 + 3] = 255
}
}
let data = NSData(bytes: &rgbaImage, length: rgbaImage.count)
let videoBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: data.length)
data.getBytes(videoBuffer, length: data.length)
VideoPreviewer.instance().push(videoBuffer, length: Int32(data.length))
}
I don't know if this is a correct way.
PS: VideoPreviewer is based on ffmpeg.
The Osmo Mobile 2 does not come with it's own camera, so the SDK is not going to return an instance of a camera - this is different than the other versions of Osmos that have a camera. You will need to build your code to interact directly with your iOS device and not through the Osmo Mobile 2.
I have the following code which displays a camera preview, retrieves a single pixel's colour from the UIImage and converts this value to a 'filtered' colour.
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
connection.videoOrientation = orientation
let videoOutput = AVCaptureVideoDataOutput()
videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue.main)
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(cvImageBuffer: pixelBuffer!)
let typeOfColourBlindness = ColourBlindType(rawValue: "deuteranomaly")
/* Gets colour from a single pixel - currently 0,0 and converts it into the 'colour blind' version */
let captureImage = convert(cmage: cameraImage)
let colour = captureImage.getPixelColour(pos: CGPoint(x: 0, y: 0))
var redval: CGFloat = 0
var greenval: CGFloat = 0
var blueval: CGFloat = 0
var alphaval: CGFloat = 0
_ = colour.getRed(&redval, green: &greenval, blue: &blueval, alpha: &alphaval)
print("Colours are r: \(redval) g: \(greenval) b: \(blueval) a: \(alphaval)")
let filteredColour = CBColourBlindTypes.getModifiedColour(.deuteranomaly, red: Float(redval), green: Float(greenval), blue: Float(blueval))
print(filteredColour)
/* #################################################################################### */
DispatchQueue.main.async {
// placeholder for now
self.filteredImage.image = self.applyFilter(cameraImage: cameraImage, colourBlindness: typeOfColourBlindness!)
}
}
Here is where the x: 0, y: 0 pixel value is converted:
import Foundation
enum ColourBlindType: String {
case deuteranomaly = "deuteranomaly"
case protanopia = "protanopia"
case deuteranopia = "deuteranopia"
case protanomaly = "protanomaly"
}
class CBColourBlindTypes: NSObject {
class func getModifiedColour(_ type: ColourBlindType, red: Float, green: Float, blue: Float) -> Array<Float> {
switch type {
case .deuteranomaly:
return [(red*0.80)+(green*0.20)+(blue*0),
(red*0.25833)+(green*0.74167)+(blue*0),
(red*0)+(green*0.14167)+(blue*0.85833)]
case .protanopia:
return [(red*0.56667)+(green*0.43333)+(blue*0),
(red*0.55833)+(green*0.44167)+(blue*0),
(red*0)+(green*0.24167)+(blue*0.75833)]
case .deuteranopia:
return [(red*0.625)+(green*0.375)+(blue*0),
(red*0.7)+(green*0.3)+(blue*0),
(red*0)+(green*0.3)+(blue*0.7)]
case .protanomaly:
return [(red*0.81667)+(green*0.18333)+(blue*0.0),
(red*0.33333)+(green*0.66667)+(blue*0.0),
(red*0.0)+(green*0.125)+(blue*0.875)]
}
}
}
The placeholder for now comment refers to the following function:
func applyFilter(cameraImage: CIImage, colourBlindness: ColourBlindType) -> UIImage {
//do stuff with pixels to render new image
/* Placeholder code for shifting the hue */
// Create a place to render the filtered image
let context = CIContext(options: nil)
// Create filter angle
let filterAngle = 207 * Double.pi / 180
// Create a random color to pass to a filter
let randomColor = [kCIInputAngleKey: filterAngle]
// Apply a filter to the image
let filteredImage = cameraImage.applyingFilter("CIHueAdjust", parameters: randomColor)
// Render the filtered image
let renderedImage = context.createCGImage(filteredImage, from: filteredImage.extent)
// Return a UIImage
return UIImage(cgImage: renderedImage!)
}
And here is my extension for retrieving a pixel colour:
extension UIImage {
func getPixelColour(pos: CGPoint) -> UIColor {
let pixelData = self.cgImage!.dataProvider!.data
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(self.size.width) * Int(pos.y)) + Int(pos.x)) * 4
let r = CGFloat(data[pixelInfo]) / CGFloat(255.0)
let g = CGFloat(data[pixelInfo+1]) / CGFloat(255.0)
let b = CGFloat(data[pixelInfo+2]) / CGFloat(255.0)
let a = CGFloat(data[pixelInfo+3]) / CGFloat(255.0)
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
How can I create a filter for the following colour range for example?
I want to take in the camera input, replace the colours to be of the Deuteranopia range and display this on the screen, in real time, using Swift.
I am using a UIImageView for the image display.
To learn how to perform filtering of video capture and real-time display of the filtered image, you may want to study the AVCamPhotoFilter sample code from Apple, and other sources such as this objc.io tutorial
In short, using a UIImage for real-time rendering is not a good idea - it's too slow. Use a OpenGL (e.g. GLKView) of Metal (e.g. MTKView). The AVCamPhotoFilter code uses MTKView and renders to intermediate buffers, but you can also render a CIImage directly using the appropriate CIContext methods, e.g. for metal https://developer.apple.com/documentation/coreimage/cicontext/1437835-render
In addition, regarding your color filter - you may want to look at the CIColorCube core image filter as shown here.
let filterName = "CIColorCrossPolynomial"
//deuteronomaly
let param = ["inputRedCoefficients" : CIVector(values: [0.8, 0.2, 0, 0, 0, 0, 0, 0, 0, 0], count: 10),
"inputGreenCoefficients" : CIVector(values: [0.25833, 0.74167, 0, 0, 0, 0, 0, 0, 0, 0], count: 10),
"inputBlueCoefficients" : CIVector(values: [0, 0.14167, 0.85833, 0, 0, 0, 0, 0, 0, 0], count: 10)]
let filter = CIFilter(name: filterName, parameters: param)
let startImage = CIImage(image: image!)
filter?.setValue(startImage, forKey: kCIInputImageKey)
let newImage = UIImage(ciImage: ((filter?.outputImage)!))
filter result:
filter result 2:
I have an odd problem. I am rendering an image from the video camera, then later on I am trying to read the color of a pixel from the image.
Most of the time I don't have a problem. But there are times when my project tries to read the pixel data, I get a segmentation fault error and everything crashes. I think it is happening in the moments when that system is updating the UIImage/Buffer and it is between frames.
Since, it is not necessary for me to have the exact color every time I call getPixelColor(), I figured I could put in a guard statement and just return UIColor.Black. But since a lot of the extension function is using Structs/Primary/Unsafe types, and I get a error from the compiler saying I cannot unwrap those in Swift.
Is there a way to graceful way to catch this segmentation fault, or do I have to do something else?
This is the function that captures the image.
func renderImageFromColorCameraSampleBuffer( _ sampleBuffer: CMSampleBuffer ) {
if let cvPixels = CMSampleBufferGetImageBuffer( sampleBuffer ) {
let coreImage = CIImage( cvPixelBuffer: cvPixels )
let context = CIContext()
let rect = CGRect(
x: 0,
y: 0,
width: CGFloat( CVPixelBufferGetWidth(cvPixels) ),
height: CGFloat( CVPixelBufferGetHeight(cvPixels) )
)
let cgImage = context.createCGImage( coreImage, from: rect )
// What to use for the system
self.currentColorImage = UIImage( cgImage: cgImage! )
// Let the View Controller Know there is a new image to display.
SRLNotifications.postDisplayCamera(image: self.currentColorImage!)
}
}
This is what I am using to read the pixel data. [From a post on StackOver
https://stackoverflow.com/a/25956283/2011541 ]
extension UIImage {
func getPixelColor(pos: CGPoint) -> UIColor {
let pixelData = self.cgImage!.dataProvider!.data
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(self.size.width) * Int(pos.y)) + Int(pos.x) ) * 4
let r = CGFloat( data[pixelInfo] ) / CGFloat(255.0)
let g = CGFloat( data[pixelInfo+1] ) / CGFloat(255.0)
let b = CGFloat( data[pixelInfo+2] ) / CGFloat(255.0)
let a = CGFloat( data[pixelInfo+3] ) / CGFloat(255.0)
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
I have 2 pictures which I want to compare, if pixel color is the same to save it.
I detect the color of the pixel by this UIImage extension function:
func getPixelColor(pos: CGPoint) -> ??? {
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(self.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(self.size.width) * Int(pos.y)) + Int(pos.x)) * 4
let r = CGFloat(data[pixelInfo]) / CGFloat(255.0)
let g = CGFloat(data[pixelInfo+1]) / CGFloat(255.0)
let b = CGFloat(data[pixelInfo+2]) / CGFloat(255.0)
let a = CGFloat(data[pixelInfo+3]) / CGFloat(255.0)
return ???
}
For example, I run the scanner on picture 1 and save it in an array? Or dictionary? And after that I run the scanner on picture 2 and when I have the information from 2 pictures to compare it with what function?
I want to see on which CGPoint the pixels colors are identical from 2 images?
UPDATE:
I update getPixelColor to return me "(pos)(r)(g)(b)(a)" and after that I created this function which left only duplicates (BEFORE USING THIS FUNCTION YOU HAVE TO .sort() THE ARRAY!)
extension Array where Element : Equatable {
var duplicates: [Element] {
var arr:[Element] = []
var start = 0
var start2 = 1
for _ in 0...self.count{
if(start2<self.count){
if(self[start] == self[start2]){
if(arr.contains(self[start])==false){
arr.append(self[start])
}
}
start+=1
start2+=1
}
}
return arr
}
}
This returns me something like this:
"(609.0, 47.0)1.01.01.01.0" I know that the color is black at this point I do x-536 to fit iPhone 5 screen and when I make an attempt to draw it again it draws something wrong... maybe I can't do it properly.. help?
have the UIImage extension return a UIColor. use this method to compare each pixel of the two images. if both pixels match, add the color to an array of arrays.
extension UIImage {
func getPixelColor(pos: CGPoint) -> UIColor {
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(self.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let pixelInfo: Int = ((Int(self.size.width) * Int(pos.y)) + Int(pos.x)) * 4
let r = CGFloat(data[pixelInfo]) / CGFloat(255.0)
let g = CGFloat(data[pixelInfo+1]) / CGFloat(255.0)
let b = CGFloat(data[pixelInfo+2]) / CGFloat(255.0)
let a = CGFloat(data[pixelInfo+3]) / CGFloat(255.0)
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
func findMatchingPixels(aImage: UIImage, _ bImage: UIImage) -> [[UIColor?]] {
guard aImage.size == bImage.size else { fatalError("images must be the same size") }
var matchingColors: [[UIColor?]] = []
for y in 0..<Int(aImage.size.height) {
var currentRow = [UIColor?]()
for x in 0..<Int(aImage.size.width) {
let aColor = aImage.getPixelColor(CGPoint(x: x, y: y))
let colorsMatch = bImage.getPixelColor(CGPoint(x: x, y: y)) == aColor
currentRow.append(colorsMatch ? aColor : nil)
}
matchingColors.append(currentRow)
}
return matchingColors
}
used like this:
let matchingPixels = findMatchingPixels(UIImage(named: "imageA.png")!, UIImage(named: "imageB.png")!)
if let colorForOrigin = matchingPixels[0][0] {
print("the images have the same color, it is: \(colorForOrigin)")
} else {
print("the images do not have the same color at (0,0)")
}
for simplicity i made findMatchingPixels() require the images be the same size, but it wouldn't take much to allow different sized images.
UPDATE
if you want ONLY the pixels that match, i'd return a tuple like this:
func findMatchingPixels(aImage: UIImage, _ bImage: UIImage) -> [(CGPoint, UIColor)] {
guard aImage.size == bImage.size else { fatalError("images must be the same size") }
var matchingColors = [(CGPoint, UIColor)]()
for y in 0..<Int(aImage.size.height) {
for x in 0..<Int(aImage.size.width) {
let aColor = aImage.getPixelColor(CGPoint(x: x, y: y))
guard bImage.getPixelColor(CGPoint(x: x, y: y)) == aColor else { continue }
matchingColors.append((CGPoint(x: x, y: y), aColor))
}
}
return matchingColors
}
Why not try a different approach?
The Core Image filter CIDifferenceBlendMode will return an all black image if passed two identical images and an image with areas of non black where two images differ. Pass that into a CIAreaMaximum which will return a 1x1 image containing the maximum pixel: if the maximum value is 0, you know you have two identical images, if the maximum is greater than zero, the two images are different.
Given two CIImage instances, imageA and imageB, here's the code:
let ciContext = CIContext()
let difference = imageA
.imageByApplyingFilter("CIDifferenceBlendMode",
withInputParameters: [
kCIInputBackgroundImageKey: imageB])
.imageByApplyingFilter("CIAreaMaximum",
withInputParameters: [
kCIInputExtentKey: CIVector(CGRect: imageA.extent)])
let totalBytes = 4
let bitmap = calloc(totalBytes, sizeof(UInt8))
ciContext.render(difference,
toBitmap: bitmap,
rowBytes: totalBytes,
bounds: difference.extent,
format: kCIFormatRGBA8,
colorSpace: nil)
let rgba = UnsafeBufferPointer<UInt8>(
start: UnsafePointer<UInt8>(bitmap),
count: totalBytes)
let red = rgba[0]
let green = rgba[1]
let blue = rgba[2]
If red, green or blue are not zero, you know the images are different!