Implementing Ordered Dithering (24 bit RGB to 3 bit per channel RGB) - image-processing

I'm writing an image editing programme, and I need functionality to dither any arbitrary 24-bit RGB image (I've taken care of loading it with CoreGraphics and such) to an image with 3 bit colour channels, then displaying it. I've set up my matrices and such, but I've not got any results from the code below besides a simple pattern that is applied to the image:
- (CGImageRef) ditherImageTo16Colours:(CGImageRef)image withDitheringMatrixType:(SQUBayerDitheringMatrix) matrix {
if(image == NULL) {
NSLog(#"Image is NULL!");
return NULL;
}
unsigned int imageWidth = CGImageGetWidth(image);
unsigned int imageHeight = CGImageGetHeight(image);
NSLog(#"Image size: %u x %u", imageWidth, imageHeight);
CGContextRef context = CGBitmapContextCreate(NULL,
imageWidth,
imageHeight,
8,
4 * (imageWidth),
CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB),
kCGImageAlphaNoneSkipLast);
CGContextDrawImage(context, CGRectMake(0, 0, imageWidth, imageHeight), image); // draw it
CGImageRelease(image); // get rid of the image, we don't want it anymore.
unsigned char *imageData = CGBitmapContextGetData(context);
unsigned char ditheringModulusType[0x04] = {0x02, 0x03, 0x04, 0x08};
unsigned char ditheringModulus = ditheringModulusType[matrix];
unsigned int red;
unsigned int green;
unsigned int blue;
uint32_t *memoryBuffer;
memoryBuffer = (uint32_t *) malloc((imageHeight * imageWidth) * 4);
unsigned int thresholds[0x03] = {256/8, 256/8, 256/8};
for(int y = 0; y < imageHeight; y++) {
for(int x = 0; x < imageWidth; x++) {
// fetch the colour components, add the dither value to them
red = (imageData[((y * imageWidth) * 4) + (x << 0x02)]);
green = (imageData[((y * imageWidth) * 4) + (x << 0x02) + 1]);
blue = (imageData[((y * imageWidth) * 4) + (x << 0x02) + 2]);
if(red > 36 && red < 238) {
red += SQUBayer117_matrix[x % ditheringModulus][y % ditheringModulus];
} if(green > 36 && green < 238) {
green += SQUBayer117_matrix[x % ditheringModulus][y % ditheringModulus];
} if(blue > 36 && blue < 238) {
blue += SQUBayer117_matrix[x % ditheringModulus][y % ditheringModulus];
}
// memoryBuffer[(y * imageWidth) + x] = (0xFF0000 + ((x >> 0x1) << 0x08) + (y >> 2));
memoryBuffer[(y * imageWidth) + x] = find_closest_palette_colour(((red & 0xFF) << 0x10) | ((green & 0xFF) << 0x08) | (blue & 0xFF));
}
}
//CGContextRelease(context);
context = CGBitmapContextCreate(memoryBuffer,
imageWidth,
imageHeight,
8,
4 * (imageWidth),
CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB),
kCGImageAlphaNoneSkipLast);
NSLog(#"Created context from buffer: %#", context);
CGImageRef result = CGBitmapContextCreateImage(context);
return result;
}
Note that find_closest_palette_colour doesn't do anything besides returning the original colour right now for testing.
I'm trying to implement the example pseudocode from Wikipedia, and I don't really get anything out of that right now.
Anyone got a clue on how to fix this up?

Use the code that I have provided here: https://stackoverflow.com/a/17900812/342646
This code converts the image to a single-channel gray-scale first. If you want the dithering to be done on a three-channel image, you can just split your image into three channels and call the function three times (once per channel).

Related

How do I blur a YUV videoframe with Agora SDK

I'm using the following method from the Advanced Video Example on Github to capture the raw video data:
- (AgoraVideoRawData *)mediaDataPlugin:(AgoraMediaDataPlugin *)mediaDataPlugin didCapturedVideoRawData:(AgoraVideoRawData *)videoRawData
I have already been able to convert the Y U V buffers to a CVPixelBuffer > CIImage and apply the blur, but i'm having trouble translating the CIImage data back into YUV buffers.
I already succeeded into setting random values to the yuv-buffers which results in a grey video frame being sent to the other user.
memset(videoRawData.yBuffer, 128, videoRawData.yStride * videoRawData.height);
memset(videoRawData.uBuffer, 128, videoRawData.uStride * videoRawData.height / 2);
memset(videoRawData.vBuffer, 128, videoRawData.vStride * videoRawData.height / 2);
Could someone point me in the right direction on how to translate CIImage data back into YUV buffers? Or if there is a more efficient way to blur a YUV videodata stream, i'm willing to try that.
I have found a solutation that works for me. I will try to post a complete answer so others might find a solution that works for them. See comments in code for more explanation.
Set these helpers somewhere in your file. This will be used later to calculate the RGB values of each color pixel:
#define Mask8(x) ( (x) & 0xFF )
#define R(x) ( Mask8(x) )
#define G(x) ( Mask8(x >> 8 ) )
#define B(x) ( Mask8(x >> 16) )
All code posted here is inside the - (AgoraVideoRawData *)mediaDataPlugin:(AgoraMediaDataPlugin *)mediaDataPlugin didCapturedVideoRawData:(AgoraVideoRawData *)videoRawData method for simplicity sake of answerring this question.
- (AgoraVideoRawData *)mediaDataPlugin:(AgoraMediaDataPlugin *)mediaDataPlugin didCapturedVideoRawData:(AgoraVideoRawData *)videoRawData
{
// create pixelbuffer from raw video data
NSDictionary *pixelAttributes = #{(NSString *)kCVPixelBufferIOSurfacePropertiesKey:#{}};
CVPixelBufferRef pixelBuffer = NULL;
CVReturn result = CVPixelBufferCreate(kCFAllocatorDefault,
videoRawData.width,
videoRawData.height,
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, // NV12
(__bridge CFDictionaryRef)(pixelAttributes),
&pixelBuffer);
if (result != kCVReturnSuccess) {
NSLog(#"Unable to create cvpixelbuffer %d", result);
}
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
unsigned char *yDestPlane = (unsigned char *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
for (int i = 0, k = 0; i < videoRawData.height; i ++) {
for (int j = 0; j < videoRawData.width; j ++) {
yDestPlane[k++] = videoRawData.yBuffer[j + i * videoRawData.yStride];
}
}
unsigned char *uvDestPlane = (unsigned char *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1);
for (int i = 0, k = 0; i < videoRawData.height / 2; i ++) {
for (int j = 0; j < videoRawData.width / 2; j ++) {
uvDestPlane[k++] = videoRawData.uBuffer[j + i * videoRawData.uStride];
uvDestPlane[k++] = videoRawData.vBuffer[j + i * videoRawData.vStride];
}
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
// create CIImage from pixel buffer
CIImage *coreImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
// apply pixel filter to image
CIFilter *pixelFilter = [CIFilter filterWithName:#"CIPixellate"];
[pixelFilter setDefaults];
[pixelFilter setValue:coreImage forKey:kCIInputImageKey];
[pixelFilter setValue:#40 forKey:#"inputScale"];
CIVector *vector = [[CIVector alloc] initWithX:160 Y:160]; // x & y should be multiple of 'inputScale' parameter
[pixelFilter setValue:vector forKey:#"inputCenter"];
CIImage *outputBlurredImage = [pixelFilter outputImage];
CIContext *blurImageContext = [CIContext contextWithOptions:nil];
CGImageRef inputCGImage = [blurImageContext createCGImage:outputBlurredImage fromRect:[coreImage extent]];
// write blurred image data to YUV buffers
NSUInteger blurredWidth = CGImageGetWidth(inputCGImage);
NSUInteger blurredHeight = CGImageGetHeight(inputCGImage);
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * blurredWidth;
NSUInteger bitsPerComponent = 8;
UInt32 * pixels = (UInt32 *) calloc(blurredHeight * blurredWidth, sizeof(UInt32));
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pixels, blurredWidth, blurredHeight, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGContextDrawImage(context, CGRectMake(0, 0, blurredWidth, blurredHeight), inputCGImage);
int frameSize = videoRawData.width * videoRawData.height;
int yIndex = 0; // Y start index
int uIndex = frameSize; // U statt index
int vIndex = frameSize * 5 / 4; // V start index: w*h*5/4
// allocate buffers to store YUV data
UInt32 *currentPixel = pixels;
char *yBuffer = malloc( sizeof(char) * ( frameSize + 1 ) );
char *uBuffer = malloc( sizeof(char) * ( uIndex + frameSize + 1 ) );
char *vBuffer = malloc( sizeof(char) * ( vIndex + frameSize + 1 ) );
// loop through each RGB pixel and translate to YUV
for (int j = 0; j < blurredHeight; j++) {
for (int i = 0; i < blurredWidth; i++) {
UInt32 color = *currentPixel;
UInt32 R = R(color);
UInt32 G = G(color);
UInt32 B = B(color);
UInt32 Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
UInt32 U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
UInt32 V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
yBuffer[yIndex++] = Y;
if (j % 2 == 0 && i % 2 == 0) {
uBuffer[uIndex++] = U;
vBuffer[vIndex++] = V;
}
currentPixel++;
}
}
// copy new YUV values to given videoRawData object buffers
memcpy((void*)videoRawData.yBuffer, yBuffer, strlen(yBuffer));
memcpy((void*)videoRawData.uBuffer, uBuffer, strlen(uBuffer));
memcpy((void*)videoRawData.vBuffer, vBuffer, strlen(vBuffer));
// cleanup
CVPixelBufferRelease(pixelBuffer);
CGImageRelease(inputCGImage);
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
free(pixels);
free(yBuffer);
free(uBuffer);
free(vBuffer);
return videoRawData;
}

Best performant way to check if an image is all white?

I'm trying to determine if a drawing currently is all white. The solution I could come up with was to scale down the image, then check pixel by pixel if it's white and return NO as soon as it finds a pixel that is not white.
It works, but I have a gut feeling it could be done in a more performant way. Here's the code:
- (BOOL)imageIsAllWhite:(UIImage *)image {
CGSize size = CGSizeMake(100.0f, 100.0f);
UIImageView *imageView = [[UIImageView alloc] initWithImage:[image scaledImageWithSize:size]];
unsigned char pixel[4 * (int)size.width * (int)size.height];
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef cgContext = CGBitmapContextCreate(
pixel,
(size_t)size.width,
(size_t)size.height,
8,
(size_t)(size.width * 4),
colorSpace,
kCGBitmapAlphaInfoMask & kCGImageAlphaPremultipliedLast);
CGContextTranslateCTM(cgContext, 0, 0);
[imageView.layer renderInContext:cgContext];
CGContextRelease(cgContext);
CGColorSpaceRelease(colorSpace);
for (int i = 0; i < sizeof(pixel); i = i + 4) {
if(!(pixel[i] == 255 && pixel[i+1] == 255 && pixel[i+2] == 255)) {
return NO;
}
}
return YES;
}
Any ideas for improvement?
Please following code for check whether UIImage is White color
- (BOOL) checkIfImage:(UIImage *)someImage {
CGImageRef image = someImage.CGImage;
size_t width = CGImageGetWidth(image);
size_t height = CGImageGetHeight(image);
GLubyte * imageData = malloc(width * height * 4);
int bytesPerPixel = 4;
int bytesPerRow = bytesPerPixel * width;
int bitsPerComponent = 8;
CGContextRef imageContext =
CGBitmapContextCreate(
imageData, width, height, bitsPerComponent, bytesPerRow, CGImageGetColorSpace(image),
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big
);
CGContextSetBlendMode(imageContext, kCGBlendModeCopy);
CGContextDrawImage(imageContext, CGRectMake(0, 0, width, height), image);
CGContextRelease(imageContext);
int byteIndex = 0;
BOOL imageExist = YES;
for ( ; byteIndex < width*height*4; byteIndex += 4) {
CGFloat red = ((GLubyte *)imageData)[byteIndex]/255.0f;
CGFloat green = ((GLubyte *)imageData)[byteIndex + 1]/255.0f;
CGFloat blue = ((GLubyte *)imageData)[byteIndex + 2]/255.0f;
CGFloat alpha = ((GLubyte *)imageData)[byteIndex + 3]/255.0f;
if( red != 1 || green != 1 || blue != 1 || alpha != 1 ){
imageExist = NO;
break;
}
}
return imageExist;
}
Calling Functions
UIImage *image = [UIImage imageNamed:#"demo1.png"];
BOOL isImageFlag=[self checkIfImage:image];
if (isImageFlag == YES) {
NSLog(#"YES it's totally White");
}else{
NSLog(#"Nope it's not White");
}
It feels like there's no speedy route that would go to the GPU and back again so the answer is really no more interesting than taking a statistical approach and using GCD to ensure multicore utilisation.
In most images, colours are more likely to be close to other similar colours. So if one pixel is white, it's more likely that its neighbouring pixel is also white. Therefore a strict linear progression through the pixels is less likely to find a white pixel quickly than is sampling points a distance apart, then sampling closer points, etc. Ideally there'd be some f(x) that took the relevant range of integers as input and returned each of them only once, such that the distance between f(x) and f(x+1) is greatest for x = 0 and then decreases monotonically.
If the image is reasonably large, and more so if you can afford to return the result asynchronously, then the cost of dispatching the task to multiple cores is likely to be outweighed by the gain of having multiple cores work on it at once.
You're fixing your image size at 100x100 pixels. I'm going to take a liberty and assume you can move up to 128x128 because it makes the f(x) easy — in that case you can just do a bit reversal.
E.g.
static inline int convolution(int input) {
// bit reverse a 14-bit number
return ((input & 0x0001) << 13) |
((input & 0x0002) << 11) |
((input & 0x0004) << 9) |
((input & 0x0008) << 7) |
((input & 0x0010) << 5) |
((input & 0x0020) << 3) |
((input & 0x0040) << 1) |
((input & 0x0080) >> 1) |
((input & 0x0100) >> 3) |
((input & 0x0200) >> 5) |
((input & 0x0400) >> 7) |
((input & 0x0800) >> 9) |
((input & 0x1000) >> 11) |
((input & 0x2000) >> 13);
}
... elsewhere ...
__block BOOL hasFoundNonWhite = NO;
const int numberOfPixels = 128 * 128;
const int pixelsPerBatch = 128;
const int numberOfBatches = numberOfPixels / pixelsPerBatch;
dispatch_apply(numberOfBatches,
dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0),
^(size_t index) {
if (hasFoundNonWhite) {
return;
}
index *= pixelsPerBatch;
for (int i = index; i < index + pixelsPerBack; i ++) {
int indexToCheck = convolution(i);
int arrayIndex = indexToCheck << 2;
if (!(pixel[arrayIndex] == 255 && pixel[arrayIndex+1] == 255 && pixel[arrayIndex+2] == 255)) {
hasFoundNonWhite = YES;
return;
}
}
});
return !hasFoundNonWhite;
Addendum: the other knee-jerk thing you'd do when dealing with a vector processing task like this is check the Accelerate framework, likely vDSP. That ends up compiling down to use the vector unit on your CPU. In this case you might reformulate the test as "sum of vector must equal size of vector * 255" (if you can make an assumption about alpha). However there is no integral sum, and converting to float probably isn't worth the cost.

Converting cv::Mat to MTLTexture

An intermediate step of my current project requires conversion of opencv's cv::Mat to MTLTexture, the texture container of Metal. I need to store the Floats in the Mat as Floats in the texture; my project cannot quite afford the loss of precision.
This is my attempt at such a conversion.
- (id<MTLTexture>)texForMat:(cv::Mat)image context:(MBEContext *)context
{
id<MTLTexture> texture;
int width = image.cols;
int height = image.rows;
Float32 *rawData = (Float32 *)calloc(height * width * 4,sizeof(float));
int bytesPerPixel = 4;
int bytesPerRow = bytesPerPixel * width;
float r, g, b,a;
for(int i = 0; i < height; i++)
{
Float32* imageData = (Float32*)(image.data + image.step * i);
for(int j = 0; j < width; j++)
{
r = (Float32)(imageData[4 * j]);
g = (Float32)(imageData[4 * j + 1]);
b = (Float32)(imageData[4 * j + 2]);
a = (Float32)(imageData[4 * j + 3]);
rawData[image.step * (i) + (4 * j)] = r;
rawData[image.step * (i) + (4 * j + 1)] = g;
rawData[image.step * (i) + (4 * j + 2)] = b;
rawData[image.step * (i) + (4 * j + 3)] = a;
}
}
MTLTextureDescriptor *textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA16Float
width:width
height:height
mipmapped:NO];
texture = [context.device newTextureWithDescriptor:textureDescriptor];
MTLRegion region = MTLRegionMake2D(0, 0, width, height);
[texture replaceRegion:region mipmapLevel:0 withBytes:rawData bytesPerRow:bytesPerRow];
free(rawData);
return texture;
}
But it doesn't seem to be working. It reads zeroes every time from the Mat, and throws up EXC_BAD_ACCESS. I need the MTLTexture in MTLPixelFormatRGBA16Float to keep the precision.
Thanks for considering this issue.
One problem here is you’re loading up rawData with Float32s but your texture is RGBA16Float, so the data will be corrupted (16Float is half the size of Float32). This shouldn’t cause your crash, but it’s an issue you’ll have to deal with.
Also as “chappjc” noted you’re using ‘image.step’ when writing your data out, but that buffer should be contiguous and not ever have a step that’s not just (width * bytesPerPixel).

Converting a 24-bit PNG image to an array of GLubytes

I'd like to do the following:
Read RGB color values from a 24 bit PNG image
Average the RGB values and store them into an array of Glubytes.
I have provided my function that I was hoping would perform these 2 steps.
My function returns an array of Glubytes, however all elements have a value of 0.
So im guessing im reading the image data incorrectly.
What am i going wrong in reading the image? (perhaps my format is incorrect).
Here is my function:
+ (GLubyte *) LoadPhotoAveragedIndexPNG:(UIImage *)image numPixelComponents: (int)numComponents
{
// Load an image and return byte array.
CGImageRef textureImage = image.CGImage;
if (textureImage == nil)
{
NSLog(#"LoadPhotoIndexPNG: Failed to load texture image");
return nil;
}
NSInteger texWidth = CGImageGetWidth(textureImage);
NSInteger texHeight = CGImageGetHeight(textureImage);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
GLubyte *indexedData = (GLubyte *)malloc(texWidth * texHeight);
GLubyte *rawData = (GLubyte *)malloc(texWidth * texHeight * numComponents);
CGContextRef textureContext = CGBitmapContextCreate(
rawData,
texWidth,
texHeight,
8,
texWidth * numComponents,
colorSpace,
kCGImageAlphaPremultipliedLast);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(textureContext,
CGRectMake(0.0, 0.0, (float)texWidth, (float)texHeight),
textureImage);
CGContextRelease(textureContext);
int rawDataLength = texWidth * texHeight * numComponents;
for (int i = 0, j = 0; i < rawDataLength; i += numComponents)
{
GLubyte b = rawData[i];
GLubyte g = rawData[i + 1];
GLubyte r = rawData[i + 2];
indexedData[j++] = (r + g + b) / 3;
}
return indexedData;
}
Here is the test image im loading (RGB colorspace in PNG format):
Do check with some logging if the parameters b,g and r are producing normal values in the last for loop. Where you made a mistake is indexedData[j++] = (r + g + b) / 3; those 3 parameters are sizeof 1 byte and you can not sum them up like that. Use a larger integer, typecast them and typecast the result back to array. (You are most likely getting overflow)
Apart from your original problem there's a major problem here (maybe even related)
for (int i = 0, j = 0; i < rawDataLength; i += numComponents)
{
GLubyte b = rawData[i];
GLubyte g = rawData[i + 1];
GLubyte r = rawData[i + 2];
indexedData[j++] = (r + g + b) / 3;
}
Namely the expression
(r + g + b)
This expression will be performed on GLubyte sized integer operations. If the sum of r+g+b is larger than the type GLubyte can hold it will overflow. Whenever you're processing data through intermediary variables (good style!) choose the variable types large enough to hold the largest value you can encounter. Another method was casting the expression like
indexedData[j++] = ((uint16_t)r + (uint16_t)g + (uint16_t)b) / 3;
But that's cumbersome to read. Also if you're processing integers of a known size, use the types found in stdint.h. You know, that you're expecting 8 bits per channel. Also you can use the comma operator in the for increment clause
uint8_t *indexedData = (GLubyte *)malloc(texWidth * texHeight);
/* ... */
for (int i = 0, j = 0; i < rawDataLength; i += numComponents, j++)
{
uint16_t b = rawData[i];
uint16_t g = rawData[i + 1];
uint16_t r = rawData[i + 2];
indexedData[j] = (r + g + b) / 3;
}

1bpp UIImage in iOS

I want to create an image that is black and white so it is 1bpp. Right now this is my code that converts an image to black and white based on a threshold:
- (UIImage *)pureBlackAndWhiteImage:(UIImage *)image value:(float)value {
unsigned char *dataBitmap = [self convertUIImageToBitmapRGBA8:image];
for (int i = 0; i < image.size.width * image.size.height * 4; i += 4) {
if ((dataBitmap[i + 0] + dataBitmap[i + 1] + dataBitmap[i + 2]) < value) {
dataBitmap[i + 0] = 0;
dataBitmap[i + 1] = 0;
dataBitmap[i + 2] = 0;
} else {
dataBitmap[i + 0] = 255;
dataBitmap[i + 1] = 255;
dataBitmap[i + 2] = 255;
}
}
image = [self convertBitmapRGBA8ToUIImage:dataBitmap withWidth:image.size.width withHeight:image.size.height];
return image;
}
However this image is still 8bpp. How can I make it so that it is 1bpp?
I don't know what those methods are that you refer to, but the second does claim to produce an image from an RGBA data set, so why do you expect 1bpp?
Frankly, I don't know if iOS can even support 1bpp. If you cannot determine for sure that the answer is no, you'll probably have to experiment.
What you need to do is first render the existing RGBA image and get access to the pixels (as it appears you have done. You will then need to malloc a chunk of memory and write the image as you want - 1 bit per pixel. When you have that data you can get a CGImageRef using:
CGImageRef CGImageCreate (
size_t width,
size_t height,
size_t bitsPerComponent,
size_t bitsPerPixel,
size_t bytesPerRow,
CGColorSpaceRef colorspace,
CGBitmapInfo bitmapInfo,
CGDataProviderRef provider,
const CGFloat decode[],
bool shouldInterpolate,
CGColorRenderingIntent intent
);
and a UIImage from that.

Resources