I'm trying to blit buffer to PVRTC texture. The reason why I'm doing it, because want to keep texture with private storage.
Here is quote from documentation.
If the texture's pixel format is a compressed format, then sourceSize
must be a multiple of the pixel format's block size or be clamped to
the edge of the texture if the block extends outside the bounds of a
texture. For a compressed format, sourceBytesPerRow is the number of
bytes from the start of one row of blocks to the start of the next row
of blocks.
Something wrong in my code, because texture looks broken after.
MTLBlitOption options = MTLBlitOptionNone;
if (_pixelFormat == MTLPixelFormatPVRTC_RGB_4BPP || _pixelFormat == MTLPixelFormatPVRTC_RGBA_4BPP) {
uint32_t blockWidth = 4;
uint32_t blockHeight = 4;
uint32_t bitsPerPixel = 4;
uint32_t blockSize = blockWidth * blockHeight;
uint32_t widthInBlocks = width / blockWidth;
uint32_t heightInBlocks = height / blockHeight;
options = MTLBlitOptionRowLinearPVRTC;
levelBytesPerRow = widthInBlocks * ((blockSize * bitsPerPixel) / 8);
}
id <MTLBuffer> buffer = [device newBufferWithBytes:[data bytes] length:[data length] options:0];
[blitEncoder copyFromBuffer:buffer
sourceOffset:0
sourceBytesPerRow:levelBytesPerRow
sourceBytesPerImage:[buffer length]
sourceSize:MTLSizeMake(width, height, 1)
toTexture:self.textureMetal
destinationSlice:0
destinationLevel:i
destinationOrigin:MTLOriginMake(0, 0, 0)
options:options];
Related
An image is a matrix of pixels. Each pixel can be represented by 4 components: Red, Green, Blue and Alpha. Each component value can be from 0 to 255. An image could be up to 4K quality, i.e. 3840 × 2160 pixels.
Implement an image repository which is a single file that can store many images (with different size) at the same time. An image within the repository file can be identified by its unique ID (4 bytes long)
The program should be able to:
Open/create a repository file
Add/Update/Delete an image into/from the opened repository file.
Extract an image out of the repository using its ID with complexity level less than O(N) - where N is the number of images within the repository.
Quickly detects if we add the same image into the repository (2 images are the same if all their pixels at the same position are the same)
Here's the code to write image to bmp file.
void writeBMP(unsigned char **Matrix, int Matrix_dimension){
FILE *out;
int ii,jj;
long pos = 1077;
out = fopen("output.bmp","wb");
// Image Signature
unsigned char signature[2] = {'B','M'};
fseek(out,0,0);
fwrite(&signature,2,1,out);
// Image file size
uint32_t filesize = 54 + 4*256 + Matrix_dimension*Matrix_dimension;
fseek(out,2,0);
fwrite(&filesize,4,1,out);
// Reserved
uint32_t reserved = 0;
fseek(out,6,0);
fwrite(&reserved,4,1,out);
// Offset
uint32_t offset = 1078;
fseek(out,10,0);
fwrite(&offset,4,1,out);
// Info header size
uint32_t ihsize = 40;
fseek(out,14,0);
fwrite(&ihsize,4,1,out);
// Image Width in pixels
uint32_t width = (uint32_t) Matrix_dimension;
fseek(out,18,0);
fwrite(&width,4,1,out);
// Image Height in pixels
uint32_t height = (uint32_t) Matrix_dimension;
fseek(out,22,0);
fwrite(&height,4,1,out);
// Number of planes
uint16_t planes = 1;
fseek(out,26,0);
fwrite(&planes,2,1,out);
// Color depth, BPP (bits per pixel)
uint16_t bpp = 8;
fseek(out,28,0);
fwrite(&bpp,2,1,out);
// Compression type
uint32_t compression = 0;
fseek(out,30,0);
fwrite(&compression,4,1,out);
// Image size in bytes
uint32_t imagesize = (uint32_t) Matrix_dimension*Matrix_dimension;
fseek(out,34,0);
fwrite(&imagesize,4,1,out);
// Xppm
uint32_t xppm = 0;
fseek(out,38,0);
fwrite(&xppm,4,1,out);
// Yppm
uint32_t yppm = 0;
fseek(out,42,0);
fwrite(&yppm,4,1,out);
// Number of color used (NCL)
uint32_t colours = 256;
fseek(out,46,0);
fwrite(&colours,4,1,out);
// Number of important color (NIC)
// value = 0 means all colors important
uint32_t impcolours = 0;
fseek(out,50,0);
fwrite(&impcolours,4,1,out);
// Colour table
unsigned char bmpcolourtable[1024];
for(ii=0; ii < 1024; ii++){
bmpcolourtable[ii] = 0;
}
jj=3;
for(ii=0; ii < 255; ii++){
bmpcolourtable[jj+1] = ii+1;
bmpcolourtable[jj+2] = ii+1;
bmpcolourtable[jj+3] = ii+1;
jj=jj+4;
}
fseek(out,54,0);
fwrite(&bmpcolourtable,256,4,out);
for(ii=0;ii<Matrix_dimension;ii++){
for(jj=0;jj<Matrix_dimension;jj++){
pos+= 1;
fseek(out,pos,0);
fwrite(&Matrix[ii][jj],(sizeof(unsigned char)),1,out);
}
}
fflush(out);
fclose(out);
}
I am trying to mipmap a texture contained in an MTLTexture object. This texture was loaded from an OpenCV Mat. I can run correctly run kernels on this texture so I know my import process is correct.
Unfortunately, the generate mipmaps function gives this rather opaque error. I get a similar error even if I change temp to be BGRA.
-[MTLDebugBlitCommandEncoder generateMipmapsForTexture:]:1074:
failed assertion `tex(MTLPixelFormatR8Uint) is not filterable.'
// create an MTL Texture
{
MTLTextureDescriptor * textureDescriptor = [MTLTextureDescriptor
texture2DDescriptorWithPixelFormat:MTLPixelFormatR8Uint
width:cols
height:rows
mipmapped:NO];
textureDescriptor.usage = MTLTextureUsageShaderRead;
_mImgTex = [_mDevice newTextureWithDescriptor:textureDescriptor];
}
{
MTLTextureDescriptor * textureDescriptor = [MTLTextureDescriptor
texture2DDescriptorWithPixelFormat:MTLPixelFormatR8Uint
width:cols
height:rows
mipmapped:YES];
textureDescriptor.mipmapLevelCount = 5;
textureDescriptor.usage = MTLTextureUsageShaderRead | MTLTextureUsageShaderWrite;
_mPyrTex = [_mDevice newTextureWithDescriptor:textureDescriptor];
}
// copy data to GPU
cv::Mat temp;
cv::cvtColor(image, temp, cv::COLOR_BGRA2GRAY);
MTLRegion region = MTLRegionMake2D(0, 0, cols, rows);
const int bytesPerPixel = 1 * 1; // 1 uint * 1 channels
const int bytesPerRow = bytesPerPixel * cols;
[_mImgTex replaceRegion:region mipmapLevel:0 withBytes:temp.data bytesPerRow:bytesPerRow];
// try to mipmap
id<MTLBlitCommandEncoder> blitEncoder = [commandBuffer blitCommandEncoder];
MTLOrigin origin = MTLOriginMake(0, 0, 0);
MTLSize size = MTLSizeMake(cols, rows, 1);
[blitEncoder copyFromTexture:_mImgTex sourceSlice:0 sourceLevel:0 sourceOrigin:origin sourceSize:size toTexture:_mPyrTex destinationSlice:0 destinationLevel:0 destinationOrigin:origin];
[blitEncoder generateMipmapsForTexture:_mPyrTex];
[blitEncoder endEncoding];
The documentation for generateMipmapsForTextures says:
Mipmap generation works only for textures with color-renderable and color-filterable pixel formats.
If you look at the "Pixel Format Capabilities" table here, you can see that R8Uint does not support Filter nor is it colour renderable (Color).
Perhaps R8Unorm (MTLPixelFormatR8Unorm) will work well for your needs. Otherwise you might need to write your own mip generation code with compute (although I'm not sure if there's a use case for mipmaps with non filterable textures).
I'm trying to get a CVPixelBuffer in RGB color space from the Apple's ARKit. In func session(_ session: ARSession, didUpdate frame: ARFrame) method of ARSessionDelegate I get an instance of ARFrame. On page Displaying an AR Experience with Metal I found that this pixel buffer is in YCbCr (YUV) color space.
I need to convert this to RGB color space (I actually need CVPixelBuffer and not UIImage). I've found something about color conversion on iOS but I was not able to get this working in Swift 3.
There's several ways to do this, depending on what you're after. The best way to do this in realtime (to say, render the buffer to a view) is to use a custom shader to convert the YCbCr CVPixelBuffer to RGB.
Using Metal:
If you make a new project, select "Augmented Reality App," and select "Metal" for the content technology, the project generated will contain the code and shaders necessary to make this conversion.
Using OpenGL:
The GLCameraRipple example from Apple uses an AVCaptureSession to capture the camera, and shows how to map the resulting CVPixelBuffer to GL textures, which are then converted to RGB in shaders (again, provided in the example).
Non Realtime:
The answer to this stackoverflow question addresses converting the buffer to a UIImage, and offers a pretty simple way to do it.
I have also stuck on this question for several days. All of the code snippet I could find on the Internet is written in Objective-C rather than Swift, regarding converting CVPixelBuffer to UIImage.
Finally, the following code snippet works perfect for me, to convert a YUV image to either JPG or PNG file format, and then you can write it to the local file in your application.
func pixelBufferToUIImage(pixelBuffer: CVPixelBuffer) -> UIImage {
let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
let context = CIContext(options: nil)
let cgImage = context.createCGImage(ciImage, from: ciImage.extent)
let uiImage = UIImage(cgImage: cgImage!)
return uiImage
}
The docs explicitly says that you need to access the luma and chroma planes:
ARKit captures pixel buffers in a planar YCbCr format (also known as YUV) format. To render these images on a device display, you'll need to access the luma and chroma planes of the pixel buffer and convert pixel values to an RGB format.
So there's no way to directly get the RGB planes and you'll have to handle this in your shaders, either in Metal or openGL as described by #joshue
You may want the Accelerate framework's image conversion functions. Perhaps a combination of vImageConvert_420Yp8_Cb8_Cr8ToARGB8888 and vImageConvert_ARGB8888toRGB888 (If you don't want the alpha channel). In my experience these work in real time.
Struggled a long while with this as well and I've ended up writing the following code, which works for me:
// Helper macro to ensure pixel values are bounded between 0 and 255
#define clamp(a) (a > 255 ? 255 : (a < 0 ? 0 : a));
- (void)processImageBuffer:(CVImageBufferRef)imageBuffer
{
OSType type = CVPixelBufferGetPixelFormatType(imageBuffer);
if (type == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange)
{
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// We know the return format of the base address based on the YpCbCr8BiPlanarFullRange format (as per doc)
StandardBuffer baseAddress = (StandardBuffer)CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer, width and height
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// Get buffer info and planar pixel data
CVPlanarPixelBufferInfo_YCbCrBiPlanar *bufferInfo = (CVPlanarPixelBufferInfo_YCbCrBiPlanar *)baseAddress;
uint8_t* cbrBuff = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1);
// This just moved the pointer past the offset
baseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
int bytesPerPixel = 4;
uint8_t *rgbData = rgbFromYCrCbBiPlanarFullRangeBuffer(baseAddress,
cbrBuff,
bufferInfo,
width,
height,
bytesPerRow);
[self doStuffOnRGBBuffer:rgbData width:width height:height bitsPerComponent:8 bytesPerPixel:bytesPerPixel bytesPerRow:bytesPerRow];
free(rgbData);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
else
{
NSLog(#"Unsupported image buffer type");
}
}
uint8_t * rgbFromYCrCbBiPlanarFullRangeBuffer(uint8_t *inBaseAddress,
uint8_t *cbCrBuffer,
CVPlanarPixelBufferInfo_YCbCrBiPlanar * inBufferInfo,
size_t inputBufferWidth,
size_t inputBufferHeight,
size_t inputBufferBytesPerRow)
{
int bytesPerPixel = 4;
NSUInteger yPitch = EndianU32_BtoN(inBufferInfo->componentInfoY.rowBytes);
uint8_t *rgbBuffer = (uint8_t *)malloc(inputBufferWidth * inputBufferHeight * bytesPerPixel);
NSUInteger cbCrPitch = EndianU32_BtoN(inBufferInfo->componentInfoCbCr.rowBytes);
uint8_t *yBuffer = (uint8_t *)inBaseAddress;
for(int y = 0; y < inputBufferHeight; y++)
{
uint8_t *rgbBufferLine = &rgbBuffer[y * inputBufferWidth * bytesPerPixel];
uint8_t *yBufferLine = &yBuffer[y * yPitch];
uint8_t *cbCrBufferLine = &cbCrBuffer[(y >> 1) * cbCrPitch];
for(int x = 0; x < inputBufferWidth; x++)
{
int16_t y = yBufferLine[x];
int16_t cb = cbCrBufferLine[x & ~1] - 128;
int16_t cr = cbCrBufferLine[x | 1] - 128;
uint8_t *rgbOutput = &rgbBufferLine[x*bytesPerPixel];
int16_t r = (int16_t)roundf( y + cr * 1.4 );
int16_t g = (int16_t)roundf( y + cb * -0.343 + cr * -0.711 );
int16_t b = (int16_t)roundf( y + cb * 1.765);
// ABGR image representation
rgbOutput[0] = 0Xff;
rgbOutput[1] = clamp(b);
rgbOutput[2] = clamp(g);
rgbOutput[3] = clamp(r);
}
}
return rgbBuffer;
}
I am using xcode and is currently trying to extract pixel values from the pixel buffer using the following code. However, when i print out the pixel values, it consists of negative values. Anyone has encountered such problem before?
part of the code is as below
- (void)captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:
(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
CVImageBufferRef Buffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(Buffer, 0);
uint8_t* BaseAddress = (uint8_t*)CVPixelBufferGetBaseAddressOfPlane(Buffer, 0);
size_t Width = CVPixelBufferGetWidth(Buffer);
size_t Height = CVPixelBufferGetHeight(Buffer);
if (BaseAddress)
{
IplImage* Temporary = cvCreateImage(cvSize(Width, Height), IPL_DEPTH_8U, 4);
Temporary->imageData = (char*)BaseAddress;
for (int i = 0; i < Temporary->width * Temporary->height; ++i) {
NSLog(#"Pixel value: %d",Temporary->imageData[i]);
//where i try to print the pixels
}
}
The issue is that imageData of IplImage is a signed char. Thus, anything greater than 127 will appear as a negative number.
You can simply assign it to an unsigned char, and then print that, and you'll see values in the range between 0 and 255, like you probably anticipated:
for (int i = 0; i < Temporary->width * Temporary->height; ++i) {
unsigned char c = Temporary->imageData[i];
NSLog(#"Pixel value: %u", c);
}
Or you can print that in hex:
NSLog(#"Pixel value: %02x", c);
The Goal:
Finding the first black pixel on the left side of an image that contains black and transparent pixels only.
What I have:
I know how to get the pixel data and have an array of black and transparent pixels (found it here : https://stackoverflow.com/a/1262893/358480 ):
+ (NSArray*)getRGBAsFromImage:(UIImage*)image atX:(int)xx andY:(int)yy count:(int)count
{
NSMutableArray *result = [NSMutableArray arrayWithCapacity:count];
// First get the image into your data buffer
CGImageRef imageRef = [image CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = malloc(height * width * 4);
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
for (int ii = 0 ; ii < count ; ++ii)
{
NSUInteger alpha = (rawData[byteIndex + 3] * 1.0) / 255.0;
byteIndex += 4;
[result addObject:[NSNumber numberWithInt:alpha]];
}
free(rawData);
return result;
}
What is the problem ?
I can not understand the order which the function "scans" the image.
What i want is to get only the columns of the image and locate the first column that has at list 1 non-transperant pixel. this way I will know how to crop the left, transparent side of the image?
How can I get the pixels by columns?
Thanks
Shani
The bytes are ordered left-to-right, top-to-bottom. So to do what you want, I think you want to loop over the rawData like this:
int x = 0;
int y = 0;
BOOL found = NO;
for (x = 0; x < width; x++) {
for (y = 0; y < height; y++) {
unsigned char alphaByte = rawData[(y*bytesPerRow)+(x*bytesPerPixel)+3];
if (alphaByte > 0) {
found = YES;
break;
}
}
if (found) break;
}
NSLog(#"First non-transparent pixel at %i, %i", x, y);
Then your first column that contains a non-transparent pixel will be column x.
Normally one would iterate over the image array from top to bottom over rows, and within each row from left to right over the columns. In this case you want the reverse: we want to iterate over each column, beginning at the left, and within the column we go over all rows and check if a black pixel is present.
This will give you the left-most black pixel:
size_t maxIndex = height * bytesPerRow;
for (size_t x = 0; x < bytesPerRow; x += bytesPerPixel)
{
for (size_t index = x; index < maxIndex; index += bytesPerRow)
{
if (rawData[index + 3] > 0)
{
goto exitLoop;
}
}
}
exitLoop:
if (x < bytesPerRow)
{
x /= bytesPerPixel;
// left most column is `x`
}
Well, this is equal to mattjgalloway, just slightly optimized, and neater too :O
Although a goto is usually permitted to abandon two loops from within the inner loop, it's still ugly. Makes me really miss those nifty flow control statements D has...
The function you provided in the example code does something different though. It starts at a certain position in the image (defined by xx and yy), and goes over count pixels going from the starting position to the right, continuing to next rows. It adds those alpha values to some array I suspect.
When passed xx = yy = 0, this will find the top-most pixel with certain conditions, not the left-most. This transformation is given by the code above. Do remind that a 2D image is simply a 1D array in memory, starting with the top row from left to right and proceeding with the next rows. Doing simple math one can iterate over rows or over columns.