iOS Metal: Jaggies Anit-aliasing - ios

I was trying to draw a half circle with renderEncoder's drawIndexedPrimitives
[renderEncoder setVertexBuffer:self.vertexBuffer offset:0 atIndex:0];
[renderEncoder drawIndexedPrimitives:MTLPrimitiveTypeTriangleStrip
indexCount:self.indexCount
indexType:MTLIndexTypeUInt16
indexBuffer:self.indicesBuffer
indexBufferOffset:0];
where the vertexBuffer and indicesBuffer for the circle were created by calculation
int segments = 10;
float vertices02[ (segments +1)* (3+4)];
vertices02[0] = centerX;
vertices02[1] = centerY;
vertices02[2] = 0;
//3, 4, 5, 6 are RGBA
vertices02[3] = 1.0;
vertices02[4] = 0;
vertices02[5] = 0.0;
vertices02[6] = 1.0;
uint16_t indices[(segments -1)*3];
for (int i = 1; i <= segments ; i++){
float degree = (i -1) * (endDegree - startDegree)/ (segments -1) + startDegree;
vertices02[i*7] = (centerX + cos([self degreesToRadians:degree])*radius);
vertices02[i*7 +1] = (centerY + sin([self degreesToRadians:degree])*radius);
vertices02[i*7 +2] = 0;
vertices02[i*7 +3] = 1.0;
vertices02[i*7 +4] = 0;
vertices02[i*7 +5] = 0.0;
vertices02[i*7 +6] = 1.0;
if (i < segments){
indices[(i-1)*3 + 0] = 0;
indices[(i-1)*3 + 1] = i;
indices[(i-1)*3 + 2] = i+1;
}
}
So I am combining 9 Triangle to form a 180 degree circle.
Then create vertexBuffer and indicesBuffer
self.vertexBuffer = [device newBufferWithBytes:vertexArrayPtr
length:vertexDataSize
options:MTLResourceOptionCPUCacheModeDefault];
self.indicesBuffer = [device newBufferWithBytes:indexArrayPtr
length:indicesDataSize
options:MTLResourceOptionCPUCacheModeDefault];
The result is like this:
I believe this is Anti-Aliasing problem from Metal of iOS. I used to create half circle in OpenGL using same technique but the edges was much smoother.
Any suggestions to tackle the problem?
Suggested by warrenm, I should set the CAMetalLayer's drawableSize equals screenSize x scale. There are improvements:
Another Suggestion by warrenm, using MTKView and setting sampleCount = 4 solved the problem:

There are a couple of things to consider here. First, you need to ensure that (when possible) the size of the grid you're rasterizing to matches the resolution of the display it will be viewed on. Second, you might need to use subpixel techniques to eke out additional smoothness, since raster techniques tend to undersample continuous functions.
In Metal, the way we match the rendered image size to the display is by ensuring that the drawable size of the Metal layer matches the pixel dimensions it will occupy on the screen. When using CAMetalLayer directly, the default behavior is for the drawable size of the layer to be the size of the layer's bounds multiplied by the layer's contentsScale property. Setting the latter to the scale of the UIScreen onto which the layer is composited will match the layer's dimensions to the screen's pixels (ignoring other transformations that might be applied to the layer or its view hierarchy).
When using MTKView, the autoResizeDrawable property determines whether the view automatically manages its layer's drawable size. This is the default behavior, but if you set this property to NO, you can manually set the drawable size to something else (e.g., use adaptive resolution rendering when fragment-bound).
In order to sample more finely, we have our choice among any number of antialiasing techniques, but perhaps the easiest of these is multisampled antialiasing (MSAA), a hardware feature that—as the name suggests—takes multiple samples for each pixel along the edges of primitives, in order to reduce the jagged effects of aliasing.
In Metal, using MSAA requires setting multisampling state (i.e., the sample count) on both the render pipeline state and the textures used for rendering. MSAA is a two-step process, where a render target that can hold the data for multiple fragments per pixel is rendered to, then a resolve step combines these samples into the final color for each pixel. When using CAMetalLayer (or drawing off-screen), you must create a texture of type MTLTextureType2DMultisample for each active color/depth attachment. These textures are configured as the texture property of their respective color/depth attachments, and the resolveTexture property is set to a texture of type MTLTextureType2D, into which the MSAA targets are resolved.
When using MTKView, simply setting the sampleCount on the view to match the sampleCount of the render pipeline descriptor is sufficient to get MetalKit to create and manage the appropriate resources. By default, the render pass descriptors you receive from a view will have an internally-managed MSAA color target set as the primary color attachment, and the current drawable's texture set as the resolve texture of that attachment. In this way, enabling MSAA with MetalKit only requires a couple of lines of code.

Related

Metal alpha blending is not working

For whatever reason I am having issues with alpha blending in metal. I am drawing to a MTKView and for every pipeline that I create I do the following:
descriptor.colorAttachments[0].blendingEnabled = YES;
descriptor.colorAttachments[0].rgbBlendOperation = MTLBlendOperationAdd;
descriptor.colorAttachments[0].alphaBlendOperation = MTLBlendOperationAdd;
descriptor.colorAttachments[0].sourceRGBBlendFactor = MTLBlendFactorSourceAlpha;
descriptor.colorAttachments[0].sourceAlphaBlendFactor = MTLBlendFactorSourceAlpha;
descriptor.colorAttachments[0].destinationRGBBlendFactor = MTLBlendFactorOneMinusSourceAlpha;
descriptor.colorAttachments[0].destinationAlphaBlendFactor = MTLBlendFactorOneMinusSourceAlpha;
However for whatever reason that is not causing alpha testing to happen. You can even check in the frame debugger and you will see vertices with an alpha of 0 that are being drawn black rather than transparent.
One thought I had is that some geometry ends up on the exact same z plane so if alpha blending does not work on the same z plane that might cause an issue. But I dont think that is a thing.
Why is alpha blending not working?
I am hoping to blend as if they were transparent glass. Think like this.
Alpha blending is an order-dependent transparency technique. This means that the (semi-)transparent objects cannot be rendered in any arbitrary order as is the case for (more expensive) order-independent transparency techniques.
Make sure your transparent 2D objects (e.g., circle, rectangle, etc.) have different depth values. (This way you can define the draw ordering yourself. Otherwise the draw ordering depends on the implementation of the sorting algorithm and the initial ordering before sorting.)
Sort these 2D objects based on their depth value from back to front.
Draw the 2D objects from back to front (painter's algorithm) using alpha blending. (Of course, your 2D objects need an alpha value < 1 to actually see some blending.)
Your blend state for alpha blending is correct:
// The blend formula is defined as:
// (source.rgb * sourceRGBBlendFactor ) rgbBlendOperation (destination.rgb * destinationRGBBlendFactor )
// (source.a * sourceAlphaBlendFactor) alphaBlendOperation (destination.a * destinationAlphaBlendFactor)
// <=>
// (source.rgba * source.a) + (destination.rgba * (1-source.a))
descriptor.colorAttachments[0].blendingEnabled = YES;
descriptor.colorAttachments[0].rgbBlendOperation = MTLBlendOperationAdd;
descriptor.colorAttachments[0].alphaBlendOperation = MTLBlendOperationAdd;
descriptor.colorAttachments[0].sourceRGBBlendFactor = MTLBlendFactorSourceAlpha;
descriptor.colorAttachments[0].sourceAlphaBlendFactor = MTLBlendFactorSourceAlpha;
descriptor.colorAttachments[0].destinationRGBBlendFactor = MTLBlendFactorOneMinusSourceAlpha;
descriptor.colorAttachments[0].destinationAlphaBlendFactor = MTLBlendFactorOneMinusSourceAlpha;

How can i prevent my object detection program from detecting multiple objects of different sizes?

So, here is my situation. I have created a object detection program which is based on color object detection. My program detects the color red and it works perfectly. But here is the problems i am facing:-
Whenever there are more than one red object in the surrounding, my program detects them and it cannot really track one object at that time(i.e it tracks other red objects of various sizes in the background. It shows me the error that "too much noise in the background". As you can see in the "threshold image" attached, it detects the round object (which is my tracking object) and my cap which is red in color. I want my program to detect only my tracking object("which is a round shaped coke cap"). How can i achieve that? Please help me out. I have my engineering design contest in few days and i have to demo my program infront of my lecturers. My program should only be able to detect and track the object which i want. Thanks
My code for the objectdetection program is a little long. So, i am hereby explaining the code as follows- I captured a frame from the webcam frame-converted it to HSV- used HSV Inrange filter to filter out the other colors but red- applied morphological operations on the filtered image. This all goes in my main function
I am using a frame resolution of 1280*720 for my webcam frame. It kind of slows down my program but it was a trade off which i had to do for performing gesture controlled operations. Anyways here is my drawobjectfunction and trackfilteredobjectfunction.
int H_MIN = 0;
int H_MAX = 256;
int S_MIN = 0;
int S_MAX = 256;
int V_MIN = 0;
int V_MAX = 256;
//default capture width and height
const int FRAME_WIDTH = 1280;
const int FRAME_HEIGHT = 720;
//max number of objects to be detected in frame
const int MAX_NUM_OBJECTS=50;
//minimum and maximum object area
const int MIN_OBJECT_AREA = 20*20;
const int MAX_OBJECT_AREA = FRAME_HEIGHT*FRAME_WIDTH/1.5;
void drawObject(int x, int y,Mat &frame){
circle(frame,Point(x,y),20,Scalar(0,255,0),2);
if(y-25>0)
line(frame,Point(x,y),Point(x,y-25),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(x,0),Scalar(0,255,0),2);
if(y+25<FRAME_HEIGHT)
line(frame,Point(x,y),Point(x,y+25),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(x,FRAME_HEIGHT),Scalar(0,255,0),2);
if(x-25>0)
line(frame,Point(x,y),Point(x-25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(0,y),Scalar(0,255,0),2);
if(x+25<FRAME_WIDTH)
line(frame,Point(x,y),Point(x+25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(FRAME_WIDTH,y),Scalar(0,255,0),2);
putText(frame,intToString(x)+","+intToString(y),Point(x,y+30),1,1,Scalar(0,255,0),2);
}
void trackFilteredObject(int &x, int &y, Mat threshold, Mat &cameraFeed){
Mat temp;
threshold.copyTo(temp);
//these two vectors needed for output of findContours
vector< vector<Point> > contours;
vector<Vec4i> hierarchy;
//find contours of filtered image using openCV findContours function
findContours(temp,contours,hierarchy,CV_RETR_CCOMP,CV_CHAIN_APPROX_SIMPLE );
//use moments method to find our filtered object
double refArea = 0;
bool objectFound = false;
if (hierarchy.size() > 0) {
int numObjects = hierarchy.size();
//if number of objects greater than MAX_NUM_OBJECTS we have a noisy filter
if(numObjects<MAX_NUM_OBJECTS){
for (int index = 0; index >= 0; index = hierarchy[index][0]) {
Moments moment = moments((cv::Mat)contours[index]);
double area = moment.m00;
//if the area is less than 20 px by 20px then it is probably just noise
//if the area is the same as the 3/2 of the image size, probably just a bad filter
//we only want the object with the largest area so we safe a reference area each
//iteration and compare it to the area in the next iteration.
if(area>MIN_OBJECT_AREA && area<MAX_OBJECT_AREA && area>refArea){
x = moment.m10/area;
y = moment.m01/area;
objectFound = true;
refArea = area;
}else objectFound = false;
}
//let user know you found an object
if(objectFound ==true){
putText(cameraFeed,"Tracking Object",Point(0,50),2,1,Scalar(0,255,0),2);
//draw object location on screen
drawObject(x,y,cameraFeed);}
}else putText(cameraFeed,"TOO MUCH NOISE! ADJUST FILTER",Point(0,50),1,2,Scalar(0,0,255),2);
}
}
Here is the link of the image; as you can see it also detects the red hat in the background along with the red cap of the coke bottle.
My observations:- Here is what i think, to achieve my desired goal of not detecting objects of unknown sizes of red color. I think i have to edit the value of maximum object area which i declared in the above program as (const int MAX_OBJECT_AREA = FRAME_HEIGHT*FRAME_WIDTH/1.5;). I think i have to change this value, that might eliminate the detection of bigger continous red pictures. But also, there is another problem some objects are not completely red in color and they have patches of red and other colors. So, if the detected area is within the range specfied in my program then my program detects those red patches too. What i mean to say is i was wearing a tshirt which has mixed colors and when i tested my program by wearing that tshirt, my program was able to detect the red color out of the other colors. Now, how do i solve this issue?
I think you can try out the following procedure:
obtain a circular kernel having roughly the same area as your object of interest. You can do it like: Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(d, d));
where d is the diameter of the disk.
perform normalized-cross-correlation or convolution of the filtered regions image with this kernel (I think normalized-cross-correlation would be better. And add an empty boarder around the kernel).
the peak of the resulting image should give you the location of the circular region in your filtered image (if you are using normalized-cross-correlation, you'll have to add the shift).
To speed things up, you can perform this at a reduced resolution.
You can filter out non-circular shapes by detecting circles in your thresholded image. OpenCV provides a built-on method to detect circles using Hough transform, more info here. You can take advantage of this function to retain only circles that have a radius in a given range.
Another possibility is to implement connected component labeling (CCL) into your demo program.
I believe that it was removed at some point in verions 2.x of OpenCV, but a basic implementation of the two-pass version is straightforward from the Wikipedia page.
CCL will assign a unique ID for each object after thresholding. You then have to implement matching between the objects at frame (T-1) and objects in frame (T) (for example based on some nearest distance criterion) and possibly trajectory filtering or smoothing, but this would definitely give you some extra-points.

Displacement Map UV Mapping?

Summary
I'm trying to apply a displacement map (Height map) to a rather simple object (Hexagonal plane) and I'm having some unexpected results. I am using grayscale and as such, I was under the impression my height map should only be affecting the Z values of my mesh. However, the displacement map I've created stretches the mesh across the X and Y planes. Furthermore, it doesn't seem to use the UV mapping I've created that all other textures are successfully applied to.
Model and UV Map
Here are reference images of my hexagonal mesh and its corresponding UV map in Blender.
Diffuse and Displacement Textures
These are the diffuse and displacement map textures I am applying to my mesh through Three.JS.
Renders
When I render the plane without a displacement map, you can see that the hexagonal plane stays within the lines. However, when I add the displacement map it clearly affects the X and Y positions of the vertices rather than affecting only the Z, expanding the plane well over the lines.
Code
Here's the relevant Three.js code:
// Textures
var diffuseTexture = THREE.ImageUtils.loadTexture('diffuse.png', null, loaded);
var displacementTexture = THREE.ImageUtils.loadTexture('displacement.png', null, loaded);
// Terrain Uniform
var terrainShader = THREE.ShaderTerrain["terrain"];
var uniformsTerrain = THREE.UniformsUtils.clone(terrainShader.uniforms);
//uniformsTerrain["tNormal"].value = null;
//uniformsTerrain["uNormalScale"].value = 1;
uniformsTerrain["tDisplacement"].value = displacementTexture;
uniformsTerrain["uDisplacementScale"].value = 1;
uniformsTerrain[ "tDiffuse1" ].value = diffuseTexture;
//uniformsTerrain[ "tDetail" ].value = null;
uniformsTerrain[ "enableDiffuse1" ].value = true;
//uniformsTerrain[ "enableDiffuse2" ].value = true;
//uniformsTerrain[ "enableSpecular" ].value = true;
//uniformsTerrain[ "uDiffuseColor" ].value.setHex(0xcccccc);
//uniformsTerrain[ "uSpecularColor" ].value.setHex(0xff0000);
//uniformsTerrain[ "uAmbientColor" ].value.setHex(0x0000cc);
//uniformsTerrain[ "uShininess" ].value = 3;
//uniformsTerrain[ "uRepeatOverlay" ].value.set(6, 6);
// Terrain Material
var material = new THREE.ShaderMaterial({
uniforms:uniformsTerrain,
vertexShader:terrainShader.vertexShader,
fragmentShader:terrainShader.fragmentShader,
lights:true,
fog:true
});
// Load Tile
var loader = new THREE.JSONLoader();
loader.load('models/hextile.js', function(g) {
//g.computeFaceNormals();
//g.computeVertexNormals();
g.computeTangents();
g.materials[0] = material;
tile = new THREE.Mesh(g, new THREE.MeshFaceMaterial());
scene.add(tile);
});
Hypothesis
I'm currently juggling three possibilities as to why this could be going wrong:
The UV map is not applying to my displacement map.
I've made the displacement map incorrectly.
I've missed a crucial step in the process that would lock the displacement to Z-only.
And of course, secret option #4 which is none of the above and I just really have no idea what I'm doing. Or any mixture of the aforementioned.
Live Example
You can view a live example here.
If anybody with more knowledge on the subject could guide me I'd be very grateful!
Edit 1: As per suggestion, I've commented out computeFaceNormals() and computeVertexNormals(). While it did make a slight improvement, the mesh is still being warped.
In your terrain material, set wireframe = true, and you will be able to see what is happening.
Your code and textures are basically fine. The problem occurs when you compute vertex normals in the loader callback function.
The computed vertex normals for the outer ring of your geometry point somewhat outward. This is most likely because in computeVertexNormals() they are computed by averaging the face normals of each neighboring face, and the face normals of the "sides" of your model (the black part) are averaged into the vertex normal calculation for those vertices that make up the outer ring of the "cap".
As a result, the outer ring of the "cap" expands outward under the displacement map.
EDIT: Sure enough, straight from your model, the vertex normals of the outer ring point outward. The vertex normals for the inner rings are all parallel. Perhaps Blender is using the same logic to generate vertex normals as computeVertexNormals() does.
The problem is how your object is constructed becuase the displacement happens along the normal vector.
the code is here.
https://github.com/mrdoob/three.js/blob/master/examples/js/ShaderTerrain.js#L348-350
"vec3 dv = texture2D( tDisplacement, uvBase ).xyz;",
This takes a the rgb vector of the displacement texture.
"float df = uDisplacementScale * dv.x + uDisplacementBias;",
this takes only red value of the vector becuase uDisplacementScale is normally 1.0 and uDisplacementBias is 0.0.
"vec3 displacedPosition = normal * df + position;",
This displaces the postion along the normal vector.
so to solve you either update the normals or the shader.

Colorizing image ignores alpha channel — why and how to fix?

Here's what I'm trying to do: On the left is a generic, uncolorized RGBA image that I've created off-screen and cached for speed (it's very slow to create initially, but very fast to colorize with any color later, as needed). It's a square image with a circular swirl. Inside the circle, the image has an alpha/opacity of 1. Outside the circle, it has an alpha/opacity of 0. I've displayed it here inside a UIView with a background color of [UIColor scrollViewTexturedBackgroundColor]. On the right is what happens when I attempt to colorize the image by filling a solid red rectangle over the top of it after setting CGContextSetBlendMode(context, kCGBlendModeColor).
That's not what I want, nor what I expected. Evidently, colorizing a completely transparent pixel (e.g., alpha value of 0) results in the full-on fill color for some strange reason, rather than remaining transparent as I would have expected.
What I want is actually this:
Now, in this particular case, I can set the clipping region to a circle, so that the area outside the circle remains untouched — and that's what I've done here as a workaround.
But in my app, I also need to be able to colorize arbitrary shapes where I don't know the clipping/outline path. One example is colorizing white text by overlaying a gradient. How is this done? I suspect there must be some way to do it efficiently — and generally, with no weird path/clipping tricks — using image masks... but I have yet to find a tutorial on this. Obviously it's possible because I've seen colored-gradient text in other games.
Incidentally, what I can't do is start with a gradient and clip/clear away parts I don't need — because (as shown in the example above) my uncolorized source images are, in general, grayscale rather than pure white. So I really need to start with the uncolorized image and then colorize it.
p.s. — kCGBlendModeMultiply also has the same flaws / shortcomings / idiosyncrasies when it comes to colorizing partially transparent images. Does anyone know why Apple decided to do it that way? It's as if the Quartz colorizing code treats RGBA(0,0,0,0) as RGBA(0,0,0,1), i.e., it completely ignores and destroys the alpha channel.
One approach that you can take that will work is to construct a mask from the original image and then invoke the CGContextClipToMask() method before rendering your image with the multiply blend mode set. Here is the CoreGraphics code that would set the mask before drawing the image to color.
CGContextRef context = [frameBuffer createBitmapContext];
CGRect bounds = CGRectMake( 0.0f, 0.0f, width, height );
CGContextClipToMask(context, bounds, maskImage.CGImage);
CGContextDrawImage(context, bounds, greyImage.CGImage);
The slightly more tricky part will be to take the original image and generate a maskImage. What you can do for that is write a loop that will examine each pixel and write either a black or white pixel as the mask value. If the original pixel in the image to color is completely transparent, then write a black pixel, otherwise write a white pixel. Note that the mask value will be a 24BPP image. Here is some code to give you the right idea.
uint32_t *inPixels = (uint32_t*) MEMORY_ADDR_OF_ORIGINAL_IMAGE;
uint32_t *maskPixels = malloc(numPixels * sizeof(uint32_t));
uint32_t *maskPixelsPtr = maskPixels;
for (int rowi = 0; rowi < height; rowi++) {
for (int coli = 0; coli < width; coli++) {
uint32_t inPixel = *inPixels++;
uint32_t inAlpha = (inPixel >> 24) & 0xFF;
uint32_t cval = 0;
if (inAlpha != 0) {
cval = 0xFF;
}
uint32_t outPixel = (0xFF << 24) | (cval << 16) | (cval << 8) | cval;
*maskPixelsPtr++ = outPixel;
}
}
You will of course need to fill in all the details and create the graphics contexts and so on. But the general idea is to simply create your own mask to filter out drawing of the red parts around the outside of the circle.

how to optimized this image processing replace all pixels on image with closest available RGB?

Im' trying to replace all pixels of input image with closest available RGB. I have a array contain color and input image. Here is my code, it give me an output image as expected, BUT it take very LONG time( about a min) to process one image. Can anybody help me improve the code? Or if you have any other suggestions, please help.
UIGraphicsBeginImageContextWithOptions(CGSizeMake(CGImageGetWidth(sourceImage),CGImageGetHeight(sourceImage)), NO, 0.0f);
//Context size I keep as same as original input image size
//Otherwise, the output will be only a partial image
CGContextRef context;
context = UIGraphicsGetCurrentContext();
//This is for flipping up sidedown
CGContextTranslateCTM(context, 0, self.imageViewArea.image.size.height);
CGContextScaleCTM(context, 1.0, -1.0);
// init vars
float d = 0; // squared error
int idx = 0; // index of palette color
int min = 1000000; // min difference
UIColor *oneRGB; // color at a pixel
UIColor *paletteRGB; // palette color
// visit each output color and determine closest color from palette
for(int y=0; y<sizeY; y++) {
for(int x=0; x<sizeX; x++) {
// desired (avg) color is one pixel of scaled image
oneRGB = [inputImgAvg colorAtPixel:CGPointMake(x,y)];
// find closest color match in palette: init idx with index
// of closest match; keep track of min to find idx
min = 1000000;
idx = 0;
CGContextDrawImage(context,CGRectMake(xx, yy, 1, 1),img);
}
}
UIImage *output = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
self.imageViewArea.image = output;
This is a similar question (with no definitive answer), but the answer there has the code for directly accessing pixels from an image.
Quantize Image, Save List of Remaining Colors
You should do that rather than use CG functions for each get and set pixel. Drawing 1 pixel of an image onto another image is a lot slower than changing 3 bytes in a array.
Also, what's in ColorDiff -- you don't need perfect diffing as long as the closest pixel has the smallest diff. There may be room for pre-processing this list so that for each palette entry you have the smallest diff to the nearest other palette entry. Then, while looping through pixels, I can quickly check to see if the next pixel is within half that distance to the color just found (because photos tend to have common colors near each other).
If that's not a match, then while looping through the palette, if I am within half this distance to any entry, there is no need to check further.
Basically, this puts a zone around each palette entry where you know for sure that this one is the closest.
The usual answer is to use a k-d tree or some other Octree structure to reduce the number of computations and comparisons that have to be done at each pixel.
I've also had success with partitioning the color space into a regular grid and keeping a list of possible closest matches for each part of the grid. For example you can divide the (0-255) values of R,G,B by 16 and end up with a grid of (16,16,16) or 4096 elements altogether. Best case is that there's only one member of the list for a particular grid element and no need to traverse the list at all.

Resources