I'm developing an App where user can draw on the screen. I started from GLPaint provided by Apple.
I want to detect if user has colored a specific area of the screen and I want to detect it each 1/15 of second (or something like that, so very fast).
I thought to create a bitmap graphics context and pass it to CGBitmapContextCreate and than
draw my view into that context and loop through all pixels.
I'm looking for something more efficient...
Any help?
UPDATE
I tried to use glReadPixels() but it is extremely slow... Maybe I miss something...
I tried to read a 100x100 area, but this method took like 1 sec to complete.
this is my code
- (void)isGameFinished
{
float x = 0;
float y = 0;
float w = 100;
float h = 100;
GLubyte* pixelColors = malloc(w * h * sizeof(GLubyte) * 3);
glReadPixels(x,y, w, h, GL_RGB, GL_UNSIGNED_BYTE, &pixelColors[0]);
for(int i=0;i<w*h;i+=3)
{
ccColor3B color=ccc3(pixelColors[i],pixelColors[i+1],pixelColors[i+2]);
NSLog(#"index: %d r:%c g:%c b:%c",i,color.r,color.g,color.b);
}
free(pixelColors);
}
For calling a method every 1/15 of a second, you can use a CADisplayLink with a frameInterval of 4 (60 FPS / 4 = 15 times per second). If you are using GLPaint, that means you are using OpenGL. You can use the function glReadPixels() to get the pixel data of a certain area. However, you need to make sure that you have access to the underlying buffer in the function that gets called by CADisplayLink. I'm not good enough at OpenGL to say how to ensure that.
Related
I was trying to draw a half circle with renderEncoder's drawIndexedPrimitives
[renderEncoder setVertexBuffer:self.vertexBuffer offset:0 atIndex:0];
[renderEncoder drawIndexedPrimitives:MTLPrimitiveTypeTriangleStrip
indexCount:self.indexCount
indexType:MTLIndexTypeUInt16
indexBuffer:self.indicesBuffer
indexBufferOffset:0];
where the vertexBuffer and indicesBuffer for the circle were created by calculation
int segments = 10;
float vertices02[ (segments +1)* (3+4)];
vertices02[0] = centerX;
vertices02[1] = centerY;
vertices02[2] = 0;
//3, 4, 5, 6 are RGBA
vertices02[3] = 1.0;
vertices02[4] = 0;
vertices02[5] = 0.0;
vertices02[6] = 1.0;
uint16_t indices[(segments -1)*3];
for (int i = 1; i <= segments ; i++){
float degree = (i -1) * (endDegree - startDegree)/ (segments -1) + startDegree;
vertices02[i*7] = (centerX + cos([self degreesToRadians:degree])*radius);
vertices02[i*7 +1] = (centerY + sin([self degreesToRadians:degree])*radius);
vertices02[i*7 +2] = 0;
vertices02[i*7 +3] = 1.0;
vertices02[i*7 +4] = 0;
vertices02[i*7 +5] = 0.0;
vertices02[i*7 +6] = 1.0;
if (i < segments){
indices[(i-1)*3 + 0] = 0;
indices[(i-1)*3 + 1] = i;
indices[(i-1)*3 + 2] = i+1;
}
}
So I am combining 9 Triangle to form a 180 degree circle.
Then create vertexBuffer and indicesBuffer
self.vertexBuffer = [device newBufferWithBytes:vertexArrayPtr
length:vertexDataSize
options:MTLResourceOptionCPUCacheModeDefault];
self.indicesBuffer = [device newBufferWithBytes:indexArrayPtr
length:indicesDataSize
options:MTLResourceOptionCPUCacheModeDefault];
The result is like this:
I believe this is Anti-Aliasing problem from Metal of iOS. I used to create half circle in OpenGL using same technique but the edges was much smoother.
Any suggestions to tackle the problem?
Suggested by warrenm, I should set the CAMetalLayer's drawableSize equals screenSize x scale. There are improvements:
Another Suggestion by warrenm, using MTKView and setting sampleCount = 4 solved the problem:
There are a couple of things to consider here. First, you need to ensure that (when possible) the size of the grid you're rasterizing to matches the resolution of the display it will be viewed on. Second, you might need to use subpixel techniques to eke out additional smoothness, since raster techniques tend to undersample continuous functions.
In Metal, the way we match the rendered image size to the display is by ensuring that the drawable size of the Metal layer matches the pixel dimensions it will occupy on the screen. When using CAMetalLayer directly, the default behavior is for the drawable size of the layer to be the size of the layer's bounds multiplied by the layer's contentsScale property. Setting the latter to the scale of the UIScreen onto which the layer is composited will match the layer's dimensions to the screen's pixels (ignoring other transformations that might be applied to the layer or its view hierarchy).
When using MTKView, the autoResizeDrawable property determines whether the view automatically manages its layer's drawable size. This is the default behavior, but if you set this property to NO, you can manually set the drawable size to something else (e.g., use adaptive resolution rendering when fragment-bound).
In order to sample more finely, we have our choice among any number of antialiasing techniques, but perhaps the easiest of these is multisampled antialiasing (MSAA), a hardware feature that—as the name suggests—takes multiple samples for each pixel along the edges of primitives, in order to reduce the jagged effects of aliasing.
In Metal, using MSAA requires setting multisampling state (i.e., the sample count) on both the render pipeline state and the textures used for rendering. MSAA is a two-step process, where a render target that can hold the data for multiple fragments per pixel is rendered to, then a resolve step combines these samples into the final color for each pixel. When using CAMetalLayer (or drawing off-screen), you must create a texture of type MTLTextureType2DMultisample for each active color/depth attachment. These textures are configured as the texture property of their respective color/depth attachments, and the resolveTexture property is set to a texture of type MTLTextureType2D, into which the MSAA targets are resolved.
When using MTKView, simply setting the sampleCount on the view to match the sampleCount of the render pipeline descriptor is sufficient to get MetalKit to create and manage the appropriate resources. By default, the render pass descriptors you receive from a view will have an internally-managed MSAA color target set as the primary color attachment, and the current drawable's texture set as the resolve texture of that attachment. In this way, enabling MSAA with MetalKit only requires a couple of lines of code.
I would like to create a brush for drawing on a PGraphics element with Processing. I would like past brush strokes to be visible. However, since the PGraphics element is loaded every frame, previous brush strokes disappear immediatly.
My idea was then to create PGraphics pg in setup(), make a copy of it in void(), alter the original graphic pg and update the copy at every frame. This produces a NullPointerException, most likely because pg is defined locally in setup().
This is what I have got so far:
PGraphics pg;
PFont font;
void setup (){
font = createFont("Pano Bold Kopie.otf", 600);
size(800, 800, P2D);
pg = createGraphics(800, 800, P2D);
pg.beginDraw();
pg.background(0);
pg.fill(255);
pg.textFont(font);
pg.textSize(400);
pg.pushMatrix();
pg.translate(width/2, height/2-140);
pg.textAlign(CENTER, CENTER);
pg.text("a", 0 , 0);
pg.popMatrix();
pg.endDraw();
}
void draw () {
copy(pg, 0, 0, width, height, 0, 0, width, height);
loop();
int c;
loadPixels();
for (int x=0; x<width; x++) {
for (int y=0; y<height; y++) {
pg.pixels[mouseX+mouseY*width]=0;
}
}
updatePixels();
}
My last idea, which I have not attempted to implement yet, is to append pixels which have been touched by the mouse to a list and to draw from this list each frame. But this seems quite complicated to me as it might result into super long arrays needing to be processed on top of the original image. So, I hope there is another way around!
EDIT: My goal is to create a smudge brush, hence a brush which kind of copies areas from one part of the image to other parts.
There's no need to manually copy pixels like that. The PGraphics class extends PImage, which means you can simply render it with image(pg,0,0); for example.
The other thing you could do is an old trick to fade the background: instead of clearing pixels completely you can render a sketch size slightly opaque rectangle with no stroke.
Here's a quick proof of concept based on your code:
PFont font;
PGraphics pg;
void setup (){
//font = createFont("Pano Bold Kopie.otf", 600);
font = createFont("Verdana",600);
size(800, 800, P2D);
// clear main background once
background(0);
// prep fading background
noStroke();
// black fill with 10/255 transparnecy
fill(0,10);
pg = createGraphics(800, 800, P2D);
pg.beginDraw();
// leave the PGraphics instance transparent
//pg.background(0);
pg.fill(255);
pg.textFont(font);
pg.textSize(400);
pg.pushMatrix();
pg.translate(width/2, height/2-140);
pg.textAlign(CENTER, CENTER);
pg.text("a", 0 , 0);
pg.popMatrix();
pg.endDraw();
}
void draw () {
// test with mouse pressed
if(mousePressed){
// slowly fade/clear the background by drawing a slightly opaque rectangle
rect(0,0,width,height);
}
// don't clear the background, render the PGraphics layer directly
image(pg, mouseX - pg.width / 2, mouseY - pg.height / 2);
}
If you hold the mouse pressed you can see the fade effect.
(changing transparency to 10 to a higher value with make the fade quicker)
Update To create a smudge brush you can still sample pixels and then manipulate the read colours to some degree. There are many ways to implement a smudge effect based on what you want to achieve visually.
Here's a very rough proof of concept:
PFont font;
PGraphics pg;
int pressX;
int pressY;
void setup (){
//font = createFont("Pano Bold Kopie.otf", 600);
font = createFont("Verdana",600);
size(800, 800, P2D);
// clear main background once
background(0);
// prep fading background
noStroke();
// black fill with 10/255 transparnecy
fill(0,10);
pg = createGraphics(800, 800, JAVA2D);
pg.beginDraw();
// leave the PGraphics instance transparent
//pg.background(0);
pg.fill(255);
pg.noStroke();
pg.textFont(font);
pg.textSize(400);
pg.pushMatrix();
pg.translate(width/2, height/2-140);
pg.textAlign(CENTER, CENTER);
pg.text("a", 0 , 0);
pg.popMatrix();
pg.endDraw();
}
void draw () {
image(pg,0,0);
}
void mousePressed(){
pressX = mouseX;
pressY = mouseY;
}
void mouseDragged(){
// sample the colour where mouse was pressed
color sample = pg.get(pressX,pressY);
// calculate the distance from where the "smudge" started to where it is
float distance = dist(pressX,pressY,mouseX,mouseY);
// map this distance to transparency so the further the distance the less smudge (e.g. short distance, high alpha, large distnace, small alpha)
float alpha = map(distance,0,30,255,0);
// map distance to "brush size"
float size = map(distance,0,30,30,0);
// extract r,g,b values
float r = red(sample);
float g = green(sample);
float b = blue(sample);
// set new r,g,b,a values
pg.beginDraw();
pg.fill(r,g,b,alpha);
pg.ellipse(mouseX,mouseY,size,size);
pg.endDraw();
}
As the comments mention, one idea is to sample colour on press then use the sample colour and fade it as your drag away from the source area. This shows simply reading a single pixel. You may want to experiment with sampling/reading more pixels (e.g. a rectangle or ellipse).
Additionally, the code above isn't optimised.
A few things could be sped up a bit, like reading pixels, extracting colours, calculating distance, etc.
For example:
void mouseDragged(){
// sample the colour where mouse was pressed
color sample = pg.pixels[pressX + (pressY * pg.width)];
// calculate the distance from where the "smudge" started to where it is (can use manual distance squared if this is too slow)
float distance = dist(pressX,pressY,mouseX,mouseY);
// map this distance to transparency so the further the distance the less smudge (e.g. short distance, high alpha, large distnace, small alpha)
float alpha = map(distance,0,30,255,0);
// map distance to "brush size"
float size = map(distance,0,30,30,0);
// extract r,g,b values
int r = (sample >> 16) & 0xFF; // Like red(), but faster
int g = (sample >> 8) & 0xFF;
int b = sample & 0xFF;
// set new r,g,b,a values
pg.beginDraw();
pg.fill(r,g,b,alpha);
pg.ellipse(mouseX,mouseY,size,size);
pg.endDraw();
}
The idea is to start simple with clear, readable code and only at the end, if needed look into optimisations.
I know this question has been asked quite a bit, but none of the solutions really fit my case. I am looking to add a second type of object to the canvas with the code shown below. I know I didn't provide much but its a quick start. Just ask for more if you think you have a hunch. This code below is in my render function.
So far I have checked that
I have enough vertices in my points array
I have enough normal vectors in my normals array
I have enough texture coordinates in my texCoords array
There are no mismatches between the vectors added when creating my terrain and my propeller.
The terrain renders just fine with the texture, lighting and all but,I am unable to get the propeller to render. I get the error I listed above. I have added multiple objects to canvases before and never run into an error like this.
//----------------------------------------- Draw Terrain ------------------------------------
var i = 0;
for(var row=0-dimension; row<dimension; row+=3){
for(var col=0-dimension; col<dimension; col+=3, i++){
var mv = mult(viewer, mult(translate(row, -1, col), mult(scale[i],rot[i])));
gl.uniformMatrix4fv(modelViewLoc, false, flatten(mv));
gl.uniformMatrix3fv(normalLoc, false, flatten(normalMatrix(mv, true)));
gl.drawArrays( gl.TRIANGLES, 0, index);
}
}
//----------------------------------------- Draw Propeller ------------------------------------
mv = mult(viewer, mult( translate(-2.1, -2.9, -.2), scalem(4,5,5)));
gl.uniformMatrix4fv(modelViewLoc, false, flatten(mv));
gl.uniformMatrix3fv(normalLoc, false, flatten(normalMatrix(mv, true)));
gl.drawArrays( gl.TRIANGLES, propellerStart, points.length);
Is there any way i can use the "Attribute 2" in the error message to track down the variable giving me this issue?
Appreciate the help!
What part don't you understand? The error is clear, whatever buffer you have attached to attribute 2 is not big enough to handle the propellerStart, points.length draw request.
So first thing is figure out which attribute is attribute 2. Do this by printing out your attribute locations. Is your points, normals, or texcoords?
You should already be looking them up somewhere with gl.getAttribLocation so print out those values, find out which one is #2.
Then go look at the size of the buffer you attached to that attribute. To do that somewhere you would have called.
gl.bindBuffer(gl.ARRAY_BUFFER, someBuffer);
gl.vertexAttribPointer(locationForAttribute2, size, type, normalize, stride, offset);
So we know it's someBuffer from the above code. We also need to know size, type, stride, and offset
Somewhere else you filled that buffer with data using
gl.bindBuffer(gl.ARRAEY_BUFFER, someBuffer);
gl.bufferData(gl.ARRAY_BUFFER, someData, ...);
So you need to find the size of someData.
sizeOfBuffer = someData.length * someData.BYTES_PER_ELEMENT
Let's it's a 1000 element Float32Array so it someData.length is 1000 and someData.BYTES_PER_ELEMENT is 4 therefore sizeOfBuffer is 4000.
Using all of that you can now check if your buffer is too small. (note: we already know it's too small since the browser told us so but if you want know how to compute it yourself)
Let's say size is 3, type is gl.FLOAT, stride is 32, offset is 12 (note: I personally never use anything but stride = 0 and offset = 0)
Let's say points.length = 50
numPoints = points.length;
bytesPerElement = size * 4; // because a gl.FLOAT is 4 bytes
realStride = stride === 0 ? bytesPerElement : stride;
bytesNeeded = realStride * (numPoints - 1) + bytesPerElement;
bytesNeeded in this case is (64 * 49) + 12 = 3148
So now we know how many bytes are needed. Does are buffer have enough data? We'll when you called draw you passed in an offset propellerStart. Let's assume it's 900 and there's the offset in the attribute so.
bufferSizeNeeded = offset + propellerStart + bytesNeeded
so bufferSizeNeeded = 12 + 900 + 3148 which is 4060. Since 4060 is > sizeOfBuffer which was 4000 you're going to get the error you got.
In any case the point is really it's up to you to figure out which buffer is used by attribute #2, then go look at why your buffer is too small. Is your offset to drawArrays wrong? Is your stride too big? Is your offset wrong in vertexAttribPointer (it's in number of bytes not number of units). Do you put the wrong size (1,2,3,4). Do you mis-calculate the number of points?
I've got a question about a PixelShader I am trying to implement, and what I currently do (this is just for debugging, and trying to figure stuff out):
int3 loc;
loc.x = (int)(In.TextureUV.x * resolution_XY.x);
loc.y = (int)(In.TextureUV.x * resolution_XY.x);
loc.z = 0;
float4 r = g_txDiffuse.Load(loc);
return float4(r.x, r.y, r.z, 1);
The point is, this is always 0,0,0,1
The texture buffer is created:
D3D11_TEXTURE2D_DESC tDesc;
tDesc.Height = 480;
tDesc.Width = 640;
tDesc.Usage = D3D11_USAGE_DYNAMIC;
tDesc.MipLevels = 1;
tDesc.ArraySize = 1;
tDesc.SampleDesc.Count = 1;
tDesc.SampleDesc.Quality = 0;
tDesc.Format = DXGI_FORMAT_R8_UINT;
tDesc.CPUAccessFlags = D3D11_CPU_ACCESS_WRITE;
tDesc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
tDesc.MiscFlags = 0;
V_RETURN(pd3dDevice->CreateTexture2D(&tDesc, NULL, &g_pCurrentImage));
I upload the texture (which should be a live display at the end) via:
D3D11_MAPPED_SUBRESOURCE resource;
pd3dImmediateContext->Map(g_pCurrentImage, 0, D3D11_MAP_WRITE_DISCARD, 0, &resource);
memcpy( resource.pData, g_Images.GetData(), g_Images.GetDataSize() );
pd3dImmediateContext->Unmap( g_pCurrentImage, 0 );
I've checked the resource.pData, the data in there is a valid 8bit monochrome image. I made sure the data coming from the camera is 8bit monochrome 640x480.
There's a few things I don't fully understand:
if I run the Map / memcpy / Unmap routine in every frame, the driver will ultimately crash, the system will be unresponsive. Is there a different way to update a complete texture every frame which should be done?
the texture I uploaded is 8bit, why is the Texture2D.load() a float4 return? Do I have to use a different method to access the texture data? I tried to .sample it, but that didn't work either. Would I have to use a int buffer or something instead?
is there a way to debug the GPU memory, to check if the memcpy worked in the first place?
The Map, memcpy, Unmap really ought not to crash unless2 you are trying to copy too much data into the texture. It would be interesting to know what "GetDataSize()" returns. Does it equal 307,200? If its more than that then there lies your problem.
Texture2D returns a float4 because thats what you've asked for. If you write float r = g_txDiffuse.Load( ... ). The 8-bits get extended to a normalised float as part of the load process. Are you sure, btw, that your calculation of "loc" is correct because as you have it now loc.x and loc.y will always be the same.
You can debug whats going on with DirectX using PIX. Its a great tool and I highly recommend you familiarise yourself with it.
I'm currently working on a XNA game prototype. I'm trying to achieve a isometric view of the game world (or is it othographic?? I'm not sure which is the right term for this projection - see pictures).
The world should a tile-based world made of cubic tiles (e.g. similar to Minecraft's world), and I'm trying to render it in 2D by using sprites.
So I have a sprite sheet with the top face of the cube, the front face and the side (visible side) face. I draw the tiles using 3 separate calls to drawSprite, one for the top, one for the side, one for the front, using a source rectangle to pick the face I want to draw and a destination rectangle to set the position on the screen according to a formula to convert from 3D world coordinates to isometric (orthographic?).
(sample sprite:
)
This works good as long as I draw the faces, but if I try to draw fine edges of each block (as per a tile grid) I can see that I get a random rendering pattern in which some lines are overwritten by the face itself and some are not.
Please note that for my world representation, X is left to right, Y is inside screen to outside screen, and Z is up to down.
In this example I'm working only with top face-edges. Here is what I get (picture):
I don't understand why some of the lines are shown and some are not.
The rendering code I use is (note in this example I'm only drawing the topmost layers in each dimension):
/// <summary>
/// Draws the world
/// </summary>
/// <param name="spriteBatch"></param>
public void draw(SpriteBatch spriteBatch)
{
Texture2D tex = null;
// DRAW TILES
for (int z = numBlocks - 1; z >= 0; z--)
{
for (int y = 0; y < numBlocks; y++)
{
for (int x = numBlocks - 1; x >=0 ; x--)
{
myTextures.TryGetValue(myBlockManager.getBlockAt(x, y, z), out tex);
if (tex != null)
{
// TOP FACE
if (z == 0)
{
drawTop(spriteBatch, x, y, z, tex);
drawTop(spriteBatch, x, y, z, outlineTexture);
}
// FRONT FACE
if(y == numBlocks -1)
drawFront(spriteBatch, x, y, z, tex);
// SIDE FACE
if(x == 0)
drawSide(spriteBatch, x, y, z, tex);
}
}
}
}
}
private void drawTop(SpriteBatch spriteBatch, int x, int y, int z, Texture2D tex)
{
int pX = OffsetX + (int)(x * TEXTURE_TOP_X_OFFRIGHT + y * TEXTURE_SIDE_X);
int pY = OffsetY + (int)(y * TEXTURE_TOP_Y + z * TEXTURE_FRONT_Y);
topDestRect.X = pX;
topDestRect.Y = pY;
spriteBatch.Draw(tex, topDestRect, TEXTURE_TOP_RECT, Color.White);
}
I tried using a different approach, creating a second 3-tiers nested for loop after the first one, so I keep the top face drawing in the first loop and the edge highlight in the second loop (I know, this is inefficient, I should also probably avoid having a method call for each tile to draw it, but I'm just trying to get it working for now).
The results are somehow better but still not working as expected, top rows are missing, see picture:
Any idea of why I'm having this problem? In the first approach it might be a sort of z-fighting, but I'm drawing sprites in a precise order so shouldn't they overwrite what's already there?
Thanks everyone
Whoa, sorry guys I'm an idiot :) I started the batch with SpriteBatch.begin(SpriteSortMode.BackToFront) but I didn't use any z-value in the draw.
I should have used SpriteSortMode.Deferred! It's now working fine. Thanks everyone!
Try tweaking the sizes of your source and destination rectangles by 1 or 2 pixels. I have a sneaking suspicion this has something to do with the way these rectangles are handled as sort of 'outlines' of the area to be rendered and a sort of off-by-one problem. This is not expert advice, just a fellow coder's intuition.
Looks like a sub pixel precision or scaling issue. Also try to ensure your texture/tile width/height is a power of 2 (32, 64, 128, etc.) as that could make the effect less bad as well. It's really hard to tell just from those pictures.
I don't know how/if you scale everything, but you should try to avoid rounding wherever possible (especially inside your drawTop() method). Every time you round some position/coordinate chances are good you might increase the error/random offsets. Try to use double (or better: float) coordinates instead of integer.