I have a problem I just discovered in chrome and used to work. Uploaded video textures to webgl from screen capture MediaStream is failing with a diagonal green line and the picture is diagonal.
If I choose to share a browser tab its a correct picture and format.
I'm not sure if it's a problem with the buffer array setup or something else. It is ok with Webcam MediaStream input. I had a feature to overlay webcam in the foreground with screen capture as the background texture but it is broken now.
The jsfiddle is here
https://jsfiddle.net/danrossi303/txpqrsgv/2/
The texture upload format is
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, screenVideo);
I figured out screen capture supplies odd dimensions. And according to Kronos the texture will produce a fault if its not even divisible by 4. It started working magically, but I came up with a simple method to format the dimensions using bitwise modulus.
https://jsfiddle.net/danrossi303/txpqrsgv/12/
function formatTextureSize(value) {
return value + (value & (4 - 1));
}
Related
I am updating some textures of the scene all the time by new images.
Problem is uploading is synchronous and texImage2D takes ~100ms. It takes so long time even if texture is not used during rendering of the next frame or rendering is switched off.
I am wondering, is there any way to upload texture data asynchronously?
Additional conditions:
I had mention there is old texture which could stay active until uploading of new one to GPU will be finished.
Solution is to use texSubImage2D and upload image to GPU by small portions. Once uploading will be finished activate your new texture and delete old one.
is there any way to upload texture data asynchronously?
no, not in WebGL 1.0. There might be in WebGL 2.0 but that's not out yet.
Somethings you might try.
make it smaller
What are you uploading? Video? Can you make it smaller?
Have you tried different formats?
WebGL converts from whatever format the image is stored in to the format you request. So for example if you load a .JPG the browser might make an RGB image. If you then upload it with gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE it has to convert the image to RGBA before uploading (more time).
Do you have UNPACK_FLIP_Y set to true?
If so WebGL has to flip your image before uploading it.
Do you have UNPACK_COLORSPACE_CONVERSION_WEBGL set to BROWSER_DEFAULT_WEBGL?
If not WebGL may have to re-decompress your image
Do you have UNPACK_PREMULTIPLY_ALPHA_WEBGL set to false or true?
Depending on how the browser normally stores images it might have
to convert the image to the format your requesting
Images have to be decompressed
are you sure your time is in "uploading" vs "decompressing"? If you switch to uploading a TypedArray of the same dimensions does it speed up?
I use Core Video texture cache for my OpenGL textures. I have an issue with rendering such textures in case of minification. Parameter GL_TEXTURE_MIN_FILTER has no effect: interpolation for minification is always the same as GL_TEXTURE_MAG_FILTER. The interesting fact is that everything works ok when I create pixel buffer object with CVPixelBufferCreateWithBytes function. The problem appears when I use CVPixelBufferCreate.
Environment:
iOS 7
OpenGL ES 2.0
iPad mini, iPad 3, iPad 4.
I've developed simple application which demonstrates this issue: https://github.com/Gubarev/iOS-CVTextureCache. The Demo application can render checkerboard (size of cell is 1x1) texture in three modes:
Regular OpenGL texture (ok).
Core Video texture, pixel buffer created with CVPixelBufferCreate (problem).
Core Video texture, pixel buffer created with CVPixelBufferCreateWithBytes (ok).
Texture is rendered two times with slight minification (achieved by using OpenGL viewport smaller than texture):
Left image rendered with minification filter GL_NEAREST, magnification filter GL_NEAREST.
Right image rendered with minification filter GL_LINEAR, magnification filter GL_NEAREST.
Image below demonstrates proper minificiation in case of regular OpenGL texture. It's clearly visible that setting for minificiation filter takes effect. Same results could be obtained when "CVPixelBufferCreateWithBytes" approach is used. The problem appear in case of "CVPixelBufferCreate" approach: both images minificated with setting for magnification filer (GL_NEAREST in particular).
I want to load 4-channel texture data from a file in iOS, so I consider the texture as a (continuous) map
[0,1]x[0,1] -> [0,1]x[0,1]x[0,1]x[0,1]
If I use the fileformat .png, XCode/iOS consider the file as an image, and so multiplies each component rgb with a (premultiplied alpha), corrupting my data. How should I solve this? Examples may be
use two textures with components rgb (3-channel)
postdivide alpha
use another file format
Of these, I consider the best solution to be to use another file format. The GL-compressed file format (PVRTC?) is not Apple-platform independent and seems to be of low resolution (4 bits) (reference).
EDIT:
If my own answer below is true, it is not possible to get the 4 channel data of png's in iOS. Since OpenGL is about creating images rather than presenting images, it should be possible to load 4-channel data in some way. png is a fileformat for images (and compression depends on all 4 channels but compression of one channel is independent of the other channels), so one may argue that I should use another file format. So which other compressed file formats should I use, which is easy to read/integrated in iOS?
UPDATE: "combinatorial" mentioned a way to load 4-channel non-premultiplied textures, so I had to give him the correct answer. However, that solution had some restrictions I didn't like. My next question is then "Access raw 4-channel data from png files in iOS" :)
I think it is a bad library design not making it possible to read 4 channel png data. I don't like systems trying to be smarter than myself.
As you considered PVRTC then using GLKit could be an option. This includes GLKTextureLoader which allows you to load textures without pre-multiplying alpha. Using for example:
+ (GLKTextureInfo *)textureWithContentsOfFile:(NSString *)fileName options:(NSDictionary *)textureOperations error:(NSError **)outError
and passing an options dictionary containing:
GLKTextureLoaderApplyPremultiplication = NO
You can simply request that Xcode not 'compress' your PNG files. Click your project in the top left, select the 'Build Settings', find 'Compress PNG Files' and set the option to 'No'.
As to your other options, postdividing isn't a bad solution but obviously you'll lose overall precision and I believe both TIFF and BMP are also supported. PVRTC is PowerVR specific so it's not Apple-specific but it's also not entirely platform independent and is specifically designed to be a lossy compression that's trivial to uncompress with little input on the GPU. You'd generally increase your texture resolution to ameliorate for the low bit per pixel count.
You should use libpng to load PNG without premultiplied colors.
It is written in C and should compile for iOS.
I've had similar problems with Android and also had to use third-party library to load PNG files with non-premultiplied colors.
This is an attempt to answer my own question.
It is not possible to load non-premultiplied .png files.
The option kCGImageAlphaLast is a valid option, but does not give a valid combination for CGBitmapContextCreate (reference). It is however a valid option for CGImageRef's.
What the build setting COMPRESS_PNG_FILES in XCode mentioned above does, is to convert .png files into some other file format and also multiply the channels rgb with a (reference). I was hoping that disabling this option would make it possible to reach the channel data in my actual .png files. But I am not sure if this is possible. The following example is an attempt to access the .png data at low level, as a CGImageRef:
void test_cgimage(const char* path)
{
CGDataProviderRef dataProvider = CGDataProviderCreateWithFilename(path);
CGImageRef cg_image = CGImageCreateWithPNGDataProvider(dataProvider, NULL, NO,
kCGRenderingIntentDefault);
CGImageAlphaInfo info = CGImageGetAlphaInfo(cg_image);
switch (info)
{
case kCGImageAlphaNone: printf("kCGImageAlphaNone\n"); break;
case kCGImageAlphaPremultipliedLast: printf("kCGImageAlphaPremultipliedLast\n"); break;
case kCGImageAlphaPremultipliedFirst: printf("kCGImageAlphaPremultipliedFirst\n"); break;
case kCGImageAlphaLast: printf("kCGImageAlphaLast\n"); break;
case kCGImageAlphaFirst: printf("kCGImageAlphaFirst\n"); break;
case kCGImageAlphaNoneSkipLast: printf("kCGImageAlphaNoneSkipLast\n"); break;
case kCGImageAlphaNoneSkipFirst: printf("kCGImageAlphaNoneSkipFirst\n"); break;
default: break;
}
}
which gives "kCGImageAlphaPremultipliedLast" with COMPRESS_PNG_FILES disabled. So I think iOS always convert .png files, even at run-time.
There is better solution, faster(about 3x~5x) and cross platform
#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"
// force 4 channel rgba, force flipy
// image: first pixel is top left, OpenGL assume first is bottom left, so need flipy
bool flipy = true;
int width, height, nrChannels;
stbi_set_flip_vertically_on_load(flipy);
unsigned char *imageData = stbi_load(path.c_str(), &width, &height, &nrChannels, 4);
// load data to your texture
// glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, bytes);
free(imageData);
Not 100% what you want, but I got around the problem using this approach: Put the alpha channel into a separate black & white png and save the original png without alpha. So space taken is about the same. Then in my texture loader load both images and combine into one texture.
I know this is a only a workaround, but at least it gives the correct result. And yes, it is very annoying, that iOS does not allow you to load textures from PNG without premultiplied alpha.
Hello people of the wasteland :),
Brief: There is a problem with GL_RGB internal texture format on iOS platform.
In my application I try to save some memory by using GL_RGB instead of GL_RGBA as an internal format.
I'm using the next code piece to achieve this. Nothing else is changed.
glTexImage2D(_textureTargetType,
0,
GL_RGB, // pixel internalFormat
texWidth, // image width
texHeight, // image height
0, // border
GL_RGBA, // pixel format
GL_UNSIGNED_BYTE, // pixel data type
bitmapData);
On MacOS these changes went fluently, no problems. But on iOS, particularly 4.3 (OpenGL ES2.0) it gives me GL_INVALID_OPERATION everytime I try to render textured polgons with this texture. As nothing except this format is changed I think the problem is in incompatibility of GL_RGB internal format with OpenGL ES2.0. This is just my guess, I'm no guru.
This doesn't work in simulator nor iPod touch 4th gen.
Thank you for any reasonable suggestion.
According to the documentation, "internalformat must match format. No conversion between formats is supported during texture image processing." See the Khronos website. OpenGL does not have this limitation, so this code will work on Mac OS, but not the more limited OpenGL ES on iOS devices.
Edit:ok, sorry, I had a simple programming error, is there a way to delete this question?
I have some compressed textures that are PVR files, but I cant seem to draw them in my iPad application using OpenGL ES.
I can draw PNG files just fine, I know the PVR files are being loaded correctly.
Are there some special OpenGL draw functions that I need to be calling to draw the PVR files?
Edit:All I get is a white image.
Any info is appreciated.
Drawing PVRTC textures should be exactly the same as any other texture format - it looks more likely that your loading code is the problem. Are any GL errors being reported during loading?
The major difference to loading uncompressed textures are in the line:
glCompressedTexImage2D(GL_TEXTURE_2D, level, GL_COMPRESSED_RGBA_PVRTC_4BPPV1_IMG, width, height, 0, size, data);
or
glCompressedTexImage2D(GL_TEXTURE_2D, level, GL_COMPRESSED_RGB_PVRTC_4BPPV1_IMG, width, height, 0, size, data);
Make sure that you're not setting a GL filter mode to use MIPmaps if they're not in the texture as well.
Searching for PVRTC in Apple's docs brings up a decent summary of how to use these textures.
After upload, PVR textures are no different from other formats. Did you forgot to skip header during data upload, or used wrong parameters for glCompressedTexImage2D? It is even possible that compression tool was unable to convert images because of wrong size or color format.