I have a SKTextureAtlas with about 90 PNG Images. Every Image has a resolution of 2000 x 70 pixel and has a size of ~1 KB.
Now I put this images from the Atlas into an array like this:
var dropBarAtlas = SKTextureAtlas(named: "DropBar")
for i in 0..<dropBarAtlas.textureNames.count{
var textuteName = NSString(format: "DropBar%i", i)
var texture = dropBarAtlas.textureNamed(textuteName)
dropFrames.addObject(texture)
}
Then I preload the array with the textures in didMoveToView:
SKTexture.preloadTextures(dropFrames, withCompletionHandler: { () -> Void in})
To play the animation with 30 fps I use SKAction.animateWithTextures
var animateDropBar = SKAction.animateWithTextures(dropFrames, timePerFrame: 0.033)
dropBar.runAction(animateDropBar)
My problem is that when I preload the textures the memory usage increases to about 300 MB.
Is there a more performant solution?
And which frame rate and image size is recommended for SKAction.animateWithTextures?
You should keep in mind that image file size (1Kb in your example) have nothing with amount of memory required for same image to be stored in RAM . You can calculate that amount of memory required with this formula:
width x height x bytes per pixel = size in memory
If you using standard RGBA8888 pixel format this means that your image will require about 0.5 megabytes in RAM memory, because RGBA8888 uses 4 bytes per pixel – 1 byte for each red, green, blue, and 1 byte for alpha transparency. You can read more here.
So what you can do, is to optimize you textures and use different texture formats. Here is another example about texture optimization.
Related
I am resizing this test picture:
Mat im = Mat::zeros(Size(832*3,832*3),CV_8UC3);
putText(im,"HI THERE",Point2i(10,90),1,7,Scalar(255,255,255),2);
by standard
cv::resize(im,out,Size(416,416),0,0,INTER_NEAREST);
and by CUDA version of resize:
static void gpuResize(Mat in, Mat &out){
double k = in.cols/416.;
cuda::GpuMat gpuInImage;
cuda::GpuMat gpuOutImage;
gpuInImage.upload(in);
const Size2i &newSize = Size(416, in.rows / k);
//cout << "newSize " << newSize<< endl;
cuda::resize(gpuInImage, gpuOutImage, newSize,INTER_NEAREST);
gpuOutImage.download(out);
}
Measuring time shows that cv::resize is ~25 times faster. What am I doing wrong? I on GTX1080ti videocard, but also observe same situation on Jetson NANO. May be there are any alternative methods to resize image faster then cv::resize with nvidia hardware acceleration?
I was doing similar things today, and had the same results on my Jetson NX running in the NVP model 2 mode (15W, 6 core).
Using the CPU to resize an image 10,000 times was faster than resizing the same image 10,000 times with the GPU.
This was my code for the CPU:
cv::Mat cpu_original_image = cv::imread("test.png"); // 1400x690 RGB image
for (size_t count = 0; count < number_of_times_to_iterate; count ++)
{
cv::Mat cpu_resized_image;
cv::resize(cpu_original_image, cpu_resized_image, desired_image_size);
}
This was my code for the GPU:
cv::cuda::GpuMat gpu_original_image;
gpu_original_image.upload(cpu_original_image);
for (size_t count = 0; count < number_of_times_to_iterate; count ++)
{
cv::cuda::GpuMat gpu_resized_image;
cv::cuda::resize(gpu_original_image, gpu_resized_image, desired_image_size);
}
My timing code (not shown above) was only for the for() loops, it didn't include imread() nor upload().
When called in a loop 10K times, my results were:
CPU: 5786.930 milliseconds
GPU: 9678.054 milliseconds (plus an additional 170.587 milliseconds for the upload())
Then I made 1 change to each loop. I moved the "resized" mat outside of the loop to prevent it from being created and destroyed at each iteration. My code then looked like this:
cv::Mat cpu_original_image = cv::imread("test.png"); // 1400x690 RGB image
cv::Mat cpu_resized_image;
for (size_t count = 0; count < number_of_times_to_iterate; count ++)
{
cv::resize(cpu_original_image, cpu_resized_image, desired_image_size);
}
...and for the GPU:
cv::cuda::GpuMat gpu_original_image;
gpu_original_image.upload(cpu_original_image);
cv::cuda::GpuMat gpu_resized_image;
for (size_t count = 0; count < number_of_times_to_iterate; count ++)
{
cv::cuda::resize(gpu_original_image, gpu_resized_image, desired_image_size);
}
The for() loop timing results are now:
CPU: 5768.181 milliseconds (basically unchanged)
GPU: 2827.898 milliseconds (from 9.7 seconds to 2.8 seconds)
This looks much better! GPU resize is now faster than CPU resize...as long as you're doing lots of work with the GPU and not a single resize. And as long as you don't continuously re-allocate temporary GPU mats, as that seems to be quite expensive.
But after all this, to go back to your original question: if all you are doing is resizing a single image once, or resizing many images once each, the GPU resize won't help you since uploading each image to the GPU mat will take longer than the original resize! Here are my results when trying that on a Jetson NX:
single image resize on CPU: 3.565 milliseconds
upload mat to GPU: 186.966 milliseconds
allocation of 2nd GPU mat and gpu resize: 225.925 milliseconds
So on the CPU the NX can do it in < 4 milliseconds, while on the GPU it takes over 400 milliseconds.
For example, for my 940M video card, the canvas created with the following code takes 500 MB of video memory
var c = document.createElement('canvas');
var ctx = c.getContext('webgl');
c.width = c.height = 4096;
At the same time, the OpenGL context of the same sizes uses only 100 MB of video memory:
glutInit(&argc, argv);
glutInitDisplayMode(GLUT_SINGLE);
int s = 4096;
glutInitWindowSize(s, s);
glutCreateWindow("Hello world :D");
Why does the WebGL use so much memory? Is it possible to reduce the amount of used memory for the same sizes of the context?
As LJ pointed out, canvas is double buffered, antialiased, has alpha and a depth buffer by default. You made the canvas 4096 x 4096 so that's
16meg * 4 (RGBA) or 64meg for one buffer
You get that times at least 4
front buffer = 1
antialiased backbuffer = 2 to 16
depth buffer = 1
So that's 256meg to 1152meg depending on what the browser picks for antialiasing.
In answer to your question you can try to not ask for a depth buffer, alpha buffer and/or antialiasing
var c = document.createElement('canvas');
var ctx = c.getContext('webgl', { alpha: false, depth: false, antialias: false});
c.width = c.height = 4096;
Whether the browser actually doesn't allocate an alpha channel or does but just ignores it is up to the browser and driver. Whether it will actually not allocate a depth buffer is also up to the browser. Passing antialias: false should at least make the 2nd buffer 1x instead of 2x to 16x.
I got an image from a bigger image by
let partialCGImage = CGImageCreateWithImageInRect(CGImage, frame)
but sometimes I got wrong RGBA value. For example, I calculated the average red values of an image, but it turned out like a gray image.
So I checked the info as follow.
image width: 64
image height: 64
image has 5120 bytes per row
image has 8 bits per component
image color space: <CGColorSpace 0x15d68fbd0> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB IEC61966-2.1)
image is mask: false
image bitmap info: CGBitmapInfo(rawValue: 8194)
image has 32 bits per pixel
image utt type: nil
image should interpolate: true
image rendering intent: CGColorRenderingIntent
Bitamp Info: ------
Alpha info mask: Ture
Float components: False
Byte oder mask: Ture
Byte order default: False
Byte order 16 little: False
Byte order 32 little: Ture
Byte order 16 big: Ture
Byte order 32 big: False
Image Info ended---------------
Then I got a really weird problem, why the width and height are both 64 pxs, and the image has 8 bits(1 byte) per component(4 bytes per pixel), but the bytes per row is 5120?
And I notice the bitmap info of the normal image is quite different, it doesn't has any byte order infomation.
I googled the different between little endian and big endian, but I got confused when they showed up together.
I really need help since my project has already delayed for 2 days because of that. Thanks!
By the way, I used following code to get the RGBA value.
let pixelData=CGDataProviderCopyData(CGImageGetDataProvider(self.CGImage))
let data:UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
var rs: [[Int]] = []
var gs: [[Int]] = []
var bs: [[Int]] = []
let widthMax = imageWidth
let heightMax = imageHeight
for indexX in 0...widthMax {
var tempR: [Int] = []
var tempG: [Int] = []
var tempB: [Int] = []
for indexY in 0...heightMax {
let offSet = 4 * (indexX * imageWidth + indexY)
**let r = Int(data[pixelInfo + offSet])
let g = Int(data[pixelInfo + 1 + offSet])
let b = Int(data[pixelInfo + 2 + offSet])**
tempR.append(r)
tempG.append(g)
tempB.append(b)
}
rs.append(tempR)
gs.append(tempG)
bs.append(tempB)
}
Ask me if you have problem with my code. Thank you for help.
The bytes-per-row is 5120 because you used CGImageCreateWithImageInRect on a larger image. From the CGImage reference manual:
The resulting image retains a reference to the original image, which means you may release the original image after calling this function.
The new image uses the same pixel storage as the old (larger) image. That's why the new image retains the old image, and why they have the same bytes-per-row.
As for why you're not getting the red values you expect: Rob's answer has some useful information, but if you want to explore deeper, consider that your bitmap info is 8194 = 0x2002.
print(CGBitmapInfo.ByteOrder32Little.rawValue | CGImageAlphaInfo.PremultipliedFirst.rawValue)
# Output:
8194
These bits determine the byte order of your bitmap. But those names aren't all that helpful. Let's figure out exactly what byte order we get for those bits:
let context = CGBitmapContextCreate(nil, 1, 1, 8, 4, CGColorSpaceCreateDeviceRGB(), CGBitmapInfo.ByteOrder32Little.rawValue | CGImageAlphaInfo.PremultipliedFirst.rawValue)!
UIGraphicsPushContext(context)
let d: CGFloat = 255
UIColor(red: 1/d, green: 2/d, blue: 3/d, alpha: 1).setFill()
UIRectFill(.infinite)
UIGraphicsPopContext()
let data = UnsafePointer<UInt8>(CGBitmapContextGetData(context))
for i in 0 ..< 4 {
print("\(i): \(data[i])")
}
# Output:
0: 3
1: 2
2: 1
3: 255
So we can see that a bitmap info of 8194 means that the byte order is BGRA. Your code assumes it's RGBA.
In addition to the question about pixelInfo, raised by Segmentation, the calculation of offSet seem curious:
let offSet = 4 * (indexX * imageWidth + indexY)
The x and y values are backwards. Also, you also cannot assume that the bytes per row is always equal to 4 times the width in pixels because some image formats pad bytes per row. Anyway, it theoretically it should be:
let offSet = indexY * bytesPerRow + indexX * bytesPerPixel
Also note that in addition to the x/y flip issue, you don't want 0 ... widthMax and 0 ... heightMax (as those will return widthMax + 1 and heightMax + 1 data points). Instead, you want to use 0 ..< widthMax and 0 ..< heightMax.
Also if you're dealing with random image files, there are other deeper problems here. For example, you can't make assumptions regarding RGBA vs ARGB vs CMYK, big endian vs little endian, etc., captured in the bitmap info field.
Rather than writing code that can deal with all of these variations in pixel buffers, Apple suggests alternative to take the image of some random configuration and render it to some consistent context configuration, and then you can navigate the buffer more easily. See Technical Q&A #1509.
First of all you haven't initialize pixelInfo variable. Second you aren't doing anything with the A value shifting everything 8bits to the left. Also i don't think you need pixelInfo and offset, these two variables are the same so keep one of them equal to what you wrote for offset
Similar to the SpriteKit Featured Game "Adventure" from WWDC, I am try to load my background image via tiles. I have created a Texture Atlas that contains 6,300 "tiles" that are each 100x100 pixels in size. The complete background image is a total of 30,000x2048 (for retina displays). The idea is that the background will move from right to left (side-scroller). The first column and the last column match so that they seem continuous.
When the application runs, it loads my initial loading screen and title images and spikes to 54MB in my memory tab with a CPU usage of 16%. This stays the same as I navigate through the menus until I choose my level, which tells a background thread to load the level assets (of which contains the aforementioned background image). The entire .atlas folder shows to be only 35.4MB. I don't believe that this is a problem since the Adventure .atlas folder (from WWDC) shows to be only 32.7MB.
Once I select the level, it loads approximately 20 of the textures in the .atlas folder before I start receiving memory warnings and it crashes the application. I've checked in Instruments for leaks and it doesn't show any memory leaks. I don't receive any compiler errors (not even the EXC_BAD_ACCESS one). I've looked at my device console and have found a few lines of where the application crashes, but it doesn't look to make much sense to me. I've also checked for Zombies, but haven't seemed to find any.
CoreLevel.m
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
// Used to determine time spent to load
NSDate *startDate = [NSDate date];
// Atlas to load
SKTextureAtlas *tileAtlas = [SKTextureAtlas atlasNamed:#"Day"];
// Make sure the array is empty, before storing the tiles
sBackgroundTiles = nil;
sBackgroundTiles = [[NSMutableArray alloc] initWithCapacity:6300];
// For each row (21 Totals Rows)
for (int y = 0; y < 21; y++) {
// For each Column (100 Total Columns)
for (int x = 1; x <= 100; x++) {
// Get the tile number (0 * 32) + 0;
int tileNumber = (y * 300) + x;
// Create a SpriteNode of that tile
SKSpriteNode *tileNode = [SKSpriteNode spriteNodeWithTexture:[tileAtlas textureNamed:[NSString stringWithFormat:#"tile_%d.png", tileNumber]]];
// Position the SpriteNode
CGPoint position = CGPointMake((x * 100), (y * 100));
tileNode.position = position;
// At layer
tileNode.zPosition = -1.0f;
tileNode.blendMode = SKBlendModeReplace;
// Add to array
[(NSMutableArray *)sBackgroundTiles addObject:tileNode];
}
}
NSLog(#"Loaded all world tiles in %f seconds", [[NSDate date] timeIntervalSinceDate:startDate]);
});
This is what seems to pertain to the crash from the Debug console:
com.apple.debugserver-300.2[9438] <Warning>: 1 +0.000000 sec [24de/1807]: error: ::read ( -1, 0x4069ec, 18446744069414585344 ) => -1 err = Bad file descriptor (0x00000009)
com.apple.debugserver-300.2[9438] <Warning>: Exiting.
com.apple.launchd[1] (UIKitApplication:tv.thebasement.Coin-Voyage[0x641d][9441]) <Notice>: (UIKitApplication:tv.thebasement.Coin-Voyage[0x641d]) Exited: Killed: 9
I don't have enough reputation to post images so here is a link to a screenshot screenshot of my allocations in Instruments:
http://postimg.org/image/j17xl39et/
Any help and advice is much appreciated! If I've left out some pertinent information, I'm glad to update.
The file size of an image file (PNG, JPG, atlas folder, etc) tells you nothing about the memory usage.
Instead you have to calculate the texture memory usage using the formula:
width * height * (color bit depth / 8) = texture size in bytes
For example an image with dimensions 4096x4096 pixels and 32 bits color depth (4 bytes) uses this much memory when loaded as a texture (uncompressed):
4096 * 4096 * 4 = 67108864 bytes (64 Megabytes)
According to your specs (6,300 tiles, each 100x100 pixels, assuming they're all unique) you're way, wayyyyyy above any reasonable limit for texture memory usage (about 1.5 Gigabytes!). Considering the atlas size of 35 Megabytes (which is huge for an atlas btw) and assuming a mere 10:1 compression ratio you may still be looking at 350+ Megabytes of texture memory usage.
I am trying to convert the camera captured image to 8 bit image. And that should be grayscale image.
I searched in forums but could able to find the way to convert to 8 bit image.
Any help or suggestion will be help ful to me.
Thanks....
You have given too little information. First of all, what is the image format your camera delivers? Is it some RAW format, jpeg, or what else?
Doing it programmatically (using C for the example):
The best way to go was to use some image loading library (e.g. SDL_image), and load the image into memory, uncompressed RGB being the target format. Once you have an uncompressed RGB image format, you could do something like
// bufPtr points to the start of the memory containing the bitmap
typedef unsigned char byte;
struct rgb { byte red, green blue; } * colorPtr = bufPtr;
for (int i = 0; i < bufSize; i++, bufPtr++) {
byte gray = (unsigned char) (((float) bufPtr->red * 0.3f +
(float) bufPtr->green * 0.59f +
(float) bufPtr->blue * 0.11f)) / 3.0f * 255.0f + 0.5f);
bufPtr->red = bufPtr->green = bufPtr->blue = gray;
}
If you don't want to code, you could e.g. use GIMP, load your image and apply desaturate from the color menu. You can install the ufraw plugin for GIMP to load images in RAW format in it. If you want to store the entire color information in 8 bits (and not use 8 bits per color channel), there is another option in GIMP to decrease the color depth.