I am using libvips to rotate the images. I am using a VM that have 3002 MB Ram and 512MB temp storage.The AWS Lambda Machine.
The command I running to rotate images is
vips rot original.jpg rotated.jpg d90
It throwing the following error
Exit Code: 1, Error Output: ERROR: wbuffer_write: write failed unix error: No space left on device
The jpg image is arround 10Mb.
Here's how libvips will rotate your jpg image.
90 degree rotate requires random access to the image pixels, but JPEG images can only be read strictly top-to-bottom, so as a first step, libvips has to unpack the JPG to a random access format. It uses vips (.v) format for this, which is pretty much a C array with a small header.
For images under 100mb (you can change this value, see below) decompressed, it will unpack to a memory buffer. For images over 100mb decompressed, it will unpack to a temporary file in /tmp (you can change this, see below).
Next, it does the rotate to the output image. It can do this as a single streaming operation, so it will typically need enough memory for 256 scanlines on the input image, and 256 on the output, so around another 30mb or so in this case, plus some more working area for each thread.
In your specific case, the input image is being decompressed to a temporary file of 30,000 x 10,000 x 3 bytes, or about 900mb. This is way over the 512mb you have in /tmp, so the operation fails.
The simplest solution is to force the loader to load via a memory buffer. If I try:
$ vipsheader x.jpg
x.jpg: 30000x10000 uchar, 3 bands, srgb, jpegload
$ time vips rot x.jpg y.jpg d90 --vips-progress --vips-leak
vips temp-3: 10000 x 30000 pixels, 8 threads, 128 x 128 tiles, 256 lines in buffer
vips x.jpg: 30000 x 10000 pixels, 8 threads, 30000 x 16 tiles, 256 lines in buffer
vips x.jpg: done in 0.972s
vips temp-3: done in 4.52s
memory: high-water mark 150.43 MB
real 0m4.647s
user 0m5.078s
sys 0m8.418s
The leak and progress flags make vips report some stats. You can see the initial decompress to the temporary file is talking 0.97s, the rotate to the output is 4.5s, it needs 150mb of pixel buffers and 900mb of disc.
If I raise the threshold, I see:
$ time VIPS_DISC_THRESHOLD=1gb vips rot x.jpg y.jpg d90 --vips-progress --vips-leak
vips temp-3: 10000 x 30000 pixels, 8 threads, 128 x 128 tiles, 256 lines in buffer
vips x.jpg: 30000 x 10000 pixels, 8 threads, 30000 x 16 tiles, 256 lines in buffer
vips x.jpg: done in 0.87s
vips temp-3: done in 1.98s
memory: high-water mark 964.79 MB
real 0m2.039s
user 0m3.842s
sys 0m0.443s
Now the second rotate phase is only 2s since it's just reading memory, but memory use has gone up to around 1gb.
This system is introduced in the libvips docs here:
http://jcupitt.github.io/libvips/API/current/How-it-opens-files.md.html
Related
I used Apple Numbers (a Spreadsheet app with styling options) to create a UX flowchart of various user interfaces of an app.
Apple Numbers has a PDF export option.
The problem is that even though some border lines in the table have been set to "none" in the export you nevertheless get small visible hairlines, see this cutout:
[
I want to to eliminate the hairlines by image processing
Before creating a flyover video over the graphics.
My basic idea is:
Convert vector to bitmap with very high resolution (oversampling, e.g. to 600 or 1200 DPI)
Then downsample to the target resolution (e.g. 150 DPI) with an algorithm which eliminates the hairlines (disappearing in the dominance of neighboring pixels) while overally still remaining as crisp and sharp as possible.
So step 1, I already figured out, by these two possibilities:
a. Apple Preview has a PDF to PNG export option where you can specify the DPI.
b. ImageMagick convert -density 600 source.pdf export.png
But for step 2 there are so many possibilities:
resample <DPI> or -filter <FilterName> -resize 25% or -scale 12.5% (when from 1200 to 150)
Please tell me by which methods (resample, resize, scale) and which of the interpolation algorithms or filters I shall use to achieve my goal of eliminating the hairlines by dissolving them into their neighboring pixels, with the rest (normal 1px lines, rendered text and symbols, etc) remaining as crisp as possible.
ImageMagick PDF tp PNG conversion with different DPI settings:
convert -density XXX flowchart.pdf flowchart-ImageMagick-XXX.png
flowchart-ImageMagick-150.png ; flowchart-ImageMagick-300.png ; flowchart-ImageMagick-600.png
Apple Preview PDF to PNG export with different DPI settings:
flowchart-ApplePreview-150.png ; flowchart-ApplePreview-300.png ; flowchart-ApplePreview-600.png
Different downscaling processings
a) convert -median 3x3 -resize 50% flowchart-ApplePreview-300.png flowchart-150-from-ApplePreview-300-median-3x3.png thanks to the hint from #ChristophRackwitz
b) convert -filter Box -resize 25% flowchart-ImageMagick-600.png flowchart-150-from-ImageMagick-600-resize-box.png
Comparison
flowchart-ApplePreview-150.png
flowchart-150-from-ApplePreview-300-median-3x3.png
✅ Hairlines gone
❌ But font is not as crisp anymore, median destroyed that.
flowchart-150-from-ImageMagick-600-resize-box.png
🆗 Overally still quite crisp
🆗 Hairline only very very faint, even only faint when zoomed in
Both variants are somehow good enough for my KenBurns / Dolly cam ride over them. Still I wished that there'd be an algorithm that keeps cripness but still eliminates 1px lines in very high DPI bitmaps. But I guess this is a Jack of all trades only in my phantasy.
Processing Durations
MacBook Pro 15'' (Mid 2014, 2,5 GHz Quad-Core Intel Core i7)
ImageMagick PDF to PNG
PDF source Ca. 84x60cm (33x23'')
300dpi -> 27s
600dpi -> 1m58s
1200dpi -> 37m34s
ImageMagic Downscaling
time convert -filter Box -resize 25% 1#600.png 1#150-from-600.png
# PNG # 39700 × 28066: 135.57s user 396.99s system 109% cpu 8:08.08 total
time convert -median 3x3 -resize 50% 2#300.png 2#150-from-300-median3x3.png
# PNG # 19850 × 14033: 311.48s user 9.42s system 536% cpu 59.76 total
time convert -median 3x3 -resize 50% 3#300.png 3#150-from-300-median3x3.png
# PNG # 19850 × 14033: 237.13s user 8.33s system 544% cpu 45.05 total
Assume I convert an 8-bit TIFF image to 16-bit with the following ImageMagick command:
$ convert 8bit-image.tif -depth 16 16bit-image.tif
The result is a file that is detected by other programs as a file with 16-bit depth:
$ identify 16bit-image.tif
16bit-image.tif TIFF 740x573 740x573+0+0 16-bit sRGB 376950B 0.000u 0:00.000
Naturally, this file does not have "true" 16 bit, since it's an 8 bit file which has simply been marked as 16 bit. It hasn't got the subtle nuances one would expect from true 16 bit. How can I distinguish a true 16 bit image from one that just "pretends"?
Best,
Bela
When you have an 8-bit image, the pixel values range from 0 to 255. For a 16-bit image, the pixel range is from 0 to 65535. So you can express more nuances in 16 bit than you can in 8 bit.
Usually, when you have a 16-bit imager in a camera, it is able to capture these nuances and map them to the full 16 bit range. An 8 bit imager will be limited to a smaller range, so when taking the picture, some information is lost compared to the 16 bit imager.
Now when you start out with an 8 bit image, that information is already lost, so converting to 16 bit will not give you greater nuance, because ImageMagick cannot invent information where there is none.
What image processing tools usually do is to fill copy the pixel values of your 8 bit image into the 16 bit image, so your 16 bit image will still contain only values in the range of [0,255]. If this is the case in your example, you can check whether the brightest pixel of your 16 bit image is greater than 255. If it is, you can assume that it is a native 16 bit image. If it isn't, it's likely that it was converted from 8 bit.
However, there is not a guarantee that the 16 bit image was really converted from 8 bit, as it could simply be a very dark native 16 bit image that only uses the darkest pixels from the 8 bit range by chance.
Edit: It is possible that someone converts the 8-bit image to 16 bit using the full 16 bit range. This could mean that a pixel of value 0 might remain at 0, a pixel at 255 might now be at 65535 and all values inbetween will be evenly distributed to the 16 bit range.
However, since no new information can be invented, there will be gaps in the pixel values used, e.g. you might have pixels of value 0, 255, 510 and so on, but values in between do not occur.
Depeding on the algorithm used for stretching the pixel range, these specific values may differ, but you would be able to spot a conversion like that by looking at the image' histogram:
It will have a distinctive comb-like structure (image taken from http://www.northlight-images.co.uk/digital-black-and-white-working-in-16-bit/)
So depending on how the conversion from 8 to 16 bit is executed, finding out whether it is a native image or not may be a bit more complicated and even then it can not be guaranteed to robustly determine whether the image was actually converted or not.
I would like to create an image with a single image tiled in width and height with graphicsmagick .
I tried this command which work :
gm montage -geometry 2x2 mypic.png mypic.png mypic.png out.png
However, I would like to repeat this pattern image a great number of time (over 100x100).
Is it possible to make that without repeating mypic.png 10000 times ? ?
I do not know GraphicsMagick. But I assume it is similar to ImageMagick, since it was a spin-off from ImageMagick. In ImageMagick, you can do that easily in two ways:
Input:
montage lena.jpg -duplicate 24 -tile 5x5 -geometry +0+0 result.jpg
convert -size 1280x1280 tile:lena.jpg result2.jpg
See the various ways to do tiling at https://imagemagick.org/Usage/canvas/#tile
I am not sure if GraphicsMagick has -duplicate, since that was introduce in ImageMagick 6.6.8-10 3/27/2011 long after they split-off.
ImageMagick has many more features than GraphicsMagick, but may be slightly slower. You may want to consider using ImageMagick rather than GraphicMagick
You don't say how large the images you are planning to make are, but if they are very large, you could run into a couple of issues.
First, JPEG is limited to 65536 x 65536 pixels, so you'll need something like bigtiff or PNG if you need larger than that.
Secondly, you can need huge amounts of memory to compose large images. For example, on this laptop I can run:
$ time convert -size 50000x50000 tile:k2.jpg result.jpg
real 6m11.366s
user 1m19.671s
sys 0m20.836s
to makes a 50k x 50k pixel JPG in about 6m.
convert will assemble the whole image before it starts writing the result. If you don't have bucketloads of RAM, it'll use a huge temporary file instead. If I look in /tmp during processing, I see:
$ ls -l /tmp
total 1199684
-rw------- 1 john john 20000000000 Dec 1 15:56 magick-9559WtN2jwPlvrMm
A 20gb temporary file. That's 50000 * 50000 * 4 * 2, so it's making a 16-bit, four channel temporary image. Because convert is spending all its time blocked in disc IO, it's rather slow.
You could consider other systems -- libvips is a streaming image processing library, so it can execute commands like this without having to make complete intermediate images. I see:
$ time vips replicate k2.jpg result.jpg 35 25
real 0m13.592s
user 0m16.383s
sys 0m1.426s
$ vipsheader result.jpg
result.jpg: 50750x51200 uchar, 3 bands, srgb, jpegload
That's copying k2.jpg 35 times horizontally and 25 times vertically to make an image slightly larger than 50k x 50k. It does not make a temporary file, and finishes in about 15 seconds. It'll have no problems going to very, very large output images -- I regularly process images of 300,000 x 300,000 pixels (though not in jpg format, obviously).
I am using libpng to convertraw image data (3 channel, 8 bit, no metadata) to PNG and store it in a buffer. I now have a problem to allocate the right amount of buffer space for writing the PNG data to it. It is clear to me, that the compressed data might be larger than the raw data (cf. the overhead for a 1x1 image)
Is there any general rule for an upper margin of the compressed data size with respect to the image size and the different filtering/compression options? If that is too generic, let's say we use PNG_COLOR_TYPE_RGB, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_FILTER_TYPE_DEFAULT.
Thank you
PNG overhead is 8 (signature) + 25 (IHDR) +12 (first IDAT) + 12 (IEND) plus 1 byte per row (filter byte), plus 12 bytes per additional IDAT when the size exceeds the zlib buffer size which is typically 8192. Zlib overhead is 6 (2-byte header and 4-byte checksum). Deflate overhead is 5 bytes plus 5 bytes per additional 32k in size.
So figure (1.02 * (3*W+1) * H) + 68.
You can decrease the 1.02 factor if you use a larger Zlib buffer size or increase it if you use a smaller buffer size. For example, a 256x256 RGB PNG compressed with a 1000000-byte buffer size (1000000 bytes per IDAT chunk) will have only one IDAT chunk and the total overhead will be around 330 bytes, or less than .2 percent, while if you compress it with a very small buffer size, for example 100 bytes, then there will be around 2000 IDAT chunks and the overhead will be about twelve percent.
See RFC-1950, RFC-1951, and RFC-2083.
You can use compressBound() in zlib to determine an upper bound on the size of the compressed data given an uncompressed data length, assuming the default zlib settings. For a specific set of different zlib settings, you can use deflateBound() after deflateInit2() has been used to establish the settings.
How to interpret texture memory information output by deviceQuery sample to know texture memory size?
Here is output of my texture memory.
Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65535),3D=(2048,2048,2048)
Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
It is a common misconception, but there is no such thing as "texture memory" in CUDA GPUs. There are only textures, which are global memory allocations accessed through dedicated hardware which has inbuilt cache, filtering and addressing limitations which lead to the size limits you see reported in the documentation and device query. So the limit is either roughly the free amount of global memory (allowing for padding and alignment in CUDA arrays) or the dimensional limits you already quoted.
The output shows that the maximum texture dimensions are:
For 1D textures 65536
For 2D textures 65536*65535
For 3D textures 2048*2048*2048
If you want the size in bytes, multiply that by the maximum number of channels (4) and the maximum sub-pixel size (4B).
(For layered textures, multiply the relevant numbers you got for the dimensions by the number of maximum layers you got.)
However, this is the maximum size for a single texture, not the available memory for all textures.