I have 100 PNG-files and each of them is 8250x4090 big. I need to append them with Imagemagick to one big PNG-file (82500 x 40900) so that I have 10 rows and 10 columns . I know how the code must look like but I get the errors: convert.exe: unable to extend cache
`C:\Row_345.png': No space left on device # error/cache.c/OpenPixelCache/3689.
convert.exe: Memory allocation failed `C:\Row_345.png' # error/png.c/WriteOnePNGImage/8725.
First question: How much space is needed (approximately)? I have 8 GB of Ram and 30 GB free SSD and it wasn't enough. The pictures have polygons and lines in up to 5 different colors. The biggest PNG is 300 KB)
Second question: Is there a way how to make it more clever so that it won't use that much space?
ImageMagick needs 8 bytes per pixel if you are using a Q16 build. A Q8 build only needs 4 bytes per pixel.
82500 * 40900 * 8 = about 27Gbytes
82500 * 40900 * 4 = about 13.5 Gbytes
The size of the PNG is irrelevant; ImageMagick stores them uncompressed.
Possibly ImageMagick is trying to hold two copies -- your 100 small images plus the large result. It may be that you'll have enough memory plus disk to run your conversion with ImageMagick-Q8.
Try doing just a single row of 10 at a time, ten times - so you get 10 rows of 10. Then do row1 plus row2. Then rows 1&2 plus row 3.
convert 1.png 2.png 3.png ... +append row1.png
convert 11.png 12.png 13.png ... +append row2.png
...
convert 91.png 92.png 93.png ... +append row10.png
Then
convert row1.png row2.png -append row1and2.png
You can add -debug cache to your ImageMagick convert command like this:
convert -debug cache 1.png 2.png 3.png ... +append row1.png
You can also look at your resource settings as to what is available to ImageMagick like this:
identify -list resource
File Area Memory Map Disk Thread Time
-------------------------------------------------------------------------------
768 1.0386GB 3.8692GiB 7.7384GiB unlimited 4 unlimited
And increase resources like this:
convert -limit memory 32MiB ...
Related
I am new to Python and am trying to parallelize a program that I somehow pieced together from the internet. The program reads all image files (usually multiple series of images such as abc001,abc002...abc015 and xyz001,xyz002....xyz015) in a specific folder and then combines images in a specified range. Most times, the number of files exceeds 10000, and my latest case requires me to combine 24000 images. Could someone help me with:
Taking 2 sets of images from different directories. Currently I have to move these images into 1 directory and then work in said directory.
Reading only specified files. Currently my program reads all files, saves names in an array (I think it's an array. Could be a directory also) and then uses only the images required to combine. If I specify a range of files, it still checks against all files in the directory and takes a lot of time.
Parallel Processing - I work with usually 10k files or sometimes more. These are images saved from the fluid simulations that I run at specific times. Currently, I save about 2k files at a time in separate folders and run the program to combine these 2000 files at one time. And then I copy all the output files to a separate folder to keep them together. It would be great if I could use all 16 cores on the processor to combine all files in 1 go.
Image series 1 is like so.
Consider it to be a series of photos of the cat walking towards the camera. Each frame is is suffixed with 001,002,...,n.
Image series 1 is like so.
Consider it to be a series of photos of the cat's expression changing with each frame. Each frame is is suffixed with 001,002,...,n.
The code currently combines each frame from set1 and set2 to provide output.png as shown in the link here.
import sys
import os
from PIL import Image
keywords=input('Enter initial characters of image series 1 [Ex:Scalar_ , VoF_Scene_]:\n')
keywords2=input('Enter initial characters of image series 2 [Ex:Scalar_ , VoF_Scene_]:\n')
directory = input('Enter correct folder name where images are present :\n') # FOLDER WHERE IMAGES ARE LOCATED
result1 = {}
result2={}
name_count1=0
name_count2=0
for filename in os.listdir(directory):
if keywords in filename:
name_count1 +=1
result1[name_count1] = os.path.join(directory, filename)
if keywords2 in filename:
name_count2 +=1
result2[name_count2] = os.path.join(directory, filename)
num1=input('Enter initial number of series:\n')
num2=input('Enter final number of series:\n')
num1=int(num1)
num2=int(num2)
if name_count1==(num2-num1+1):
a1=1
a2=name_count1
elif name_count2==(num2-num1+1):
a1=1
a2=name_count2
else:
a1=num1
a2=num2+1
for x in range(a1,a2):
y=format(x,'05') # '05' signifies number of digits in the series of file name Ex: [Scalar_scene_1_00345.png --> 5 digits], [Temperature_section_2_951.jpg --> 3 digits]. Change accordingly
y=str(y)
for comparison_name1 in result1:
for comparison_name2 in result2:
test1=result1[comparison_name1]
test2=result2[comparison_name2]
if y in test1 and y in test2:
a=test1
b=test2
test=[a,b]
images = [Image.open(x) for x in test]
widths, heights = zip(*(i.size for i in images))
total_width = sum(widths)
max_height = max(heights)
new_im = Image.new('RGB', (total_width, max_height))
x_offset = 0
for im in images:
new_im.paste(im, (x_offset,0))
x_offset += im.size[0]
output_name='output'+y+'.png'
new_im.save(os.path.join(directory, output_name))
I did a Python version as well, it's not quite as fast but it is maybe closer to your heart :-)
#!/usr/bin/env python3
import cv2
import numpy as np
from multiprocessing import Pool
def doOne(params):
"""Append the two input images side-by-side to output the third."""
imA = cv2.imread(params[0], cv2.IMREAD_UNCHANGED)
imB = cv2.imread(params[1], cv2.IMREAD_UNCHANGED)
res = np.hstack((imA, imB))
cv2.imwrite(params[2], res)
if __name__ == '__main__':
# Build the list of jobs - each entry is a tuple with 2 input filenames and an output filename
jobList = []
for i in range(1000):
# Horizontally append a-XXXXX.png to b-XXXXX.png to make c-XXXXX.png
jobList.append( (f'a-{i:05d}.png', f'b-{i:05d}.png', f'c-{i:05d}.png') )
# Make a pool of processes - 1 per CPU core
with Pool() as pool:
# Map the list of jobs to the pool of processes
pool.map(doOne, jobList)
You can do this a little quicker with libvips. To join two images left-right, enter:
vips join left.png out.png result.png horizontal
To test, I made 200 pairs of 1200x800 PNGs like this:
for i in {1..200}; do cp x.png left$i.png; cp x.png right$i.png; done
Then tried a benchmark:
time parallel vips join left{}.png right{}.png result{}.png horizontal ::: {1..200}
real 0m42.662s
user 2m35.983s
sys 0m6.446s
With imagemagick on the same laptop I see:
time parallel convert left{}.png right{}.png +append result{}.png ::: {1..200}
real 0m55.088s
user 3m24.556s
sys 0m6.400s
You can do that much faster without Python, and using multi-processing with ImageMagick or libvips.
The first part is all setup:
Make 20 images, called a-000.png ... a-019.png that go from red to blue:
convert -size 64x64 xc:red xc:blue -morph 18 a-%03d.png
Make 20 images, called b-000.png ... b-019.png that go from yellow to magenta:
convert -size 64x64 xc:yellow xc:magenta -morph 18 b-%03d.png
Now append them side-by-side into c-000.png ... c-019.png
for ((f=0;f<20;f++))
do
z=$(printf "%03d" $f)
convert a-${z}.png b-${z}.png +append c-${z}.png
done
Those images look like this:
If that looks good, you can do them all in parallel with GNU Parallel:
parallel convert a-{}.png b-{}.png +append c-{}.png ::: {1..19}
Benchmark
I did a quick benchmark and made 20,000 images a-00000.png...a-019999.png and another 20,000 images b-00000.png...b-019999.png with each image 1200x800 pixels. Then I ran the following command to append each pair horizontally and write 20,000 output images c-00000.png...c-019999.png:
seq -f "%05g" 0 19999 | parallel --eta convert a-{}.png b-{}.png +append c-{}.png
and that takes 16 minutes on my MacBook Pro with all 12 CPU cores pegged at 100% throughout. Note that you can:
add spacers between the images,
write annotation onto the images,
add borders,
resize
if you wish and do lots of other processing - this is just a simple example.
Note also that you can get even quicker times - in the region of 10-12 minutes if you accept JPEG instead of PNG as the output format.
android version upgrade to 5.x, I got a png image through screenshot.
when I use imagemagick to resize it, I got an error like this
identify: Extra compressed data. `a.png' # warning/png.c/MagickPNGWarningHandler/1777.
identify: Extra compression data. `a.png' # warning/png.c/MagickPNGWarningHandler/1777.
convert: Extra compressed data. `a.png' # warning/png.c/MagickPNGWarningHandler/1777.
convert: Extra compression data. `a.png' # warning/png.c/MagickPNGWarningHandler/1777.
use pngcheck info:
File: a.png (251221 bytes)
chunk IHDR at offset 0x0000c, length 13
1440 x 2560 image, 24-bit RGB, non-interlaced
chunk sBIT at offset 0x00025, length 3
red = 8 = 0x08, green = 8 = 0x08, blue = 8 = 0x08
chunk IDAT at offset 0x00034, length 251149
zlib: deflated, 32K window, maximum compression
chunk IEND at offset 0x3d54d, length 0
No errors detected in a.png (4 chunks, 97.7% compression).
My soft version:
ImageMagick 6.7.9-0
PNG* PNG rw- Portable Network Graphics (libpng 1.2.52)
See http://www.libpng.org/ for details about the PNG format.
PNG24* PNG rw- opaque 24-bit RGB (zlib 1.2.8,1.2.1.2)
I tryed ImageMagick-6.9.1-6 and libpng-1.6.17 zlib-1.2.8, all are newest version, but got error yet
please try this png image , and help me...
http://pan.baidu.com/s/1mg3mkCo
thanks
Add the option -quiet to your convert or identify command so you don't get an error generated.
"Extra compressed data" is just a warning. You may ignore it.
The result of your "convert" operation should be OK; ImageMagick won't propagate the extra data.
Note that "pngcheck" doesn't decompress the IDAT chunks so it won't report about the extra data that is evidently due to a bug in the screenshot application.
I'm trying to convert a 16 bit greyscale PNG to a raw file. The image size is 640*480.
First, identify:
$ identify image.png
image.png PNG 640x480 640x480+0+0 16-bit PseudoClass 65536c 299KB 0.000u 0:00.000
I'm expecting the result file to be 640*480*2 bytes in size.
Attempt 1:
$ convert image.png -depth 16 image.raw
This gives a file size of 330805 bytes. Its first 16 bytes look like:
0x00000000: 89504E47 0D0A1A0A 0000000D 49484452 .PNG........IHDR
Attempt 2:
$ convert image.png -depth 16 image.rgb
This gives a file size of 1843200 bytes, which is 640*480*2*3.
I'm running imagemagick version 6.7.7-10 on Ubuntu 14.04.
Any ideas?
Updated Answer
It occurred to me since answering you, that there is a simpler method of doing what you want, that takes advantage of ImageMagick's little-used stream tool, to stream raw pixel data around.
In effect, you can use this command
stream -map r -storage-type short image.png image.raw
which will read the Red channel (-map r), which is the same as the Green and Blue channels if your image is greyscale, and write it out as unsigned 16-bit shorts (-storage-type short) to the output file image.raw.
This is cleaner than my original answer - though should give identical results.
Original Answer
If you write an RGB raw file, you will get 3 channels - R, G and B. Try writing a PGM (Portable Greymap) like this...
convert image.png -depth 16 pgm:-
P5
640 480
65535
<binary data> < binary data>
The PGM format is detailed here, but suffice to say that there is header with a P followed by a digit describing the actual subtype, then a width and height and then a MAX VALUE that describes the range of the pixel intensities. In your case, the MAX VALUE is 65535 rather than 255 because your data are 16-bit.
You can the strip the header like this:
convert image.png -depth 16 pgm:- | tail -c 614400 > file.raw
If you are converting lots of files of different sizes and dislike the hard-coded 614400, and are using bash, you can get ImageMagick to tell you the size (height * width * 2 bytes/pixel) and use that like this:
bytes=$(identify -format "%[fx:h*w*2]" image.png)
convert image.png -depth 16 pgm:- | tail -c $bytes > file.raw
gray might be the format you want:
convert image.png -depth 16 image.gray
This command stores each pixel in 2 bytes and nothing else in the file.
Here I provide a minimal synthetic example: https://superuser.com/questions/294270/how-to-view-raw-binary-data-as-an-image-with-given-width-and-height/978432#978432
.raw is not really a "pixel only" format: it does contain some metadata: https://en.wikipedia.org/wiki/Raw_image_format#File_contents
In this post, i was wondering why my png files were badly displayed on retina displays.
I finaly found that the problem came from the PNG file itself: when I open it and save it again with photoshop or something else, the problem disapear.
As this post proposed, I used sips command to see what exactly were formed my PNG file. I have the original-image.png (with the glitch) and the photoshoped-image.png
The command
sips original-image.png -g all
Gives me
pixelWidth: 256
pixelHeight: 256
typeIdentifier: public.png
format: png
formatOptions: default
dpiWidth: 72.000
dpiHeight: 72.000
samplesPerPixel: 3
bitsPerSample: 8
hasAlpha: no
space: RGB
And
sips photoshoped-image.png -g all
Gives me
pixelWidth: 256
pixelHeight: 256
typeIdentifier: public.png
format: png
formatOptions: default
dpiWidth: 72.000
dpiHeight: 72.000
samplesPerPixel: 4
bitsPerSample: 8
hasAlpha: yes
space: RGB
profile: HD 709-A
So 3 differences :
samplePerPixel
hasAlpha
the photoshoped file has a profile.
But these properies are read-only in sips and I wonder how can I change them to understand exactly where the bug comes from.
Any idea ?
So using sips you can output a different file. Take the photoshop file and start modifying it. First remove the profile, then remove the alpa channel (which will affect the first two variables).
Its quite possible that this image works. PNG has many options, and the original image may have some other feature not visible using these tools. Photoshop is obviously re-writing the image completely, using the RGB values as the only common attribute between the files.
I suspect that when you do the above, that image will work too. There is just something odd about the originals.
In any case, you make it easier on iOS if you use pngs with an alpha channel, as it will convert them to have one if the base image does not have one.
On some files, this works:
sips -s format png '/Volumes/HD/Optimized PNG/TXT - Section Depth copy.png' --out '/Volumes/HD/Optimized PNG/TXT - Section Depth copy-.PNG'
/Volumes/HD/Optimized PNG/TXT - Section Depth copy.png
/Volumes/HD/Optimized PNG/TXT - Section Depth copy-.PNG
mis-bhayward61p-swk:~ zav$
But also, sometimes it doesn't:
sips -s format png --setProperty hasAlpha 0 '/Volumes/HD/Optimized PNG/Subsection copy 2/Section Depth Text.png' --out '/Volumes/HD/Optimized PNG/Subsection copy 2/Section Depth Text-.PNG'
/Volumes/HD/Optimized PNG/Subsection copy 2/Section Depth Text.png
Error: Cannot do --setProperty hasAlpha on file
/Volumes/HD/Optimized PNG/Subsection copy 2/Section Depth Text-.PNG
mis-bhayward61p-swk:~ zav$
Hope this gets you a little farther.
I have a large Tiff image that I want to chop into 512x512 tiles and write to disk.
In the past I've used ImageMagick like so:
convert -crop 512x512 +repage image_in.tif image_out_%d.tif
But recently this hasn't been working, processes running out of memory, etc.
Is there a similar command in VIPS? I know there's a CLI but I can't find an example or useful explanation in the documentation, and I'm still trying to figure out the nip2 GUI thing. Any help appreciated. :)
libvips has a operator which can do this for you very quickly. Try:
$ vips dzsave wtc.tif outdir --depth one --tile-size 512 --overlap 0 --suffix .tif
That's the DeepZoom writer making a depth 1 pyramid of tif tiles. Look in outdir_files/0 for the output tiles. There's a chapter in the docs talking about how to use dzsave.
It's a lot quicker than IM for me:
$ time convert -crop 512x512 +repage huge.tif x/image_out_%d.tif
real 0m5.623s
user 0m2.060s
sys 0m2.148s
$ time vips dzsave huge.tif x --depth one --tile-size 512 --overlap 0 --suffix .tif
real 0m1.643s
user 0m1.668s
sys 0m1.000s
Where huge.tif is a 10,000 by 10,000 pixel uncompressed RGB image. Plus it'll process any size image in only a small amount of memory.
I am running into the same issue. It seems that VIPS does not have a built-in command like the one from imagemagick above, but you can do this with some scripting (Python-code snippet):
for x in xrange(0, tiles_per_row):
xoffset = x * tile_size
for y in xrange(0, tiles_per_row):
yoffset = y * tile_size
filename = "%d_%d_%d.png" % (zoom, x, y)
command = "vips im_extract_area %s %s %d %d %d %d" % (base_image_name, filename, xoffset, yoffset, tile_size, tile_size)
os.system(command)
However you won't get the same speed as with imagemagick cropping...