We have plenty of png images in DAM. We want to compress these images in such a way that their quality/resolution should not compromise. I have gone through several options and want to know the best one.
ImageMagick
https://helpx.adobe.com/experience-manager/6-3/assets/using/best-practices-for-imagemagick.html
We are on AEM 6.0 and wondering whether this features will work well in 6.0 version. Moreover, This tool have some vulnerabilities as well.
DAM Update Asset Workflow
We can update the workflow to add our custom code block and by using JAVA7 IO api we can perform the compression. Also can we add a new process step at first stage so that can compress the images as first operation when this workflow initiate. Any other thoughts are welcome.
3rd Party API
Are there any third party API where using CURL we can compress the images apart from TinyPNG.
Have started using ImageMagick on 6.0:
EPS thumbnails (powered by ImageMagick) Step is as follows :
Web enabled rendition process step is as follows :
Problem :
Image size does not getting compress and size increased from 206KB to 208KB.
AEM Logs says :
13.04.2018 15:05:43.111 *INFO* [JobHandler: /etc/workflow/instances/server0/2018-04-13_1/update_asset_439:/content/dam/A.jpg/jcr:content/renditions/original] com.day.cq.dam.core.process.CommandLineProcess execute: executing command line ["C:\Program Files\ImageMagick-6.9.9-Q16\convert.exe" -depth 8 -define jpeg:size=319x319 A.jpg -thumbnail 319x319 cq5dam.thumbnail.319.319.png] for asset [/content/dam/A.jpg]. (Size is 208KB of thumbnail generated)
If I run Locally, I can see the file size reduced to 130KB.
"C:\Program Files\ImageMagick-6.9.9-Q16\convert.exe" -depth 8 -define jpeg:size=319x319 A.jpg -thumbnail 319x319 cq5dam.thumbnail.319.319.png
Any Idea why image in not compressing on AEM?
You need to write a servlet which will dynamically compress the size of the image such a way that the quality and resolution is maintained.
public class YourServlet extends AbstractImageServlet {
protected Layer createLayer(AbstractImageServlet.ImageContext imageContext)
throws RepositoryException, IOException {
Layer resized = ImageHelper.resize(layer, new Dimension(), new Dimension(0, 0),
new Dimension(768, 768));
}
protected void writeLayer(SlingHttpServletRequest request, SlingHttpServletResponse response,
AbstractImageServlet.ImageContext context, Layer layer, double quality)
throws IOException, RepositoryException {
super.writeLayer(request, response, context, layer, QUALITY);
}
}
Related
I want to automatically manipulate images with ImageMagick (or any other tool that can be run on an Ubuntu server or called via an API). In order to get exactly the result I want, I have tinkered with Photoshop and exported the XMP metadata to document the manipulation. It looks like this:
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.3-c007 1.136881, 2010/06/10-18:11:35 ">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
crs:Version="7.0"
crs:ProcessVersion="6.7"
crs:Sharpness="38"
crs:LuminanceSmoothing="0"
crs:VignetteAmount="-17"
crs:VignetteMidpoint="32"
crs:SplitToningShadowHue="0"
crs:SplitToningShadowSaturation="0"
crs:SplitToningHighlightHue="0"
crs:SplitToningHighlightSaturation="0"
crs:SplitToningBalance="-15"
crs:ParametricShadows="-14"
crs:ParametricDarks="-16"
crs:ParametricLights="+4"
crs:ParametricHighlights="+31"
crs:ParametricShadowSplit="25"
crs:ParametricMidtoneSplit="50"
crs:ParametricHighlightSplit="75"
crs:SharpenRadius="+0.5"
crs:SharpenDetail="100"
crs:SharpenEdgeMasking="17"
crs:PostCropVignetteAmount="0"
crs:GrainAmount="24"
crs:GrainSize="2"
crs:GrainFrequency="6"
crs:LensProfileEnable="0"
crs:LensManualDistortionAmount="-2"
crs:PerspectiveVertical="0"
crs:PerspectiveHorizontal="0"
crs:PerspectiveRotate="0.0"
crs:PerspectiveScale="100"
crs:Exposure2012="-0.05"
crs:Contrast2012="0"
crs:Highlights2012="+27"
crs:Shadows2012="+18"
crs:Whites2012="+29"
crs:Blacks2012="+1"
crs:Clarity2012="+6"
crs:ToneCurveName2012="Linear"
crs:LensProfileSetup="Auto"
crs:HasSettings="True">
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
<crs:ToneCurvePV2012Red>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012Red>
<crs:ToneCurvePV2012Green>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012Green>
<crs:ToneCurvePV2012Blue>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012Blue>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
Now I would like to apply exactly the same manipulation with a tool that can be run server-side. Is it possible ? Can I achieve exactly the same result with ImageMagick ? Does Photoshop expose an API, or maybe a SDK allowing me to do that ?
I think it would be a pretty massive ask of ImageMagick to achieve identical results, or even close, to those achieved by Photoshop.
Adobe puts massive development effort into ACR (Camera Raw) and things like lens profiles require vast amounts of testing and management of databases of lens characteristics on different cameras which are just not part of the remit or objective of ImageMagick.
Also, ImageMagick delegates the RAW processing to UFRAW and I don't believe that supports advanced features like "Clarity" and "Luminance Smoothing".
Photoshop does expose an API which is scriptable - Adobe Photoshop Scripting in JavaScript, AppleScript and VBScript.
One way to accomplish this for RAW photos is to use the free Adobe DNG Converter (available for both Windows and macOS, but the Windows version might run with wine).
When you ask it to convert a RAW to a DNG you can have it embed a full size JPEG preview in the DNG. It will respect whatever settings are in the XMP when generating the JPEG preview. You can ask it to convert a DNG to a second DNG as a way to update the embed preview. And it works from the command line.
Then use something like ExifTool to edit the XMP metadata, and to extract the embed JPEG preview.
I've used this with reasonable success to create a barebones RAW converter:
https://github.com/ncruces/RethinkRAW
I encountered a problem when I want to read an image using the OpenCV function imread().
The image is Ok and I can show it in the image display software.
But when I use the imdecode() to get the image data, the data returns NULL.
I will upload the image and the code and hope some one could help me
Mat img = imread(image_name);
if(!img.data) return -1;
The image's link is here: http://img3.douban.com/view/photo/raw/public/p2198361185.jpg
PS: The image_name is all right.
I guess OpenCV cannot decode this image. So is there any way to decode this image using OpenCV?, like add new decode library. By the way, I can read this image using other image library such as freeImage.
Your image is in .gif and it is not supported by OpenCV as of now.
Note OpenCV offers support for the image formats Windows bitmap (bmp),
portable image formats (pbm, pgm, ppm) and Sun raster (sr, ras). With
help of plugins (you need to specify to use them if you build yourself
the library, nevertheless in the packages we ship present by default)
you may also load image formats like JPEG (jpeg, jpg, jpe), JPEG 2000
(jp2 - codenamed in the CMake as Jasper), TIFF files (tiff, tif) and
portable network graphics (png). Furthermore, OpenEXR is also a
possibility.
Source - Click here
You can use something like this, to perform the conversion.
I was able to load your image using imread using this. Also, you can check out FreeImage.
You can also try to use the library gif2numpy. It converts a gif image to a numpy image which then can be loaded by OpenCV:
import cv2, gif2numpy
np_images, extensions, image_specs = gif2numpy.convert("yourgifimage.gif")
cv2.imshow("np_image", np_images[0])
cv2.waitKey()
The library can be found here: https://github.com/bunkahle/gif2numpy It is not dependent on PIL or pillow for this like imageio.
There are two methods to read an image in OpenCV, one is using Mat the other one using IplImage. I see you have used the former one. You can try with the second argument of imread also:
image = imread("image.jpg", CV_LOAD_IMAGE_COLOR); // Read the file
else use IplImage
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc_c.h"
#include <opencv2/core/core.hpp>
IplImage* src = 0;
if( (src = cvLoadImage("filename.jpg",1)) == 0 )
{
printf("Cannot load file image %s\n", filename);
}
If they don't work please check if you have installed libjpeg, libtiff and other dependencies for reading an image in OpenCV.
Hope it would help.
I have a Rails rake task that is processing a batch of images. It strips out the white background (using RMagick), replaces it with a transparent layer, writes it to a tempfile and then saves it as a PNG on Amazon S3 (using Paperclip).
It works for the bulk of the images. However, it runs into an error for at least 1 image. Can someone help me figure out why and how to fix it?
Code sample:
require 'RMagick'
require 'tempfile'
include Magick
task :task_name => :environment do
x = Item.find(128) # image 128 is the one giving me trouble
sourceImage = Image.read(x.image_link_hires)
processedImage = sourceImage[0].transparent("white")
tempImageFile = Tempfile.new(["processed_image",".png"])
processedImage.write("png:" + tempImageFile.path)
x.image_transparent = tempImageFile
x.save!
end
The error message:
rake aborted! Validation failed: Image transparent C:/Users/Roger/AppData/Local/Temp/processed_image20130107-8640-1ck71i820130107-8640-i6p91w.png is not recognized by the 'identify' command., Image transparent C:/Users/Roger/AppData/Local/Temp/processed_
image20130107-8640-1ck71i820130107-8640-i6p91w.png is not recognized by the 'identify' command.
This message appears upon running the last line (the save operation).
Tempfile problem with small files?
I think the error has something to do with Tempfile not actually writing a file to the temp path. This error may have to do with small filesize? The specific image that it's having trouble with has an usually amount of white space, so the resulting filesize after processing is about 30k for an 800x800 pixel image.
How can I verify if this is the case? And if it is, how can I work around it?
Other observations:
When I write the trouble image to a normal file (rather than Tempfile), it saves successfully locally.
The task works fine for other images, which tend to be much bigger (~1-2MB)
After processedImage.write, I've checked tempImageFile.size. It says that it's 30kb as expected.
When I observe the temp file directory when the rake task runs, I can see the temp files being created when the task is run other images successfully. The files seem to show up when processedImage.write runs. However, for the trouble image, I don't see temp files ever being created.
Thanks for any advice.
Update 7 Jan 2013
I've investigated this more. I reran #1 above, but attempted to save onto S3 with Paperclip. This generated the same error message.
So now I believe the issue is that this is a small file in terms of bytes (32kb), but with a decent height and width (800x800). Paperclip is trying to save a thumbnail version of it, which is 90x90. Typically this generates a filesize that is <1% the original, which I assume is the source of the errors.
If anyone has an elegant workaround / fix for this, I'd appreciate hearing about it.
OpenCV says something like
Corrupt JPEG data: premature end of data segment
or
Corrupt JPEG data: bad Huffman code
or
Corrupt JPEG data: 22 extraneous bytes before marker 0xd9
when loading a corrupt jpeg image with imread().
Can I somehow catch that? Why would I get this information otherwise?
Do I have to check the binary file on my own?
OpenCV (version 2.4) does not overwrite the basic error handling for libjpeg, making them 'uncatchable'. Add the following method to modules/highgui/src/grfmt_jpeg.cpp, right below the definition of error_exit():
METHODDEF(void)
output_message( j_common_ptr cinfo )
{
char buffer[JMSG_LENGTH_MAX];
/* Create the message */
(*cinfo->err->format_message) (cinfo, buffer);
/* Default OpenCV error handling instead of print */
CV_Error(CV_StsError, buffer);
}
Now apply the method to the decoder error handler:
state->cinfo.err = jpeg_std_error(&state->jerr.pub);
state->jerr.pub.error_exit = error_exit;
state->jerr.pub.output_message = output_message; /* Add this line */
Apply the method to the encoder error handler as well:
cinfo.err = jpeg_std_error(&jerr.pub);
jerr.pub.error_exit = error_exit;
jerr.pub.output_message = output_message; /* Add this line */
Recompile and install OpenCV as usual. From now on you should be able to catch libjpeg errors like any other OpenCV error. Example:
>>> cv2.imread("/var/opencv/bad_image.jpg")
OpenCV Error: Unspecified error (Corrupt JPEG data: 1137 extraneous bytes before marker 0xc4) in output_message, file /var/opencv/opencv-2.4.9/modules/highgui/src/grfmt_jpeg.cpp, line 180
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
cv2.error: /var/opencv/opencv-2.4.9/modules/highgui/src/grfmt_jpeg.cpp:180: error: (-2) Corrupt JPEG data: 1137 extraneous bytes before marker 0xc4 in function output_message
(I've submitted a pull request for the above but it got rejected because it would cause issues with people reading images without exception catching.)
Hope this helps anyone still struggling with this issue. Good luck.
It could be easier to fix the error in the file instead of trying to repair the loading function of OpenCV. If you are using Linux you can use ImageMagick to make reparation to a set of images (is usual to have it installed by default):
$ mogrify -set comment 'Image rewritten with ImageMagick' *.jpg
This command changes a property of the file leaving the image data untouched. However, the image is loaded and resaved, eliminating the extra information that causes the corruption error.
If you need more information about ImageMagick you can visit their website: http://www.imagemagick.org/script/index.php
You cannot catch it if you use imread(). However there is imdecode() function that is called by imread(). Maybe it gives you more feedback. For this you would have to load the image into memory on your own and then call the decoder.
It boils down to: You have to dig through the OpenCV sources to solve your problem.
i had to deal with this recently and found a solution over here
http://artax.karlin.mff.cuni.cz/~isa_j1am/other/opencv/
i just need to make 2 edits # $cv\modules\highgui\src\grfmt_jpeg.cpp.
--- opencv-1.0.0.orig/otherlibs/highgui/grfmt_jpeg.cpp 2006-10-16 13:02:49.000000000 +0200
+++ opencv-1.0.0/otherlibs/highgui/grfmt_jpeg.cpp 2007-08-11 09:10:28.000000000 +0200
## -181,7 +181,7 ##
m_height = cinfo->image_height;
m_iscolor = cinfo->num_components > 1;
- result = true;
+ result = (cinfo->err->num_warnings == 0);
}
}
## -405,8 +405,9 ##
icvCvt_CMYK2Gray_8u_C4C1R( buffer[0], 0, data, 0, cvSize(m_width,1) );
}
}
- result = true;
+
jpeg_finish_decompress( cinfo );
+ result = (cinfo->err->num_warnings == 0);
}
}
I am using opencv python package to read some image and also met this error message. This error can not be catch by Python. But if you want to find which image is corrupted without recompiling opencv as #Robbert suggested, you can try the following method.
First, you can pinpoint the directory where the corrupt images reside, which is fairly easy. Then you go to the directory, and use mogrify command line tool provided by ImageMagick to change the image meta info, as suggest by #goe.
mogrify -set comment "errors fixed in meta info" -format png *.jpg
The above command will convert the original jpg image to png format and also clean the original image to remove errors in meta info. When you run mogrify command, it will also output some message about which image is corrupted in the directory so that you can accurately find the corrupted image.
After that, you can do whatever you want with the original corrupted jpg image.
Any one stumbles upon this post and reads this answer.
I had to get hold of a corrupted image file.
These websites can help you corrupt your file
Corrupt a file - The file corrupter you were looking for!
CORRUPT A FILE ONLINE
Corrupt my File
First and the third website was not that much useful.
Second website is interesting as I could set the amount of file that I need to corrupt.
OpenCV version I used here is 3.4.0
I used normal cv2.imread(fileLocation)
fileLocation Location of corrupted image file
OpenCV didn't show any error message for any of the corrupted files used here
First and Third website only gave one file and both had None stored in them, when I tried to print them
Second website did let me decide the amount of file that was needed to be corrupted
Corruption% Opencv message on printing the image
4% None
10% None
25% None
50% None Corrupt JPEG data: 3 extraneous bytes before marker 0x4f
75% None Corrupt JPEG data: 153 extraneous bytes before marker 0xb2
100% Corrupt JPEG data: 330 extraneous bytes before marker 0xc6 None
I guess the only check we have to make here would be
if image is not None:
Do your code or else pop an error
You can redirect stderr to a file, then after imread, search for the string "Huffman" inside that file. After searching the file, empty it. It works for me and now I am able to discard corrupted images and just process good ones.
If you load your image with imdecode, you can check errno :
std::vector<char> datas();
//Load yout image in datas here
errno = 0;
cv::Mat mat = cv::imdecode(datas, -1);
if (errno != 0)
{
//Error
}
(tested on OpenCV 3.4.1)
I found that the issue is in libjpeg. If OpenCV uses it, it gets error
Corrupt JPEG data: 22 extraneous bytes before marker 0xd9
You can try my solution to solve it. It disables JPEG during compilation. After that OpenCV cannot read/write, but it works.
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_SHARED_LIBS=OFF -D BUILD_EXAMPLES=OFF -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D WITH_JPEG=OFF -D WITH_IPP=OFF ..
I found an easy solution without the need to recompile openCV.
You can use imagemagick to detect the same errors, however it returns an error as expected. See the description here: https://stackoverflow.com/a/66283167/2887398
Given a user uploaded image, I need to create various thumbnails of it for display on a website. I'm using ImageMagick and trying to make Google PageSpeed happy. Unfortunately, no matter what quality value I specify in the convert command, PageSpeed is still able to suggest compressing the image even further.
Note that http://www.imagemagick.org/script/command-line-options.php?ImageMagick=2khj9jcl1gd12mmiu4lbo9p365#quality mentions:
For the JPEG ... image formats,
quality is 1 [provides the] lowest
image quality and highest compression
....
I actually even tested compressing the image using 1 (it produced an unusable image, though) and PageSpeed still suggests that I can still optimize such image by "losslessly compressing" the image. I don't know how to compress an image any more using ImageMagick. Any suggestions?
Here's a quick way to test what I am talking about:
assert_options(ASSERT_BAIL, TRUE);
// TODO: specify valid image here
$input_filename = 'Dock.jpg';
assert(file_exists($input_filename));
$qualities = array('100', '75', '50', '25', '1');
$geometries = array('100x100', '250x250', '400x400');
foreach($qualities as $quality)
{
echo("<h1>$quality</h1>");
foreach ($geometries as $geometry)
{
$output_filename = "$geometry-$quality.jpg";
$command = "convert -units PixelsPerInch -density 72x72 -quality $quality -resize $geometry $input_filename $output_filename";
$output = array();
$return = 0;
exec($command, $output, $return);
echo('<img src="' . $output_filename . '" />');
assert(file_exists($output_filename));
assert($output === array());
assert($return === 0);
}
echo ('<br/>');
}
The JPEG may contain comments, thumbnails or metadata, which can be removed.
Sometimes it is possible to compress JPEG files more, while keeping the same quality. This is possible if the program which generated the image did not use the optimal algorithm or parameters to compress the image. By recompressing the same data, an optimizer may reduce the image size. This works by using specific Huffman tables for compression.
You may run jpegtran or jpegoptim on your created file, to reduce it further in size.
To minimize the image sizes even more, you should remove all meta data. ImageMagick can do this by adding a -strip to the commandline.
Have you also considered to put your thumbnail images as inline-d base64 encoded data into your HTML?
This can make your web page load much faster (even though the size gets a bit larger), because it saves the browser from running multiple requests for all the image files (the images) which are referenced in the HTML code.
Your HTML code for such an image would look like this:
<IMG SRC="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAAM4AAABJAQMAAABPZIvnAAAABGdBTUEAALGPC/xh
BQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAA
OpgAABdwnLpRPAAAAAZQTFRFAAAA/wAAG/+NIgAAAAF0Uk5TAEDm2GYAAAABYktH
RACIBR1IAAAACXBIWXMAAABIAAAASABGyWs+AAAB6ElEQVQ4y+3UQY7bIBQG4IeQ
yqYaLhANV+iyi9FwpS69iGyiLuZYpepF6A1YskC8/uCA7SgZtVI3lcoiivkIxu/9
MdH/8U+N6el2pk0oFyibWyr1Q3+PlO2NqJV+/BnRPMjcJ9zrfJ/U+zQ9oAvo+QGF
d+npPqFQn++TXElkrEpEJhAtlTBR6dNHUuzIMhFnEhxAmJDkKxlmt7ATXDDJYcaE
r4Txqtkl42VYSH+t9KrD9b5nxZeog/LWGVHprGInGWVQUTvjDWXca5KdsowqyGSc
DrZRlGlQUl4kQwpUjiSS9gI9VdECZhHFQ2I+UE2CHJQfkNxTNKCl0RkURqlLowJK
1h1p3sjc0CJD39D4BIqD7JvvpH/GAxl2/YSq9mtHSHknga7OKNOHKyEdaFC2Dh1w
9VSJemBeGuHgMuh24EynK03YM1Lr83OjUle38aVSfTblT424rl4LhdglsUag5RB5
uBJSJBIiELSzaAeIN0pUlEeZEMeClC4cBuH6mxOlgPjC3uLproUCWfy58WPN/MZR
86ghc888yNdD0Tj8eAucasl2I5LqX19I7EmEjaYjSb9R/G1SYfQA7ZBuT5H6WwDt
UAfK1BOJmh/eZnKLeKvZ/vA8qonCpj1h6djfbqvW620Tva36++MXUkNDlFREMVkA
AAAldEVYdGRhdGU6Y3JlYXRlADIwMTItMDgtMjJUMDg6Mzc6NDUrMDI6MDBTUnmt
AAAAJXRFWHRkYXRlOm1vZGlmeQAyMDEyLTA4LTIyVDA4OjM3OjQ1KzAyOjAwIg/B
EQAAAA50RVh0bGFiZWwAImdvb2dsZSJdcbX4AAAAAElFTkSuQmCC"
ALT="google" WIDTH=214 HEIGHT=57 VSPACE=5 HSPACE=5 BORDER=0 />
And you would create the base64 encoded image data like this:
base64 -i image.jpg -o image.b64
Google performs those calculations based on it's WebP image format (https://developers.google.com/speed/webp/).
Despite giving performance gains though, it is currently supported only by chrome and opera (http://caniuse.com/webp)