Google PageSpeed & ImageMagick JPG compression - image-processing

Given a user uploaded image, I need to create various thumbnails of it for display on a website. I'm using ImageMagick and trying to make Google PageSpeed happy. Unfortunately, no matter what quality value I specify in the convert command, PageSpeed is still able to suggest compressing the image even further.
Note that http://www.imagemagick.org/script/command-line-options.php?ImageMagick=2khj9jcl1gd12mmiu4lbo9p365#quality mentions:
For the JPEG ... image formats,
quality is 1 [provides the] lowest
image quality and highest compression
....
I actually even tested compressing the image using 1 (it produced an unusable image, though) and PageSpeed still suggests that I can still optimize such image by "losslessly compressing" the image. I don't know how to compress an image any more using ImageMagick. Any suggestions?
Here's a quick way to test what I am talking about:
assert_options(ASSERT_BAIL, TRUE);
// TODO: specify valid image here
$input_filename = 'Dock.jpg';
assert(file_exists($input_filename));
$qualities = array('100', '75', '50', '25', '1');
$geometries = array('100x100', '250x250', '400x400');
foreach($qualities as $quality)
{
echo("<h1>$quality</h1>");
foreach ($geometries as $geometry)
{
$output_filename = "$geometry-$quality.jpg";
$command = "convert -units PixelsPerInch -density 72x72 -quality $quality -resize $geometry $input_filename $output_filename";
$output = array();
$return = 0;
exec($command, $output, $return);
echo('<img src="' . $output_filename . '" />');
assert(file_exists($output_filename));
assert($output === array());
assert($return === 0);
}
echo ('<br/>');
}

The JPEG may contain comments, thumbnails or metadata, which can be removed.
Sometimes it is possible to compress JPEG files more, while keeping the same quality. This is possible if the program which generated the image did not use the optimal algorithm or parameters to compress the image. By recompressing the same data, an optimizer may reduce the image size. This works by using specific Huffman tables for compression.
You may run jpegtran or jpegoptim on your created file, to reduce it further in size.

To minimize the image sizes even more, you should remove all meta data. ImageMagick can do this by adding a -strip to the commandline.
Have you also considered to put your thumbnail images as inline-d base64 encoded data into your HTML?
This can make your web page load much faster (even though the size gets a bit larger), because it saves the browser from running multiple requests for all the image files (the images) which are referenced in the HTML code.
Your HTML code for such an image would look like this:
<IMG SRC="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAAM4AAABJAQMAAABPZIvnAAAABGdBTUEAALGPC/xh
BQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAA
OpgAABdwnLpRPAAAAAZQTFRFAAAA/wAAG/+NIgAAAAF0Uk5TAEDm2GYAAAABYktH
RACIBR1IAAAACXBIWXMAAABIAAAASABGyWs+AAAB6ElEQVQ4y+3UQY7bIBQG4IeQ
yqYaLhANV+iyi9FwpS69iGyiLuZYpepF6A1YskC8/uCA7SgZtVI3lcoiivkIxu/9
MdH/8U+N6el2pk0oFyibWyr1Q3+PlO2NqJV+/BnRPMjcJ9zrfJ/U+zQ9oAvo+QGF
d+npPqFQn++TXElkrEpEJhAtlTBR6dNHUuzIMhFnEhxAmJDkKxlmt7ATXDDJYcaE
r4Txqtkl42VYSH+t9KrD9b5nxZeog/LWGVHprGInGWVQUTvjDWXca5KdsowqyGSc
DrZRlGlQUl4kQwpUjiSS9gI9VdECZhHFQ2I+UE2CHJQfkNxTNKCl0RkURqlLowJK
1h1p3sjc0CJD39D4BIqD7JvvpH/GAxl2/YSq9mtHSHknga7OKNOHKyEdaFC2Dh1w
9VSJemBeGuHgMuh24EynK03YM1Lr83OjUle38aVSfTblT424rl4LhdglsUag5RB5
uBJSJBIiELSzaAeIN0pUlEeZEMeClC4cBuH6mxOlgPjC3uLproUCWfy58WPN/MZR
86ghc888yNdD0Tj8eAucasl2I5LqX19I7EmEjaYjSb9R/G1SYfQA7ZBuT5H6WwDt
UAfK1BOJmh/eZnKLeKvZ/vA8qonCpj1h6djfbqvW620Tva36++MXUkNDlFREMVkA
AAAldEVYdGRhdGU6Y3JlYXRlADIwMTItMDgtMjJUMDg6Mzc6NDUrMDI6MDBTUnmt
AAAAJXRFWHRkYXRlOm1vZGlmeQAyMDEyLTA4LTIyVDA4OjM3OjQ1KzAyOjAwIg/B
EQAAAA50RVh0bGFiZWwAImdvb2dsZSJdcbX4AAAAAElFTkSuQmCC"
ALT="google" WIDTH=214 HEIGHT=57 VSPACE=5 HSPACE=5 BORDER=0 />
And you would create the base64 encoded image data like this:
base64 -i image.jpg -o image.b64

Google performs those calculations based on it's WebP image format (https://developers.google.com/speed/webp/).
Despite giving performance gains though, it is currently supported only by chrome and opera (http://caniuse.com/webp)

Related

jpg images without extension aren't displayed

from kivy.uix.image import Image
self.img = Image(source="image") # This works when image is an PNG image
self.img = Image(source="image.jpg") # This works when image.jpg is a JPG image
self.img = Image(source="image") # This doesn't work when image is a JPG image
I need to specify images without extention for the app to be generic (working with more image types). Can I achieve it somehow?
Kivy is using "imghdr" to determine the image type here, and as a fallback it uses the file extension here.
That explains why the image loads fine when it has a file extension, even though "imghdr" can't find the file type in the file's content.
I tested on a list of JPEG files, and each time "imghdr" was able to detect the file type each time. That is done here im imghdr. Notably, "imghdr" does not consider the file extension.
$ python
>>> import os, imghdr
... for f in os.listdir('.'):
... print('%s -- %s' % (f, imghdr.what(f)))
Maybe the JPEG file is missing the "JFIF" or "Exif" string that imghdr is looking for? You could use hexedit to see if one of those string is present at Byte 6 of the image file.

Upload an image with my own size to vimeo

Is it possible to upload an image with my own size to vimeo?
For instance, I want my thumbnail_medium to be 400x400, but every time I am uploading the 400x400 I am getting the 200x150 after retrieving the video data by the url https://vimeo.com/api/v2/video/myVideoIdGoesHere.json.
Thank you.
Try replacing '200x150.webp' at the end of thumbnail_medium with '400x400'
That postfix is what determines the size. I'd recommend indexing on '_' and appending the dimensions you desire (the '.webp' is unnecessary). Something like this...
const newURL = oldURL.substring(0, oldURL.indexOf('_')) + '_400x400'
If you look at thumbnail_small, the only difference in the url is a change of the postfix

Image Compression In AEM

We have plenty of png images in DAM. We want to compress these images in such a way that their quality/resolution should not compromise. I have gone through several options and want to know the best one.
ImageMagick
https://helpx.adobe.com/experience-manager/6-3/assets/using/best-practices-for-imagemagick.html
We are on AEM 6.0 and wondering whether this features will work well in 6.0 version. Moreover, This tool have some vulnerabilities as well.
DAM Update Asset Workflow
We can update the workflow to add our custom code block and by using JAVA7 IO api we can perform the compression. Also can we add a new process step at first stage so that can compress the images as first operation when this workflow initiate. Any other thoughts are welcome.
3rd Party API
Are there any third party API where using CURL we can compress the images apart from TinyPNG.
Have started using ImageMagick on 6.0:
EPS thumbnails (powered by ImageMagick) Step is as follows :
Web enabled rendition process step is as follows :
Problem :
Image size does not getting compress and size increased from 206KB to 208KB.
AEM Logs says :
13.04.2018 15:05:43.111 *INFO* [JobHandler: /etc/workflow/instances/server0/2018-04-13_1/update_asset_439:/content/dam/A.jpg/jcr:content/renditions/original] com.day.cq.dam.core.process.CommandLineProcess execute: executing command line ["C:\Program Files\ImageMagick-6.9.9-Q16\convert.exe" -depth 8 -define jpeg:size=319x319 A.jpg -thumbnail 319x319 cq5dam.thumbnail.319.319.png] for asset [/content/dam/A.jpg]. (Size is 208KB of thumbnail generated)
If I run Locally, I can see the file size reduced to 130KB.
"C:\Program Files\ImageMagick-6.9.9-Q16\convert.exe" -depth 8 -define jpeg:size=319x319 A.jpg -thumbnail 319x319 cq5dam.thumbnail.319.319.png
Any Idea why image in not compressing on AEM?
You need to write a servlet which will dynamically compress the size of the image such a way that the quality and resolution is maintained.
public class YourServlet extends AbstractImageServlet {
protected Layer createLayer(AbstractImageServlet.ImageContext imageContext)
throws RepositoryException, IOException {
Layer resized = ImageHelper.resize(layer, new Dimension(), new Dimension(0, 0),
new Dimension(768, 768));
}
protected void writeLayer(SlingHttpServletRequest request, SlingHttpServletResponse response,
AbstractImageServlet.ImageContext context, Layer layer, double quality)
throws IOException, RepositoryException {
super.writeLayer(request, response, context, layer, QUALITY);
}
}

Imagemagick with greyscale images and 24 bit depth

I am using the Imagick library to work with ImageMagick in PHP. I am first reading an (JPEG) image from an external server with:
$img = new Imagick();
$img->readImage($source);
And then upload it to my Amazon S3 bucket with the following code:
$s3 = new AmazonS3();
$s3->create_object(BUCKET, $destination_path, array(
'body' => $img->getImageBlob(),
'length' => $img->getImageSize(),
'acl' => AmazonS3::ACL_PUBLIC,
'contentType' => 'image/jpeg'
));
Everything seems to be working fine, the files appear in my storage bucket and I can view them in my browser. However, when handling greyscale images, ImageMagick converts the image from 24 bit depth to 8 bit depth. I would like them to keep their 24 bit depth, how could I achieve this? I've tried the following, without success:
$img->setImageType(imagick::IMGTYPE_TRUECOLOR);
For colorized images, everything works fine, images keep their 24 bit depth.
Edit:
It seems that ImageMagick changes the image type from 6 (truecolor) to 2 (greyscale). Trying to overwrite this does not work, as tested with the following code:
$img = new Imagick();
$img->readImage($source);
$img->setImageType(6);
echo $img->getImageType();
which outputs 2
Use setType() before you load the image. So:
$img = new Imagick();
$img->setType(6); //or use imagick::IMGTYPE_TRUECOLOR instead of 6
$img->loadImage($source);
This will output an image with truecolor, even if the loaded picture uses graycolors only.

How can I catch corrupt JPEGs when loading an image with imread() in OpenCV?

OpenCV says something like
Corrupt JPEG data: premature end of data segment
or
Corrupt JPEG data: bad Huffman code
or
Corrupt JPEG data: 22 extraneous bytes before marker 0xd9
when loading a corrupt jpeg image with imread().
Can I somehow catch that? Why would I get this information otherwise?
Do I have to check the binary file on my own?
OpenCV (version 2.4) does not overwrite the basic error handling for libjpeg, making them 'uncatchable'. Add the following method to modules/highgui/src/grfmt_jpeg.cpp, right below the definition of error_exit():
METHODDEF(void)
output_message( j_common_ptr cinfo )
{
char buffer[JMSG_LENGTH_MAX];
/* Create the message */
(*cinfo->err->format_message) (cinfo, buffer);
/* Default OpenCV error handling instead of print */
CV_Error(CV_StsError, buffer);
}
Now apply the method to the decoder error handler:
state->cinfo.err = jpeg_std_error(&state->jerr.pub);
state->jerr.pub.error_exit = error_exit;
state->jerr.pub.output_message = output_message; /* Add this line */
Apply the method to the encoder error handler as well:
cinfo.err = jpeg_std_error(&jerr.pub);
jerr.pub.error_exit = error_exit;
jerr.pub.output_message = output_message; /* Add this line */
Recompile and install OpenCV as usual. From now on you should be able to catch libjpeg errors like any other OpenCV error. Example:
>>> cv2.imread("/var/opencv/bad_image.jpg")
OpenCV Error: Unspecified error (Corrupt JPEG data: 1137 extraneous bytes before marker 0xc4) in output_message, file /var/opencv/opencv-2.4.9/modules/highgui/src/grfmt_jpeg.cpp, line 180
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
cv2.error: /var/opencv/opencv-2.4.9/modules/highgui/src/grfmt_jpeg.cpp:180: error: (-2) Corrupt JPEG data: 1137 extraneous bytes before marker 0xc4 in function output_message
(I've submitted a pull request for the above but it got rejected because it would cause issues with people reading images without exception catching.)
Hope this helps anyone still struggling with this issue. Good luck.
It could be easier to fix the error in the file instead of trying to repair the loading function of OpenCV. If you are using Linux you can use ImageMagick to make reparation to a set of images (is usual to have it installed by default):
$ mogrify -set comment 'Image rewritten with ImageMagick' *.jpg
This command changes a property of the file leaving the image data untouched. However, the image is loaded and resaved, eliminating the extra information that causes the corruption error.
If you need more information about ImageMagick you can visit their website: http://www.imagemagick.org/script/index.php
You cannot catch it if you use imread(). However there is imdecode() function that is called by imread(). Maybe it gives you more feedback. For this you would have to load the image into memory on your own and then call the decoder.
It boils down to: You have to dig through the OpenCV sources to solve your problem.
i had to deal with this recently and found a solution over here
http://artax.karlin.mff.cuni.cz/~isa_j1am/other/opencv/
i just need to make 2 edits # $cv\modules\highgui\src\grfmt_jpeg.cpp.
--- opencv-1.0.0.orig/otherlibs/highgui/grfmt_jpeg.cpp 2006-10-16 13:02:49.000000000 +0200
+++ opencv-1.0.0/otherlibs/highgui/grfmt_jpeg.cpp 2007-08-11 09:10:28.000000000 +0200
## -181,7 +181,7 ##
m_height = cinfo->image_height;
m_iscolor = cinfo->num_components > 1;
- result = true;
+ result = (cinfo->err->num_warnings == 0);
}
}
## -405,8 +405,9 ##
icvCvt_CMYK2Gray_8u_C4C1R( buffer[0], 0, data, 0, cvSize(m_width,1) );
}
}
- result = true;
+
jpeg_finish_decompress( cinfo );
+ result = (cinfo->err->num_warnings == 0);
}
}
I am using opencv python package to read some image and also met this error message. This error can not be catch by Python. But if you want to find which image is corrupted without recompiling opencv as #Robbert suggested, you can try the following method.
First, you can pinpoint the directory where the corrupt images reside, which is fairly easy. Then you go to the directory, and use mogrify command line tool provided by ImageMagick to change the image meta info, as suggest by #goe.
mogrify -set comment "errors fixed in meta info" -format png *.jpg
The above command will convert the original jpg image to png format and also clean the original image to remove errors in meta info. When you run mogrify command, it will also output some message about which image is corrupted in the directory so that you can accurately find the corrupted image.
After that, you can do whatever you want with the original corrupted jpg image.
Any one stumbles upon this post and reads this answer.
I had to get hold of a corrupted image file.
These websites can help you corrupt your file
Corrupt a file - The file corrupter you were looking for!
CORRUPT A FILE ONLINE
Corrupt my File
First and the third website was not that much useful.
Second website is interesting as I could set the amount of file that I need to corrupt.
OpenCV version I used here is 3.4.0
I used normal cv2.imread(fileLocation)
fileLocation Location of corrupted image file
OpenCV didn't show any error message for any of the corrupted files used here
First and Third website only gave one file and both had None stored in them, when I tried to print them
Second website did let me decide the amount of file that was needed to be corrupted
Corruption% Opencv message on printing the image
4% None
10% None
25% None
50% None Corrupt JPEG data: 3 extraneous bytes before marker 0x4f
75% None Corrupt JPEG data: 153 extraneous bytes before marker 0xb2
100% Corrupt JPEG data: 330 extraneous bytes before marker 0xc6 None
I guess the only check we have to make here would be
if image is not None:
Do your code or else pop an error
You can redirect stderr to a file, then after imread, search for the string "Huffman" inside that file. After searching the file, empty it. It works for me and now I am able to discard corrupted images and just process good ones.
If you load your image with imdecode, you can check errno :
std::vector<char> datas();
//Load yout image in datas here
errno = 0;
cv::Mat mat = cv::imdecode(datas, -1);
if (errno != 0)
{
//Error
}
(tested on OpenCV 3.4.1)
I found that the issue is in libjpeg. If OpenCV uses it, it gets error
Corrupt JPEG data: 22 extraneous bytes before marker 0xd9
You can try my solution to solve it. It disables JPEG during compilation. After that OpenCV cannot read/write, but it works.
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_SHARED_LIBS=OFF -D BUILD_EXAMPLES=OFF -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D WITH_JPEG=OFF -D WITH_IPP=OFF ..
I found an easy solution without the need to recompile openCV.
You can use imagemagick to detect the same errors, however it returns an error as expected. See the description here: https://stackoverflow.com/a/66283167/2887398

Resources