We are trying to increase the page score (google) for our website. One of the options to do this is "Image optimization".
As we have a huge number of images in the DAM, how can we compress/optimize them? Does AEM have any such tool to achieve this?
ImageMagick is one of the tool to achieve this. Do we need to integrate that with AEM or we'll have to re-upload all the images after compressing them using the tool?
Any suggestions?
In contrast to CSS, JS and HTML files which can be gzipped using dispatcher, images can be compressed only by reducing quality or resizing them.
It is a quite common case for AEM projects and there are a couple of options to do that, some of them are coming out-of-the-box and do not even require programming:
You can extend DAM Update Asset with CreateWebEnabledImageProcess Workflow Process Step. It allows you to generate new image rendition with parameters like size, quality, mime-type. Depending on workflow launcher configuration, this rendition can be generated during creation or modification of assets. You can also trigger the workflow to be run on chosen or all assets.
In case that CreateWebEnabledImageProcess configuration is not sufficient for your requirements, you can implement your own Workflow Process Step and generate proper rendition programmatically, using for example ImageHelper or some Java framework for images transformation. That might be also needed if you want to generate the compressed images on the fly, for example, instead of generating rendition for each uploaded image, you can implement servlet attached to proper selectors and image extensions (i.e. imageName.mobile.png) which return the compressed image.
Eventually, integration with ImageMagick is possible, Adobe documentation describes how it can be achieved using CommandLineProcess Workflow Process Step. However, you need to be aware of security vulnerabilities related to this mentioned in the documentation.
It is also worth to mention that if your client needs more advanced solutions for images transformation in the future, then integration with Dynamic Media can also be considered as a possibility, however, this is the most costly solution.
There are many ways to optimise Images in AEM. Here I will go through 3 of those ways.
1) Using DAM Update Asset Workflow.
This is an out of the box workflow in AEM, Where on upload of images renditions get created . You can use those renditions path in img src attribute.
2) Using ACS commons Image transformer
Install ACS commons Package , Use Image transformer Servlet config to generate optimised or transformed images acc to requirement. For more Info on this check ACS AEM commons.
3) Using Google PageSpeed in dispatcher level
If you want to reduce the size of image, Google PageSpeed is an option to consider. Install PageSpeed in dispatcher level and add image optimise rules to achieve your requirement.
This rule Insights detects the images on the page that can be optimized to reduce their filesize without significantly impacting their visual quality.
for more info check here Optimising Images
AEM offers options for "image optimisation" but this is a broad topic so there is no "magic" switch you can turn to "optimise" your images. It all boils down to the amount of kilo- or megabytes that are transferred from AEM to the users browser.
The size of an asset is influenced by two things:
Asset dimension (width and height).
Compression.
The biggest gains can be achieved by simply reducing the assets dimensions. AEM does that already. If you have a look at your assets renditions you will notice that there is not just the so called original rendition but several other renditions with different dimensions.
MyImage.jpg
└── jcr:content
└── renditions/
├── cq5dam.thumbnail.140.100.png
├── cq5dam.thumbnail.319.319.png
├── cq5dam.thumbnail.48.48.png
└── original
The numbers in the renditions name are the width and height of the rendition. So there is a version of MyImage.jpg that has a width of 140px and a height of 100px and so on.
This is all done by the DAM Update Asset workflow when the image is uploaded and can be modified to generate more renditions with different dimensions.
But generating images with different dimensions is only half of the story. AEM has to select the rendition with the right dimension at the right moment. This is commonly referred to as "responsive images". The AEM image component does not support "responsive" images out of the box and there are several ways to implement this feature.
The gist of it is that your image component has to contain a list of URLs for different sized renditions. When the page is rendered client side JavaScript determines which rendition is the best for current screen size and adds the URL to the img tags src attribute.
I would recommend that you have a look at the fairly new AEM Core components which are not included with AEM. Those core components contain an image component that supports responsive images. You can read more about those here:
AEM Core Components Image Component (GitHub)
AEM Core Components Documentation
Usually, components like that will not use "static" renditions that were already generated by the DAM Update Asset workflow but will rely on a Adaptive Image Servlet. This servlet basically gets the asset path and the target width and will return the asset in the requested width. To avoid doing this over and over you should allow the Dispatcher to cache the resulting image.
Those are just the basic things you can do. There are a lot of other things that can be done but all of them with less and less gains in terms of "optimisation".
I had the same need, and I looked at ImageMagick too and researched various options. Ultimately I customized the workflows that we use to create our image renditions to integrate with another tool. I modified them to use the Kraken.io API to automatically send the rendition images AEM produced to Kraken where they would be fully web-optimized (using the default Kraken settings). I used their Java integration library to get the basic code for this integration. So eventually I ended up with properly web-optimized images for all the generated renditions (and the same could be done to the original) that were automatically optimized during a workflow without authors having to manually re-upload images. This API usage required a Kraken license.
So I believe the answer is that at this time AEM does not provide a feature to achieve this, and your best bet is to integrate with another tool that does it (custom code).
TinyPng.com was another image optimization service that looked like it would be good for this need and that also had an API.
And for the record, I also submitted this as a feature request to our AEM rep. It seems like a glaring product gap to me, and something I am surprised hasn't been built into the product yet to allow customers to make images fully web-optimized.
Related
After studying many articles and some questions on StackOverflow, I know that I will need to write a customized parser plugin for the purpose and I also know how to do this, but I am stuck at how to proceed.
In fact I am confused with the "flow chart" of the system, which perhaps needs too much in depth study of the Nutch crawling and parsing mechanism. Where to start? Customising the HTML parsing process, then parsing the img tags on the relevant pages and finally completing the process with tools like JSoup etc.
For example let I have to crawl the web and collect all the images of some specific brand item. The images search will take place by the file name and the surrounding text (this makes it necessary to include text parsing as well).
How should the system flow chart look like to start writing the customized plugin for?
I am using Nutch 1.12 and Solr 6.3 integrated...
Let me start by saying that what you're trying to do is not an easy task, but let's go step by step:
Assuming that you don't have all the URLs of the images before the crawl begins, you need to crawl the entire web, but you only need to keep the images in your index (and all the associated metadata). For this particular issue you can use the mimetype-plugin one of the sample configurations does a simplistic version of this (block everything and show only the images).
You need to extract metadata about the image (size, colors, etc.) the good news is that Tika already parse the images and detect a lot of metadata. And you'll need to write a custom parse filter for extracting all the additional data that you want.
Also you'll need to extract the text around the image, this is not really hard in an HtmlParseFilter, the tricky part is how would you relate this content with the image metadata. One way you can accomplish this is writing a custom ScoringPlugin to share the data from the original HTML page (where the text is) to the actual NutchDocument for the image itself (keep in mind that this are processed in different Nutch steps). One other chance is to index this as two separated documents (image metadata + metadata extracted from the HTML), and do a group/join on the query side of your application (web application for instance).
Some additional notes, this particular use case is not really straightforward to implement at the moment with Nutch's out-of-the-box features, but is definitively doable. I built an image search engine based on Nutch and Solr following the previous approach.
please bear with me as I'm not trying to frustrate anyone with inane questions, and I did google search this but I couldn't really find anything recent or helpful.
I am a novice programmer and I am using a classic asp web application. I just enabled the users to upload and download images, but I'm quickly regretting it as it's eating up all of the router bandwidth. I am finding my solution inadequate, so I wanted to start over.
My desire is threefold with this functionality:
Compression. I understand that this is impossible to do BEFORE uploading without some kind of Java/Silverlight/Flash portion of the application to handle uploads, correct? What is the common way most places go about this? Just allow regular file uploads and compress once they are on the server?
Resizing. I want to resize all images before they are uploaded to a reasonable size, instead of just telling users that try and upload huge camera images that they can't upload. I figure I just want to let them upload and have it resize for them before uploading. Does this functionality exist already?
Changing filetype. I want to allow users to upload all image file types but make them .jpg on the server after the upload.
With these three requirements, how hard is it to implement something like this in just pure code and libraries? Would it be better to just use a 3rd party plugin, such as ASPjpeg or ASPupload? Have you encountered something similar, and what was your solution?
Thanks.
Take a look at ASPJpeg and ASPUpload from Persits. We use these components to upload a full size image (can be png even though the library is "ASPJpeg"), resize it to several different sizes we need on our site, then store the resized images on the server in a variety of folders. The ASPUpload component is a little tricky but if you follow their sample code you'll be fine.
I never found a good component for decompressing uploaded zip files and had to write my own, which I've since abandoned. In the end with upload speeds increasing and storage getting so cheap, it started to matter less and less that the files were compressed before being uploaded.
EDIT: Just noticed you mentioned these components in your question. Consider this an endorsement of your idea to use them. :-)
I am interested in building a Rails based system for handling the display and organization of large amounts of photos. This is sort of like Flickr but smaller. Each photo will have metadata associated with it. Photos will be shown in a selectable list and grid view. It would be nice to be able to load images as they are needed as well (as this would probably speed things up).
At the moment I have a test version of my database working by images loading from the assets/images directory but it is beginning to run slow when displaying several images (200-600 images). This is due to the way I have my view setup. I am using a straight loop to display the images in both list and grid layouts.
I also manually resized the thumbnails and a medium sized image from a full sized source image. I am investigating other resizing methods. Any advice is appreciated here as well.
As I am new to handling the images this way, could someone point me in a direction based on experience designing and implementing something like Flickr?
I am investigating the following tools:
Paperclip
http://railscasts.com/episodes/134-paperclip
Requirements: ImageMajick
attachment_fu
http://clarkware.com/blog/2007/02/24/file-upload-fu#FileUploadFu
Requirement: One of the following: ImageScience, RMagick, miniMagick, ImageMajick?
CarrierWave
http://cloudinary.com/blog/ruby_on_rails_image_uploads_with_carrierwave_and_cloudinary
http://cloudinary.com/blog/advanced_image_transformations_in_the_cloud_with_carrierwave_cloudinary
I'd go with Carrierwave anyday. It is very flexible and has lot of useful strategies. It generates it's on Uploader class and has all nifty and self explanatory features such as automatic generation of thumbnails (as specified by you), blacklisting, formatting image, size constraints etc; which you can put to your use.
This Railscast by Ryan Bates - http://railscasts.com/episodes/253-carrierwave-file-uploads is very useful, if you haven't seen it already.
Paperclip and CarrierWave are totally appropriate tools for the job, and the one you choose is going to be a matter of personal preference. They both have tons of users and active, ongoing development. The difference is whether you'd prefer to define your file upload rules in a separate class (CarrierWave), or if you'd rather define them inline in your model (Paperclip).
I prefer CarrierWave, but based on usage it's clear plenty of people feel otherwise.
Note that neither gem is going to do anything for your slow view with 200-600 images. These gems are just for handling image uploads, and don't help you with anything beyond that.
Note also that Rails is really pretty bad at handling file uploads and file downloads, and you should avoid this where possible by letting other services (a cdn, your web server, s3, etc) handle these for you. The central gotcha is that if you handle a file transfer with rails, your entire web application process is busy for the duration of the transfer. (For related discussion on this topic, see: Best Ruby on Rails Architecture for Image Heavy App).
I'm using Google's Custom Search API to dynamically provide web search results. I very intensely searched the API's docs and could not find anything that states it grants you access to Google's site image previews, which happen to be stored as base64 encodes.
I want to be able to provide image previews for sites for each of the urls that the Google web search API returns. Keep in mind that I do not want these images to be thumbnails, but rather large images. My question is what is the best way to go about doing this, in terms of both efficiency and cost, in both the short and long term.
One option would be to crawl the web and generate and store the images myself. However this is way beyond my technical ability, and plus storing all of these images would be too expensive.
The other option would be to dynamically fetch the images right after Google's API returns the search results. However where/how I fetch the images is another question.
Would there be a low cost way of me generating the images myself? Or would the best solution be to use some sort of site thumbnailing service that does this for me? Would this be fast enough? Would it be too expensive? Would the service provide the image in the correct size for me? If not, how could I change the size of the image?
I'd really appreciate answers that are comprehensive and for any code examples to be in ruby using rails.
So as you pointed out in your question, there are two approaches that I can see to your issue:
Use an external service to render and host the images.
Render and host the images yourself.
I'm no expert in field, but my Googling has so far only returned services that allow you to generate thumbnails and not full-size screenshots (like the few mentioned here). If there are hosted services out there that will do this for you, I wasn't able to find them easily.
So, that leaves #2. For this, my first instinct was to look for a ruby library that could generate an image from a webpage, which quickly led me to IMGKit (there may be others, but this one looked clean and simple). With this library, you can easily pass in a URL and it will use the webkit engine to generate a screenshot of the page for you. From there, I would save it to wherever your assets are stored (like Amazon S3) using a file attachment gem like Paperclip or CarrierWave (railscast). Store your attachment with a field recording the original URL you passed to IMGKit from WSAPI (Web Search API) so that you can compare against it on subsequent searches and use the cached version instead of re-rendering the preview. You can also use the created_at field for your attachment model to throw in some "if older than x days, refresh the image" type logic. Lastly, I'd put this all in a background job using something like resque (railscast) so that the user isn't blocked when waiting for screenshots to render. Pass the array of returned URLs from WSAPI to background workers in resque that will generate the images via IMGKit--saving them to S3 via paperclip/carrierwave, basically. All of these projects are well-documented, and the Railscasts will walk you through the basics of the resque and carrierwave gems.
I haven't crunched the numbers, but you can against hosting the images yourself on S3 versus any other external provider of web thumbnail generation. Of course, doing it yourself gives you full control over how the image looks (quality, format, etc.), whereas most of the services I've come across only offer a small thumbnail, so there's something to be said for that. If you don't cache the images from previous searches, then your costs reduces even further, since you'll always be rendering the images on the fly. However I suspect that this won't scale very well, as you may end up paying a lot more for server power (for IMGKit and image processing) and bandwidth (for external requests to fetch the source HTML for IMGKit). I'd be sure to include some metrics in your project to attach some exact numbers to the kind of requests you're dealing with to help determine what the subsequent costs would be.
Anywho, that would be my high-level approach. I hope it helps some.
Screen shotting web pages reliably is extremely hard to pull off. The main problem is that all the current solutions (khtml2png, CutyCapt, Phantom.js etc) are all based around QT which provides access to an embedded Webkit library. However that webkit build is quite old and with HTML5 and CSS3, most of the effects either don't show, or render incorrectly.
One of my colleagues has used most, if not all, of the current technologies for generating screenshots of web pages for one of his personal projects. He has written an informative post about it here about how he now uses a SaaS solution instead of trying to maintain a solution himself.
The TLDR version; he now uses URL2PNG to do all his thumbnail and full size screenshots. It isn't free, but he says that it does the job for him. If you don't want to use them, they have a list of their competitors here.
I'm looking for a free, preferably open source, http image processing server. I.e. I would send it a request like this:
http://myimageserver/rotate?url=http%3A%2F%2Fstackoverflow.com%2FContent%2FImg%2Fstackoverflow-logo-250.png&angle=90
and it would return that image rotated. Features wanted:
Server-side caching
Several operations/effects (like scaling, watermarking, etc). The more the merrier.
POST support to supply the image (instead of the server GETting it).
Different output formats (PNG, JPEG, etc).
Batch operations
It would be something like this, but free and less SOAPy. Is there anything like this or am I asking too much?
The ImageResizing.Net library is both a .NET library and an IIS module. It's an image server or an image library, whichever you prefer.
It's open-source, under an MIT-style license, and is supported by plugins.
It has excellent performance, and supports 3 pipelines: GDI+, Windows Imaging Components, and FreeImage. WIC is the fastest, and can do some operations in under 15ms. It supports disk caching (for up to 1 million files), and is CDN compatible (Amazon CloudFront is ideal).
It has a very human-friendly URL syntax. Ex. image.jpg?width=100&height=100&mode=crop.
It supports resizing, cropping, padding, rotation, PNG/GIF/JPG output, borders, watermarking, remote URLs, Amazon S3, MS SQL, Amazon CloudFront, batch operations, image filters, disk caching, and lots of other cool stuff, like seam carving.
It doesn't support POST delivery of images, but that's easy to do with a plugin. And don't you typically want to store images that are delivered via POST instead of just replying to the POST command with the result?
[Disclosure: I'm the author of ImageResizer]
Apache::ImageMagick, you install that - and also Apache along with mod_perl. This is the standard setup, check docs, there are alternatives. This is probably as turn-key as it gets.
Sample conf:
<Location /img>
PerlFixupHandler Apache::ImageMagick
PerlSetVar AIMCacheDir /tmp/your/cache/directory
</Location>
Your requests could look like:
http://domain/img/test.gif/Frame?color=red
More docs are here!
While not an out of the box solution, check out ImageMagick. There is a perl interface for it, so combine that with some fairly simple cgi scripts, or mod_perl and it should do the trick.
You can use LibGD or ImageMagick to build a service like that fairly easily. They both have many language bindings.
You could make this with Google App Engine -- they provide image processing routines and will host for free within some bounds.
Here are some examples of people doing things like this already
http://appgallery.appspot.com/results?q=image
I found this product, it seems to match my requirements
Try Nginx image processing server with OpenResty and Lua. It uses ImageMagick C API. Openresty comes with LuaJIT. It has amazing performance in terms of speed. Checkout some benchmarks for LuaJIT and Openresty.