I'm writing an MVC application which serves transformed versions of user-uploaded images (rotated, cropped, and watermarked.) Once the transformations are specified, they're not likely to change, so I'd like to aggressively cache the generated output, and serve it as efficiently as possible.
The transformation options are stored in the database and used by the image servers to create the images on demand; only the original uploads are stored permanently. Caching the generated images in a local directory allows IIS 7 to pick them up without touching the ASP.NET process, i.e. by matching the route. Static images in images/ take precedence over the dynamic MVC route /images/{id}.jpg.
My concern at this point is when the user actually changes the transformation options -- the images need to be re-generated, but I'd like to avoid manually deleting them. I'm storing a last-modified field in the database, so I could append that timestamp to the URL, e.g. http://images.example.com/images/153453543.jpg?m=123542345453. This would work if the caching was handled by the ASP.NET process, which could vary the output cache by the parameter m, but seeing as I need to serve large quantities of images I'd rather avoid that.
Is there an intelligent way to get the IIS to discard static files if some condition is met?
If you don't want your ASP.NET code to be invoke every time someone requests an image then I would recommend that you delete the images when updating the transformations. It is a relatively "free" operation since it just a cache and they will be regenerated when needed.
You might be concerned about tracking if the transformation is actually changed when the user updates image properties but how often will the user make changes at all. Does it matter if you need to regenerate an image a bit to often?
You could include the timestamp in the filename itself, e.g.
http://images.example.com/images/153453543_20091124120059.jpg.
That way you could avoid to delete the images when updating. However, you would leave a trail of old outdated files...
Why not run the process to generate the physical image whenever those settings are changed, rather than on each request?
Related
We recently upgraded our app to Rails 5.2 to make use of the active storage direct upload feature.
Following this guide to integrate direct upload with our existing JS drag and drop, we've been able to get the upload working. We take the signed ID returned, and add it to hidden fields . Then on form submission, we create a new record and associate the blobs signed_id to create the association.
However, if the user doesn't go through with the form submission, is there a recommended way to ensure that the blobs/files without model associations get purged? The tricky part seems how to determine when to purge the blob.
Purging process depends on you underlying storage, for example on S3 you can define object expiration policy for temporary blobs, on filesystem - periodically delete all files from temporary folder that are older than some limit.
As for temporary blob age that should be purged - this also depends on your application. Obviously it should be longer than user is filling the form plus some margin. If you do not have a problem with these lingering a bit longer - you can make the threshold somewhere around 24 hours or even more and purge once a day, so users will not encounter lost file for sure.
I went through the same questioning, and ended up concluding there is no ideal way: since it depends on the user absence of input, it is to be expected that the Blob can only be purged after a certain arbitrary timeout.
It can be a cron-like job for example.
Removing all dangling Blobs can be done through a one-liner though:
ActiveStorage::Blob.unattached.each(&:purge)
(Note: I spent quite some time on the MD5 computation too, if it's your case, take a look at the blog article I posted on MD5 computation in javascript)
Reading Workbox strategies documentation, and I found I can't imagine the situation using "cache-first" strategies in Workbox.
There is "Stale-While-Revalidate" strategies which uses cache first, and in background, updates cache file through Network. If you change the target file, it is useful because when access next time, the App uses latest file which last time cached. If you don't have any changes, there is no disadvantage, I think.
What is the main purpose using "cache-first" strategies in Workbox?
Thanks in advance.
(This answer isn't specific to Workbox, though Workbox makes it easier to use these strategies, instead of "rolling them by hand.")
I'd recommend using a cache-first strategy when you're sure that the content at a given URL won't change. If you're confident of that, then the extra network request made in the stale-while-revalidate strategy is just a waste—why bother with that overhead?
The scenarios in which you should have the highest confidence that the content at a URL won't change is if the URL contains some explicit versioning information (e.g. https://example.com/librbaries/v1.0.0/index.js) or a hash of the underlying contents (e.g. https://example.com/libraries/index.abcd1234.js).
It can sometimes make sense to use a cache-first strategy when a resource might update, but your users are not likely to "care" about the update, and the cost of retrieving that the update is high. For example, you could argue that using a cache-first strategy for the images used in the body of an article is a reasonable tradeoff, even if the image URLs aren't explicitly versioned. Seeing an out of date image might not be the worst thing, and you would not force your users to download a potentially large image during the revalidation step.
I am fetching images from my server and need to store them on the disk. I was wondering what is the best way to cache these images to prevent excessive requests and traffic. From doing some research it seems that there are many options available but I am not sure which to use.
Core Data
Store in the Cache Resources Folder in the file directory
After storing these is it best to use a NSCache class to put these data into memory for quick access or is Core Data quick enough?
Based on my experience, you could use SDWebImage, which caches the images you request based on their url, so the next time you "request" it, it will check if it is in cache, if so it won't make the request and it will load it from it instead.
I'm not sure why would you need to store the image, maybe you could tell us the reason and see if we can help any further.
I've had great results using FastImageCache by Path.
What Fast Image Cache Does
Stores images of similar sizes and styles together
Persists image data to disk
Returns images to the user significantly faster than traditional methods
Automatically manages cache expiry based on recency of usage
Utilizes a model-based approach for storing and retrieving images
Allows images to be processed on a per-model basis before being stored into the cache
37 signals suggests id partitioning to accomplish this thing..
http://37signals.com/svn/archives2/id_partitioning.php
Any suggestions would be more than welcome.
Thanks.
We use Paperclip for storing our files. It can do what you want pretty easily.
We use partitioning by date so an image uploaded today would end up in 2009/12/10/image_12345.jpg. The path is stored in the db for reference and the path to the image folder (the parent of 2009) is placed in some config file. If we need to change things later it makes it very easy.
You can map by virtually everything. We use mapping by user on our designs, but it's a HR system so it makes sense (there's no way the user will have 32k file entries) and the files are clearly connected with user. On Media Library parts of the system dividing by date or ID will be more useful.
The catch is, you should store some part of file path in database table (as suggested before). Will it be date, or user hash/name (often also divided, eg u/user j/john j/jo/john etc). Then you don't have to worry about changing division system, as this will only require database update.
Stored in the database or file system ?
And I need several different sizes. like 128*128, 96*96, 64*64 and so on.
What is the best way to upload user portrait?
Not knowing all of your constraints, I'd upload the 128x128 picture and then create all the other portraits on the fly.
I don't think you need to worry about storing the images in the DB, specially if you're running SQLServer 2008 (and you use the new FILESTREAM type).
Definitively it depends of the amount of images you need to store.
If you store them in the file system you just need to keep the URL or location of the image the database. To change the size you can do it in real time using components to achieve the change depending of the language you're using. For .NET ASPjpeg is a very good one, but you can manipulate the image with the System.Drawing.Imaging class. No need to manipulate database filestream or BLOB fields.
In the other hand, storing images in the database can make it too big in order to backup or download depending of the amount of records, even if you are working with SQL server, but you have everything in the same place. Maintenance is faster.
The problem with storing images in the file system is the cleaning procedure, if you delete a record, you need to delete the image in the file system too in order to prevent of keeping garbage.