Rails 3 - downloading large amounts of picture from the web - ruby-on-rails

I am solving the question about downloading a large amounts of picture from the web. I have XML file with links to images on web sites and I have to download this images. Every image have ca ±3MB and the count of images is tens of thousands, so is not possible to store these images on hosting (100.000 x 3MB)...
And these images I need to display on sites. I don't worked yet with so large amounts of data yet, so I would like to ask you, what could be the best idea for displaying these images on "my" page.
My first ideas:
- store only links into my database table and then for displaying images use just image_tag URL_OF_IMAGE
- some a way of caching images/links of images (I don't know specifically)
Can you help me, please, what you think will be the fastest way for displaying images from foreigners sources?
Thank you in advance,
M.

Amazon S3, but as ezkl says, make sure you have licensing rights to the images or you'll land yourself in trouble (and in any case it is wrong to use the content without permission).

Related

Generating Thumbnails on Client

My team and I are building an iOS application. We allow technicians in the field to upload images for certain issues they are resolving on technical equipment. It will be important to zoom in (so keep quality relatively high) when these images are uploaded to S3.
Recently we decided to add thumbnails because it will be much faster when others browse the iOS app, rather than downloading a 1.5-2.5mb image.
My co-worker decided the best way to handle this is to generate a 200-500kb thumbnail in iOS then upload the image and the thumbnail to s3.
I voiced my concern that some of our technicians may be in some parts of the world where internet is slow and data usage is limited. So doing all this additional work on the device and uploading makes no sense to me. However the team considers this a good solution and will move forward. I've shown them easy examples of how to generate thumbnails from S3 and Lambda automatically on the server... allowing us to either upload higher fidelity images with the additional bandwith or just increase the speed of the app by uploading much less. Sometimes a user may upload as many as 100 images... meaning an additional 20-50mb...
Anyways I wanted to hear some answers about how you guys think the best way to handle this is, mainly for my own sanity check.
I don't completely comprehend the intricacies of your project, but from experience, I have one word for you - Cloudinary. As opposed to S3, which a general purpose Cloud storage solution, cloudinary is designed to handle images.
We have a 200,000 hits a day online classified app that handles tens of thousands of photos daily. And cloudinary provides an extremely mean solution for all our needs. We have uploads by users from their mobile and desktop devices, bookmarking of those images, CDN based serving, and thumbnail generation.
Did I mention they have thumbnail generation built in? They have lots of other features as well, including
Resize and Crop
Optimized JPEG Custom Crop
Face Thumbnail
Rotated Circular Thumbnail
Zoom Effects and Zoom Image Overlay
Watermark Image
Optimized WebP
Overlay, Border, Shadow Text Overlay, Border, Shadow etc.
The admin console is pretty kickass too, with all of the above features available for you to configure over the cloud. And it fits well with pretty much any application (we use it in our internal Ruby, Go, NodeJS services, our Web Application and our iOS and Android apps as well).
I'm not paid to sell Cloudinary to you, but I can vouch that if it is image based services I needed, I would go for Cloudinary any day over S3. Major players like EBay and TED etc. use it for their image requirements.

What are some options for handling image uploading/compression in ASP?

please bear with me as I'm not trying to frustrate anyone with inane questions, and I did google search this but I couldn't really find anything recent or helpful.
I am a novice programmer and I am using a classic asp web application. I just enabled the users to upload and download images, but I'm quickly regretting it as it's eating up all of the router bandwidth. I am finding my solution inadequate, so I wanted to start over.
My desire is threefold with this functionality:
Compression. I understand that this is impossible to do BEFORE uploading without some kind of Java/Silverlight/Flash portion of the application to handle uploads, correct? What is the common way most places go about this? Just allow regular file uploads and compress once they are on the server?
Resizing. I want to resize all images before they are uploaded to a reasonable size, instead of just telling users that try and upload huge camera images that they can't upload. I figure I just want to let them upload and have it resize for them before uploading. Does this functionality exist already?
Changing filetype. I want to allow users to upload all image file types but make them .jpg on the server after the upload.
With these three requirements, how hard is it to implement something like this in just pure code and libraries? Would it be better to just use a 3rd party plugin, such as ASPjpeg or ASPupload? Have you encountered something similar, and what was your solution?
Thanks.
Take a look at ASPJpeg and ASPUpload from Persits. We use these components to upload a full size image (can be png even though the library is "ASPJpeg"), resize it to several different sizes we need on our site, then store the resized images on the server in a variety of folders. The ASPUpload component is a little tricky but if you follow their sample code you'll be fine.
I never found a good component for decompressing uploaded zip files and had to write my own, which I've since abandoned. In the end with upload speeds increasing and storage getting so cheap, it started to matter less and less that the files were compressed before being uploaded.
EDIT: Just noticed you mentioned these components in your question. Consider this an endorsement of your idea to use them. :-)

How to load images faster from Azure Blob?

I've been trying to upload some images to azure blob and then using ImageReader in Azure ML studio to read them from the blob. The problem is that ImageReader takes a lot of time to load images and I need it in real time.
I also tried making a csv of 4 images (four rows) containing 800x600 pixels as columns (500,000 cols. approx) and tried simple Reader. Reader took 31 mins to read the file from the blob.
I want to know the alternate methods of loading and reading images in Azure ML studio. If anyone know any other method or can share a helpful and relevant link.
Please share if i can speed up ImageReader by any means.
Thanks
Look at the Azure CDN http://azure.microsoft.com/en-us/services/cdn/ , after which the blobs will get an alternative url. My blob downloads became about 4 times faster after switching.

iOS App Backend Provider

I've recently submitted my iOS Quiz app to Apple but noticed that the file size for the app is pretty big (about 150 MB). Users would need to be connected to wifi in order to download it per Apple's rules. My quiz app is set up so users are given 4 choices and shown an image and must guess the correct answer from the image shown to them. How would I minimize the file size for my app so that it isn't so large? Is there a way I can host the images on a server without losing the functionality of my app? I heard of something like Backend Services but know nothing about it. If anyone can guide me in the right direction that would be awesome, thanks!
You can check out a free back end service like Parse, it could do the trick for you, especially because you dont have a lot (besides images I guess) that'll be on the server side.
This also helped me start with using it.
Good luck :)
I'm assuming you have all the quiz data (questions and images) within your app bundle?
You can shrink it next to nothing if you move all your questions and images to a backend server and serve the questions and images (links) using simple JSON Structure.
You can build your own backend (Java/PHP/etc..) or look into using Parse.
use JPEG images whenever possible. PNGs costs more space. Do not place jpeg to xcassets, since they will be converted to PNGs. If your pictures should be transparent - it is better to use Webp or JPNG format.
You may use CloudKit to host your data in a public database. You won't need any backend knowledge to do that. This tutorial will help you understand the basics. WWDC videos covers some more, i suggest you to look at WWDC 2014, Introducing CloudKit and WWDC 2015, CloudKit Tips and Tricks.

iOS app additional content download

I have an iOS app which has a lot of images and sounds in it, hence the build size is growing rapidly and can no longer fit in the 50MB limit for 3G download. I would like to upload those images and sounds to an online server and download them from the application on demand. Can anyone please recommend some online storage (for example Amazon S3) and give suggestions for best practices about this issue?
Thank you!
I upload my files to an "unlimited site" with Network solutions, however if you don't want to pay for anything like this just upload the photos to photo bucket or something and right click on them and click "Get image address" then use this URL to download the images to your app. (Check to make sure it doesn't violate their terms of use though, I don't know if it does or not!)
As for best practices, I probably use the absolute worst method which is displaying it in a UIWebView off-screen and taking a picture of it using CoreGraphics then saving that picture haha! I'll leave you with this link for a probably better solution to your second half of the question, that being "Best practices": ios store URL images in offline mode(not connected to internet)

Resources