I want to build a simple dummy app like OfferUp where users can upload 5 images per listing.
I would normally use AWS S3 with Cloudfront to serve these images.
Can I do the same with Backblaze B2? Is this an intended usecase for the product?
Yev from Backblaze here -> Yes, you can absolutely use Backblaze B2 to host images, and we have a bandwidth partnership with Cloudflare (https://help.backblaze.com/hc/en-us/articles/217666928-Using-Backblaze-B2-with-the-Cloudflare-CDN) if you also need a CDN solution.
Related
I am uploading images via CarrierWave in my Rails 4 app, to an AWS S3 Bucket. I also have Cloudfront setup, which currently serves up all of my statis assets (Excl. Public uploads).
How do I serve uploaded images via Cloudfront instead of S3, even though they are stored in an S3 Bucket? I have found tutorials like this, but since I already have a CloudFront distribution running, I was wondering if I should add another one for my Public Image uploads or is there a way to add it to my Current distribution.
You can add the bucket as an additional custom origin to your existing Cloudfront distribution.
You can then use path patterns to determine which prefixes (e.g. /images/uploads/*) should route to the alternate origin.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesPathPattern
Since creating distributions doesn't cost anything except a few minutes of your time while you wait for the distribution to become globally available, I'd suggest creating a new distribution for experimentation before adding this to your production distribution... but this is definitely doable.
I plan to use S3 + Cloudfront for hosting static images for an e-commerce web app. Each user-uploaded image can be used for several end points.
Example user actions in the back office :
Uploads image flower.jpg in his media library
Creates a flower product with id 1
Creates another flower product with id 2
Assign image flower.jpg to illustrate both product
I was thinking about a convention over configuration mechanism such as :
Uploaded images have a unique name, like flower.jpg in this case
When used to illustrate any item, use a convention like : point p1.jpg and p2.jpg to flower.jpg, the same way symlinks work
All three following URLs would return the same file :
http://aws-s3/my_app/flower.jpg
http://aws-s3/my_app/p1.jpg
http://aws-s3/my_app/p2.jpg
Can I do that with AWS ?
I did not find any such thing in the API docs, except for the temporary public URL, which comes with two no-go : 1, they expire, 2, I cannot chose the URL
Can I do that with another CDN ?
Thanks.
I believe that to accomplish such a thing your best bet is going to be to use EC2 (pricing) with S3.
My reasoning is that S3, as you say, doesn't allow for redirect URLs. To accomplish what you want, you would need to actually upload the file to each place, which would greatly increase your costs.
However, what you can do is use EC2 as a webserver. I'll leave it up to you to decide on your configuration, but you can do anything on EC2 you could do on any server - like set up redirects.
For reference, here's a good tutorial on setting up Apache on Ubuntu Server, and here's one on setting up Apache redirects.
I think, now you can use S3, Route53 and Cloudfront to achieve the same. The question seems to however aged, but thought it may be useful for someone looking now.
I am creating an iOS app with S3 currently without distributions (CloudFront) as a test before I divulge into creating a full pledged app. In the S3 Management Console, I have made my bucket in Singapore, where I live, so CloudFront isn't really needed for this demo. I have to set an endpoint like this:
[s3Client setEndpoint: [AmazonEndpoints s3Endpoint: AP_SOUTHEAST_1]];
Which points to Singapore, endpoint is the place the bucket needs to send the data off to right? (Where the user is)
So now I have two questions
If I am using CloudFront, do I need to set an endpoint? How do I even use CloudFront in iOS, I generate a signed URL then what?
If a user is using the app in a random country lets say, what endpoint, if I need to set (with CloudFront), would I set it to? Would I find their current country via the locale and find which endpoint it is closest to?
Thanks!
A set of files in CloudFront is called a "distribution." When you set up a distribution, you specify one or more "origins", which is/are the canonical source of the files you're serving to your users.
In your case, create a new distribution and specify the S3 bucket as the origin. Then in your application, you'd reference it as: http://xxxxxxx.cloudfront.net/hello.png rather than http://mybucket.s3.amazonaws.com/hello.png. Cloudfront will automatically fetch hello.png from the S3 bucket the first time someone requests it and cache it.
CloudFront automatically (and near-instantaneously) detects which edge location is closest to the user by routing them based on network latency. You don't have to do any of these calculations yourself.
I'd recommend that you read the caveats that I've listed here though before using CloudFront in your app.
I agree with #jamieb. You should create a new Cloudfront distribution and set the S3 bucket as the origin. Then, you will no longer use the s3 bucket link, you will now use the cloudfront link to view the image. Cloudfront will pull image from S3 and store it as a cache for however long you determine. For example, if the image is going to be looked at constantly by different people in the same region, you are going to want it cached in the edge location in that region, so when a new user in that region looks it up, they get the image much more quickly.
I have seen quite a few code samples/plugins that promote uploading assets directly to S3. For example, if you have a user object with an avatar, the file upload field would load directly to S3.
The only way I see this being possible is if the user object is already created in the database and your S3 bucket + path is something like
user_avatars.domain.com/some/id/partition/medium.jpg
But then if you had an image tag that tried to access that URL when an avatar was not uploaded, it would yield a bad result. How would you handle checking for existence?
Also, it seems like this would not work well for most has many associations. For example, if a user had many songs/mp3s, where would you store those and how would you access them.
Also, your validations will be shot.
I am having trouble thinking of situations where direct upload to S3 (or any cloud) is a good idea and was hoping people could clarify either proper use cases, or tell me why my logic is incorrect.
Why pay for storage/bandwidth/backups/etc. when you can have somebody in the cloud handle it for you?
S3 (and other Cloud-based storage options) handle all the headaches for you. You get all the storage you need, a good distribution network (almost definitely better than you'd have on your own unless you're paying for a premium CDN), and backups.
Allowing users to upload directly to S3 takes even more of the bandwidth load off of you. I can see the tracking concerns, but S3 makes it pretty easy to handle that situation. If you look at the direct upload methods, you'll see that you can force a redirect on a successful upload.
Amazon will then pass the following to the redirect handler: bucket, key, etag
That should give you what you need to track the uploaded asset after success. Direct uploads give you the best of both worlds. You get your tracking information and it unloads your bandwidth.
Check this link for details: Amazon S3: Browser-Based Uploads using POST
If you are hosting your Rails application on Heroku, the reason could very well be that Heroku doesn't allow file-uploads larger than 4MB:
http://docs.heroku.com/s3#direct-upload
So if you would like your users to be able to upload large files, this is the only way forward.
Remember how web servers work.
Unless you're using a sort of async web setup like you could achieve with Node.JS or Erlang (just 2 examples), then every upload request your web application serves ties up an entire process or thread while the file is being uploaded.
Imagine that you're uploading a file that's several megabytes large. Most internet users don't have tremendously fast uplinks, so your web server spends a lot of time doing nothing. While it's doing all of that nothing, it can't service any other requests. Which means your users start to get long delays and/or error responses from the server. Which means they start using some other website to get the same thing done. You can always have more processes and threads running, but each of those costs additional memory which eventually means additional $.
By uploading straight to S3, in addition to the bandwidth savings that Justin Niessner mentioned and the Heroku workaround that Thomas Watson mentioned, you let Amazon worry about that problem. You can have a single-process webserver effectively handle very large uploads, since it punts that actual functionality over to Amazon.
So yeah, it's more complicated to set up, and you have to handle the callbacks to track things, but if you deal with anything other than really small files (and even in those cases), why cost yourself more money?
Edit: fixing typos
I would like to know if Google App Engine can be used as a Content delivery network like aws S3. I'm running a RoR app on Heroku and I would like store my uploaded files on GAE instead of s3.
If it's possible what would be the best way to do it?
http://24ways.org/2008/using-google-app-engine-as-your-own-cdn
It won't be able to host files over 1MB though.
Make sure to read through the comments on that blog post as well, some have concerns about the terms of service.
GAE in itself isn't meant to be a CDN... that doesn't, however, stop you from writing a CDN application on top of it. The only limit you'll need to worry about is the 50 MB limit on the size of the blobstore. Such an app will have to provide a URL that you can hit to get the upload URL, which could then be used to upload the file. The download url can also be generated with the upload URL, and used to access the content.