At the end of Chapter 11 of The Rails Tutorial by Michael Hartl I successfully managed to enable user uploads to Amazons S3 service by creating a bucket, using IAM to set a user and granting the user an AmazonS3FullAccess policy. It feels dirty and very insecure to allow an unknown user on my website to have full access to a bucket for image upload on my website and I'm not sure if I should feel this way. I created a custom policy at
http://awspolicygen.s3.amazonaws.com/policygen.html
Which is the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1445501067518",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::bucketname"
}
]
}
I am not confident in my solution and could not find any answers googling for the best way to go about this. I am using carrierwave (with intentions of using carrierwave_direct for my own project), fog, and mini_magick gems.
The best and probably the most secure way of allowing users to upload files to your site (ie. S3) is to use Browser-Based Post Uploads.
This lets users upload directly to S3 without having to go through your servers. On your servers you simply create a request signature using your access keys.
You can read more about it here:
Browser Based Uploads Using Post
I'm not familiar with carrierwave myself but you may find this useful:
Uploading directly to S3 in rails
Related
At the moment my Rails 6 React app has user uploaded images (avatars, profile wallpapers, etc) stored in S3, inside a public bucket for local development (not facilitated by active storage because it was not playing nice with vips for image processing). The reason its set to public was for ease of set up, now that all of the functionality is complete, for the stagging (and soon production) I would like to add sensible bucket policies. I don't currently have CloudFront set up but I do intend to add that in the near term, for right now I'm using the bucket asset URL to serve assets. I have created two separate buckets, one for images that will be displayed in the app and one for content that is never to be publicly displayed which will be used for internal purposes.
The question I have is, for the content that is in the bucket reserved for viewable content, do I have to make it public (disable that setting in the AWS console that disables public access), then create a policy that allows GET request from wherever (*), then restriction POST, PUT, DELETE, requests to the arn ID of the EC2 instance that's hosting the rails application. The AWS documentation has confused me, it gives me the impression that you never want to enable public access to a bucket, and that policies alone are how you surface bucket content. When I take that approach I keep getting access denied in the UI when I have attempted to do that.
EDIT:
I'm aware that signed URLs can be used, but it is my current understanding that there is a nontrivial speed hit to the UX of the application if you have to generate a signed URL for every image (this app is image heavy). There are also SEO concerns given that all the image URLs would effectively be temporary.
Objects in Amazon S3 are private by default. You can grant access to an object in several ways:
A Bucket Policy that can grant access to everyone ('Public'), or to specific IP addresses or users
An IAM Policy on an IAM User or IAM Group that grants access to that user or group -- however, they would need to access via an AWS SDK so that they can authenticate the call (eg when an application makes a request to S3, it would make an authenticated API call)
An Access Control List (ACL) on the object, which can make the object public without requiring the bucket to be public
By using an Amazon S3 pre-signed URL, which is a time-limited URL that provides temporary access to a private object
Given your use-case, an S3 pre-signed URL would be the best choice since the content is kept private but the application can generate a link that provides temporary access to the object. This can also be done with CloudFront.
Generating the pre-signed URL only takes a few lines of code and does not involve an API call to AWS. It is simply creating a hash of the request using your Secret Key, and then appending that hash as a 'signature'. Therefore, there is effectively no speed impact of generating pre-signed URLs for all of your images.
I don't see how SEO would be impacted by using pre-signed URLs. Only actual web pages (HTML) are tracked in SEO -- images are not relevant. Also, the URLs point to the normal image, but have some parameters at the end of the URL so they could be tracked the same as a non-signed URL.
No it does not have to be public. If you don't want to use CloudFront, the other option is to use S3 presigned RLs.
In Chapter 11.4.4 'Image upload in production' of Michael Hartl' Rails Tutorial it is suggested to use Amazon Web Services S3 as a cloud storage service. In a note at the bottom of the page, the author himself defines this section of the book as "challenging" and also suggests that it "can be skipped without loss of continuity".
Indeed one of the most challenging parts of this section is to find a suitable IAM policy that can be attached to the IAM user at AWS in order to grant to the IAM user read and write permissions on the S3 bucket.
I found that this at Stackoverflow is a common issue: see for instance 'Trying to set up Amazon's S3 bucket: 403 Forbidden error & setting permissions'.
In effect, Amazon Web Services's solution for applications that need read and write permissions on a single S3 bucket does not work, and the user who tries to upload images receives a 403 forbidden status from the AWS server at Heroku.
The predefined 'AmazonS3FullAccess' policy works indeed, however it should not be considered as a definitive solution, because granting too many permissions to the IAM user, and therefore to the application, is not required and also, I suppose, can be a security bug.
What then is the correct IAM policy?
If the 'AmazonS3FullAccess' policy works, there should definitely be some action in it that is essential for the working of the application.
Apart from the ListBucket, PutObject, GetObject and DeleteObject actions whose presence seems logical, I found that PutObjectAcl is also necessary.
From the AWS's Access Control List (ACL) Overview:
"Amazon S3 Access Control Lists (ACLs) enable you to manage access to buckets and objects. Each bucket and object has an ACL attached to it as a subresource. It defines which AWS accounts or groups are granted access and the type of access. When a request is received against a resource, Amazon S3 checks the corresponding ACL to verify the requester has the necessary access permissions. When you create a bucket or an object, Amazon S3 creates a default ACL that grants the resource owner full control over the resource"
Moreover, from the documentation on 'PutObjectAcl':
"This implementation of the PUT operation uses the acl subresource to set the access control list (ACL) permissions for an object that already exists in a bucket... The ACL of an object is set at the object version level. By default, PUT sets the ACL of the current version of an object."
I could not find explanations in my request of support from the Amazon forums on why PutObjectAcl is necessary, but from my understanding probably the operation of creating the ACL of an object is made the first time the object is uploaded in the bucket and should be explicitly requested by the application (or IAM user) that makes the upload. You can find a copy of my policy in the above Amazon Forums discussion and below:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::<bucket-name>"
},
{
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::<bucket-name>/*"
}
]
}
Consider also Heroku's suggestions on the choice of your bucket name: avoid for instance punctuations.
We develop a rails-based healthcare application. What is the best way to configure our s3 implementation so that only the authenticated user has access to the image?
From the Documentation,you should use one of Amazon's "canned" ACLs.
Amazon accepts the following canned ACLs:
:private
:public_read
:public_read_write
:authenticated_read
:bucket_owner_read
:bucket_owner_full_control
You can specify a the ACL at bucket creation or later update a bucket.
# at create time, defaults to :private when not specified
bucket = s3.buckets.create('name', :acl => :public_read)
# replacing an existing bucket ACL
bucket.acl = :private
Wanted to post an updated answer to this question, as the S3 API has changed (slightly) since 2015. Here's a link to the updated ACL section of the S3 Docs. Further, the above answer reflects the use of the Ruby SDK, which not everyone uses.
Canned ACL's are predefined grants supported by S3 that have specific grantees and permissions in place. Canned ACL's can be sent via the SDK, as demonstrated in the above answer, or in an HTTP request by using the x-amz-acl request header for new resources, or with the request header or body for existing resources.
The canned ACL's are as follows. Unless otherwise specified, the bucket owner has FULL_CONTROL in addition to the other permissions listed:
private: No other user is granted access (default)
public-read: AllUsers group gets READ access
public-read-write: AllUsers group gets READ and WRITE access (not recommended)
aws-exec-read: Amazon EC2 gets READ access to GET an Amazon Machine Image (bundle)
authenticated-read: AuthenticatedUsers group gets READ access
bucket-owner-read: Bucket owner gets READ access. Ignored when creating a bucket
bucket-owner-full-control: Both object and bucket owner get FULL_CONTROL over object. Ignored when creating a bucket
log-delivery-write: LogDelivery group gets WRITE and READ_ACP permissions
Also noted in the docs: you can specify only one canned ACL in your request
I want to setup a video&image viewing function on my site. My idea is deploy everything video,image to amazon s3. I know I should use devise to setup a user signup feature. But I still have few concern about the security issue and usage charge problems
1.Is devise safe?
2.How can I guarantee only user signed in and can only access the video/images on my amazon-s3 via only my sites while they signed in?
3.This is the most most difficult problem.. Can we keep track of a user's usage? let say I dont want each user in my sites accessing more than 100mb/day contents from s3, anyway to acheive this features??
Thankyou in advance!
Devise is most certainly a framework that will allow you to use best practices to authorize and authenticate users (e.g. by doing things like using very strong encryption methods when storing passwords). But "safe" is a little subjective -- think of Devise as a very good toolbox that will allow you to easily do things that will make your site safe.
Guaranteeing that users will only access data via your site means that you cannot set the default S3 permissions that make content in S3 buckets readable by all. I am pretty sure S3 is pretty basic in terms of permissions. Instead consider a gem like CarrierWave that makes it easy to move files around, including streaming file from S3 through your server to the user, thus giving you hooks to authenticate by user. This is also a hook for measuring number of bits transferred.
If I recall, CarrierWave (or maybe Fog?) gives you a way to query the S3 buckets similarly to how you would in a filessystem, so you can check for size.
I have opened this as an issue on Github (http://github.com/thoughtbot/paperclip/issues/issue/225) but on the chance that I'm just doing this wrong, I thought I'd also ask about it here. If someone can tell me where I'm going wrong, I can close the issue and save the Paperclip guys some trouble.
Issue:
When using S3 for storage, and you wish your bucket to allow access to other users to whom you have granted access, Paperclip appears to overwrite the permissions on the bucket, removing access to these users.
Process for duplication:
Create a bucket in S3 and set up a Rails app with Paperclip to use this bucket for storage
Add a user (for example, aws#zencoder.com, the user for the video encoding service Zencoder) to the bucket, and grant this user List and Read/Write permissions.
Upload a file.
Refresh the permissions. The user you added will be gone. As well, a user "Everyone" with read permissions will have been added.
The end result is that you cannot, so far as I can tell, retain desired permissions on your bucket when using Paperclip and S3.
Can anyone help?
Try explicitly setting :s3_permissions => :public_read
Seems to work for me.