I have a ruby on rails application where users can upload their avatar or change it. First I stored the images in Amazon s3 but then I realized that content contents were being served slowly and decided to use Amazon cloudfront.
There is no problem for uploading and getting avatar. However, I can see that an updated photo changes immediately but I expect to invalidate it through cloudfront api. And uploading an image takes a lot of time.
At this point I can't decide whether I use cloudfront correctly or not.
This my carrierwave.rb file inside config/initializer:
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws'
config.fog_credentials = {
provider: 'AWS',
aws_access_key_id: 'key',
aws_secret_access_key: 'value',
region: 'us-east-1'
}
config.storage :fog
config.asset_host = 'http://images.my-domain.com'
config.fog_directory = 'bucket_name'
config.fog_public = true
config.fog_attributes = { cache_control: "public, max-age=315576000" }
end
I can't see what I'm missing ? How can I be assure that I'm using cloudfront properly ?
Thanks.
Your images aren't being stored in CloudFront, they're being served through CloudFront's CDN.
First request for an image served through CF looks like this:
Browser -> CloudFront -> S3
|
Browser <- CloudFront <-
The second request for an image just looks like this:
Browser -> CloudFront
|
Browser <-
The second request never hit's CF because CF has cached the result for that URL.
NOW, your avatar's updating immediately is simply probably because it's being uploaded to S3 and resulting in a new URL, and thusly, an immediate update. This is how you want it to work.
Related
I am using ruby 2.4.0p0 and Rails 5.2.3
In the production.rb file I have done the following setting:
# Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
config.force_ssl = true
if Rails.application.config.force_ssl
Rails.application.routes.default_url_options[:protocol] = 'https'
end
But still the resource are getting rendered on http rather then https do I need to do any thing extra, please provide the desired thing to be done to get all assets getting loaded from s3 loads over https.
The website is live here at: https://tukaweb.com/asset/garments
The s3 resources are at http
ex: http://tukaweb.s3.amazonaws.com/uploads/three_d_garment/thumbnail/7/Womens_Dress_35-41_Thumbnail.png?X-Amz-Expires=600&X-Amz-Date=20200918T060705Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIRDA3IQIVTEPMN6Q%2F20200918%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=1792bd4cc2437abd950b7d16d360d09e64423bdef89f41c24a5386d35e982dfa
need them over https.
The required change should be done inside the carrierwave.rb inside the webapp/config/initializers directory modified the settings as:
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws'
config.fog_credentials = {
provider: 'AWS',
aws_access_key_id: 'XXXXXXXXXX',
aws_secret_access_key: 'xxxxxxxxxx',
use_iam_profile: false,
region: 'us-west-2', # optional, defaults to 'us-east-1'
# host: 'ec2-xx-xxx-xx-xx.us-west-2.compute.amazonaws.com', # optional, defaults to nil
:endpoint => 'https://s3.amazonaws.com',
}
config.fog_directory = 'tukaweb' # required
config.fog_public = false # optional, defaults to true
# config.fog_attributes = { cache_control: "public, max-age=#{365.days.to_i}" } # optional, defaults to {}
end
The line which is responsible for changing s3 resource to be downloaded from https instead of http
:endpoint => 'https://s3.amazonaws.com' ## earlier it was 'http://s3.amazonaws.com'
Force SSL only works for the incoming requests to the rail's routes. If you have an image link set to http://image-domain.com/image it's going to use the http, and you'll get a mixed content warning. You need to ensure anything external to the app's routes is going to be using SSL or a secure connection as well.
First thing I do when I see a mixed content warning is to do a global search of the codebase for http:// to find everywhere that isn't using https://. I may or may not do a global find + replace depending on what I see, there are cases where it needs to be http:// or it won't work right (if the site doesn't have an https:// version).
Next thing is to work out what is causing the insecure url, here it is S3, so I would be looking at what uses S3, and working out how I can tell it to use SSL or a secure connection.
Note: The other answer does well explaining what your actual issue is, but this may be more useful to others for general troubleshooting of mixed content issues, and would lead to the same result.
It seems like I finally figured how to setup Rails + Paperclip + AWS CDN + Heroku.
Everything seems to be working. Both CSS and js files load from cdn, as well as images.
Unfortunately sharing functionality is broken. Open graph can't parse image url. I assume it's because links are in this format https:////drex16ydhdd8s.cloudfront.net/...rest_of_url
Originally, long time ago, I've configured CDN link to be //drex16ydhdd8s.cloudfront.net. I understand I need to remove slashes in front of the link, make it drex16ydhdd8s.cloudfront.net instead.
The problem is, if I do it, Heroku gives me Application Error. (displays their static page)
Logs don't display anything helpful, other than it seems it goes over memory limit pretty much immediately.
I've contacted Heroku support, but their response was
You should not need any slashes, it should just be a host name. (As seen in the documentation for config.action_controller.asset_host.)
If removing the slashes causes errors, you'll want to debug those errors.
I tried to do it locally, everything seems to work as expected.
environments/production.rb
config.action_controller.asset_host = ENV.fetch("ASSET_HOST", ENV.fetch("APPLICATION_HOST"))
config.paperclip_defaults = {
storage: :s3,
s3_protocol: :https,
s3_region: ENV["AWS_REGION"],
url: ":s3_alias_url",
path: "/:class/:attachment/:id_partition/:style/:filename",
s3_host_alias: ENV.fetch("ASSET_HOST"),
s3_credentials: {
bucket: ENV["S3_BUCKET_NAME"],
access_key_id: ENV["AWS_ACCESS_KEY_ID"],
secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"]
},
default_url: "https://s3.amazonaws.com/ezpoisk/missing-small.png"
}
env variable
ASSET_HOST = //drex16ydhdd8s.cloudfront.net
on CDN I have 2 befaviors
/assets/* - that points to domain name
default (*) - that points to s3 bucket
Does anyone have any ideas?
Solution.
I had in production.rb
config.assets.compile = true
I'm not strong on details here, I just remember that I made a note on this line to possibly remove it when switching to cdn.
After digging through this answer I have decided to try it out. So I
removed the lined,
deployed,
all works fine.
Tried updating cdn link then.
At first, same issue persisted, url for some reason was /drex16ydhdd8s.cloudfront.net, but after few seconds it now seems
to be all good.
Just switched over to Refile for image uploads on my Rails application.
I have the uploads going directly to my s3 bucket. I have two buckets configured (with same settings), one for testing and one for production.
Everything in my local development works fine, and uploading to my bucket in production works, but on all the uploaded images I get the following when rendering them on a webpage.
Resource interpreted as Image but transferred with MIME type text/html:
Also in production, the images are not showing up.
I've looked into permissions for the buckets, but they seem to be good to go. I've also looked at others questions/answers regarding this warning, but have been unable to find any that pertain here.
If any code is needed please let me know.
config/initializers/refile.rb
require 'refile/backend/s3'
aws = {
access_key_id: Rails.application.secrets.aws['access_key_id'],
secret_access_key: Rails.application.secrets.aws['secret_access_key'],
bucket: Rails.application.secrets.aws['s3_bucket_name'],
use_ssl: true
}
Refile.cache = Refile::Backend::S3.new(max_size: 5.megabytes, prefix: 'cache', **aws)
Refile.store = Refile::Backend::S3.new(prefix: 'store', **aws)
Gist of the image helper
attachment_image_tag(avatar, :image, :fill, size, size)
Thanks for taking a look.
I have created new music application and I store all mp3 files on Amazon S3. Before moving to S3 I used store them on server file system itself. It used to cache files and on consecutive reload of page files weren't downloaded from server. But after moving to S3 everytime I load page it downloads files from S3. This not only making my app slow but every request to S3 is money.
I found some documentation on cache-control and I tried them all but no success. I might be missing something here. Any help is appreciated. Thanks.
Here is my code for uploading mp3 files on S3. I use CarrierWave with Rails.
CarrierWave.configure do |config|
config.fog_credentials = {
:provider => 'AWS',
:aws_access_key_id => MyAppConfig.config['aws']['aws_access_key'],
:aws_secret_access_key => MyAppConfig.config['aws']['aws_secret_key'],
}
config.fog_directory = MyAppConfig.config['aws']['aws_bucket_name']
config.fog_public = false
config.storage = :fog
config.fog_attributes = {'Cache-Control'=>'max-age=315576000'}
end
If you're using signed URLs, which you say you are in the comments, and not reusing those signed URLs then there is no way to cache these requests.
Amazon Web Services cannot override your Web browser's internal cache system. When two URIs are unique, as they are with signed URLs, then your Web browser treats them as two distinct resources on the Internet.
For example, let's take:
http://www.example.com/song1.mp3
http://www.example.com/song2.mp3
These are two discrete URIs. Even if song1.mp3 and song2.mp3 had the same Etag and Content-Length HTTP response headers, they're still two different resources.
The same is true if we merely alter their query strings:
http://www.example.com/song1.mp3?a=1&b=2&c=3
http://www.example.com/song1.mp3?a=1&b=2&c=4
These are still two discrete URIs. They will not reference one another for purposes of caching. This is the principle behind using query strings to override caching.
No amount of fiddling with HTTP headers will ever get you the cache behavior you're seeking.
Take a look at http://www.bucketexplorer.com/documentation/amazon-s3--how-to-set-cache-control-header-for-s3-object.html
Set Cache-Control for already uploaded file on S3 using Update Metadata:
1) Run Bucket Explorer and login with your credentials.
2) After listing all Buckets, select any S3 Bucket.
3) It will list all objects of selected S3 Bucket.
4) Select any file and right click on the objects and select “Update Metadata” option.
5) Add Key and Value in Metadata attributes. Enter Key: “Cache-Control” with Value: “max-age = (time for which you want your object to be accessed from cache in seconds)”
6) Click on Save button. It will update metadata as Cache-Control on all selected S3 objects.
Example to set max-age: For time limit of 15 days = 3600 * 24 * 15 = 1296000 sec. Set Key = “Cache-Control” value = “max-age=1296000”
Note: If object is HTML file, set Key: “Cache-Control” and value: max-age = (time for which you want your object to be accessed from cache in seconds), must-revalidate “i.e. Key: “Cache-Control” value: max-age = “2592000, must re-validate” for 30 days. “must re-validate” string must be added after the time in second as value.
Assuming you have properly set the cache control headers and you are using signed URL's than you will need to hold onto the signed URL for a given file and re-render the exact same URL in subsequent page loads.
If you have not set the cache control headers or you want them to change based on who is making the request you can set them before signing your URL with: &response-cache-control=value or &response-expires=value.
I currently using carrierwave with fog to store and upload images using an s3 bucket but the images load much slower than they should. These images load almost instantly when stored as part of the application - but stored with carrierwave and fog it takes a few seconds.
Is this a problem with my s3 setup or carrierwave/fog? My carrierwave config is the following:
CarrierWave.configure do |config|
config.fog_credentials = {
:provider => 'AWS', # required
:aws_access_key_id => '***', # required
:aws_secret_access_key => '***', # required
}
config.cache_dir = "#{Rails.root}/tmp/uploads" # To let CarrierWave work on heroku
config.fog_directory = 'bucketname' # required NB: having '.' in the bucket name creates an untrusted certificate
config.fog_public = false # optional, defaults to true
config.fog_attributes = {'Cache-Control'=>'max-age=315576000'} # optional, defaults to {}
end
I do have my s3 bucket configured for the US and I'm located in Australia so that might pose a few problems - but my heroku app is also configured to the US and it loads the same images blazingly quick when they're stored as part of the app itself. Maybe aws isn't the best solution?
Anyway any solutions on how I can improve the speed of image load time would be great. It just seems unnecessarily slower than it should be.
It sounds like you want to use CloudFront, Amazon's CDN (content delivery network) service that integrates with S3. Using a CDN will globally replicate the content you're storing in CDN (for a price), which should improve your load times.
After you set up a CloudFront account and link it to S3, add a line like the following to your CarrierWave configuration:
config.asset_host = "http://1234567.cloudfront.net"
With the URL that you get during CloudFront setup.
Unfortunately it looks like you may also need to set config.fog_public = true for Carrierwave to be able to use Amazon's CDN.