I've got a rails app hosted on Heroku for which I'm trying to update the robot.txt file.
The local file, which is at /public/robots.txt, reads:
User-agent: *
Disallow: /admin/
However when I deploy the app to Heroku, the robots file is seemingly not updated. The remote version reads:
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-agent: *
Disallow: /
The live url is at http://www.cowboypicks.com/robots.txt.
Running curl -I http://www.cowboypicks.com/robots.txt yields:
HTTP/1.1 200 OK
Age: 2317078
Cache-Control: public, max-age=2592000
Content-length: 200
Content-Type: text/plain
Date: Wed, 30 Apr 2014 17:01:43 GMT
Last-Modified: Thu, 03 Apr 2014 14:21:08 GMT
Status: 200 OK
X-Content-Digest: 10c5b29b9aa0c6be63a410671662a29a796bc772
X-Rack-Cache: fresh
Connection: keep-alive
Indicating the file hasn't been updated since 3 April, however it has been updated today (30 April). Even stranger, when I run heroku run bash followed by cat public/robots.txt I get:
User-agent: *
Disallow: /admin/
Indicating the file is being updated on Heroku but it's showing an older (I assume cached) version for some reason. There is some caching on the app using dalli/memcache, but I wouldn't have thought that would affect static files? Any ideas on how I could further debug the problem?
It turned out that Dalli had indeed been caching the robots.txt file in production. The expiry date was set in production.rb with the line:
config.static_cache_control = "public, max-age=2592000"
Running the following from the rails console flushed the cache and sorted the problem:
Rails.cache.dalli.flush_all
Related
Nginx 1.10.1 Rails 5.0.1. The asset pipeline is making both the zipped and native versions of the css and js files, but only the uncompressed is served to the browser. I can see both versions in public/assets and I can use curl to retrieve the zipped version by appending '.gz' to the css/js asset url delivered to me.
I am using a CDN (AWS CloudFront) but have tested without the CDN and anyway it should still point me to the zipped version, right?
nginx has --with-http_gzip_static_module. Using this answer as a guide, my nginx config (edited) has:
http {
server {
listen 80;
server_name idoimaging.com www.idoimaging.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name idoimaging.com www.idoimaging.com;
root /var/www/idoimaging/current/public;
location ~ ^/(assets)/ {
gzip_static on;
}
}
}
I've also tried /assets/ as the regex in location. In my production.rb:
# Have also tried setting this to false
config.public_file_server.enabled = ENV['RAILS_SERVE_STATIC_FILES'].present?
config.middleware.insert_before(Rack::Sendfile, Rack::Deflater)
config.assets.compress = true
config.assets.js_compressor = Uglifier.new(mangle: false)
I'm testing with Chrome with cache disabled, also for speed with curl https://idoimaging.com | grep assets which I believe should have the same behaviour as the browser?
I don't know why nginx will not serve the gzip'ed js/css asset files when they are present.
EDIT: I've also seen this guide that takes another approach: Use Rails as the static asset server instead of nginx. Would this be a better approach?
Given that Rails is producing the gzipped assets, we can be confident that the issue resides solely with nginx. So let's focus there!
I believe Nginx is faster at serving static assets than Rails is, so I'd stay away from using it as a static asset server in any event.
Now looking at the URL you provided (https://idoimaging.com), your server appears to be providing gzipped files. So the issue is just with your testing approach (assuming this is the correct URL, and you have not changed the server configuration since this post).
Your curl command doesn't include the Accept-Encoding: gzip header, which tells the server your client is capable of handling gzipped files. Without it, Nginx will serve the uncompressed versions. You can see the difference in commands and outputs in this gist. The difference is in the Content-Length and Content-Encoding response headers.
If you're seeing something different, let me know!
Edit 1
That's odd, CloudFront appears to have cached redirects for both your CSS and your JS.
richardseviora:Richards-MacBook-Pro#~> curl "https://cdn.idoimaging.com/assets/application-0cd41e63d35c1e5a7ab76ded23fbaf2ef1d1b786144134a80a1dfa9c765cff0d.css" -I -H "accept-encoding: gzip"
HTTP/1.1 301 Moved Permanently
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Server: nginx/1.10.1
Date: Thu, 23 Feb 2017 03:30:49 GMT
Location: https://idoimaging.com/assets/application-0cd41e63d35c1e5a7ab76ded23fbaf2ef1d1b786144134a80a1dfa9c765cff0d.css
Age: 942
X-Cache: Hit from cloudfront
Via: 1.1 d8b73f8fefd106d5c95f11977e132c46.cloudfront.net (CloudFront)
X-Amz-Cf-Id: ao8PwibmSj1JhmfbmuNfC2gYi9x-RTcCrJDAqLWAUIyOjP_3qYTGQA==
# It should look like this instead.
richardseviora:Richards-MacBook-Pro#~> curl -I -H "accept-encoding: gzip" "http://cdn.sweatrecord.com/assets/application-b932da0ddcf53d3650da5135b083224e863b349c784f3d1e3ca992b36ce3e31d.css"
HTTP/1.1 200 OK
Content-Type: text/css
Connection: keep-alive
Accept-Ranges: bytes
Content-Encoding: gzip
Date: Thu, 23 Feb 2017 03:50:13 GMT
Last-Modified: Mon, 30 Jan 2017 16:29:44 GMT
Server: Apache
Vary: Accept-Encoding,Origin
X-Cache: Miss from cloudfront
Via: 1.1 8b5947aba7280333032d4dcdd80b3489.cloudfront.net (CloudFront)
X-Amz-Cf-Id: FN9FyKl0RCpNTTqBwb0WyQhbDd-rEyyQ05eCtaFCD8YaH_FtjG7Q8Q==
This is Nginx issue, but I'm not sure where exactly because CloudFront will cache 301s.
I've website at http://revoniaga.com which is run RoR but the font won't load where it said "blocked from loading by Cross-Origin Resource Sharing policy". It cause by Amazon CloudFront. I've try everything but still same result.
First, I put my font assets under /app/assets/font
then in my css (vendor/assets/stylesheets/font-awesome.css), I use something like this:
#font-face {
font-family: 'FontAwesome';
src: url('/assets/fontawesome-webfont.eot?v=4.3.0');
//and so on for other font format
}
At my /etc/nginx/sites-available/revoniaga_production, I put
location ~* \.(eot|otf|svg|ttf|woff|woff2)$ {
add_header Access-Control-Allow-Origin *;
}
Then when I curl:
curl -I https://d3bkb7gt2ds4m6.cloudfront.net/assets/fontawesome-webfont.woff2
It said
HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 168
Connection: keep-alive
Server: nginx/1.6.2
Date: Wed, 18 Mar 2015 09:34:29 GMT
Age: 242
X-Cache: Error from cloudfront
Via: 1.1 404e3b476748051f3f9bc690b72173b4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 3yUNjkFQHC83FlbEwVMBSddpQND_4GbXauyeTtDyKawS6GzftUwXRA==
I'm also restarting my server "service nginx restart" but still nothing happen
Here is my CloudFront which is the default setting
Everything work fine in my dev env
I couldn't figure why this happening. Please help
Thanks in advance
Problem solved by clean install the server, upgrade ruby to 2.2.1 plus.. looks like the issue come from the server itself. my setting before was fine..
We're hosting a Rails app on Heroku, and having some trouble serving some static assets out of the public folder. (I know, this isn't recommended. We actually use CloudFront with a custom origin of our main application so we're getting the benefit of a CDN still. But that's irrelevant for this issue.)
Some assets serve up fine and exactly as expected. Ex: /404.html, /favicon.png
Others return an HTTP 406 error: /video-js/swf/video-js-4.1.0.swf
Rails 4.1.1
Ruby 2.1.2
The rails_12factor gem is present.
production.rb file sets config.serve_static_assets = true
The file definitely exists.. Using the console on Heroku:
File.size("/app/public/video-js/swf/video-js-4.1.0.swf")
=> 14059
But when I try to access the file, I get an HTTP 406 Not Acceptable response
curl -I http://my-rails-application.com/video-js/swf/video-js-4.1.0.swf
returns:
HTTP/1.1 406 Not Acceptable
Content-length: 0
Content-Type: text/html; charset=utf-8
Date: Wed, 04 Jun 2014 02:25:37 GMT
Status: 406 Not Acceptable
X-Request-Id: d184b097-7326-49cb-ad05-539459f7df08
X-Runtime: 0.532988
Connection: keep-alive
I tried adding a Mime Type for swf files (Mime::Type.register "application/x-shockwave-flash", :swf) and now I get a 404 instead of a 406, but that's still not very useful.
To recap, html, png, jpg, ico, and txt files all are served perfectly. swf, ttf, woff and a few others aren't being served out of the public folder.
What could cause some file types in the public folder to work properly, but not others?
I am running my producction environment with following settings, and it appears that it is minifying but is not gzipping/compressing the response.
# Disable Rails's static asset server (Apache or nginx will already do this)
config.serve_static_assets = false
# Compress JavaScripts and CSS
config.assets.compress = true
# Don't fallback to assets pipeline if a precompiled asset is missed
config.assets.compile = false
# Generate digests for assets URLs
config.assets.digest = true
Here is my js file, and the headers dont contain gzip or deflate???
$ curl -I http://sctest-perf.herokuapp.com/assets/application-1d21d488644b11e5610fa26bacdbc868.js
HTTP/1.1 200 OK
Age: 0
Cache-Control: public, max-age=31536000
Content-length: 213394
Content-Type: application/javascript
Date: Fri, 08 Jun 2012 20:34:58 GMT
Etag: "1d21d488644b11e5610fa26bacdbc868"
Last-Modified: Wed, 30 May 2012 16:38:32 GMT
Status: 200 OK
X-Content-Digest: 914bc6f80b0619be63c1302821ccc24a082ace53
X-Rack-Cache: miss, store
X-Request-Id: 76cb22ef23b73c440878569eec42ed4d
X-Runtime: 0.001314
X-Ua-Compatible: IE=Edge,chrome=1
Connection: keep-alive
You need to edit the config.ru file, located in your project root:
require ::File.expand_path('../config/environment', __FILE__)
# Add beautifully gzipped responses
use Rack::Deflater
run YourApp::Application
You could also try using one of these gems which are supposed to help optimize the serving of assets process further.
https://github.com/romanbsd/heroku-deflater
https://github.com/mattolson/heroku_rails_deflate
If I'm not mistaken using use Rack::Deflater also compresses images which wastes resources and isn't useful.
My Ruby on Rails application uses the following controller code to generate a sitemap.xml file:
class SitemapController < ApplicationController
layout nil
def index
headers['Content-Type'] = 'application/xml'
last_post = Post.last
if stale?(:etag => last_post, :last_modified => last_post.updated_at.utc)
respond_to do |format|
format.xml { #posts = Post.sitemap } # sitemap is a named scope
end
end
end
end
My understanding is that the stale? method should ensure a HTTP 304 Not Modified response if the content hasn't changed. However, whenever I test this using curl or a web browser I always get an HTTP 200:
$ curl --head localhost:3000/sitemap.xml
HTTP/1.1 200 OK
Connection: close
Date: Mon, 13 Apr 2009 15:50:00 GMT
Last-Modified: Wed, 08 Apr 2009 16:52:07 GMT
X-Runtime: 100
ETag: "5ff2ed60ddcdecf291e7191e1ad540f6"
Cache-Control: private, max-age=0, must-revalidate
Content-Type: application/xml; charset=utf-8
Content-Length: 29318
Am I using the stale? method correctly? Is it even possible to test this locally?
it is likely that your Rails code is just fine but curl is not sending the If-Modified-Since header when you perform your test. From the curl docs:
TIME CONDITIONS
HTTP allows a client to specify a time
condition for the document it
requests. It is If-Modified-Since or
If-Unmodified-Since. Curl allow you to
specify them with the -z/--time-cond
flag.
For example, you can easily make a
download that only gets performed if
the remote file is newer than a local
copy. It would be made like:
curl -z local.html
http://remote.server.com/remote.html
Or you can download a file only if the
local file is newer than the remote
one. Do this by prepending the date
string with a '-', as in:
curl -z -local.html
http://remote.server.com/remote.html
You can specify a "free text" date as
condition. Tell curl to only download
the file if it was updated since
yesterday:
curl -z yesterday
http://remote.server.com/remote.html
Curl will then accept a wide range of
date formats. You always make the date
check the other way around by
prepending it with a dash '-'.