Trying to debug why we're getting cache misses. In config/environments/production.rb we have asset fallback to true:
config.assets.compile = true
Every once in a while we see CPU surges and the culprit is nodejs, which we believe is firing due to this line (it's compiling assets when there's a miss). Is there a way to log, either to the production log, or it's own, when an asset is missed, so we can figure out why it's not being added to the precompile locus?
Thanks for any help,
Kevin
The directive you mentioned is for asset compiling, not caching. The config.rb directive for caching is (https://guides.rubyonrails.org/caching_with_rails.html)
config.action_controller.perform_caching = true|false
This will not affect the caching of your static assets though, such as images, javascript files, or stylesheets. For that, you need to look at your web server settings and see why your static assets aren't being cached properly. You indicated that you get a spike once in a while. Are the spikes consistently spaced apart? If so then it may just be that your cache time to live is set too low and you need to bump it up. Do you see your spikes right after a deploy? If so, your assets may not be precompiled in that environment. You may even have both problems.
Related
The situation
I couldn't get my vendor assets to precompile in heroku without specifying each individual file to precompile in config/initializers/assets so resorted to setting
config.assets.compile = true
Note: I didn't require vendor assets in application.js because I'm calling them on a per page basis when they are needed.
Anyhow I setup a Cloudfront account and now everything is working as it does in development. But on deploy to Heroku, there is a warning and a link that leads to a StackOverflow post, warning against setting config.assets.compile to true.
Compile Set to True in Production If you have enabled your application
to config.assets.compile = true in production, your application might
be very slow. This was best described in a stack overflow post:
When you have compile on, this is what happens: Every request for a
file in /assets is passed to Sprockets. On the first request for each
and every asset it is compiled and cached in whatever Rails is using
for cache (usually the filesystem). On subsequent requests Sprockets
receives the request and has to look up the fingerprinted filename,
check that the file (image) or files(css and js) that make up the
asset were not modified, and then if there is a cached version serve
that.
This setting is also known to cause other run-time instabilities and
is generally not recommended. Instead we recommend either precompiling
all of your assets on deploy (which is the default) or if that is not
possible compiling assets locally.
My question is, Since I'm now using Cloudfront, does that cover me from what they are warning about, slowness etc.?
Thanks in advance for any advice :)
Yes, you're likely covered. The request for an asset will hit CloudFront first. After the first request, it will be cached and shouldn't hit your server for compilation. Every time your assets change, however, they'll have to recompile, which is of course very slow.
I'm trying to figure out, whether I should be using config.assets.prefixin my development environment or not.
When I'm using localhost, for development, are there any (dis-)advantages of doing this? When the local server itself isn't caching and is on another domain (production-domain vs localhost), I fail to see the disadvantages? Especially for hard-coding some paths in CSS and Javascript, which will then always return 404's on development..
I've been using config.assets.prefix = "/dev-assets" as pr. reccomendation of another developer, who isn't working with us anymore.
Is this a problem, that anyone else have thought about and taken a standpoint at?
The biggest reason of using config.assets.prefix, emerges from the use of local asset precompilation. There are multiple reasons of precompiling locally and storing in source control. From http://guides.rubyonrails.org/asset_pipeline.html#local-precompilation
You may not have write access to your production file system.
You may be deploying to more than one server, and want to avoid duplication of work.
You may be doing frequent deploys that do not include asset changes.
Local compilation allows you to commit the compiled files into source control, and deploy as normal.
But if you use this, you will get into trouble in development mode because rails will serve your precompiled assets found in /public/assets. Therefore, you often set config.assets.prefix = '/dev-assets' so you don't need to precompile every time you want to see effects of local changes in /app/assets.
This month, I upgraded from Rails 3.0 to Rails 3.1 - this week I tried to launch the server in production mode - today I have hit a wall !
I am unable to get my production environment server to serve up my public assets (JavaScript and CSS) via the asset pipeline, unless I set config.assets.compile = truein my environment.rb file, which for speed reasons I obviously don't want to do.
I have a large number of JS and CSS files, each of which tends to get used on one or two different pages. This means creating a single "manifest" file doesn't fit my usage, as each page wants something slightly different. I also expect some of the CSS won't sit together nicely. Therefore I have gone down the approach of "just get it working", looking to tidy up the large quantity of CSS / JS later.
Here is the production.rb file:
Implicit::Application.configure do
...
config.consider_all_requests_local = false
config.action_controller.perform_caching = true
# I set this to true, as I am testing this locally, just running a local Thin server
config.serve_static_assets = true
config.assets.compress = true
# Setting this to false removes the issue - but is SLOW
config.assets.compile = true
config.assets.digest = true
# This is overkill - but does get EVERYTHING precompiled for now
config.assets.precompile += %w( *.css *.js )
config.action_dispatch.x_sendfile_header = nil
...
end
This is quite a new area for me, and so I've spent much of today toggling these booleans and stop/starting the local Thin / Rails server to try them out. But the only value that's made a solid difference is the compile flag.
My application.rb file is pretty much standard, and has config.assets.enabled = true and config.assets.initialize_on_precompile = false in it, the latter from a heroku post (and a guess, again).
I have a fully populated public/assets directory, created with the bundle exec rake assets:precompile command, that takes about 20 mins to run on my oldish laptop (5 years), probably something to do with that "catch all" precompile regex, although with that line commented it still takes over 10 mins (!)
With the compile flag set to true, I can see copies of these assets getting created in my /tmp/cache directory - this is clearly the application creating it's own "compiled copy" of the assets.
With the compile flag set to false, I am confronted with the error message (in the logs, unless I set requests to local, then I see it on the rendered error page) of jquery.reveal isn't precompiled. However, when I go to http://localhost:3000/assets/jquery.reveal.js the javascript file is served up.
The line of my application layout causing this is:
<%= javascript_include_tag "application", "jquery.reveal" %>
I have tried changing that jquery.reveal to jquery.reveal.js with no change. Removing it fixes the index page, except that the jquery.reveal functionality is gone of course ! So clearly the application.js is being served up correctly. I just can't figure out why jquery.reveal isn't, as I can see the precompiled files in the public/assets directory.
This is an interesting issue, and I think there may be two bugs - the one you've linked and another: the file is being being compiled to the wrong name. It might be worth trying to generate a minimal test case that you can submit with a bug report.
The workaround for this - and I suspect that this is why few people seem to have the problem - is to use a secondary manifest (linking libraries only via a manifest seems to be an evolving best-practice).
Create one called home.js and require just that one file to it.
This isn't a bad approach overall. These extra manifests have to be added to the precompile array (see the guide), and allow multiple libraries to be shared over several pages or sections without having to link them each time.
Answering my own question here, but looks like it might be a bug in parsing assets with "periods" such as jquery.reveal
An issue report (https://github.com/rails/rails/issues/3398) reports this and highlights a commit (https://github.com/sstephenson/sprockets/commit/4ba5b32764a9073671df5e77355df6ed2bb3d3c9) which occurs just after Sprockets 2.0.3 - the default version that rails 3.1.3 relies upon. Upgrading Sprocket would therefore involve moving onto 3.2-stable rails - a bit bleeding edge for this newbie !
But the bug report does say this only occurs when config.assets.compile = true - and I did see whilst that was the case in my code that jquery.reveal was being dynamically compiled to jquery-8fu...8g.reveal.js (instead of jquery.reveal-8fu...8g.js).
So perhaps this ISN'T the answer. At least I hope it isn't. But will certainly continue to look at this period issue, as the "non-period" containing CSS / JS files are being served up just fine, as far as I can tell.
The default Rails app installed by rails new has config.assets.compile = false in production.
And the ordinary way to do things is to run rake assets:precompile before deploying your app, to make sure all asset pipeline assets are compiled.
So what happens if I set config.assets.compile = true in production?
I wont' need to run precompile anymore. What I believe will happen is the first time an asset is requested, it will be compiled. This will be a performance hit that first time (and it means you generally need a js runtime in production to do it). But other than these downsides, after the asset was lazily compiled, I think all subsequent access to that asset will have no performance hit, the app's performance will be exactly the same as with precompiled assets after this initial first-hit lazy compilation. is this true?
Is there anything I'm missing? Any other reasons not to set config.assets.compile = true in production? If I've got a JS runtime in production, and am willing to take the tradeoff of degraded performance for the first access of an asset, in return for not having to run precompile, does this make sense?
I wrote that bit of the guide.
You definitely do not want to live compile in production.
When you have compile on, this is what happens:
Every request for a file in /assets is passed to Sprockets. On the first request for each and every asset it is compiled and cached in whatever Rails is using for cache (usually the filesystem).
On subsequent requests Sprockets receives the request and has to look up the fingerprinted filename, check that the file (image) or files(css and js) that make up the asset were not modified, and then if there is a cached version serve that.
That is everything in the assets folder and in any vendor/assets folders used by plugins.
That is a lot of overhead as, to be honest, the code is not optimized for speed.
This will have an impact on how fast asset go over the wire to the client, and will negatively impact the page load times of your site.
Compare with the default:
When assets are precompiled and compile is off, assets are compiled and fingerprinted to the public/assets. Sprockets returns a mapping table of the plain to fingerprinted filenames to Rails, and Rails writes this to the filesystem. The manifest file (YML in Rails 3 or JSON with a randomised name in Rails 4) is loaded into Memory by Rails at startup and cached for use by the asset helper methods.
This makes the generation of pages with the correct fingerprinted assets very fast, and the serving of the files themselves are web-server-from-the-filesystem fast. Both dramatically faster than live compiling.
To get the maximum advantage of the pipeline and fingerprinting, you need to set far-future headers on your web server, and enable gzip compression for js and css files. Sprockets writes gzipped versions of assets which you can set your server to use, removing the need for it to do so for each request.
This get assets out to the client as fast as possible, and in the smallest size possible, speeding up client-side display of the pages, and reducing (with far-future header) requests.
So if you are live compiling it is:
Very slow
Lacks compression
Will impact render time of pages
Versus
As fast as possible
Compressed
Remove compression overheard from server (optionally).
Minimize render time of pages.
Edit: (Answer to follow up comment)
The pipeline could be changed to precompile on the first request but there are some major roadblocks to doing so. The first is that there has to be a lookup table for fingerprinted names or the helper methods are too slow. Under a compile-on-demand senario there would need to be some way to append to the lookup table as each new asset is compiled or requested.
Also, someone would have to pay the price of slow asset delivery for an unknown period of time until all the assets are compiled and in place.
The default, where the price of compiling everything is paid off-line at one time, does not impact public visitors and ensures that everything works before things go live.
The deal-breaker is that it adds a lot of complexity to production systems.
[Edit, June 2015] If you are reading this because you are looking for a solution for slow compile times during a deploy, then you could consider precompiling the assets locally. Information on this is in the asset pipeline guide. This allows you to precompile locally only when there is a change, commit that, and then have a fast deploy with no precompile stage.
To have less overhead with Pre-compiling thing.
Precompile everything initially with these settings in production.rb
# Precompile *all* assets, except those that start with underscore
config.assets.precompile << /(^[^_\/]|\/[^_])[^\/]*$/
you can then simply use images and stylesheets as as "/assets/stylesheet.css" in *.html.erb
or "/assets/web.png"
For anyone using Heroku:
If you deploy to Herkou, it will do the precompile for you automatically during the deploy if compiled assets are not included (i.e. public/assets not committed) so no need for config.assets.compile = true, or to commit the precompiled assets.
Heroku's docs are here. A CDN is recommended to remove the load on the dyno resource.
It won't be the same as precompiling, even after that first hit: because the files aren't written to the filesystem they can't be served directly by the web server. Some ruby code will always be involved, even if it just reads a cache entry.
Set config.asset.compile = false
Add to your Gemfile
group :assets do
gem 'turbo-sprockets-rails3'
end
Install the bundle
Run rake assets:precompile
Then Start your server
From the official guide:
On the first request the assets are compiled and cached as outlined in development above, and the manifest names used in the helpers are altered to include the MD5 hash.
Sprockets also sets the Cache-Control HTTP header to max-age=31536000. This signals all caches between your server and the client browser that this content (the file served) can be cached for 1 year. The effect of this is to reduce the number of requests for this asset from your server; the asset has a good chance of being in the local browser cache or some intermediate cache.
This mode uses more memory, performs poorer than the default and is not recommended.
Also, precompile step is not trouble at all if you use Capistrano for your deploys. It takes care of it for you. You just run
cap deploy
or (depending on your setup)
cap production deploy
and you're all set. If you still don't use it, I highly recommend checking it out.
Because it is opening a directory traversal vulnerability - https://blog.heroku.com/rails-asset-pipeline-vulnerability
This question is similar to Why do I get “no route matches” for requests to the asset pipeline?.
I have a rails 3.0 application that I upgraded to 3.1 and converted to use the new asset pipeline (thanks to RailsCasts #282 and #279).
In production mode, I'm seeing the application-<digest>.js and application-<digest>.css. Great! And if I look at the source of those files, I see they are compressed. Yee-haw! So that means the asset pipeline is working, right?
However, if I add ?debug_assets=1 to the URL so that I may view individual files, some of them are producing ActionController::RoutingError (No route matches [GET] "/assets/<filename>-<digest>.js"), and same goes for some CSS files. But not all, just some, and I can't figure out what makes some files do this and others not.
I've cleared out tmp/cache/* and restarted Passenger. I've bumped config.assets.version. I've restarted memcached. None of these seem to resolve it. But what's odd is this only comes up when I'm using ?debug_assets=1 in the URL; without it, I see just one JS and CSS file, all compressed and minified.
I don't use precompiled assets, by the way. But just for grins, I performed a rake assets:precompiled, and whaddya know? The ?debug_assets=1 now shows all JS and CSS files, and none of them are 404'd.
So I guess the question you might have is, "Why not just use precompiled assets and not worry about missing assets from lazy load?" Good point. Answer: I just like to make sure I understand what I am doing, what's happening, and that I'm doing things correctly.
application.rb:
config.assets.enabled = true
config.assets.version = '1.2'
production.rb:
config.assets.compress = true
config.assets.compile = true
config.assets.digest = true
config.assets.js_compressor = :uglifier
config.assets.css_compressor = :scss
development.rb:
config.assets.compress = false
# I keep this off during development because I want
# to make sure the compression isn't breaking my JS
config.assets.debug = false
If you precompile your assets and set compile to false, debug is disabled because you've told Rails to not use Sprockets at all, but assumes that the files can be served by nginx based on mappings in the asset pipeline manifest.
When compile is true (like you have) then requests for these assets (and the debug request) are sent back to Sprockets to be processed if the files are missing (which without being precompiled is the case).
I would have assumed Sprockets would serve the individual files for each digested name. This behavior sounds buggy to me, although I don't think it is intended to use debug in production anyway.