Our app (1 repo) has a Rails backend and an Angular frontend. As such, the deployment process has an npm install, bower install, grunt build --force at some point. The problem is that it takes a long time to deploy, since these commands are still executed even though we are just updating Rails-related things.
Is there some kind of hook so that I can check that if the folder containing frontend code has changes, then npm install? Or should we just split the repo into two repos with their own deploy processes each?
apistrano-faster-assets plugins enables such functionality for plain Rails assets.
You might want to check the core task to see how that's done and adapt or copy-paste the code for your use.
Here's my attempt to extract only the relevant steps and provide some more comments:
class PrecompileRequired < StandardError; end
begin
# get the previous release
latest_release = capture(:ls, '-xr', releases_path).split[1]
# precompile if this is the first deploy
raise PrecompileRequired unless latest_release
# create a 'Pathname' object for latest_relase
latest_release_path = releases_path.join(latest_release)
# execute raises if there is a diff
execute(:diff, '-Naur', release_path.join('path/to/frontend/code'), latest_release_path.join('path/to/frontend/code')) rescue raise(PrecompileRequired)
info("Skipping asset precompile, no asset diff found")
# copy over all of the assets from the last release
execute(:cp, '-r', latest_release_path.join('public', fetch(:assets_prefix)), release_path.join('public', fetch(:assets_prefix)))
rescue PrecompileRequired
# execute compile command here
end
Related
I'm upgrading a large, commercial (proprietary) Rails 6 application to Rails 7. We never used Webpacker, and are instead going directly from bundled gems for things like Bootstrap, to the "Rails 7 way".
It turns out that the "no Node" workflow for Rails 7 has no good answer for components that consist of both a CSS and JS component. In our case, the most obvious offender there is Bootstrap. Faced with maintaining the JS "half" of Bootstrap through import maps and the CSS "half" through something like the old Bootstrap gem or manual vendoring (and yes, there really is no other solution without Node here) we end up back at a full Node workflow.
This is coming together. All front-end components that provide CSS and/or JS were already happily available in NPM, so now that's all managed via package.json & Yarn, with bin/dev driving Sass & esbuild compilation of the SCSS and JS components pulled from either app/assets, app/javascript or node_modules/...; the asset pipeline manifest.js contains only references to the build and images folders inside app/assets as a result.
It feels like a bit of backwards step with all the heavyweight manual maintenance of lists of filenames (wildcard imports no longer supported) along with the complexity of the multiple processes now running under Foreman vs just having things synchronously processed in Sprockets on a per-request basis, but with all that stuff being deprecated/abandonware, it was clearly time to update.
This all works fine in dev & production mode, but what about test? We use RSpec; in CI, there's no built assets and developers don't want to have to remember to run esbuild or assets:precompile or whatever every time they're about to run rspec. Apart from anything else, it's quite slow.
What's the official, idiomatic Rails 7 solution in a Yarn/Node-based workflow specifically using cssbundling-rails and jsbundling-rails, when you want to run tests with up to date assets?
This is pretty ropey but better than nothing for now; it'll ensure CI always builds assets and also ensure that local development always has up-to-date assets, even if things have been modified when e.g. bin/dev isn't running.
# Under Rails 7 with 'cssbundling-rails' and/or the 'jsbundling-rails' gems,
# entirely external systems are used for asset management. With Sprockets no
# longer synchronously building assets on-demand and only when the source files
# changed, compiled assets might be (during local development) or will almost
# always be (CI systems) either out of date or missing when tests are run.
#
# People are used to "bundle exec rspec" and things working. The out-of-box gem
# 'cssbundling-rails' hooks into a vanilla Rails "prepare" task, running a full
# "css:build" task in response. This is quite slow and generates console spam
# on every test run, but points to a slightly better solution for RSpec.
#
# This class is a way of packaging that solution. The class wrapper is really
# just a namespace / container for the code.
#
# First, if you aren't already doing this, add the folllowing lines to
# "spec_helper.rb" somewhere *after* the "require 'rspec/rails'" line:
#
# require 'rake'
# YourAppName::Application.load_tasks
#
# ...and call MaintainTestAssets::maintain! (see that method's documentation
# for details). See also constants MaintainTestAssets::ASSET_SOURCE_FOLDERS and
# MaintainTestAssets::EXPECTED_ASSETS for things you may want to customise.
#
class MaintainTestAssets
# All the places where you have asset files of any kind that you expect to be
# dynamically compiled/transpiled/etc. via external tooling. The given arrays
# are passed to "Rails.root.join..." to generate full pathnames.
#
# Folders are checked recursively. If any file timestamp therein is greater
# than (newer than) any of EXPECTED_ASSETS, a rebuild is triggered.
#
ASSET_SOURCE_FOLDERS = [
['app', 'assets', 'stylesheets'],
['app', 'javascript'],
['vendor']
]
# The leaf files that ASSET_SOURCE_FOLDERS will build. These are all checked
# for in "File.join(Rails.root, 'app', 'assets', 'builds')". Where files are
# written together - e.g. a ".js" and ".js.map" file - you only need to list
# any one of the group of concurrently generated files.
#
# In a standard JS / CSS combination this would just be 'application.css' and
# 'application.js', but more complex applications might have added or changed
# entries in the "scripts" section of 'package.json'.
#
EXPECTED_ASSETS = %w{
application.js
application.css
}
# Call this method somewhere at test startup, e.g. in "spec_helper.rb" before
# tests are actually run (just above "RSpec.configure..." works reasonably).
#
def self.maintain!
run_build = false
newest_mtime = Time.now - 100.years
# Find the newest modificaftion time across all source files of any type -
# for simplicity, timestamps of JS vs CSS aren't considered
#
ASSET_SOURCE_FOLDERS.each do | relative_array |
glob_path = Rails.root.join(*relative_array, '**', '*')
Dir[glob_path].each do | filename |
next if File.directory?(filename) # NOTE EARLY LOOP RESTART
source_mtime = File.mtime(filename)
newest_mtime = source_mtime if source_mtime > newest_mtime
end
end
# Compile the built asset leaf names into full file names for convenience.
#
built_assets = EXPECTED_ASSETS.map do | leaf |
Rails.root.join('app', 'assets', 'builds', leaf)
end
# If any of the source files are newer than expected built assets, or if
# any of those assets are missing, trigger a rebuild task *and* force a new
# timestamp on all output assets (just in case build script optimisations
# result in a file being skipped as "already up to date", which would cause
# the code here to otherwise keep trying to rebuild it on every run).
#
run_build = built_assets.any? do | filename |
File.exist?(filename) == false || File.mtime(filename) < newest_mtime
end
if run_build
Rake::Task['javascript:build'].invoke()
Rake::Task[ 'css:build'].invoke()
built_assets.each { | filename | FileUtils.touch(filename, nocreate: true) }
end
end
end
(EDIT) As a commenter below points out, you'll need to make sure Rake tasks are loaded in your spec_helper.rb, e.g.:
require 'rake'
Rails.application.load_tasks
Both jsbundling-rails and cssbundling-rails append themselves into a rake task called test:prepare.
There are a few ways to cause test:prepare to run, depending on your overall build process.
Call it directly:
bundle exec rails test:prepare test
Or, if running rspec outside of the rails command:
bundle exec rails test:prepare && bundle exec rspec
Use a test task that already calls test:prepare.
Curiously, only some test tasks call (depend on) test:prepare, while others (including the default test task) don't. Example:
bundle exec rails test:all
Make test:prepare a dependency on your preferred test task.
For example, if you normally use the spec task by running bundle exec rails spec, add this to a new or existing task file (such as lib/tasks/tests.rake):
task spec: ['css:build', 'javascript:build']
Background
test:prepare is an empty task defined by Rails. Both cssbundling-rails and jsbundling-rails add themselves as dependencies of that task.
In general, test:prepare is a useful place to add any kind of dependency needed to run your tests, with the caveat that only some of Rails' default test tasks depend on it. But as mentioned above, you can always call it directly or add your own dependencies.
In most cases, calling test:prepare is going to be equivalent to calling both css:build and javascript:build, which is why I showed test:prepare in most of the above examples. On occasion, other gems or your app may an extended test:prepare with additional commands as well, in which case those will run as well (and would likely be wanted).
Also note that assets:precompile also depends on css:build and javascript:build. In my experience, test:prepare (or css:build and javascript:build separately) runs faster than assets:precompile, likely because we're running a lightweight configuration of sprockets-rails (as opposed to propshaft) and assets:precompile runs the entire sprockets compilation process.
Capistrano v2 had two helpful tasks: cap deploy:update_code would do a full deployment to a new releases/ directory, but not change the current symlink or start/restart the server (so the server keeps running the current version without interruption). And cap deploy:update did the same thing plus changing the current symlink, but didn't start/restart the server. These were useful to shake out issues e.g. with asset compilation, before doing a real deploy.
Those two "update" tasks are gone in Capistrano v3. Is there an equivalent way to do a full deploy without changing the current symlink or restarting the server?
A custom task list this should do it:
task :deploy_without_symlink do
set(:deploying, true)
%w{ starting started
updating updated }.each do |task|
invoke "deploy:#{task}"
end
end
You can look at the code here: https://github.com/capistrano/capistrano/blob/master/lib/capistrano/tasks/framework.rake#L58 to see what deploy triggers. And the Publishing task per https://github.com/capistrano/capistrano/blob/master/lib/capistrano/tasks/deploy.rake#L38 is what changes the symlinks. So by omitting everything afterwards, you get what you are looking for.
I would like to run cap development deploy:setup and cap development deploy to setup the dev environment for my rails app. In order to do this, I will have to remove the project folder from the remote machine, is there a way to automate this in some fashion using cap.
I basically want to remove the app folder in remote machine before I do a full deploy.
The Capistrano app folder is structured like this:
releases - Contains separate folders (named using commit IDs or timestamps) for each deployment made so far
current - Linked to the latest deployment folder under releases
shared - Shared logs, gem files, and tmp which are used between deployments
Now do you really want to clean the entire app folder? That could be dangerous since you'll lose logs.
current is just a symbolic link, it points to the latest release only.
If you want to clean up old releases, check this answer. You can set keep_releases to 1.
If you want to delete the shared folder as well, then you have to write your own Capistrano hook as #Strik3r mentioned.
before 'deploy:setup', 'removecode'
task :removecode do
run "rm -rf <path>", :shell => fetch(:rvm_shell)
end
add this code in to your deploy.rb file, this will call before deploy:setup
in this way you can create a task and do what ever you want to do
I am looking into using capybara-webkit to do somewhat close-to-reality tests of app. This is absolutely neccessary as the app features a very rich JS-based UI and the Rails part is mostly API calls.
The question is: is there any tools to integrate into testing pipeline which could instrument Javascript code and report its coverage? The key here is the ability to integrate into testing workflow (just like rcov/simplecov) easily – I don't like the idea do it myself with jscoverage or analogue :)
Many thanks in advance.
This has now been added to JSCover (in trunk) - the related thread at JSCover is here.
I managed to get JSCover working in the Rails + Capybara pipeline, but
it did take quite a bit of hacking to get it to work
These changes are now in JSCover's trunk and will be part of version 1.0.5. There's working examples (including a Selenium IDE recorded example) and documentation in there too.
There is some additional work needed to get the branch detection to
work since that uses objects that cannot be easily serialized to JSON
This is a function to do this which is used in the new code.
Anyway, the end result works nicely
I agree. This makes JSCover useable by higher level tools that don't work well with iFrames or multiple windows which are avoided by this approach. It also means code coverage can be added to existing Selenium tests with two adjustments:
Make the tests run through the JSCover proxy
Save the coverage report at the end of the test suite
See JSCover's documentation for more information. Version 1.0.5 containing these changes should be released in a few days.
Update: Starting from JSCover version 1.05 the hacks I outlined in my previous answer are no longer needed. I've updated my answer to reflect this.
I managed to get JSCover working in the Rails + Capybara pipeline, but it did take some hacking to get it to work. I built a little rake task that:
uses the rails asset pipeline to generate the scripts
calls the java jar to instrument all the files and generate an empty report into a temp dir
patches the jscover.js script to operate in "report mode" (simply add jscoverage_isReport=true at the end)
copies the result to /public/assets so the tests pick it up without needing any changes and so the coverage report can be opened automatically in the browser
Then I added a setup task to clear out the browser's localStorage at the start of the tests and a teardown task that writes out the completed report at the end.
def setup
unless $startup_once
$startup_once=true
puts 'Clearing localStorage'
visit('/')
page.execute_script('localStorage.removeItem("jscover");')
end
end
def teardown
out=page.evaluate_script("typeof(_$jscoverage)!='undefined' && jscoverage_serializeCoverageToJSON()")
unless out.blank? then
File.open(File.join(Rails.root,"public/assets/jscoverage.json"), 'w') {|f| f.write(out) }
end
end
Anyway, the end result works nicely, the advantage of doing it this way is that it also works on headless browsers so it can also be included in CI.
*** Update 2: Here is a rake task that automates the steps, drop this in /lib/tasks
# Coverage testing for JavaScript
#
# Usage:
# Download JSCover from: http://tntim96.github.io/JSCover/ and move it to
# ~/Applications/JSCover-1
# First instumentalize the javascript files:
# rake assets:coverage
# Then run browser tests
# rake test
# See the results in the browser
# http://localhost:3000/assets/jscoverage.html
# Don't forget to clean up instrumentalization afterwards:
# rake assets:clobber
# Also don't forget to re-do instrumentalization after changing a JS file
namespace :assets do
desc 'Instrument all the assets named in config.assets.precompile'
task :coverage do
Rake::Task["assets:coverage:primary"].execute
end
namespace :coverage do
def jscoverage_loc;Dir.home+'/Applications/JSCover-1/';end
def internal_instrumentalize
config = Rails.application.config
target=File.join(Rails.public_path,config.assets.prefix)
environment = Sprockets::Environment.new
environment.append_path 'app/assets/javascripts'
`rm -rf #{tmp=File.join(Rails.root,'tmp','jscover')}`
`mkdir #{tmp}`
`rm -rf #{target}`
`mkdir #{target}`
print 'Generating assets'
require File.join(Rails.root,'config','initializers','assets.rb')
(%w{application.js}+config.assets.precompile.select{|f| f.is_a?(String) && f =~ /\.js$/}).each do |f|
print '.';File.open(File.join(target,f), 'w') {|ff| ff.write(environment[f].to_s) }
end
puts "\nInstrumentalizing…"
`java -Dfile.encoding=UTF-8 -jar #{jscoverage_loc}target/dist/JSCover-all.jar -fs #{target} #{tmp} #{'--no-branch' unless ENV['C1']} --local-storage`
puts 'Copying into place…'
`cp -R #{tmp}/ #{target}`
`rm -rf #{tmp}`
File.open("#{target}/jscoverage.js",'a'){|f| f.puts 'jscoverage_isReport = true' }
end
task :primary => %w(assets:environment) do
unless Dir.exist?(jscoverage_loc)
abort "Cannot find JSCover! Download from: http://tntim96.github.io/JSCover/ and put in #{jscoverage_loc}"
end
internal_instrumentalize
end
end
end
I was deploying my code in openshift rails but it did not take the assets properly.
It's neither loading the javascripts nor the images.
How to make it work?
The first thing you will want to check is if the assets exist on your application. This script is run before the build.sh script (mentioned next). It attempts to create a symlink for your public/assets folder (here is the reasoning behind this). Even if you had that directory, the rake task would still work; the assets would just be wiped every push.
Your assets should be compiling whenever you push to your git repository. This is taken care of by this script in the Ruby 1.9 cartridge (this is run by the service and you have no control over it). As you can see, it will run bundle exec rake assets:precompile as long as you have a Gemfile (which should be the case for all Rails apps).
With all that being said, the first thing you should do is check your .openshift/action_hooks to make sure you are not running anything that might be overwriting your public/assets directory. You can compare them against the ones here.
The next thing you should do is actually check the directory on your OpenShift host. You can do this by SSHing into your app (instructions are here). Then check your public/assets directory. Note: Some of the output has been shortened with .....
# First we make sure it is a symlink.
> file $OPENSHIFT_REPO_DIR/public/assets
..../app-root/runtime/repo/public/assets: symbolic link to `..../data//assets'
# Then we see if there is anything in it
> ls $OPENSHIFT_REPO_DIR/public/assets
.... (should have a bunch of.js, .css, etc files)
If that directory is empty, maybe there is a problem with compiling the assets. You should pay attention to the output when you git push and see if there is any indication that it is failing (you may want to capture the output using tee like: git push 2>&1 | tee git_push.log). If there are assets in the directory, check your logs by following these steps.
If you're still having problems, swing by our IRC channel and somebody should be able to help in person.