I've just set up a Ruby on Rails app on Heroku, but, I'm experiencing a CI setup performance problem, which is quite annoying.
During the setup of our unit tests, Heroku is always re-installing our gems, producing logs like:
Fetching activestorage 6.0.2.2
Installing activestorage 6.0.2.2
This is not the case when setting up our app, as it shows lines like this instead:
Using activestorage 6.0.2.2
In both cases, bundler looks to be called with the correct arguments:
(for CI setup)
Running: bundle install --without development --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
(for app setup)
Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
In both cases, we're having the line:
Bundled gems are installed into `./vendor/bundle`
So, I'm wondering: do you know why Heroku does not cache gems between CI test setups, while it's the case during consequent builds of our app?
If yes, is there a way to force the reuse of gems previously installed in our CI pipeline?
This is taking a lot of time and can benefit from that.
Thanks a lot
After asking Heroku's team, it was a bug in the ruby's buildpack.
This is now fixed, and works well.
Buildpack creators can cache data between CI build, it was just turned off during one of their upgrade.
Heroku does not cache gems between CI test setups
Heroku filesystem is an ephemeral filesystem. When you run the tests, a dyno is spun up with a brand new operating system. Then heroku install all the dependencies and starts your tests.
I think it is a common scenario with all the CI platforms. Everytime we run tests, they will spin up a new environment from start by installing all the dependencies.
One justification for this may be that tests change a lot of data on different levels. So the CI providers start from scratch so that previous test residue data can't interfere with current run.
Related
I'm trying to understand some of the details behind bundling for deployment that I can't wrap my head around. I've read a few posts on here such as this one:
What does Rails 3's Bundler "bundle install --deployment" exactly do?
and I feel I understand what it should do. On my computer, I ran bundle install initially and have been developing a project. However, I wanted to see if I could run it in deployment just to get a feel as to how a production server like Heroku sets up the application.
Therefore, I started by running bundle install --deployment, which correctly installs all my gems into the local vendor/bundle local directory. However, when I run bundle show [GEM], I'm still seeing the path to my system gem. I feel it should be showing a path to the local folder, but it's not.
Can someone clear up on what my misconception is?
Have a look at the description of the two on Bundler's site.
Running bundle install --deployment is to be run in the production environment, but will grab the gems from rubygems when run. Read more here under the 'Deploying Your Application' heading for the purpose of the --deployment flag.
I'm running multiple of my Rails apps on Heroku. I use most of the standards including Gemfile and Bundler. Gemfile.lock is commited with Git.
However, when I push to Heroku, instead of reading Gemfile.lock, it seems to just run bundle install --without development:test --path vendor/bundle --binstubs bin/ from afresh, which means it updates the gems every time I push, causing discrepancies between my dev and prod environments.
I get no error message, but that's not the behavior I expect. What am I doing wrong?
Thanks!
I think it does this s part of it's 'build everything from the ground up in case the developers system has any differences" philosophy.
For example if you were running "ibuntu" (made up) and your environment had different dependencies between gems due to the way ruby is implemented on your system. Its safer just to build everything from scratch o their system and make sure that any dependencies that come up in their stack get met.
I have an interesting error when installing gems directly from github (:git => 'whatever').
Firstly, when I remove all gems and run bundle install command, I get the following:
Installing gem1
Installing gem2
Using gem3 (the one from github)
Then when I want to check what I've got I see the following by using gem list:
gem1 (x.x.x)
gem2 (y.y.y)
No gem3... now, looking closer to the file system, I see the following:
ls -l ~/.rvm/gems/ruby-1.9.3-p125/gems
gem1
gem2
So where is gem 3? Not where I'm expecting it to be:
ls -l ~/.rvm/gems/ruby-1.9.3-p125/bundler/gems
gem3-213213213
So it goes under bundler/gems and is not visible to gem list... and by Capistrano deploy, which gives me following:
git://github.com/author/gem3.git (at master) is not checked out. Please run `bundle install`
I'm more worried about Capistrano unable to deploy... Anyone has any clues?
Bundler gets its gems from various sources on your system. As long as they are the correct version, it will pull them in.
When deploying, it has more strict/conservative behavior.
From bundle help install, in the section about Deployment Mode, which is used when the --deployment flag is specified:
Gems are installed to vendor/bundle not your default system loca-
tion
In development, it's convenient to share the gems used in your
application with other applications and other scripts run on the
system.
In deployment, isolation is a more important default. In addition,
the user deploying the application may not have permission to
install gems to the system, or the web server may not have permis-
sion to read them.
As a result, bundle install --deployment installs gems to the ven-
dor/bundle directory in the application. This may be overridden
using the --path option.
A rails 3.0.x project I am working with uses a gem that contains a large amount of test data that isn't needed for the heroku deployment. I would like to exclude this from the heroku slug, as it adds a few dozen megs to the slug (and has pushed us past the 100mb size limit several times, our slug is large for other reasons.)
I've tried doing this using the .slugignore mechanism, but I can't find a way to have it exclude files in gems as opposed to files in the app. This is a rails 3.0.x application running on the bamboo stack, but I would upgrade to rails 3.1 and/or the cedar stack if there was a workaround / procedure in those versions.
Other suggestions about fixing this that aren't 'make the huge gem smaller' are also great and extremely welcome.
On cedar stack the gems are installed into your vendor folder (the actual path is something like vendor/bundle/ruby/1.9.1/gems/), so I believe you can slugignore the subpaths you need, though I haven't tried it.
In general use case this will not work because .slugignore matched files are being deleted before installing gem files. See the following bit of heroku deployment output:
-----> Deleting 2 files matching .slugignore patterns.
-----> Ruby/Rails app detected
-----> Installing dependencies using Bundler version 1.3.0.pre.5
Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
Using rake (10.0.3)
Using Platform (0.4.0)
Using open4 (1.3.0)
...
I have a couple of rails websites in production on a ubuntu webserver of ours..
We're running Ubuntu 10.10 + Passenger and Apache.. and ruby enterprise edition version 1.8.7 (No RVM),
I've been having a problem with bundler on both the sites..
Whenever i restart the app by touching restart.txt or by rebooting the server, The user sees an error saying passenger could not be started because it couldn't find nokogiri (a requirement of one of the test gems) so i try and run 'bundle install' which fails cause it tries to install nokogiri..
I've tried running
bundle install --without development:test
And various other variations..
I think one of them worked until i restarted the app and we were back to square one again..
So for the moment i resigned myself to commenting out all gems in the development and test group..
Any ideas?
Thanks
Daniel
Oh i forgot to mention.. I'm deploying using capistrano :)
You have to provide a space-separated list for the --without parameter, see http://gembundler.com/man/bundle-install.1.html
You're missing a space. You should do:
bundle install --without=development:test
Try:
bundle install --without development test
or even:
bundle install --path vendor/bundle --without development test