Capistrano deploy:migrate and db:migrate run all migrations every time - ruby-on-rails

So, I'm diddling around with rails (ruby 1.9.3p392, rails 3.2, sqlite3 db) and I'm trying to deploy the ubiquitous blog tutorial code to a "production" server (apache, passenger, ubuntu). My deploy.rb looks like this:
require 'bundler/capistrano'
require 'rvm/capistrano'
load 'deploy/assets'
set :rvm_ruby_string, ENV['GEM_HOME'].gsub(/.*\//,"")
set :rvm_type, :user
set :user, 'blah'
set :application, 'railsTest'
set :domain, 'www.blah.com'
set :applicationdir, "/home/sean/public/blah.com/public"
set :scm, 'git'
set :repository, "ssh://blah#1.1.1.1/home/blah/public/bla.com/public/capDep.git"
#set :git_enable_submodules, 1 # if you have vendored rails
set :branch, 'master'
set :git_shallow_clone, 1
set :scm_verbose, true
set :use_sudo, false
# roles (servers)
role :web, domain
role :app, domain
role :db, domain, :primary => true
# deploy config
set :deploy_to, applicationdir
set :deploy_via, :export
set :migrate_target, :latest
# additional settings
default_run_options[:pty] = true # Forgo errors when deploying from windows
#ssh_options[:keys] = %w(/home/blah/.ssh/id_rsa)
ssh_options[:forward_agent] = true
# if you want to clean up old releases on each deploy uncomment this:
# If you are using Passenger mod_rails uncomment this:
namespace :deploy do
task :start do ; end
task :stop do ; end
task :restart, :roles => :app, :except => { :no_release => true } do
run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
end
end
#after "deploy:update_code", "deploy:migrate"
Now, I am sure that must look like a big hot mess to those who know what they are doing with capistrano, but I am an utter rube. In the end, despite my inadequacies, the deploy seems to work, because when I run the following
cap deploy:setup
cap deploy
my app is up and running and, just because I can, I add a few rows to a table in the db via the web ui that was created for me by rails. Now, I get bold and create a migration, adding a column to a table. I push my changes to git. To my horror, when I run
cap deploy
ALL the migrations are run, which recreates the tables, thus destroying all my data. I have repeated this painful process several times. My schema_migrations table looks like this:
20130620210004
20130620220229
20130628213331
20130628214946
20130628223002
What am I missing here?
UPDATE: I recently gave #TheMahrvin's suggestion regarding running deploy:migrations at the command line and removing it from the deploy.rb. It didn't work... once again, all migrations were run. My muse must have whispered something in my ear, because I decided to try running db:migrate on the server itself. I was astonished to see this output after running just "rake":
20130717230110 CreateHighScores
20130717230342 CreateGames
20130717231041 AddGameTypeToGame
20130717233707 AddGamePublisherToGame
20130717234124 AddGameRatingToGame
20130731210558 AddGameMechanicToGame
Only the last migrations should be pending. So, perhaps this isn't a problem with Capistrano at all (I've updated the title of this question to reflect that). So, why are the previous migrations still being flagged as pending? I know they were run in the past, both because I saw them in the output and verified the db schema after they ran.
UPDATE #2: Setup another migration and ssh'd into the server and cd'd my way to the "current" directory, which if I understand capistrano at all (fat chance) is where the current files are. Running
bundle exec rake db:migrate:status
got me:
Status Migration ID Migration Name
--------------------------------------------------
down 20130717230110 Create high scores
down 20130717230342 Create games
down 20130717231041 Add game type to game
down 20130717233707 Add game publisher to game
down 20130717234124 Add game rating to game
down 20130731210558 Add game mechanic to game
down 20130731212454 Add publish year to game
down 20130731214515 Add game rank to game
down 20130731214928 Add game abbr to game
down 20130731215749 Add crazy field to game
I can't help feeling that there is something profoundly wrong with what I am trying to do.

OK, figured it out... though how anybody else in the stackosphere was supposed to do the same based on the red herrings in my original question is beyond me.
The problem was that my production database was set to
db/production.sqlite3
Because it was a sqlite database in the main project directory, it was getting axed every time I ran
cap deploy
Then, when I would run
cap deploy:migrate
It would find an empty database and think all migrations needed to be run. I solved this by changing the database path to
/my_absolute_path/shared/db/production.sqlite3
Thanks to #TheMahvin and anyone else who attempted to take on the hopeless task of answering my poorly worded question!
H/T to this question, which made the scales fall from my eyes:
Capistrano Deploy Wipes Database?

I haven't seen:
after "deploy:update_code", "deploy:migrate"
before. Try deleting that line and run:
bundle exec cap deploy:migrations (deploys code and migrations)
or
bundle exec cap deploy:migrate (runs the migrate rake task on the server)
instead.
The rest of your deploy.rb seems OK to me, though I don't know anything about the rvm/capistrano integration or the windows adjustment.

How did you "add a few rows to a table in the db"?
I suspect your data loss results from mixing migrations and your own db changes.
Rails expects you to do all the database changes via migrations.
There's some debate on migrations in general in the Rails community, but right now (especially if you're a beginner) always use migrations to change your database. That way you have a complete blueprint for your db allowing you to deploy on several machines from scratch without fiddling with your db and to make sure other contributors have the same db to work with.
I don't know a lot about these kinds of internals, but from what I understand your data loss resulted something like that:
After your manual changes Rails couldn't match the db-layout to the result of any migration (via the timestamps in your migrations and your schema respectively) thus treating the db as if it was new. To get to the state defined by all migrations, all of them needed to be executed, including the migrations that create tables, thus discarding everything in them.
I hope this helps,
Andy

Related

Rails Automatic Builds with Capistrano

I used to be a .NET guy, and enjoyed using a nightly build system (continuous integration) called CruiseControl.NET to automatically deploy my applications to my staging environment each night.
Now that I've made the jump to Ruby on Rails and GitHub, I've found myself a little confused about how to setup an equivalent automated, nightly build system. I want to do things right, the Rails way, but I could use a push in the right direction.
Here's what I'm using...
Ruby on Rails 3.2.9 (with asset pipeline)
RVM
Apache + Passenger
MySQL
Ubuntu 12.04 (staging server OS)
GitHub (SCM)
I'm looking for a system/solution that fulfills these requirements: (Ideally utilizing Capistrano...)
Deploy the latest commit from my 'master' branch in my GitHub repository to my staging server.
One-click build on-demand: I want to click a button or link to force a deploy (or re-deploy) at any time
Have the capability to run custom commands on the staging server as a part of the deploy (i.e. 'bundle install', restart Apache, etc...)
Automatic deploy daily or after a commit on GitHub (optional)
And because I'd never ask this question without doing some research first, here are some relevant resources I've found, but haven't been able to decipher:
Automatic Deployment via Git
Deploying Rails 3 apps with Capistrano
Suggestions, anyone? Thanks!
David
Well, it turns out there is alot of good information out there regarding how to use Capistrano for this purpose (including Prakash's reference), but none of it seems to be fully comprehensive.
After many hours of picking through guide, after forum, after stack overflow question, I managed to fulfill most of my goals. To save time for others, I will try to provide answers and information that I found in the context of my original question.
UPDATE:
It turns out I was looking for Jenkins all along: it's a perfect (an even an improved) analog to the CruiseControl build server application I had used before. Jenkins is a web application that kicks off builds on a scheduled basis or, with plugins, commit events. It doesn't include functionality to actually deploy my Rails app, so that's where Capistrano steps in. Using Jenkins' "shell execute" build task to trigger Capistrano deploys, I was able to accomplish all of my goals above.
Take a look at this guide for more detail.
ORIGINAL POST:
First off, Capistrano can work for the purpose of build system, however, it is not at all similar to CruiseControl.
CruiseControl is:
A web application, that is run on a machine that serves as a 'build server'
Any user may visit the web app to run builds; the user does not have to have anything setup or configured on their end... it's just a webpage.
The interface also provides functions for logs, blame, scheduled builds, etc...
Capistrano is:
Not an application or service, but a gem that works like rake; it bears similar functionality to the Ant and Nant scripts used by CruiseControl for the purpose of deploying.
Capistrano is run on a user's local machine (typically the Rails app folder), and requires configuration on their end to run
It directly connects from the user's machine to the remote server via SSH to perform deploys (this may not be preferable if you don't want to grant users SSH access to the remote server)
Commands are run through a console i.e. cap deploy (this is as close as it gets to 'one click' deploys.)
It does produce logs, perform rollbacks etc.
It does not kick-off scheduled builds on its own (using cron was suggested)
In regards to my original requirements...
Deploy the latest commit from my 'master' branch in my GitHub repository to my staging server.
Configuring Capistrano's deploy.rb file will accomplish this.
One-click build on-demand: I want to click a button or link to force a deploy (or re-deploy) at any time
All Capistrano deploys are done through 'force': you run 'cap deploy' in your console manually
Have the capability to run custom commands on the staging server as a part of the deploy (i.e. 'bundle install', restart Apache, etc...)
Configuring Capistrano's deploy.rb file will accomplish this. Most of these commands are included out of the box.
Automatic deploy daily or after a commit on GitHub (optional)
I haven't figured this one out yet... a cron job might be the best way to do this.
Setting up Capistrano
Start with this tutorial from GitHub first. In your Rails App folder, you should end up with a Capfile and config/deploy.rb file.
To save you some time, copy and paste these files, and tweak the settings to your needs.
The files below are configured for:
Deployment to a test environment
GitHub
RVM based Rails
Using an explicit version of Ruby and gemset
Including untracked files (i.e. a database.yml file you didn't commit to SCM)
Using Rails asset pipeline
Running migrate and seed with each deploy
Restarting Apache based Passenger with each deploy
Capfile
# Set this if you use a particular version of Ruby or Gemset
set :rvm_ruby_string, 'ruby-1.9.3-p286#global'
#set :rvm_ruby_string, ENV['GEM_HOME'].gsub(/.*\//,"") # Read from local system
require "bundler/capistrano"
# Uncomment this if you're using RVM
require "rvm/capistrano"
load 'deploy'
# Uncomment if you are using Rails' asset pipeline
load 'deploy/assets'
load 'config/deploy' # remove this line to skip loading any of the default tasks
config/deploy.rb
# BEGIN RUBY CONFIG
# You can manually override path variables here
# set :default_environment, {
# 'PATH' => "/usr/local/bin:/bin:/usr/bin:/bin:/<ruby-dir>/bin",
# 'GEM_HOME' => '<ruby-dir>/lib/ruby/gems/1.8',
# 'GEM_PATH' => '<ruby-dir>lib/ruby/gems/1.8',
# 'BUNDLE_PATH' => '<ruby-dir>/lib/ruby/gems/1.8/gems'
# }
# This changes the default RVM bin path
# set :rvm_bin_path, "~/bin"
# If your remote server doesn't have a ~/.rvm directory, but is installed
# at the /usr/local/rvm path instead, use this line
set :rvm_type, :system
# END RUBY CONFIG
default_run_options[:pty] = true # Must be set for the password prompt
# from git to work
# BEGIN MULTIPLE ENVIRONMENT DEPLOYS
# Read the following URL if you need to deploy to different environments (test, development, etc)
# https://github.com/capistrano/capistrano/wiki/2.x-Multistage-Extension
# set :stages, %w(production test)
# set :default_stage, "test"
# require 'capistrano/ext/multistage'
# END MULTIPLE ENVIRONMENT DEPLOYS
# BEGIN APPLICATION VARS
set :application, "yourapp"
set :rails_env, 'test'
# END APPLICATION VARS
# BEGIN PATH DEFINITIONS
set(:releases_path) { File.join(deploy_to, version_dir) }
set(:shared_path) { File.join(deploy_to, shared_dir) }
set(:current_path) { File.join(deploy_to, current_dir) }
set(:release_path) { File.join(releases_path, release_name) }
# END PATH DEFINITIONS
# BEGIN SCM VARS
set :repository, "git#github.com:yourgithubuser/yourrepository.git" # Your clone URL
set :scm, "git"
set :scm_username, "yourgithubuser"
set :scm_password, proc{Capistrano::CLI.password_prompt('GitHub password:')} # The deploy user's password
set :branch, "master"
# END SCM VARS
# BEGIN SERVER VARS
set :user, "ubuntu" # The server's user for deploys
role :web, "dev.#{application}" # The location of your web server i.e. dev.myapp.com
role :app, "dev.#{application}" # The location of your app server i.e. dev.myapp.com
role :db, "dev.#{application}", :primary => true # The location of your DB server i.e. dev.myapp.com
set :deploy_to, "/home/#{user}/www/#{application}"
set :deploy_via, :remote_cache
# Uncomment this if you want to store your Git SSH keys locally and forward
# Else, it uses keys in the remote server's .ssh directory
# ssh_options[:forward_agent] = true
# END SERVER VARS
# BEGIN ADDITIONAL TASKS
before "deploy:start" do
deploy.migrate
deploy.seed
end
before "deploy:restart" do
deploy.migrate
deploy.seed
end
# Some files that the Rails app needs are too sensitive to store in SCM
# Instead, manually upload these files to your <Rails app>/shared folder
# then the following code can be used to generate symbolic links to them,
# so that your rails app may use them.
# In this example below, I link 'shared/config/database.yml' and 'shared/db/seed_data/moderators.csv'
before "deploy:assets:precompile" do
run ["ln -nfs #{shared_path}/config/settings.yml #{release_path}/config/settings.yml",
"ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml",
"mkdir -p #{release_path}/db/seed_data",
"ln -nfs #{shared_path}/db/seed_data/moderators.csv #{release_path}/db/seed_data/moderators.csv",
"ln -fs #{shared_path}/uploads #{release_path}/uploads"
].join(" && ")
end
namespace :deploy do
# Define seed task (call 'cap deploy:seed')
desc "Reload the database with seed data"
task :seed do
run "cd #{current_path}; bundle exec rake db:seed RAILS_ENV=#{rails_env}"
end
# If you are using Passenger mod_rails uncomment this:
task :start do ; end
task :stop do ; end
task :restart, :roles => :app, :except => { :no_release => true } do
run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
end
end
# END ADDITIONAL TASKS
Yes. Capistrano is the tool you are looking for. Using Capistrano with the system cron should enable you to implement a build/deploy system with all four of the mentioned requirements. Rather easily at that.
The best resource to learn about Capistrano is the Deploying Rails book. It has two chapters on Capistrano, the first one covering the basic usage, and the second covering some advanced usage concepts. There is also a Capistrano Case study at the end which explores further config options with a real life deploy script example.
FYI: I finished reading the two chapters yesterday in a couple of hours and tweeted about it ;-)
I'd say go with Capistrano for the deploy part.
The CI part (build when you do a git push) would be best implemented by using a CI server, such as travis (if you're on Github) or Jenkins if you have a private repo.
For the one-click-build, go with either capistrano directly (cap deploy from command line) or a simple CI hook which runs all tests, then runs cap deploy.

capistrano, :db role, what's it for?

As far as I can tell, the capistrano :db role is used only to run migrations.
(Thus, in most cases it probably shouldn't actually be the server that runs your database. Why would you have a ruby/rails stack there (or allow ssh logins there)? it's just whatever server you want to actually execute the rails migrations).
And only the server identified as db role with :primary => true is used to run migrations.
So any other servers identified as 'db' role but without :primary => true... are used for nothing at all? So why does the default deploy.rb created by capify . encourage you to list them? What would you even list here?
Anything I'm missing?
Obviously, the name of role :db is misleading. As you pointed out, Capistrano defines it (with :primary => true) as a host that we execute rake db:migrate on, but database servers are not always running on such hosts. We can alter the schema of remote database server through a Rails app. The correct role name for this kind of host is not :db.
Inferring from the comments on lib/capistrano/configuration/roles.rb, the original meaning of the role :db is a host on which database servers are running. We are expected to login to the :db hosts and do some tasks.
The designers of Capistrano should have defined :migration role or something else for the deploy:migrate task. But the association between the :db role with this task was defined six years ago with 9a6d2fb and has not been changed since then.
Generally speaking, the users of Capistrano can define roles and associate them with tasks freely. The deploy:migrate task is provided just as a recipe for Rails developers. Unfortunately, this recipe includes a misconception about how we do the database migration and is used widely for a long time.
You may have a setup that includes a master database server and multiple slave servers. In some cases, you could have an capistrano task that needs to be run on all database servers. In others, you may want to run a task (for instance, a migration) only on the master server and allow the changes to propagate to the slave instances.

Capistrano and Git, Ruining my life. "Unable to resolve revision for [HEAD] on repository ..."

I searched all of the relevant Capistrano issues, but couldn't find something that even elucidated anything here for me.
git version 1.6.4.2
Capistrano v2.5.10
Basically, when I run my cap deploy.rb script, it connects to the server, starts executing the deploy:update task, then in the deploy:update_code task:
*** [deploy:update_code] rolling back
* executing "rm -rf /home/user_name/public_html/project_name/releases/20091223094358; true"
servers: ["project_name.com"]
It fails with the following error:
/Library/Ruby/Gems/1.8/gems/capistrano-2.5.10/lib/capistrano/recipes/deploy/scm/git.rb:231:in `query_revision': Unable to resolve revision for 'master' on repository 'ssh://git#slice_ip:path_to_git_repository'. (RuntimeError)
Here's my deploy script, I've tried including and omitting:
set :branch 'master'
I also just thought my path to the repository was off, but i've tried just about every permutation (absolute, not absolute, .git suffix, no suffix). There's definitely a bare git repository at the path i'm pointing to.
**I do have multiple projects being hosted on one slice. The other projects is also a rails project, but is running SVN. Capistrano deployments work fine.
Any pointers in the right direction or any ideas would help reduce the amount of drinking I am planning on doing if I can't figure this out. (Paths / IPs obfuscated, dont hack me bro!)
set :application, "project1"
set :user, "username"
set :repository, "ssh://git#67.24.9.133/home/git/project1.git"
set :branch, "master"
set :port, 696969
set :deploy_to, "/home/username/public_html/#{application}"
set :scm, :git
role :app, application
role :web, application
role :db, application, :primary => true
# deployment via remote client (workstation)
set :deploy_via, :copy
set :runner, user
# mod_rails
namespace :deploy do
desc "Restarting mod_rails with restart.txt"
task :restart, :roles => :app, :except => { :no_release => true } do
run "touch #{current_path}/tmp/restart.txt"
end
[:start, :stop].each do |t|
desc "#{t} task is a no-op with mod_rails"
task t, :roles => :app do ; end
end
end
This was the most relevant post (extremely relevant even), but I couldn't really figure out what they were saying the fix is. I'm pretty new with git / capistrano configs.
https://capistrano.lighthouseapp.com/projects/8716/tickets/56-query_revision-unable-to-resolve-revision-for-head-on-repository
Ok I seemed to have fixed it.
Basically, since I have 2 separate repositories on the remote server, I think the "git" user was failing because I hadn't registered an ssh keypair for the git user. That explains why one of my deploy.rb scripts was working properly, while this one wasn't.
In the link I posted in the question, one of the commenters pointed out the issue:
https://capistrano.lighthouseapp.com/projects/8716/tickets/56-query%5Frevision-unable-to-resolve-revision-for-head-on-repository
Note this error is also displayed if
you are using multiple github keys per
http://capistrano.lighthouseapp....
and you do not have these keys and a
corresponding entry in your
.ssh/config on the workstation you're
running the deploy from. so the
ls-remote is run locally. is there a
way to reference the repository at
github.com for this request while the
remote deploy uses
git#github-project1:user/project1.git
Also, see the following link for more details, since the whole ssh issue would apply even if you're not using github.
http://github.com/guides/multiple-github-accounts
Both your workstation and your server must be able to reach the repository at the address specified, if not then you may have to set :local_repository to how you access it from your workstaion, and :repository to be how your servers should access it.
For me Capistrano deployments with Git only seem to work when setting set :copy_cache, true
I've only used capistrano with git once, but never used or seen the use of ssh:// in the repository definition.
Try using set :repository, "git#67.24.9.133/home/git/project1.git" instead
Make sure the branch you are deploying from exists.
set :branch, "upgrade-to-2013.4.3"
is not equal to
set :branch, "upgrade-to-2013.3.4"

Why is Capistrano not checking out the latest version of my code from SVN?

I'm using Capistrano and Rails 2.3.4. I've already done a deploy:cold to the remote server. Now on my local box I changed a layout file and committed it to the repository (I am using Netbeans 6 as my IDE). I type cap deploy and Capistrano runs through it's commands and tells me that it's checked out and deployed the most recent version of my code. On the server, however, the changes aren't there and when I looked at the layout file, it was using the old version not the one I just committed and supposedly deployed.
Anyone experience this?
EDIT: The weird thing is that I changed some image files and those were updated on the server, but the HTML layout I modified was not. Could it just be a cookies issue?
EDIT2: I checked the repository itself (I am using ProjectLocker) and sure enough the code is in there, modified. The issue is only that Capistrano is NOT checking it out even though it says that it is, and it's not reporting any errors.
Here is my deploy.rb file (scrubbed, of course):
# Application
set :application, "myapp"
set :deploy_to, "/var/www/html/#{application}"
# Settings
default_run_options[:pty] = true
set :use_sudo, true
# Servers
set :user, "deploy"
set :domain, "111.111.111.111"
set :runner, "deploy"
server domain, :app, :web
role :db, domain, :primary => true
# SVN
set :repository, "http://myhosting.com/svn/myapp/trunk"
set :scm_username, "wayne#mysite.com"
set :scm_password, "secret"
set :checkout, "export"
# Passenger
namespace :passenger do
desc "Restart Application"
task :restart do
run "touch #{current_path}/tmp/restart.txt"
end
end
after :deploy, "passenger:restart"
It works sometimes, it seems. For instance I made some changes to code earlier and it checked it out fine. I had an issue with it not checking out my database.yml file either; I was forced to edit it on the server.
Just a shot in the dark here, but did you actually look at the layout file or did you hit the website and then look at the webpage source?
If you are in production the layout will be cached (config.action_controller.perform_caching = true), you need to reboot the server. This does not occur in development mode by default, since the above setting is set to false.
See this for more info.

Deploy from Git using Capistrano without a hard reset?

I've an issue at the moment where we are running a CMS within a site
(browsercms) that lets the user upload files. However, every time I
do a deploy Capistrano runs a hard reset thus nuking any uploaded
files.
Does anyone have any suggestions as to how to prevent the hard reset,
and just do a pull, or a way of moving the uploaded files elsewhere,
without having to change the application code?
This might not be the right approach.
You should include your 'images' folder in your .gitignore and symlink the $current_release/images folder to $shared/images.
This may be done automatically on every deployment if you put in your deploy.rb:
task :link_imgs do
run "ln -s #{shared_path}/photos #{release_path}/photos"
end
after "deploy:update_code", :link_imgs
I've done the same with my CMS and it works like a charm
This doesn't quite meet your criteria of "without having to change the application code".
However after running into a similar issue I shifted my uploaded image from /public/images to /public/system/images the /public/system directory is not 'versioned' by each capistrano deployment so the images survive.
Could it be the capistrano 'versioning' causing the problem (instead of a git reset)?
cap deploy calls deploy:update and deploy:restart
deploy:update makes the versioning, copying stuff
deploy:restart does the true restart, overload it at your convenince, usually in your config/deploy.rb file
namespace :deploy do
desc "Softly restart the server"
task :restart, :roles => :app, :except => { :no_release => true } do
my_own.restart_recipe
end
end

Resources