I've inherited control of a website that's run using ruby on rails. I don't know much about ruby or ruby on rails, and the original developer only really left instructions on how to make some basic changes, without much in-depth information on the site or its internal structure.
I was making some basic changes today, and somehow during the process, the website went down, and it displays the message
"Web application could not be started, Passenger Phusion has listed more information about the error below"
and states that
database configuration does not specify adapter (ActiveRecord::AdapterNotSpecified).
I didn't make any edits to database.yml, but this is what it currently looks like:
development:
adapter: mysql2
database: line_dev
encoding: utf8
username: root
# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
adapter: mysql2
database: line_test
encoding: utf8
username: root
from what i've seen on the ruby on rails tutorials, it seems like the database.yml should also have a production database, something like:
production:
adapter: mysql2
database: line_production #A new database?
encoding: utf8
username: root
I'm very hesitant to make changes to the internal parts of the website, so I wanted to know if I'm on the right track here, or if this is some entirely different problem. Any advice would be greatly appreciated.
Here's a brief summary of what I did change:
I was trying to add a new file to /public, I cloned the github repository in my desktop, added the files in the github application, and then synced. The files showed up in the public folder, but when I tried to open them on the website, it couldn't find them. Some stack overflow solutions on a similar problems were to hard reset, and to git rebase --origin. I have a really loose understanding of what those do, but not really any idea if they could have any repercussions to the way the website was operating.
On further inspection I believe I may have been trying to push from the production environment... Here's a list of commands given in the production env:
git pull remote branch
git pull origin/master
git fetch
git push (failed)
git config
git merge origin/master
git push (failed)
git reset --hard HEAD~
git merge origin/master (?)
git reset --hard origin/master
git merge
git pull --rebase
git pull origin master
git rebase --interactive
git rebase --abort
git reset --hard (SHA of commit before shit went down)
For a lot of these commands, git status was reporting that my branch was ahead of origin/master by # commits, so I was blindly entering solutions from stackoverflow to fix that. I'm just looking through reflog/terminal history for these, so I'm not quite sure what messages were received from the commands.
I'm not proud.
Related
I have a versionated schema.rb that reflects the master branch migrations.
Whenever I go to another branch and run a new migration in that branch, my database structure is changed.
If I go back to master branch and run db:migrate, that migration changes from the another branch is added to the schema, despite it not belonging to the master branch.
I know this is the way rails works, but how do you devs deal with this?
I heard some people just schema:load to have local database synched with schema, but since it erases all data, it's not a good solution for me because I use a heavy sql dump that helps the development (it is very painful and prone to risks to develop without this data)
Having a production dump for development (using the browser/rails server) is great, but you should probably have a test environment setup that uses a minimal seeded database.
You can get this by doing RAILS_ENV=test rake db:schema:load on an empty (new) DB.
What I'd normally do is something like:
git checkout db/schema.rb
RAILS_ENV=test rake db:reset
RAILS_ENV=test rake db:migrate
and load fixtures as needed. That will leave you with a schema that should just have your migration changes and a changed timestamp. (Unless you have stuff like setting changes hardcoded into your schema, in which case they'll be lost - we monkeypatched ActiveRecord::SchemaDumper to avoid this).
After you've made your migration and done this, commit the schema.rb and migration file before you apply the migration to development. You can then discard the schema.rb with the unwanted changes.
Found this question as I've scoured the web dealing with exactly the same issue - large production dump, different migrations on different branches.
The options seem to be:
Meticulously make sure your migrations are reversible and roll them back if needed before switching branches - can be done, but not practical for me as I was given a nasty database and have needed to use raw sql to rework it.
Fresh dump from production every time a branch is switched - not practical for local development
Generate a new database and seed it with sample data for every git branch - not great for realistic testing in staging
I've been doing #1 for a long time and am looking for a better way. I've decided to do #2 when needed for staging, since it doesn't get touched too often, and do #3 locally.
The local solution is the interesting part - here's what I have (will edit as I improve on it):
database.yml - we want to work on a different db depending on what git branch we're on (tweaked from another stackoverflow answer I've lost the link to):
<%
branch = `git rev-parse --abbrev-ref HEAD`.strip rescue nil
use_single_db = !branch || branch == 'master'
branch_spec = (use_single_db ? "" : "_#{branch}").underscore.gsub(/[\.\/\-]/, '_')
%>
default: &default
host: '127.0.0.1'
adapter: mysql2
encoding: utf8
pool: 10
port: 3306
username: 'root'
password: 'password'
database: tmp_<%= branch_spec %>
development:
<<: *default
This does mean, every time we create a new branch, we'll have to create a database and seed it. You'll need to populate seeds.rb for your use case - tedious, but necessary. (You mentioned that there's quite a bit of data you need for development - you can run all sorts of ruby in this file, perhaps you load the big stuff you need from a csv or something?) One thing I make sure to do in seeds.rb is to populate the
schema_migrations table with the current migrations on the branch
(and any other appropriate edits to seeds.rb) for when that is mysteriously NOT loaded from structure.sql. More on that at the
end.
conn = ActiveRecord::Base.connection
schema_migrations_versions = [
...
20160407183816,
20220317151510,
20220414173833,
20220425184402,
20220711221447,
20220712172621,
]
if conn.execute("SELECT * FROM schema_migrations").to_a.length == 0
conn.execute("INSERT INTO schema_migrations (version) VALUES (#{schema_migrations_versions.join('),(')})")
end
To actually do the creation/seeding, run the following (I made a bash script so that this is a quick, one-time command):
bin/rails db:create
bin/rails db:structure:load # you might need db:schema:load if you're using schema.rb
bin/rails db:environment:set RAILS_ENV=development
mv db/migrate db/migrate_x
bin/rails db:seed
mv db/migrate_x db/migrate
Why the hacky mv command you ask? We have a chicken-and-egg problem:
The schema.rb or structure.sql file is the 'source of truth' for the database structure at any given time - presumably, any migrations you have committed to the repo have been run on the database and this file has been updated with them. When you run db:structure:load, it creates all your tables as if all the migrations have been run.
Great, let's seed the database. But then Rails complains that you haven't run your migrations on it yet, because the schema_migrations table is empty, because we haven't seeded the database yet. This is supposed to get added from structure.sql but I find that most of the time (all the time?) it doesn't actually get added, and I haven't been able to figure out why.
Ok, so run migrations first. But unless you carefully create your migrations to check whether they should be run ("create if not exists" type stuff), your migrations will fail because that table exists, that column exists, or doesn't exist... can't do it.
It is possible to set a rails setting to just not check for migrations, but it seems equally annoying to temporarily edit the config just to seed the database, so I have opted for moving the migrations somewhere rails can't find them for the duration of the seeding.
This seems more like a git problem than a Rails one. Git allows for branch changes with new files, since they would cause no conflict with the master branch, or whatever branch you're checking out. In Rails, creating a migration creates a new untracked file in the db/migrations directory. Since it is a new file, you are able to checkout the master branch with no issue.
~/code/node/server (test)
$ git checkout -b dev
Switched to a new branch 'dev'
~/code/node/server (dev)
$ touch test.txt
~/code/node/server (dev)
$ git checkout test
Switched to branch 'test'
~/code/node/server (test)
$ git status
On branch test
Untracked files:
(use "git add <file>..." to include in what will be committed)
test.txt
nothing added to commit but untracked files present (use "git add" to track)
In this example, I start on branch 'test' and then checkout a new branch called 'dev'. I then create a test file and make sure not to start tracking it. Then checkout the 'test' branch again. (Notice no issue here). git status reveals that the untracked file followed me back to the 'test' branch.
If you want the migration file to stay local to the non-master branch where it was created, you can do a few things:
You can stage and commit this file to the non-master branch. After you do this, the file will reside on that branch and not follow you back to the master branch (or whatever other branch you decide to checkout).
You can stage this file, and then stash it:
git add path_to/your_file
git stash
to retrieve the stash later you can git stash pop
for more info on git stash, read here
Delete the file.
If you do accidentally migrate on a branch you didn't mean to, you can always rollback the migration using bundle exec rake db:rollback until you get to the desired schema state.
Hopefully this helps, and happy coding!
I am working on a rails app with quite a few git branches and many of them include db migrations. We try to be careful but occasionally some piece of code in master asks for a column that got removed/renamed in another branch.
What would be a nice solution to "couple" git branches with DB states?
What would these "states" actually be?
We can't just duplicate a database if it's a few GBs in size.
And what should happen with merges?
Would the solution translate to noSQL databases as well?
We currently use MySQL, mongodb and redis
EDIT: Looks like I forgot to mention a very important point, I am only interested in the development environment but with large databases (a few GBs in size).
When you add a new migration in any branch, run rake db:migrate and commit both the migration and db/schema.rb
If you do this, in development, you'll be able to switch to another branch that has a different set of migrations and simply run rake db:schema:load.
Note that this will recreate the entire database, and existing data will be lost.
You'll probably only want to run production off of one branch which you're very careful with, so these steps don't apply there (just run rake db:migrate as usual there). But in development, it should be no big deal to recreate the database from the schema, which is what rake db:schema:load will do.
If you have a large database that you can't readily reproduce, then I'd recommend using the normal migration tools. If you want a simple process, this is what I'd recommend:
Before switching branches, rollback (rake db:rollback) to the state before the branch point. Then, after switching branches, run db:migrate. This is mathematically correct, and as long as you write down scripts, it will work.
If you forget to do this before switching branches, in general you can safely switch back, rollback, and switch again, so I think as a workflow, it's feasible.
If you have dependencies between migrations in different branches... well, you'll have to think hard.
Here's a script I wrote for switching between branches that contain different migrations:
https://gist.github.com/4076864
It won't solve all the problems you mentioned, but given a branch name it will:
Roll back any migrations on your current branch which do not exist on the given branch
Discard any changes to the db/schema.rb file
Check out the given branch
Run any new migrations existing in the given branch
Update your test database
I find myself manually doing this all the time on our project, so I thought it'd be nice to automate the process.
Separate Database for each Branch
It's the only way to fly.
Update October 16th, 2017
I returned to this after quite some time and made some improvements:
I've added another namespace rake task to create a branch and clone the database in one fell swoop, with bundle exec rake git:branch.
I realize now that cloning from master is not always what you want to do so I made it more explicit that the db:clone_from_branch task takes a SOURCE_BRANCH and a TARGET_BRANCH environment variable. When using git:branch it will automatically use the current branch as the SOURCE_BRANCH.
Refactoring and simplification.
config/database.yml
And to make it easier on you, here's how you update your database.yml file to dynamically determine the database name based on the current branch.
<%
database_prefix = 'your_app_name'
environments = %W( development test )
current_branch = `git status | head -1`.to_s.gsub('On branch ','').chomp
%>
defaults: &defaults
pool: 5
adapter: mysql2
encoding: utf8
reconnect: false
username: root
password:
host: localhost
<% environments.each do |environment| %>
<%= environment %>:
<<: *defaults
database: <%= [ database_prefix, current_branch, environment ].join('_') %>
<% end %>
lib/tasks/db.rake
Here's a Rake task to easily clone your database from one branch to another. This takes a SOURCE_BRANCH and a TARGET_BRANCH environment variables. Based off of #spalladino's task.
namespace :db do
desc "Clones database from another branch as specified by `SOURCE_BRANCH` and `TARGET_BRANCH` env params."
task :clone_from_branch do
abort "You need to provide a SOURCE_BRANCH to clone from as an environment variable." if ENV['SOURCE_BRANCH'].blank?
abort "You need to provide a TARGET_BRANCH to clone to as an environment variable." if ENV['TARGET_BRANCH'].blank?
database_configuration = Rails.configuration.database_configuration[Rails.env]
current_database_name = database_configuration["database"]
source_db = current_database_name.sub(CURRENT_BRANCH, ENV['SOURCE_BRANCH'])
target_db = current_database_name.sub(CURRENT_BRANCH, ENV['TARGET_BRANCH'])
mysql_opts = "-u #{database_configuration['username']} "
mysql_opts << "--password=\"#{database_configuration['password']}\" " if database_configuration['password'].presence
`mysqlshow #{mysql_opts} | grep "#{source_db}"`
raise "Source database #{source_db} not found" if $?.to_i != 0
`mysqlshow #{mysql_opts} | grep "#{target_db}"`
raise "Target database #{target_db} already exists" if $?.to_i == 0
puts "Creating empty database #{target_db}"
`mysql #{mysql_opts} -e "CREATE DATABASE #{target_db}"`
puts "Copying #{source_db} into #{target_db}"
`mysqldump #{mysql_opts} #{source_db} | mysql #{mysql_opts} #{target_db}`
end
end
lib/tasks/git.rake
This task will create a git branch off of the current branch (master, or otherwise), check it out and clone the current branch's database into the new branch's database. It's slick AF.
namespace :git do
desc "Create a branch off the current branch and clone the current branch's database."
task :branch do
print 'New Branch Name: '
new_branch_name = STDIN.gets.strip
CURRENT_BRANCH = `git status | head -1`.to_s.gsub('On branch ','').chomp
say "Creating new branch and checking it out..."
sh "git co -b #{new_branch_name}"
say "Cloning database from #{CURRENT_BRANCH}..."
ENV['SOURCE_BRANCH'] = CURRENT_BRANCH # Set source to be the current branch for clone_from_branch task.
ENV['TARGET_BRANCH'] = new_branch_name
Rake::Task['db:clone_from_branch'].invoke
say "All done!"
end
end
Now, all you need to do is run bundle exec git:branch, enter in the new branch name and start killing zombies.
Perhaps you should take this as a hint that your development database is too big? If you can use db/seeds.rb and a smaller data set for development then your issue can be easily solved by using schema.rb and seeds.rb from the current branch.
That assumes that your question relates to development; I can't imagine why you'd need to regularly switch branches in production.
I was struggling with the same issue. Here is my solution:
Make sure that both schema.rb and all migrations are checked in by all developers.
There should be one person/machine for deployments to production. Let's call this machine as the merge-machine. When the changes are pulled to the merge machine, the auto-merge for schema.rb will fail. No issues. Just replace the content with whatever the previous contents for schema.rb was (you can put a copy aside or get it from github if you use it ...).
Here is the important step. The migrations from all developers will now be available in db/migrate folder. Go ahead and run bundle exec rake db:migrate. It will bring the database on the merge machine at par with all changes. It will also regenerate schema.rb.
Commit and push the changes out to all repositories (remotes and individuals, which are remotes too). You should be done!
This is what I have done and I'm not quite sure that I have covered all the bases:
In development (using postgresql):
sql_dump db_name > tmp/branch1.sql
git checkout branch2
dropdb db_name
createdb db_name
psql db_name < tmp/branch2.sql # (from previous branch switch)
This is a lot faster than the rake utilities on a database with about 50K records.
For production, maintain the master branch as sacrosanct and all migrations are checked in, shema.rb properly merged. Go through your standard upgrade procedure.
You want to preserve a "db environment" per branch. Look at smudge/clean script to point to different instances. If you run out of db instances, have the script spin off a temp instance so when you switch to a new branch, it's already there and just needs to be renamed by the script. DB updates should run just before you execute your tests.
Hope this helps.
I totally experience the pita you are having here. As I think about it, the real issue is that all the branches don't have the code to rollback certain branches. I'm in the django world, so I don't know rake that well. I'm toying with the idea that the migrations live in their own repo that doesn't get branched (git-submodule, which I recently learned about). That way all the branches have all the migrations. The sticky part is making sure each branch is restricted to only the migrations they care about. Doing/keeping track of that manually would be a pita and prone to error. But none of the migration tools are built for this. That is the point at which I am without a way forward.
I would suggest one of two options:
Option 1
Put your data in seeds.rb. A nice option is to create your seed data via FactoryGirl/Fabrication gem. This way you can guarantee that the data is in sync with the code if we assume, that the factories are updated together with the addition/removal of columns.
After switching from one branch to another, run rake db:reset, which effectively drops/creates/seeds the database.
Option 2
Manually maintain the states of the database by always running rake db:rollback/rake db:migrate before/after a branch checkout. The caveat is that all your migrations need to be reversible, otherwise this won't work.
If you do a git pull, you should already have the latest schema, affected by any migrations that came in via the pull, but your database tables may not be updated
So, you do need to run the migrations after pulling, but this will often change db/schema.rb
If all you've done is pull and migrate, there's no reason you should be responsible for committing any of the resultant schema changes as they don't technically belong to you, and they may end up being extraneous/incorrect
Resetting the schema diff makes the most sense
Here is my step by step version of what to do before creating a new branch
Switch to your parent/base branch
Pull the latest code
Run bundle exec rake db:migrate to update your schema.rb file locally
Do a git checkout db/schema.rb to throw away the changes brought by db:migrate if any
Create your new branch and switch to it
Make sure to commit your changes before switching to another branch
Adapted from here
On development environment:
You should work with rake db:migrate:redo to test if your script are reversible, but keep in mind always should have a seed.rb with the data population.
If you work with git, you seed.rb should be change with an migration change, and the execution of db:migrate:redo for the begining (load the data for a new development on other machine or new database)
Apart of ´change´, with yours up's and down's methods your code always be cover scenarios for the "change" in this moment and when start from zero.
I ran into a merge conflict in file db/development.sqlite3.
warning: Cannot merge binary files: db/development.sqlite3 (HEAD vs. nest-questions)
Auto-merging db/development.sqlite3
CONFLICT (content): Merge conflict in db/development.sqlite3
Normally I would open file (in Cloud9 IDE), manually edit out the conflict, and done.
But I cannot read anything in this file - it just looks like gibberish.
The app is in development. Can i just wipe out the file, then run rake db:reset, or will cause issues? I've never manually modified this file before.
Has anyone resolved a conflict in db/development.sqlite3 before and can advise?
You won't be able to resolve a merge of db/development.sqlite3 by hand. It's a binary format of the SQLite database that Rails is using behind the scenes when you are using rails console, rails server or an equivalent app server.
The typical approach is to make sure that these .sqlite3 files are ignored by git (since you don't usually wish to persist your database to your source code repository).
To resolve:
git rm db/*.sqlite3
git commit -m "Removing SQLite files from git"
rm .gitignore
wget https://raw.githubusercontent.com/github/gitignore/master/Rails.gitignore
mv Rails.gitignore .gitignore
git add .gitignore
git commit -m "Ignoring files that Rails recommends"
The sqlite files and many others will be ignored by this technique. Note that if the files in question are already in your repository, you'll be prompted to work with them as usual. You'll need to remove them from the repo using git rm before these changes take effect.
You can wipe out the file with db:reset, etc. as you mention above, but the problem will recur until you take the above (or similar) steps.
We have inherited a Rails project hosted on Heroku. The problem is that I don't understand how to get a copy of the database.yml or other various config files that have been ignored in the .gitignore file.
Sure I can look through the history but it seems like a hip shot when comparing it to what should be on the production server. Right now we are having to make changes to the staging environment which makes things a bit arduous and less efficient than having a local dev environment so we can really get under the hood. Obviously, having an exact copy of the config is a good place to start.
With other hosts it's easy to get access to all the files on the server using good ol' ftp.
This might be more easily addressed with some sort of git procedure that I am less familiar with.
Heroku stores config variables to the ENV environment
heroku config will display these list of variables
heroku config:get DATABASE_URL --app YOUR_APP
Will return the database url that Heroku as assigned to your app, using this string one can deduce the parameters necessary to connect to your heroku applications database, it follows this format
username:password#database_ip:port/database_name
This should provide you with all the values you'll need for the heroku side database.yml, its a file that is created for you each time you deploy and there is nothing it that can't be gotten from the above URL.
gitignoring the database.yml file is good practice
It's not something you can do - entries added to .gitignore means they've been excluded from source control. They've never made it into Git.
As for database.yml you can just create the file in app\config\database.yml with local settings, it should look something like;
development:
adapter: postgresql
host: 127.0.0.1
username: somelocaluser
database: somelocaldb
It's safe to say though, that if the file is not in Git then it's not on Heroku since that's the only way you can get files to Heroku. Any config is likely to be in environment variables - heroku config will show you that.
Im developing a rails app. Following this tutorial here:
Tutorial
I finished a part and wanted to push all the changes up to heroku and view them there. I can view the site on my local machine. On heroku all I see is this:
I typed in the following commands after I made my changes, saw the site was working on my local computer.
$ git add .
$ git commit -m "Finish layout and routes"
$ git checkout master
$ git push
$ git push heroku
$ heroku open
I even looked at the heroku logs but couldn't make sense of whats going wrong! There is no content in my database, its empty. I just have views setup.
Why did you checkout to master? Are you working on another branch?
If that is the case, you will need to merge your changes to master before pushing your code (supposing you are on master which is already the case after the checkout):
git merge other-branch
Also don't forget to migrate your database on heroku.
heroku rake db:migrate
EDIT
To find out your current branch, type:
git branch
It will mark the current branch with a '*'.
If you have removed the public/index.html file, you'll have to do git rm public/index.html as well, then commit and push to Heroku. If you delete a file locally, git won't pick up on that without doing the git rm, and Heroku's knowledge of your app is 100% through Git.
I had a similarly strange (but unrelated) problem when I had an app that was storing uploaded files on Heroku. Every time I'd do a push they'd all go away. It confused me greatly until I realized that Heroku essentially wipes every time you do a push, and anything that isn't in git isn't kept. A good reason to use S3 or similar for uploaded file storage.