I ran into a merge conflict in file db/development.sqlite3.
warning: Cannot merge binary files: db/development.sqlite3 (HEAD vs. nest-questions)
Auto-merging db/development.sqlite3
CONFLICT (content): Merge conflict in db/development.sqlite3
Normally I would open file (in Cloud9 IDE), manually edit out the conflict, and done.
But I cannot read anything in this file - it just looks like gibberish.
The app is in development. Can i just wipe out the file, then run rake db:reset, or will cause issues? I've never manually modified this file before.
Has anyone resolved a conflict in db/development.sqlite3 before and can advise?
You won't be able to resolve a merge of db/development.sqlite3 by hand. It's a binary format of the SQLite database that Rails is using behind the scenes when you are using rails console, rails server or an equivalent app server.
The typical approach is to make sure that these .sqlite3 files are ignored by git (since you don't usually wish to persist your database to your source code repository).
To resolve:
git rm db/*.sqlite3
git commit -m "Removing SQLite files from git"
rm .gitignore
wget https://raw.githubusercontent.com/github/gitignore/master/Rails.gitignore
mv Rails.gitignore .gitignore
git add .gitignore
git commit -m "Ignoring files that Rails recommends"
The sqlite files and many others will be ignored by this technique. Note that if the files in question are already in your repository, you'll be prompted to work with them as usual. You'll need to remove them from the repo using git rm before these changes take effect.
You can wipe out the file with db:reset, etc. as you mention above, but the problem will recur until you take the above (or similar) steps.
Related
I have a versionated schema.rb that reflects the master branch migrations.
Whenever I go to another branch and run a new migration in that branch, my database structure is changed.
If I go back to master branch and run db:migrate, that migration changes from the another branch is added to the schema, despite it not belonging to the master branch.
I know this is the way rails works, but how do you devs deal with this?
I heard some people just schema:load to have local database synched with schema, but since it erases all data, it's not a good solution for me because I use a heavy sql dump that helps the development (it is very painful and prone to risks to develop without this data)
Having a production dump for development (using the browser/rails server) is great, but you should probably have a test environment setup that uses a minimal seeded database.
You can get this by doing RAILS_ENV=test rake db:schema:load on an empty (new) DB.
What I'd normally do is something like:
git checkout db/schema.rb
RAILS_ENV=test rake db:reset
RAILS_ENV=test rake db:migrate
and load fixtures as needed. That will leave you with a schema that should just have your migration changes and a changed timestamp. (Unless you have stuff like setting changes hardcoded into your schema, in which case they'll be lost - we monkeypatched ActiveRecord::SchemaDumper to avoid this).
After you've made your migration and done this, commit the schema.rb and migration file before you apply the migration to development. You can then discard the schema.rb with the unwanted changes.
Found this question as I've scoured the web dealing with exactly the same issue - large production dump, different migrations on different branches.
The options seem to be:
Meticulously make sure your migrations are reversible and roll them back if needed before switching branches - can be done, but not practical for me as I was given a nasty database and have needed to use raw sql to rework it.
Fresh dump from production every time a branch is switched - not practical for local development
Generate a new database and seed it with sample data for every git branch - not great for realistic testing in staging
I've been doing #1 for a long time and am looking for a better way. I've decided to do #2 when needed for staging, since it doesn't get touched too often, and do #3 locally.
The local solution is the interesting part - here's what I have (will edit as I improve on it):
database.yml - we want to work on a different db depending on what git branch we're on (tweaked from another stackoverflow answer I've lost the link to):
<%
branch = `git rev-parse --abbrev-ref HEAD`.strip rescue nil
use_single_db = !branch || branch == 'master'
branch_spec = (use_single_db ? "" : "_#{branch}").underscore.gsub(/[\.\/\-]/, '_')
%>
default: &default
host: '127.0.0.1'
adapter: mysql2
encoding: utf8
pool: 10
port: 3306
username: 'root'
password: 'password'
database: tmp_<%= branch_spec %>
development:
<<: *default
This does mean, every time we create a new branch, we'll have to create a database and seed it. You'll need to populate seeds.rb for your use case - tedious, but necessary. (You mentioned that there's quite a bit of data you need for development - you can run all sorts of ruby in this file, perhaps you load the big stuff you need from a csv or something?) One thing I make sure to do in seeds.rb is to populate the
schema_migrations table with the current migrations on the branch
(and any other appropriate edits to seeds.rb) for when that is mysteriously NOT loaded from structure.sql. More on that at the
end.
conn = ActiveRecord::Base.connection
schema_migrations_versions = [
...
20160407183816,
20220317151510,
20220414173833,
20220425184402,
20220711221447,
20220712172621,
]
if conn.execute("SELECT * FROM schema_migrations").to_a.length == 0
conn.execute("INSERT INTO schema_migrations (version) VALUES (#{schema_migrations_versions.join('),(')})")
end
To actually do the creation/seeding, run the following (I made a bash script so that this is a quick, one-time command):
bin/rails db:create
bin/rails db:structure:load # you might need db:schema:load if you're using schema.rb
bin/rails db:environment:set RAILS_ENV=development
mv db/migrate db/migrate_x
bin/rails db:seed
mv db/migrate_x db/migrate
Why the hacky mv command you ask? We have a chicken-and-egg problem:
The schema.rb or structure.sql file is the 'source of truth' for the database structure at any given time - presumably, any migrations you have committed to the repo have been run on the database and this file has been updated with them. When you run db:structure:load, it creates all your tables as if all the migrations have been run.
Great, let's seed the database. But then Rails complains that you haven't run your migrations on it yet, because the schema_migrations table is empty, because we haven't seeded the database yet. This is supposed to get added from structure.sql but I find that most of the time (all the time?) it doesn't actually get added, and I haven't been able to figure out why.
Ok, so run migrations first. But unless you carefully create your migrations to check whether they should be run ("create if not exists" type stuff), your migrations will fail because that table exists, that column exists, or doesn't exist... can't do it.
It is possible to set a rails setting to just not check for migrations, but it seems equally annoying to temporarily edit the config just to seed the database, so I have opted for moving the migrations somewhere rails can't find them for the duration of the seeding.
This seems more like a git problem than a Rails one. Git allows for branch changes with new files, since they would cause no conflict with the master branch, or whatever branch you're checking out. In Rails, creating a migration creates a new untracked file in the db/migrations directory. Since it is a new file, you are able to checkout the master branch with no issue.
~/code/node/server (test)
$ git checkout -b dev
Switched to a new branch 'dev'
~/code/node/server (dev)
$ touch test.txt
~/code/node/server (dev)
$ git checkout test
Switched to branch 'test'
~/code/node/server (test)
$ git status
On branch test
Untracked files:
(use "git add <file>..." to include in what will be committed)
test.txt
nothing added to commit but untracked files present (use "git add" to track)
In this example, I start on branch 'test' and then checkout a new branch called 'dev'. I then create a test file and make sure not to start tracking it. Then checkout the 'test' branch again. (Notice no issue here). git status reveals that the untracked file followed me back to the 'test' branch.
If you want the migration file to stay local to the non-master branch where it was created, you can do a few things:
You can stage and commit this file to the non-master branch. After you do this, the file will reside on that branch and not follow you back to the master branch (or whatever other branch you decide to checkout).
You can stage this file, and then stash it:
git add path_to/your_file
git stash
to retrieve the stash later you can git stash pop
for more info on git stash, read here
Delete the file.
If you do accidentally migrate on a branch you didn't mean to, you can always rollback the migration using bundle exec rake db:rollback until you get to the desired schema state.
Hopefully this helps, and happy coding!
I managed to delete to following files with git clean in my rails app:
Removing .DS_Store
Removing .bundle/
Removing config/application.yml
Removing db/development.sqlite3
Removing log/development.log
Removing public/system/profiles/avatars/000/000/011/
Removing public/system/profiles/avatars/000/000/012/
Removing tmp/
Are there any chances of getting back these files somehow or being able to copy from somewhere else? (Git deletes files permanently, so trash is empty. I also tried w/ disk drill but id didn't work out either.)
How can I protect my gitignored files properly to avoid these kinda situations in the future? Should I copy them sometimes or there are other programs to handle this?
Do I have to recreate my rails app or somebody knows a better option?
I think you can try to find these files in previous commits.
Next time remove using command git rm --cached file_name
I was able to pull it off. I ran bundle install (creates the files within .bundle) and reinstalled sqlite and figaro. After that I ran rake db:migrate which created the development.sqlite3 file. I lost the development data, but it is not a huge problem at all. For figaro I had to retype the keys and passwords (Sendgrid, Stripe, etc.) into the new application.yml file. Temporary and log files don't seem to be important. Thanks for the answers. This $git clean is pretty dangerous, watch out!
I've inherited control of a website that's run using ruby on rails. I don't know much about ruby or ruby on rails, and the original developer only really left instructions on how to make some basic changes, without much in-depth information on the site or its internal structure.
I was making some basic changes today, and somehow during the process, the website went down, and it displays the message
"Web application could not be started, Passenger Phusion has listed more information about the error below"
and states that
database configuration does not specify adapter (ActiveRecord::AdapterNotSpecified).
I didn't make any edits to database.yml, but this is what it currently looks like:
development:
adapter: mysql2
database: line_dev
encoding: utf8
username: root
# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
adapter: mysql2
database: line_test
encoding: utf8
username: root
from what i've seen on the ruby on rails tutorials, it seems like the database.yml should also have a production database, something like:
production:
adapter: mysql2
database: line_production #A new database?
encoding: utf8
username: root
I'm very hesitant to make changes to the internal parts of the website, so I wanted to know if I'm on the right track here, or if this is some entirely different problem. Any advice would be greatly appreciated.
Here's a brief summary of what I did change:
I was trying to add a new file to /public, I cloned the github repository in my desktop, added the files in the github application, and then synced. The files showed up in the public folder, but when I tried to open them on the website, it couldn't find them. Some stack overflow solutions on a similar problems were to hard reset, and to git rebase --origin. I have a really loose understanding of what those do, but not really any idea if they could have any repercussions to the way the website was operating.
On further inspection I believe I may have been trying to push from the production environment... Here's a list of commands given in the production env:
git pull remote branch
git pull origin/master
git fetch
git push (failed)
git config
git merge origin/master
git push (failed)
git reset --hard HEAD~
git merge origin/master (?)
git reset --hard origin/master
git merge
git pull --rebase
git pull origin master
git rebase --interactive
git rebase --abort
git reset --hard (SHA of commit before shit went down)
For a lot of these commands, git status was reporting that my branch was ahead of origin/master by # commits, so I was blindly entering solutions from stackoverflow to fix that. I'm just looking through reflog/terminal history for these, so I'm not quite sure what messages were received from the commands.
I'm not proud.
I'm using the cloud9 IDE with Ruby on Rails and having an issue with git.
I have a folder in my workspace called .c9 which contains all the cloud9 related files including it's regularly changing metadata.
I've added an ignore to my gitignore but keeps pushing to my bitbucket repository anyway. Worse when I branch off the metadata contained within goes out of sync and git insist i have to resolve the issue before I merge back.
I have no idea what I'm doing wrong here.
My gitignore looks like this:
# Ignore the c9 files.
/.c9
I have also tried
.c9, /.c9/*, .c9/* but no joy...
I've checked to see that it is actually in my workspace with la returning:
.c9/
.gitignore
Gemfile.lock
README.md
Todo.md
bin/
config.ru
lib/
public/
test/
vendor
.git/
Gemfile
Guardfile
Rakefile
app/
config/
db/
log/
spec/
tmp/
Any thoughts on where I'm going wrong?
I can confirm as per Jubobs comment above my solution in this case was resolved by running:
git rm --cached .c9/ -r
and then ensuring my gitignore featured the following lines
#Ignore all .c9 specific files<br>
/.c9/
I agree this is perhaps a duplicate of this question here: .DS_Store still appears in git status despite being in .gitignore
However as I went looking for a cloud9 specific answer perhaps this is still of use to some.
I don't want to add schema.rb to .gitignore, because I want to be able to load a new database schema from that file. However, keeping it checked in is causing all sorts of spurious conflicts that are easily resolved by a fresh db:migrate:reset.
Basically I want a way to:
Keep schema.rb in the repository for deploy-time database setup
Keep schema.rb in '.gitignore' for general development
There would be one or two people responsible for updating schema.rb and knowing that it was correct.
Is there a way I can have my cake and eat it, too?
I'm afraid the magic solution you're looking for does not exist. This file is normally managed in version control, then for any conflicts on the version line just choose the later of the two dates. As long as you're also running all of the associated migrations nothing should get out of sync this way. If two developers have caused modifications to a similar area of schema.rb and you get conflicts in addition to the version then you are faced with a normal merge conflict resolution, but in my opinion these are normally easy to understand and resolve. I hope this helps some!
One other thing you can do is use:
git update-index --assume-unchanged /path/schema.rb
This will keep the file in the repository but won't track changes. you can switch the tracking anytime by using:
git update-index --no-assume-unchanged /path/schema.rb
What has worked really well for me is to delete and .gitignore schema.rb and then have it regenerated for each developer when they rake db:migrate.
You can still achieve what you wanted without migrating from 0 and risking broken migrations from years ago by simply doing a "roll-up" of the migrations periodically. You can do this by:
Run all outstanding migrations with rake db:migrate
Taking the contents of your schema.rb in the ActiveRecord::Schema.define block
Paste it into your initial_schema migration inside def up (overwriting what's already there)
Delete all other migrations
Now your initial_schema migration is your starting point for new systems and you don't have to worry about conflicts in schema.rb that may not be resolved correctly. It's not magical, but it works.
Would it be sufficient to do a rake db:dump in a pre-commit git hook?
The following won't necessarily fix (1) or (2), but it might take care of the merging issue, and then maybe (1) and (2) go away.
Instead of using .gitignore, use separate branches: Develop which omits schema.rb and Test and Deploy which include schema.rb. Only make code changes in the Develop branches and never merge from Test into Develop. Keep schema.rb in a separate branch:
Developer A
Develop --------
Local Schema \ Your Repo
Test ---------> Dev A
---------> Dev B
Developer B / Master
Develop -------- Schema
Local Schema Test
Test Deploy
In Git, branches are pointers to collections of file contents, so they can include or exclude particular files as well as track file versions. This makes them flexible tools for building your particular workflow.
You could define a merge strategy.
I've found this solution, but dont remember the source
[merge "railsschema"]
name = newer Rails schema version
driver = "ruby -e '\n\
system %(git), %(merge-file), %(--marker-size=%L), %(%A), %(%O), %(%B)\n\
b = File.read(%(%A))\n\
b.sub!(/^<+ .*\\nActiveRecord::Schema\\.define.:version => (\\d+). do\\n=+\\nActiveRecord::Schema\\.define.:version => (\\d+). do\\n>+ .*/) do\n\
%(ActiveRecord::Schema.define(:version => #{[$1, $2].max}) do)\n\
end\n\
File.open(%(%A), %(w)) {|f| f.write(b)}\n\
exit 1 if b.include?(%(<)*%L)'"
put this "somewhere" and
git-config --global core.attributesfile "somewhere"
I built a gem to solve this problem.
It sorts columns, index names and foreign keys, removes excess whitespace and runs Rubocop for some formatting to unify the output of your schema.rb file.
https://github.com/jakeonrails/fix-db-schema-conflicts
After you add it to your Gemfile you just run rake db:migrate or rake db:schema:dump like normal.
Commit schema.rb file.
Run git pull (or continue with what you're doing)
Every time you migrate the database, the schema.rb file updates and appears in git status. When working on something and occasionally doing git pull, this can be annoying because you have to commit schema.rb file before pulling to resolve conflict. This means that every time you migrate the database, you need to commit schema.rb file.
schema.rb should be tracked Git, of course.
I've just released this gem that can solve an issue with "conflicts" between branches for good.
The idea of that gem is simple. It keeps all migrated migrations inside tmp folder so that Git ignores them. It's just only your local story. These files are needed to roll back the "unknown" migrations being in another branch. Now, whenever you have an inconsistent DB schema due to running migrations in some other branch just run rails db:migrate inside the current branch and it will fix the issue automatically. The gem does all this magic automatically for you.