I am trying to deploy a new release of a project to my staging server, but the schema_migrations table in the database is inexplicably empty.
It is now trying to run all migrations while deploying, causing issues since the other tables exist and are intact.
Instead of dropping/recreating the database and losing all my data(though inconvenient, a valid option), is it possible to generate the schema_migrations table without dropping?
Two points:
There's nothing wrong with deleting migrations that aren't needed anymore, you're not supposed to keep them around forever.
You can manually update the schema_migrations table if you have to.
As far as (1) goes, I delete migrations (provided they've been applied everywhere) that are old than a month or so. Keeping around years of clutter in db/migrate is pointless. Migrating down and then back up tends to be more of a mess than it is worth for all but the most recent migrations; if you need to test something old, dump your current development database and start with a fresh one based on an older schema.rb. Besides, if you really need an old migration then you can always dig it out of revision control or simply recreate it based on diffs between versions of db/schema.rb (or db/structure.sql).
Of course, if you don't keep db/schema.rb in revision control then you're doing it wrong and you a looming disaster on your hands.
For (2), the schema_migrations table is very simple:
create table schema_migrations (
version varchar not null primary key
)
So you can connect to PostgreSQL using psql, create the table by hand, and then do a bunch of insert into schema_migrations (version) values (...) to cover the historic migrations that should have been run in your staging environment, and then db:migrate as normal to get things up to date. Then you can spend as much time as you want doing a post mortem to figure out what happened to your staging database's schema_migrations table.
If you go this route, you might want to have a look at the ar_internal_metadata table if your version of Rails uses it.
You will, of course, backup your staging database before doing any of this. And you will want to examine your production database to make sure it isn't suffering the same problems.
Sometimes you have to get your hands dirty to make things go.
Related
QUESTION:
Is there any way I can avoid running current migrations on a remote database already configured correctly, while ALSO being allowed to apply future migrations to it?
Context:
I have an existing rails app with plenty of migrations, that, up until now, has been using a local postgres database. A remote postgres database which SHOULD mostly match up to the structure of my local database exists.
When I try to connect to it, I get a "pending migrations" error. Attempting to run my migrations gets errors about tables already existing.
I want to skip all current migrations, but UNLIKE a lot of similar questions I'm seeing, I want to make sure my future migrations will work on this remote database.
Edit:
I followed this answer:
Rails 4 how to ignore pending migrations
And inserted my own current scheme version number into the remove database's schema_migrations table. But the pending migration remains.
I can confirm trying to run the migration gets me an error of a table already existing. This table is the FIRST migration past the remote databases previously most recent version in the schema table. It appears to be ignoring my inserted current most recent version.
You have to add all migration version numbers that you want to ignore to schema_migrations
Other technique that may be also applicable - migration squashing with something like squasher or manually. Point is in combining all old migrations that are not to be rolled back into one.
Since doing migrations with rails + git is type of a pain, a new thorn has sprung..
Before I am doing any harm to my prod DB, would the following situation cause havoc? If so, how would I handle it?
I am working a long-term feature in a separate branch (feature/long-term). This feature is an overhaul of a lot of components and it will take awhile to complete. This feature has new migrations, which were migrated to the localhost DB.
meanwhile, I need to fix/add a migration to the prod system via another branch (feature/quick-fix). This has a migration file with date later than the feature/long-term migration.
The migrations of the quick-fix and the long-term have nothing to do with each other, they do not collide and work on separate tables. It doesn't matter what order they are run.
If I merge feature/quick-fix to master and db:migrate and in a few days/weeks merge feature/long-term the migration files order would be the long-term first.
Would this affect the DB in some way? (the prod DB is important, so I don't want to reset)
What you described is a very common development workflow (especially so in teams with more members) and it's perfectly safe for your production DB.
Rails, as of version 2.1, is smart enough to keep a list of all migrations ever run, instead of just the latest migration version run. This information is stored on a separate table aptly named schema_migrations.
So, if you push a new migration today, say 20140527_quick_fix.rb, and a month after that you push a new (but with an older timestamp) one 20140101_long_term_feature.rb, Rails will still know that the latter was never run in your production environment so during rake db:migrate it will process it, as you would expect. The newest won't be run again of course as the Rails would know that it has already been processed.
From the official documentation:
Rails versions 2.0 and prior used to create a table called schema_info when using migrations. This table contained the version of the schema as of the last applied migration.
Starting with Rails 2.1, the schema_info table is (automatically) replaced by the schema_migrations table, which contains the version numbers of all the migrations applied.
As a result, it is now possible to add migration files that are numbered lower than the current schema version: when migrating up, those never-applied “interleaved” migrations will be automatically applied, and when migrating down, never-applied “interleaved” migrations will be skipped.
I restored my database from the latest dump and tried to run rake tests. Unfortunately 30 migrations were pending. My first idea was to comment out each of 30 migrations code and run 'rake db:migrate' but there must be a simpler solution. I use Rails 2.3.14 and Postgresql 9.1.3.
If you're restoring a database from a dump, the schema_migrations table should restore along with the rest of the tables.
This seems to indicate your schema_migrations table may not be getting backed up which would lead to the problem you have now.
The ideal solution would be to restore a backup that has all the tables in it correctly -- including schema_migrations.
Even if you decide to find a way around this in the short-term, in the long-term the correct solution is to modify your backup scripts to get all the tables you need, including schema_migrations.
In terms of what to do now, the ideal solution is probaby to backup just that one table (schema_migrations) from your database and import that data into the database you're trying to load now. Then your migrations should no longer be pending.
Doing that with a simple table dump and load script should be fine. The simple postgres gui PgAdmin ( http://www.pgadmin.org/ ) may also provide some basic tools for dumping then loading a single table.
Kevin is correct. However, he is missing a critical point.
When you restore from a backup it restores the schema_migrations table which tracks which migrations need to be run. If those thirty migrations had been run on the database you restored from, they would not have run.
However, your code is thirty migrations ahead of the snapshot of your database represented by the backup.
This can happen to me if I deploy, then grab the production backup right away. Although the migrations have run on production, I'm getting the backup from before office hours prior to my deployment. I usually like to wait a day and get the next day's backup.
Or, don't worry about it. Your backup is before those thirty migrations, but then they were applied, so the migrations have made sure your schema matches the version of your code. That's a good thing.
Don't sweat it, and refresh again tomorrow, when the backup has your changes.
You could also manually add the timestamps of the missing migrations in the db table like:
INSERT INTO "public"."schema_migrations"("version") VALUES ('20201212012345')
That should have the same effect as temporarily outcommenting the 'create' instructions in the migration files. If you run migrations as part of a deploy process from git, commenting out would mean you had to push those changes to git.
If you just work on staging / development env, directly fixing the db might be nicer than pushing those changes, possibly confusing other deploys or devs.
In section Rails Database Migrations of Ruby on Rails Guides, there is one line saying that
The db:reset task will drop the database, recreate it and load the current
schema into it. This is not the same as running all the migrations.
Can anyone tell me where exactly they are different and why it is more error prone to replay the migration history?
I'm fairly new to Ruby on Rails. Thanks in advance.
The schema file contains the current structure of your database. When you load it, you are guaranteed to have the exact schema in your db that is in the file. Migrations were designed to make incremental changes in the database. You may add a table, then some columns, and then remove the table in three separate migrations. There's no need to go through all this when the schema already knows that the table no longer exists.
On why they are error prone, I'm not totally sure. The one thing I can think of is that migrations can be used to make changes to data and not just the structure.
Running rake db:reset will rebuild the structure of your database from schema.db, which essentially works as a cached version of your migrated database structure. Running all your migrations, on the other hand, applies the migrations one by one, which may include arbitrary code to accomodate for changes to the database (e.g. prepopulate an added counter cache column).
It can be more error prone to replay the migration history, since it is the product of changes to both the structure and data of the database. If the developers haven't been careful, it might not apply cleanly to a fresh environment (e.g. the migration assumes an old version of a model). On the other hand, schema.db can get out of sync if you edit a migration once you've migrated (a useful trick to avoid migration explosion during development). In that case, you need to run rake db:migrate:reset.
I have several dozens Rails DB migrations which were written over a year. Is there a way to aggregate them to one migration so that I will just see a full DDL statement for the database as it exists now? I just need the current snaphot without all the history of how we got to it.
It is possible, but probably not a good idea to aggregate the migrations!
Maybe ask:
Why do you want to do this?
How often do you really need to migrate all the way to VERSION=0 and then back up again?
Is something really broken? (if not, then don't fix it)
I've had the same problem once.. I ended up just re-ordering my migrations, because changes in the schema caused it to not correctly migrate up/down anymore. I would be hesitant to do that again.
If you have migrations which just add fields or indexes, then maybe you can combine them with the main migration for the model -- but beware that you can't reproduce old situations anymore, e.g. older DB-dumps may not be compatible with what migration number they should be compatible with -- that is probably the biggest argument against aggregating...
Technically, you can dump the schema and then load it directly - that is one way:
rake db:schema:dump
then create a single new migration with the contents of the schema dump file db/schema.rb
Here are some similar questions:
Rebase Rails migrations in a long running project
Deleting/"Rebasing" rails migrations
Way to "flatten" Rails migrations?
Should I flatten Rails migrations?
P.S.: I found it useful to stick with the old migration numbering scheme, where the migrations do not use timestamps - for me this works better (is easier to see in which order they are).
e.g. in your config/application.rb file:
config.active_record.timestamped_migrations = false
You should never be using all the migrations to get a database up and running. The current schema.rb is always what the DB looks like 'presently'.
It's good practice to periodically just truncate your migrations if you have a ton of them in there. We finally did that with one of our larger applications, removing a good 50 migrations from the folder because the only thing that matters is schema.rb. Migrations are just that, a way to migrate and make changes to an existing state of the database. They should only ever have to be run once.
You can simply load the current schema into the DB.
rake db:schema:load RAILS_ENV=[production, test, etc.]
This will take the schema.rb file's version of the schema, and load it into the DB without running individual migrations.
NOTE: if you have migrations that put data into the DB (e.g. default values, for example), that data will not be added to the DB.
If you need to load default values into your DB, that might be better done via a custom rake task, independent of migrations.