In Rails, migrations have a down method by default for reverting a migration. In what scenario would I ever want to revert a migration, though?
Some thoughts:
Whether in development or production, I always have a snapshot of my database to go back to, before I even run the migrations. Especially for migrations which perform data conversion, I find in most cases that reverting a snapshot is even faster than reverting a migration. (So I would never do it in a rush!)
If a migration were to fail, it would either:
fail with an exception on a non-transactional database, and thus leave the database broken, or
fail with an exception and roll back the transaction, and thus there would be no need to revert otherwise.
If the changes made are in production (or late in development), and later turn out to be a mistake, I would fix my mistake in a new migration. I would not revert the old one. In development, I'd simply delete the migration.
I also find that the down method introduces extra code in which I repeat myself, and thus may introduce new bugs. This is against the DRY principle.
So I'm curious about the pros, because I can't think of any.
In development, it is easy and fast to incrementally "improve" migrations by using the down method automatically. Eg
Create a migration and migrate to it
Realize you need to make a change
Migrate to the ver prior to your new migration by using db:migrate with a version
Improve/fix your migration
Rerun the migration task
Your method of taking snapshots works fine. But rails includes the same effect auto-magically using the "down" migration techniques. Works with all db's, tastes great
Added:
For production, I agree that a down migration shouldn't be needed. But sometimes mistakes happen and you need to roll back. The down migration path gives you a first, and quick opportunity to fix things in an emergency situation during an upgrade that goes wrong.
-- it is much faster to try a down migration in an emergency than to restore the db using a checkpoint.
The "down" migration used for DB Rollbacks is in place so that every action has an equal and opposite action. It takes the onus off of the developer to maintain database snapshots, and allows them to use code to achieve the same ends. As Larry K said,
they're good for situations like so:
Add a database column called 'resubmitted', it's a boolean.
Product owner says they can resubmit multiple times, so change that column needs to be an int
Now, if you're 10 or 15 migrations deep, it's easier to just write a new one instead of losing all the dev data in the new tables/columns by doing a rollback. However, if you've just written that migration, it's cleaner and less cluttered to roll back, change the migration, and re-run it.
The other extremely useful feature of rollbacks is this:
Developer 1 has his own Dev DB. He writes a migration and runs it.
Developer 1 commits his migration to source control
Developer 2 has her own Dev DB. She writes a migration and runs it.
Developer 2 updates from source control
Developer 2 tries to run migrations but can't, as her local DB says "the latest migration has already been run", since her migration (the latest one), has technically already been run. Now she needs to rollback, then do a db:migrate to get all migrations in her local DB.
The idea of running a down migration in production horrifies me. Back when the preferred way to rollback all migrations was rake db:migrate VERSION=0 I would do this all the time in development. However, then I got paranoid that because it was committed to muscle memory I would accidentally type this on a production server when I meant to simply migrate.
Because of this paranoia, I add the following to all of my down methods.
def self.down
if Rails.env.production?
raise ActiveRecord::IrreversibleMigration
else
drop_table :foo_bars
end
end
This way, it still works in development, but I can't accidentally nuke my production database from orbit while half-asleep at 2:00 AM.
When a migration hasn't had the expected result, it is better to roll back and rewrite it than to keep the failed migration in the code.
Related
I added a migration in branch "add_dogs" with migration db/migrate/20221220155010_create_dogs.rb, and ran db:migrate.
Later on, I changed branches (without a merge), and ultimately abandoned the "new_dogs" branch.
Later later on, I checked out "add_cats" branch with db/migrate/20221101010101_create_cats.rb, and ran db:migrate. So far, all is well.
But then I tweak the "add_cats" migration (before committing anything), and ran db:rollback so I can run it again. I get this error:
ActiveRecord::UnknownMigrationVersionError:
No migration with version number 20221220155010.
I can still run db:migrate on new migrations just fine, but not db:rollback or db:migrate:redo.
This makes sense, because the database has a record of applying 20221220155010, but that migration file no longer exists, so there is no way to roll it back.
How can I get past this?
Here are three ways to deal with a missing migration file, depending on your needs and access:
For a quick temporary fix, you can roll back just the migration you're currently editing so you can run it again. This may be useful if the other migration is still in the pipeline on the other branch and both eventually will get merged.
rake db:migrate:down VERSION=20230101010101
// This is the version of the migration you WANT to rollback, not the missing one.
If the missing migration will never come back, you want a permanent fix. The simplest way is to remove that record from the database. You can do this from your favorite SQL client, rails console, etc. (I suppose you could even write a migration to do that, but that seems mighty sketchy.)
DELETE FROM schema_migrations WHERE version = '20221220155010'
-- This is the version of the migration that is MISSING, not the one you are working on.
If you don't have direct access to the database for whatever reason, you can give Rails a placebo to rollback. Ensure the timestamp in the filename matches the missing migration's version number.
Create a file named db/migrations/20221220155010_just_kidding.rb:
class JustKidding < ActiveRecord::Migration
def change
# nothing to see here.
end
end
Then, rails db:rollback will roll back that no-op migration and delete 20221220155010 from the schema_migrations table. You can now delete the placebo migration forever and you'll be in good shape as far as running migrations and rollbacks.
However...don't forget that the effects of the old migration are still in your schema. Maybe you're stuck with a new, unused 'dogs' table or an extra column on a table. Maybe that's benign on your dev box, but you certainly don't want that cruft on a production environment. All the advice in this answer assumes you're on a throw-away environment and that the effects of the old migration aren't a problem. Tearing down your whole database and rebuilding may become a more attractive option in this case.
One of the realy take-aways here is... don't let this happen in the first place! Ideally, you should rollback any new, uncommitted migrations before changing away from a branch. But...things happen...
p.s. If there is a way to do this from the command line, I'd love to learn it. I'm imagining something like rails db:migrate:delete VERSION=20230101010101 might be handy in a hackish kind of way.
I took a couple courses on rails but a few things are unclear to me regarding migrations:
1.) If I generate a migrations and run it, no matter how few operations I perform does rails still create a whole entirely new version of my schema? Is there anyway to view each version's schema before migrations back down?
2.) If I do not define the down method in a migration and I try and rollback, will the rollback do nothing?
3.) Should I delete migrations after I run them?
you dont need to delete the migration file after migrate if you run a migrate to create a table without defining a down method , if you tried rake db:rollback it will revert the last migrate you did , you can see more in here Migration
Migrate does incremental changes to your schema. It knows (unless you stuff it up) what state your schema is in and runs those migrations that haven't yet been run.
e.g. if you do a drop create and migrate and then migrate a second time, the second one does nothing, because they've all be done.
No down indeed means rollback will do nothing.
The only time you should delete a migration is if you are completely undoing a change. ie you added a model and then decided you didn't need it.
Any other approach would mean you couldn't achieve your schema from scratch.
e.g. you add a model and migrate
then you realise you need a relation and do that. delete the migration that added the table, things go horribly wrong.
Is having a long list of migration files often common when building a web application? I seem to be adding up a long list of migration files because I keep forgetting or keep thinking of adding an extra column to a 'already' migrated database table.
Having a long list of migration files is normal. It's one of the best features of rails. Think of them as layers(like an onion) that you stack on top of each other. If you add a new column or table and then you decide that you don't want it anymore you can rollback(peel away) the latest changes. As long as you have the migration files you can move back and forth easily(don't recommend moving much but you get the point). REMEMBER DO NOT DELETE migration files once they are raked unless you do a rollback. When you rollback and delete a migration file make absolutely SURE you are at the right layer(rollback point).
why? because for example when someone clones your app and runs your migration file it goes through all the migration files from beginning to end. if something in the middle is messed up or deleted you won't be able to create the database because it goes through ALL the steps. Hope it helps.
It may be bad habit, but I migrate down using rake db:migrate VERSION=0, then change the respective migration that has (for instance) the users, and finally migrate the database using rake db:migrate. That way i have less of a mess, and know exactly which migration does what to what model. It's cleaner, but I guess this technique can be used only at the beginning of the webapp.
Hope this helps.
One more point to be noted, from rails docs
If you need to create the application database on another system, you should be using db:schema:load, not running all the migrations from scratch. The latter is a flawed and unsustainable approach (the more migrations you'll amass, the slower it'll run and the greater likelihood for issues).
It's strongly recommended to check this file(schema.rb) into your version control system.
IMO, if you do the migrations correctly it should have to have a long list of migrations. Because every little change you have to do via migration. As you said, proper way is to add a new migration when you need to alter a table.
So as I mentioned I believe having more and more migrations means you are doing it correct. (Because most of the time when you need alterations to to an existing table you simple cannot drop the table and re-create it).
But having said that it always a good idea to run rake db:migrate every now and then (for an isolated db) just to make sure your migrations are working as a group.
In other words I'd like to know the best way to make sure that self.down actually rolls back self.up before running the migration in question.
What can I do if I need to rollback a migration but self.down doesn't serve the purpose?
Which is the best practice when dealing with potentially destructive migrations?
Just a database backup?
Thanks,
Duccio.
You should be developing on a development database which should not contain live data. Therefore it should not matter if the data is destroyed as you can easily generate it again?
A database backup might be appropriate if you find yourself in a situation where your development data is important but not ideal.
Typically migrations should contain only schema changes. In that case it should be very safe & easy to run the migrations in the dev/test environment. If something goes wrong you can alway re-create the database and populate it with some test data. But if you have some data related migrations to be tested, things might go wrong when you actually run them on production.
In that case as you mentioned database backup is what you should rely on. Come with a proper & quick restore mechanism before deploying.
To make sure the migrations behave as you want you should experiment in your development environment.
Run the command
rake -T
to show you the available tasks such as
rake db:migrate
or
rake db:rollback
Each migration runs inside a transaction. Keep that in mind. That means that if something would go wrong inside a single migration, the migration is rolled back (and if there are any following they are not executed).
To test migrations, be it up or down I insert a lot of puts-statements, to check everything has worked has supposed to, and then in my last line I raise an exception. This will make rails think the migration has failed, and it will rollback the operation (as if it never happened).
When I am sure everything works as it should, I remove the raise line and let the migration really work.
In your case, you would test with the raise, remove the raise and NOT run it again I assume :)
Hope this helps.
I have been running a big Rails application for over 2 years and, day by day, my ActiveRecord migration folder has been growing up to over 150 files.
There are very old models, no longer available in the application, still referenced in the migrations. I was thinking to remove them.
What do you think? Do you usually purge old migrations from your codebase?
The Rails 4 Way page 177:
Sebastian says...
A little-known fact is that you can remove old migration files (while
still keeping newer ones) to keep the db/migrate folder to a
manageable size. You can move the older migrations to a
db/archived_migrations folder or something like that. Once you do trim
the size of your migrations folder, use the rake db:reset task to
(re-)create your database from db/schema.rb and load the seeds into
your current environment.
Once I hit a major site release, I'll roll the migrations into one and start fresh. I feel dirty once the migration version numbers get up around 75.
I occasionally purge all migrations, which have already been applied in production and I see at least 2 reasons for this:
More manageable folder: it is easier to spot a new migration.
Cleaner text search results: global text search within a project does not lead to tons of useless matches because of some 3-year-old migration when someone added or removed some column which anyway does not exist anymore.
They are relatively small, so I would choose to keep them, just for the record.
You should write your migrations without referencing models, or other parts of application, because they'll come back to you haunting ;)
Check out these guidelines:
http://guides.rubyonrails.org/migrations.html#using-models-in-your-migrations
Personally I like to keep things tidy in the migrations files. I think once you have pushed all your changes into prod you should really look at archiving the migrations. The only difficulty I have faced with this is that when Travis runs it runs a db:migrate, so these are the steps I have used:
Move historic migrations from /db/migrate/ to /db/archive/release-x.y/
Create a new migration file manually using the version number from the last run migration in the /db/archive/release-x.y directory and change the description to something like from_previous_version. Using the old version number means that it won't run on your prod machine and mess up.
Copy the schema.rb contents from inside the ActiveRecord::Schema.define(version: 20141010044951) do section and paste into the change method of your from_previous_version changelog
Check all that in and Robert should be your parent's brother.
The only other consideration would be if your migrations create any data (my test scenarios contain all their own data so I don't have this issue)
Why? Unless there is some kind of problem with disk space, I don't see a good reason for deleting them. I guess if you are absolutely certain that you are never going to roll back anything ever again, than you can. However, it seems like saving a few KB of disk space to do this wouldn't be worth it. Also, if you just want to delete the migrations that refer to old models, you have to look through them all by hand to make sure you don't delete anything that is still used in your app. Lots of effort for little gain, to me.
See http://edgeguides.rubyonrails.org/active_record_migrations.html#schema-dumping-and-you
Migrations are not a representation of the database: either structure.sql or schema.rb is. Migrations are also not a good place for setting/initializing data. db/seeds or a rake task are better for that kind of task.
So what are migrations? In my opinion they are instructions for how to change the database schema - either forwards or backwards (via a rollback). Unless there is a problem, they should be run only in the following cases:
On my local development machine as a way to test the migration itself and write the schema/structure file.
On colleague developer machines as a way to change the schema without dropping the database.
On production machines as a way to change the schema without dropping the database.
Once run they should be irrelevant. Of course mistakes happen, so you definitely want to keep migrations around for a few months in case you need to rollback.
CI environments do not ever need to run migrations. It slows down your CI environment and is error prone (just like the Rails guide says). Since your test environments only have ephemeral data, you should instead be using rake db:setup, which will load from the schema.rb/structure.sql and completely ignore your migration files.
If you're using source control, there is no benefit in keeping old migrations around; they are part of the source history. It might make sense to put them in an archive folder if that's your cup of coffee.
With that all being said, I strongly think it makes sense to purge old migrations, for the following reasons:
They could contain code that is so old it will no longer run (like if you removed a model). This creates a trap for other developers who want to run rake db:migrate.
They will slow down grep-like tasks and are irrelevant past a certain age.
Why are they irrelevant? Once more for two reasons: the history is stored in your source control and the actual database structure is stored in structure.sql/schema.rb. My rule of thumb is that migrations older than about 12 months are completely irrelevant. I delete them. If there were some reason why I wanted to rollback a migration older than that I'm confident that the database has changed enough in that time to warrant writing a new migration to perform that task.
So how do you get rid of the migrations? These are the steps I follow:
Delete the migration files
Write a rake task to delete their corresponding rows in the schema_migrations table of your database.
Run rake db:migrate to regenerate structure.sql/schema.rb.
Validate that the only thing changed in structure.sql/schema.rb is removed lines corresponding to each of the migrations you deleted.
Deploy, then run the rake task from step 2 on production.
Make sure other developers run the rake task from step 2 on their machines.
The second item is necessary to keep schema/structure accurate, which, again, is the only thing that actually matters here.
It's fine to remove old migrations once you're comfortable they won't be needed. The purpose of migrations is to have a tool for making and rolling back database changes. Once the changes have been made and in production for a couple of months, odds are you're unlikely to need them again. I find that after a while they're just cruft that clutters up your repo, searches, and file navigation.
Some people will run the migrations from scratch to reload their dev database, but that's not really what they're intended for. You can use rake db:schema:load to load the latest schema, and rake db:seed to populate it with seed data. rake db:reset does both for you. If you've got database extensions that can't be dumped to schema.rb then you can use the sql schema format for ActiveRecord and run rake db:structure:load instead.
Yes. I guess if you have completely removed any model and related table also from database, then it is worth to put it in migration. If model reference in migration does not depend on any other thing, then you can delete it. Although that migration is never going to run again as it has already run and even if you don't delete it from existing migration, then whenever you will migrate database fresh, it cause a problem.
So better it to remove that reference from migration. And refactore/minimize migrations to one or two file before big release to live database.
I agree, no value in 100+ migrations, the history is a mess, there is no easy way of tracking history on a single table and it adds clutter to your file finding. Simply Muda IMO :)
Here's a 3-step guide to squash all migrations into identical schema as production:
Step1: schema from production
# launch rails console in production
stream = StringIO.new
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, stream); nil
stream.rewind
puts stream.read
This is copy-pasteable to migrations, minus the obvious header
Step 2: making the migrations without it being run in production
This is important. Use the last migration and change it's name and content. ActiveRecord stors the datetime number in it's schema_migrations table so it knows what it has run and not. Reuse the last and it'll think it has already run.
Example: rename 20161202212203_this_is_the_last_migration -> 20161202212203_schema_of_20161203.rb
And put the schema there.
Step 3: verify and troubleshoot
Locally, rake db:drop, rake db:create, rake db:migrate
Verify that schema is identical. One issue we encountered was datetime "now()" in schema, here's the best solution I could find for that: https://stackoverflow.com/a/40840867/252799