Is having a long list of migration files often common when building a web application? I seem to be adding up a long list of migration files because I keep forgetting or keep thinking of adding an extra column to a 'already' migrated database table.
Having a long list of migration files is normal. It's one of the best features of rails. Think of them as layers(like an onion) that you stack on top of each other. If you add a new column or table and then you decide that you don't want it anymore you can rollback(peel away) the latest changes. As long as you have the migration files you can move back and forth easily(don't recommend moving much but you get the point). REMEMBER DO NOT DELETE migration files once they are raked unless you do a rollback. When you rollback and delete a migration file make absolutely SURE you are at the right layer(rollback point).
why? because for example when someone clones your app and runs your migration file it goes through all the migration files from beginning to end. if something in the middle is messed up or deleted you won't be able to create the database because it goes through ALL the steps. Hope it helps.
It may be bad habit, but I migrate down using rake db:migrate VERSION=0, then change the respective migration that has (for instance) the users, and finally migrate the database using rake db:migrate. That way i have less of a mess, and know exactly which migration does what to what model. It's cleaner, but I guess this technique can be used only at the beginning of the webapp.
Hope this helps.
One more point to be noted, from rails docs
If you need to create the application database on another system, you should be using db:schema:load, not running all the migrations from scratch. The latter is a flawed and unsustainable approach (the more migrations you'll amass, the slower it'll run and the greater likelihood for issues).
It's strongly recommended to check this file(schema.rb) into your version control system.
IMO, if you do the migrations correctly it should have to have a long list of migrations. Because every little change you have to do via migration. As you said, proper way is to add a new migration when you need to alter a table.
So as I mentioned I believe having more and more migrations means you are doing it correct. (Because most of the time when you need alterations to to an existing table you simple cannot drop the table and re-create it).
But having said that it always a good idea to run rake db:migrate every now and then (for an isolated db) just to make sure your migrations are working as a group.
Related
I'm working on a project that's accumulated hundreds of migrations, and I'm unsure what to do with them long term. I suppose they're no hurting anything, but it seems strange to keep around a bunch of old files for incremental migrations, some of them creating tables that are later removed.
So far, I've seen three possibilities:
Leave them alone. They're not hurting anything.
Just delete them. I don't see much harm in doing this, since a new developer would be probably be starting with schema load anyway, not migrations.
Delete them all, create a new one with a timestamp matching an old merge, and create a new merge from your schema. This seems very clean, but I'm not sure who would actually use it.
I'm inclined to just delete them, but I'm curious if there's a big pitfall I'm missing.
In my opinion, as soon as every database on the project, especially the production, are at least at version '201xxxxxxxx', it should be fine to delete migrations before that version. They are not technically necessary anymore.
After that, if you want to play archaeology with your database history, you can still use your version control system.
With Git for example, you can use the following commands to have a quick look over the past :
git log --name-only db/migrate/ #to list commit involving migrations + migration filename
git show xxxxx db/migrate #to see the code of commit xxxxx's migration(s)
Alternatively, you can browse repository history of schema.rb, identify a commit and see the corresponding migration content with the command above.
If you prefer to have a lighter db/migrate and use Version control, I would go for a little cleanup.
If you find it more convenient to have the whole migrations history directly available because it's easier to browse, I would go for option 1.
Note: it is very likely that old migrations do not make sense with the current application code. For example, some migrations may refer to class or methods that don't exist anymore.
Using version control to checkout the application at the time the migration was written could also avoid some confusion.
Personally, I lean toward option 1: It's generally true that at some point in any project, the schema is what matters, and the migrations are just a curiosity, but you're right when you say they're not hurting anything. Theoretically the old migrations could be useful for someone who wanted to go back and see how the database was organized at some point in the past.
I don't know of any serious pitfalls in deleting them, but I also don't see an advantage to doing so, unless it's the saving of time scrolling past them when you want to edit a new migration.
I don't think the effort of putting together a single migration that duplicates the schema is beneficial - it's extra work, and that's what the schema is for anyway.
If you work on a project large enough for long enough, there will come a time when you look at all of those extra migrations with disdain and wonder how so many can exist.
You think to yourself, "just delete them...they don't do anything." This is a perfectly logical and normal thought process (especially as a Rails developer, we just love to minimize code and make things efficient!), but don't let the dark side tempt you.
By deleting migrations, you are deleting the historical record of your application's data model, and worse, the logical path you took to get to your current model. This history can help you remember why you did what you did, and didn't do what you didn't do.
Yes, we've all been guilty of deleting migrations from time to time. But you must resist the temptation as the net benefit would only be a few kB and a cleaner migration folder.
Remember:
Those who delete migrations are condemned to repeat them!
In one of our former projects we dropped all migrations, created a new one which would truncate schema_migrations table with manual sql and then copied db/schema.rb contents to it.
Surely this migration is irreversible, however it allowed us to get rid of hundreds of old valueless migrations but still be able to re-create db not from db schema only but from migrations as well.
I wrote a few migrations for my Rails 3 app, but I would like to change the order of the migrations. How can I change the migration order or sequence? Is it as simple as renaming the migration file with what appears to be the timestamp?
I know this is an odd question, but basically, I made a mess of my migrations and removed some old migrations and now I need to drop a table before creating a new one. I also know I can include the drop statement in the create-the-new-table migration, but I'm curious to know how to reorder migrations.
Yes, it runs the migrations which have not been run in the order of the prefix. In earlier versions of rails, maybe 2.1 or 2.2, they used to be numbered starting with 01, but they switched to timestamps.
There is a table which keeps track of which migrations have run. The intentions is, multiple developers my have added migrations, and checked them in version control later. So, there may be a migration which has not run, but is numbered before the highest numbered migration which has run.
If you change the migration sequence, (and I have) it's better to first down version before all the migrations you are re-sequencing. Use the VERSION option with db:migrate. If the highest numbered migration that you want to keep (not run the down) is 20120318143249, call it this way.
rake db:migrate VERSION=20120318143249
I often run the down on a migration, and re-run it until I get the details of the migration to my satisfaction. Sometimes I re-order them, when I want to work on one of them, and I want it to be the last.
Yes, the prefix on the filename is what determines the order of execution. However, keep in mind this will only change your current system if you wipe your DB/start over.
I was having a similar issue because the migrations were not happening in the right order.
After reading Mitch's answered I decided to rename the migrations to sequence the dates the way I needed.
After this fix I ran rails db:migrate and it worked.
Not the most elegant solution but it worked so I thought I'd share it.
I have several dozens Rails DB migrations which were written over a year. Is there a way to aggregate them to one migration so that I will just see a full DDL statement for the database as it exists now? I just need the current snaphot without all the history of how we got to it.
It is possible, but probably not a good idea to aggregate the migrations!
Maybe ask:
Why do you want to do this?
How often do you really need to migrate all the way to VERSION=0 and then back up again?
Is something really broken? (if not, then don't fix it)
I've had the same problem once.. I ended up just re-ordering my migrations, because changes in the schema caused it to not correctly migrate up/down anymore. I would be hesitant to do that again.
If you have migrations which just add fields or indexes, then maybe you can combine them with the main migration for the model -- but beware that you can't reproduce old situations anymore, e.g. older DB-dumps may not be compatible with what migration number they should be compatible with -- that is probably the biggest argument against aggregating...
Technically, you can dump the schema and then load it directly - that is one way:
rake db:schema:dump
then create a single new migration with the contents of the schema dump file db/schema.rb
Here are some similar questions:
Rebase Rails migrations in a long running project
Deleting/"Rebasing" rails migrations
Way to "flatten" Rails migrations?
Should I flatten Rails migrations?
P.S.: I found it useful to stick with the old migration numbering scheme, where the migrations do not use timestamps - for me this works better (is easier to see in which order they are).
e.g. in your config/application.rb file:
config.active_record.timestamped_migrations = false
You should never be using all the migrations to get a database up and running. The current schema.rb is always what the DB looks like 'presently'.
It's good practice to periodically just truncate your migrations if you have a ton of them in there. We finally did that with one of our larger applications, removing a good 50 migrations from the folder because the only thing that matters is schema.rb. Migrations are just that, a way to migrate and make changes to an existing state of the database. They should only ever have to be run once.
You can simply load the current schema into the DB.
rake db:schema:load RAILS_ENV=[production, test, etc.]
This will take the schema.rb file's version of the schema, and load it into the DB without running individual migrations.
NOTE: if you have migrations that put data into the DB (e.g. default values, for example), that data will not be added to the DB.
If you need to load default values into your DB, that might be better done via a custom rake task, independent of migrations.
How do you go about changing column names and types in your rails app? Do you create a new migration to make the changes, or do you rollback, edit your migration file, and then migrate again?
What's the "proper" way to do this in Rails?
It sort of depends on when this happened in your development cycle, If you recently made the change and haven't pushed it out into a public repo, then you indeed might want to do the rollback thing and then edit the migration files and migrate again, just to keep things clean. But if it's a change to a migration that's a few migrations back then you should create a new migration that changes the rows and columns to the "new" old values.
Well, to undo a migration you typically want to roll it back:
bundle exec rake db:rollback
Where VERSION= can also be specified. If you wanted to change it to something entirely new, you would make a new migration. Typically you shouldn't be touching old migrations at all.
Rails gives you the choice of creating a migration and changing it a various of ways. So there is no "ruby developers" technique of choice.
However, every time you create a migration, a file is created, and it represents the history of your development. There are a lot of cases that a simpler file should be uselfull, like a code that other ruby beginners would have to look at and modify. On other cases may be necessary to an advanced developer understand changes and improvements made on the code, specially if it is a code running for a long time (some years maybe).
I have been running a big Rails application for over 2 years and, day by day, my ActiveRecord migration folder has been growing up to over 150 files.
There are very old models, no longer available in the application, still referenced in the migrations. I was thinking to remove them.
What do you think? Do you usually purge old migrations from your codebase?
The Rails 4 Way page 177:
Sebastian says...
A little-known fact is that you can remove old migration files (while
still keeping newer ones) to keep the db/migrate folder to a
manageable size. You can move the older migrations to a
db/archived_migrations folder or something like that. Once you do trim
the size of your migrations folder, use the rake db:reset task to
(re-)create your database from db/schema.rb and load the seeds into
your current environment.
Once I hit a major site release, I'll roll the migrations into one and start fresh. I feel dirty once the migration version numbers get up around 75.
I occasionally purge all migrations, which have already been applied in production and I see at least 2 reasons for this:
More manageable folder: it is easier to spot a new migration.
Cleaner text search results: global text search within a project does not lead to tons of useless matches because of some 3-year-old migration when someone added or removed some column which anyway does not exist anymore.
They are relatively small, so I would choose to keep them, just for the record.
You should write your migrations without referencing models, or other parts of application, because they'll come back to you haunting ;)
Check out these guidelines:
http://guides.rubyonrails.org/migrations.html#using-models-in-your-migrations
Personally I like to keep things tidy in the migrations files. I think once you have pushed all your changes into prod you should really look at archiving the migrations. The only difficulty I have faced with this is that when Travis runs it runs a db:migrate, so these are the steps I have used:
Move historic migrations from /db/migrate/ to /db/archive/release-x.y/
Create a new migration file manually using the version number from the last run migration in the /db/archive/release-x.y directory and change the description to something like from_previous_version. Using the old version number means that it won't run on your prod machine and mess up.
Copy the schema.rb contents from inside the ActiveRecord::Schema.define(version: 20141010044951) do section and paste into the change method of your from_previous_version changelog
Check all that in and Robert should be your parent's brother.
The only other consideration would be if your migrations create any data (my test scenarios contain all their own data so I don't have this issue)
Why? Unless there is some kind of problem with disk space, I don't see a good reason for deleting them. I guess if you are absolutely certain that you are never going to roll back anything ever again, than you can. However, it seems like saving a few KB of disk space to do this wouldn't be worth it. Also, if you just want to delete the migrations that refer to old models, you have to look through them all by hand to make sure you don't delete anything that is still used in your app. Lots of effort for little gain, to me.
See http://edgeguides.rubyonrails.org/active_record_migrations.html#schema-dumping-and-you
Migrations are not a representation of the database: either structure.sql or schema.rb is. Migrations are also not a good place for setting/initializing data. db/seeds or a rake task are better for that kind of task.
So what are migrations? In my opinion they are instructions for how to change the database schema - either forwards or backwards (via a rollback). Unless there is a problem, they should be run only in the following cases:
On my local development machine as a way to test the migration itself and write the schema/structure file.
On colleague developer machines as a way to change the schema without dropping the database.
On production machines as a way to change the schema without dropping the database.
Once run they should be irrelevant. Of course mistakes happen, so you definitely want to keep migrations around for a few months in case you need to rollback.
CI environments do not ever need to run migrations. It slows down your CI environment and is error prone (just like the Rails guide says). Since your test environments only have ephemeral data, you should instead be using rake db:setup, which will load from the schema.rb/structure.sql and completely ignore your migration files.
If you're using source control, there is no benefit in keeping old migrations around; they are part of the source history. It might make sense to put them in an archive folder if that's your cup of coffee.
With that all being said, I strongly think it makes sense to purge old migrations, for the following reasons:
They could contain code that is so old it will no longer run (like if you removed a model). This creates a trap for other developers who want to run rake db:migrate.
They will slow down grep-like tasks and are irrelevant past a certain age.
Why are they irrelevant? Once more for two reasons: the history is stored in your source control and the actual database structure is stored in structure.sql/schema.rb. My rule of thumb is that migrations older than about 12 months are completely irrelevant. I delete them. If there were some reason why I wanted to rollback a migration older than that I'm confident that the database has changed enough in that time to warrant writing a new migration to perform that task.
So how do you get rid of the migrations? These are the steps I follow:
Delete the migration files
Write a rake task to delete their corresponding rows in the schema_migrations table of your database.
Run rake db:migrate to regenerate structure.sql/schema.rb.
Validate that the only thing changed in structure.sql/schema.rb is removed lines corresponding to each of the migrations you deleted.
Deploy, then run the rake task from step 2 on production.
Make sure other developers run the rake task from step 2 on their machines.
The second item is necessary to keep schema/structure accurate, which, again, is the only thing that actually matters here.
It's fine to remove old migrations once you're comfortable they won't be needed. The purpose of migrations is to have a tool for making and rolling back database changes. Once the changes have been made and in production for a couple of months, odds are you're unlikely to need them again. I find that after a while they're just cruft that clutters up your repo, searches, and file navigation.
Some people will run the migrations from scratch to reload their dev database, but that's not really what they're intended for. You can use rake db:schema:load to load the latest schema, and rake db:seed to populate it with seed data. rake db:reset does both for you. If you've got database extensions that can't be dumped to schema.rb then you can use the sql schema format for ActiveRecord and run rake db:structure:load instead.
Yes. I guess if you have completely removed any model and related table also from database, then it is worth to put it in migration. If model reference in migration does not depend on any other thing, then you can delete it. Although that migration is never going to run again as it has already run and even if you don't delete it from existing migration, then whenever you will migrate database fresh, it cause a problem.
So better it to remove that reference from migration. And refactore/minimize migrations to one or two file before big release to live database.
I agree, no value in 100+ migrations, the history is a mess, there is no easy way of tracking history on a single table and it adds clutter to your file finding. Simply Muda IMO :)
Here's a 3-step guide to squash all migrations into identical schema as production:
Step1: schema from production
# launch rails console in production
stream = StringIO.new
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, stream); nil
stream.rewind
puts stream.read
This is copy-pasteable to migrations, minus the obvious header
Step 2: making the migrations without it being run in production
This is important. Use the last migration and change it's name and content. ActiveRecord stors the datetime number in it's schema_migrations table so it knows what it has run and not. Reuse the last and it'll think it has already run.
Example: rename 20161202212203_this_is_the_last_migration -> 20161202212203_schema_of_20161203.rb
And put the schema there.
Step 3: verify and troubleshoot
Locally, rake db:drop, rake db:create, rake db:migrate
Verify that schema is identical. One issue we encountered was datetime "now()" in schema, here's the best solution I could find for that: https://stackoverflow.com/a/40840867/252799