QUESTION:
Is there any way I can avoid running current migrations on a remote database already configured correctly, while ALSO being allowed to apply future migrations to it?
Context:
I have an existing rails app with plenty of migrations, that, up until now, has been using a local postgres database. A remote postgres database which SHOULD mostly match up to the structure of my local database exists.
When I try to connect to it, I get a "pending migrations" error. Attempting to run my migrations gets errors about tables already existing.
I want to skip all current migrations, but UNLIKE a lot of similar questions I'm seeing, I want to make sure my future migrations will work on this remote database.
Edit:
I followed this answer:
Rails 4 how to ignore pending migrations
And inserted my own current scheme version number into the remove database's schema_migrations table. But the pending migration remains.
I can confirm trying to run the migration gets me an error of a table already existing. This table is the FIRST migration past the remote databases previously most recent version in the schema table. It appears to be ignoring my inserted current most recent version.
You have to add all migration version numbers that you want to ignore to schema_migrations
Other technique that may be also applicable - migration squashing with something like squasher or manually. Point is in combining all old migrations that are not to be rolled back into one.
Related
my issue appears to pertain to mainly docker.
I got frustrated with a table that I had so I deleted it plus all associated migrations from my rails migrations.
I did make a zip file of repository before I did this so that I wouldn't lose all my work, I'm not sure if this can be affecting anything but I thought I would put it out there just in case because all of those files do technically exist somewhere else.
When I do rails db:migrate:status those migrations no longer appear on the list, but they still exist in schema.rb
What I have tried:
I deleted my branch that the original migration is in and started in a new one.
I tried just deleting it from schema.rb because it didn't exist anymore, but when I tried to create a table with the same name (because I need the table to be named that) it says the migration failed because that table already exists.
So then I restarted my docker container with docker restart <container name*> and tried running the migration again... same error.
Next I rebuilt my database, same error.
After that failed I deleted the docker container and rebuilt it from scratch.. same error.
I then deleted the repository from my computer, deleted the docker image, shut down and restarted my computer. Re-cloned the repository, rebuilt the container and still the same error.
It just keeps reappearing in the schema.
Sure I could create a new table with a different name, but I don't want the schema for a table that doesn't exist.
What should my next move be?
If I understood correctly, you had run migrations to update your schema.rb file, then you removed (dropped) a database table, maybe using your database software directly, and then you deleted the related migration files.
I don't think this has anything to do with Docker but everything with how Rails works with databases.
First, Rails will not know if you remove tables or make any other changes to your database directly using your database software and SQL (DDL), so the schema file is not updated based on it.
Second, Rails will not know about removing any migration files from the disk, and the schema file is not updated based on your changes.
You will always have to use migrations to keep your database structure up-to-date.
Your options
All of the below options only operate on your local database.
Option 1: Resetting and recreating your local database using Rails
Assumptions: you can delete your local database
You could try resetting your database with
bin/rails db:reset
that basically runs bin/rails db:drop db:setup that is described as:
The bin/rails db:setup command will create the database, load the schema, and initialize it with the seed data.
This should recreate the database structure using your schema.rb and it should bring you back the frustrating table.
Option 2: Resetting and recreating your local database by restoring it from a backup
Assumptions: you have a backup, you can delete your local database
You could try restoring or importing the latest backup using your database software. Most likely, you would need to drop the database, first, but it depends on the tools used and their options available.
This should result in the structure you have in your schema.rb and it should give you back the frustrating table with any data you had in the backup snapshot.
Option 3: Recreating only the missing table in your local database by importing it from a backup
Assumptions: you have a backup, you do not want to delete your local database
You could try restoring or importing only the missing table using your database software.
This should result in the structure you have in your schema.rb and it should give you back the frustrating table. It will also import any data for that table you had in your backup snapshot.
Depending on your table definition(s), you may need to update something in the data, though, especially if other tables reference this one or this one has references to others. The data you imported might be wrong so you may want to set some references to NULL, for example, or fix them, whichever suits you better.
Option 4: Recreating only the missing table in your local database manually
Assumptions: you do not have a backup, you do not want to delete your local database
You could try recreating just the frustrating table manually without any data.
You could do this based on your schema.rb by translating it to SQL DDL yourself. A faster approach would be to ask one of your team members or colleagues, if any, who has an up-to-date version of the database structure to share you the SQL DDL for that table so that you could run it locally using your database software, for example.
This would result in the structure you have in your schema.rb and it should give you back the frustrating table.
Next steps: Finally, no matter which option you decided to implement to restore your database structure, you should create a new migration in which you call drop_table to remove (drop) the frustrating table correctly. That will remove the table definition from the schema.rb as well so that you can later create a new migration to recreate the table with the same name.
I am trying to deploy a new release of a project to my staging server, but the schema_migrations table in the database is inexplicably empty.
It is now trying to run all migrations while deploying, causing issues since the other tables exist and are intact.
Instead of dropping/recreating the database and losing all my data(though inconvenient, a valid option), is it possible to generate the schema_migrations table without dropping?
Two points:
There's nothing wrong with deleting migrations that aren't needed anymore, you're not supposed to keep them around forever.
You can manually update the schema_migrations table if you have to.
As far as (1) goes, I delete migrations (provided they've been applied everywhere) that are old than a month or so. Keeping around years of clutter in db/migrate is pointless. Migrating down and then back up tends to be more of a mess than it is worth for all but the most recent migrations; if you need to test something old, dump your current development database and start with a fresh one based on an older schema.rb. Besides, if you really need an old migration then you can always dig it out of revision control or simply recreate it based on diffs between versions of db/schema.rb (or db/structure.sql).
Of course, if you don't keep db/schema.rb in revision control then you're doing it wrong and you a looming disaster on your hands.
For (2), the schema_migrations table is very simple:
create table schema_migrations (
version varchar not null primary key
)
So you can connect to PostgreSQL using psql, create the table by hand, and then do a bunch of insert into schema_migrations (version) values (...) to cover the historic migrations that should have been run in your staging environment, and then db:migrate as normal to get things up to date. Then you can spend as much time as you want doing a post mortem to figure out what happened to your staging database's schema_migrations table.
If you go this route, you might want to have a look at the ar_internal_metadata table if your version of Rails uses it.
You will, of course, backup your staging database before doing any of this. And you will want to examine your production database to make sure it isn't suffering the same problems.
Sometimes you have to get your hands dirty to make things go.
Since doing migrations with rails + git is type of a pain, a new thorn has sprung..
Before I am doing any harm to my prod DB, would the following situation cause havoc? If so, how would I handle it?
I am working a long-term feature in a separate branch (feature/long-term). This feature is an overhaul of a lot of components and it will take awhile to complete. This feature has new migrations, which were migrated to the localhost DB.
meanwhile, I need to fix/add a migration to the prod system via another branch (feature/quick-fix). This has a migration file with date later than the feature/long-term migration.
The migrations of the quick-fix and the long-term have nothing to do with each other, they do not collide and work on separate tables. It doesn't matter what order they are run.
If I merge feature/quick-fix to master and db:migrate and in a few days/weeks merge feature/long-term the migration files order would be the long-term first.
Would this affect the DB in some way? (the prod DB is important, so I don't want to reset)
What you described is a very common development workflow (especially so in teams with more members) and it's perfectly safe for your production DB.
Rails, as of version 2.1, is smart enough to keep a list of all migrations ever run, instead of just the latest migration version run. This information is stored on a separate table aptly named schema_migrations.
So, if you push a new migration today, say 20140527_quick_fix.rb, and a month after that you push a new (but with an older timestamp) one 20140101_long_term_feature.rb, Rails will still know that the latter was never run in your production environment so during rake db:migrate it will process it, as you would expect. The newest won't be run again of course as the Rails would know that it has already been processed.
From the official documentation:
Rails versions 2.0 and prior used to create a table called schema_info when using migrations. This table contained the version of the schema as of the last applied migration.
Starting with Rails 2.1, the schema_info table is (automatically) replaced by the schema_migrations table, which contains the version numbers of all the migrations applied.
As a result, it is now possible to add migration files that are numbered lower than the current schema version: when migrating up, those never-applied “interleaved” migrations will be automatically applied, and when migrating down, never-applied “interleaved” migrations will be skipped.
I restored my database from the latest dump and tried to run rake tests. Unfortunately 30 migrations were pending. My first idea was to comment out each of 30 migrations code and run 'rake db:migrate' but there must be a simpler solution. I use Rails 2.3.14 and Postgresql 9.1.3.
If you're restoring a database from a dump, the schema_migrations table should restore along with the rest of the tables.
This seems to indicate your schema_migrations table may not be getting backed up which would lead to the problem you have now.
The ideal solution would be to restore a backup that has all the tables in it correctly -- including schema_migrations.
Even if you decide to find a way around this in the short-term, in the long-term the correct solution is to modify your backup scripts to get all the tables you need, including schema_migrations.
In terms of what to do now, the ideal solution is probaby to backup just that one table (schema_migrations) from your database and import that data into the database you're trying to load now. Then your migrations should no longer be pending.
Doing that with a simple table dump and load script should be fine. The simple postgres gui PgAdmin ( http://www.pgadmin.org/ ) may also provide some basic tools for dumping then loading a single table.
Kevin is correct. However, he is missing a critical point.
When you restore from a backup it restores the schema_migrations table which tracks which migrations need to be run. If those thirty migrations had been run on the database you restored from, they would not have run.
However, your code is thirty migrations ahead of the snapshot of your database represented by the backup.
This can happen to me if I deploy, then grab the production backup right away. Although the migrations have run on production, I'm getting the backup from before office hours prior to my deployment. I usually like to wait a day and get the next day's backup.
Or, don't worry about it. Your backup is before those thirty migrations, but then they were applied, so the migrations have made sure your schema matches the version of your code. That's a good thing.
Don't sweat it, and refresh again tomorrow, when the backup has your changes.
You could also manually add the timestamps of the missing migrations in the db table like:
INSERT INTO "public"."schema_migrations"("version") VALUES ('20201212012345')
That should have the same effect as temporarily outcommenting the 'create' instructions in the migration files. If you run migrations as part of a deploy process from git, commenting out would mean you had to push those changes to git.
If you just work on staging / development env, directly fixing the db might be nicer than pushing those changes, possibly confusing other deploys or devs.
In section Rails Database Migrations of Ruby on Rails Guides, there is one line saying that
The db:reset task will drop the database, recreate it and load the current
schema into it. This is not the same as running all the migrations.
Can anyone tell me where exactly they are different and why it is more error prone to replay the migration history?
I'm fairly new to Ruby on Rails. Thanks in advance.
The schema file contains the current structure of your database. When you load it, you are guaranteed to have the exact schema in your db that is in the file. Migrations were designed to make incremental changes in the database. You may add a table, then some columns, and then remove the table in three separate migrations. There's no need to go through all this when the schema already knows that the table no longer exists.
On why they are error prone, I'm not totally sure. The one thing I can think of is that migrations can be used to make changes to data and not just the structure.
Running rake db:reset will rebuild the structure of your database from schema.db, which essentially works as a cached version of your migrated database structure. Running all your migrations, on the other hand, applies the migrations one by one, which may include arbitrary code to accomodate for changes to the database (e.g. prepopulate an added counter cache column).
It can be more error prone to replay the migration history, since it is the product of changes to both the structure and data of the database. If the developers haven't been careful, it might not apply cleanly to a fresh environment (e.g. the migration assumes an old version of a model). On the other hand, schema.db can get out of sync if you edit a migration once you've migrated (a useful trick to avoid migration explosion during development). In that case, you need to run rake db:migrate:reset.