We need to modify (such as add/remove table, add/remove/change column in current table) the mysql production database with the progress of development of the rails 3.1 app. What's the best way to do that? There is valuable data in production database and they should be retained after the modification.
Thanks.
Not sure what you have tried but Rails Database Migrations is what you are looking for.
The only way I believe if i understand your question is - to keep your data and schema copy handy and maintain them using some standard naming convention like dd-mm-yy-timestamp. this will allow you to take hot backup/cold backup. while in rails you can play with migrations to and from. but not keep data as well. if the data is a seed data then it can be returned and reversed.
The solution is other hand is more handy.you can write cron jobs and call shell scripts to make sure your operation is not manual and keeps the backup copy at safe location.
Related
Problem: I'm creating a Rails app, single dev, running staged/prod servers on Heroku, not publicly released yet. Reworking my DB infrastructure, since I've done several migrations since creating tables. I know it's somewhat trivial, but I'm trying to get things cleaned up before initial launch:
Redo indexes.
Reorder/rename fields. I would prefer avoiding tables with timestamp fields randomly sandwiched in the middle and PostgreSQL doesn't allow simple field reordering (for this reason, I may standardize timestamps as the first fields moving forward, so future migrations aren't so noticeable).
Possible Solution(s): I'll need to drop my schema and reload a clean copy of it. I can:
Edit schema.rb structure for existing tables to my liking.
(?) Manually edit the [VERSION] timestamp in schema.rb.
(?) Edit latest migration file, duplicate schema.rb.
Run rails db:schema:load-esque (likely with additional db:reset-esque steps to drop the existing schema/structure first).
Delete older migration files.
Question #1: See 2.-3. Aside from the elephant in the room that this method isn't generally recommended long-term, when does rails db:schema:dump have a use case?, since it's essentially what I'm doing by hand? I don't believe it would generate models tables not generated through Rails beforehand, so that could get messy (without running rails generate model --skip-migration). Does it create a new migration, or at minimum does it update the schema.rb timestamp so as not to look backwards at prior migrations? Otherwise, I would think :dump would be unconventional to Rails' own system.
Question #2: I know it will break staged/production servers once I push the changes (again, I'll have to run step 5. on them or just replace my Heroku apps with fresh copies). However, would this method also break these too, and/or break future Rails migration steps? I'd rather make sure whatever I build can be launched cleanly without requiring additional steps by hand that I could have avoided.
As someone who as ditched old, embarrassing migrations for a flat, updated schema part way through an app development process, I only ever regretted it.
Migrations only run on deployment, so there's no real speed reason to squash them, combine them, rename them, or remove them (unless you are totally ditching whatever is in that migration).
AND, ALL your migrations only run once on your first deploy. From then on, only future migrations will be run. So the overhead is a one-time thing.
Having a few (hundred) migration files is really no big deal.
"But what about the fact that migration #3 adds a column which is later removed by migration #45?" That's how we write software: over time, with changes. It's fine.
If you need to redo indexes and rename fields, write another migration. Call it "CleanupBeforeProductionDeploy" if it'll give you that wonderful sense of cleaned up code.
Reordering columns is useless. This isn't Excel. Don't bother.
If you need to display fields in a certain order, use SQL or .map or .pluck or I'm sure a dozen other Ruby or RoR solutions, that's what they are for.
You've done good work, it sounds like your app is nearly ready to deploy. Congratulations, that's a serious milestone. So many of us start scratching something out and never push it across the finish line. Seriously, you should feel good.
Don't procrastinate with meaningless fussing that will just lead to errors.
Go push the code and be happy.
Rant over. If I haven't convinced you, here are some tips to at least keep you safe / sane.
The purpose of the schema is a shortcut for creating a new dB from scratch and just jumping to the end without running every migration.
The schema dump is used, for example, by Heroku when you are creating a backup of your production database.
(Just an aside, I use parity to get production data from my apps into my development environments so I can work with "real" data).
You could also use a schema dump from the actual database to create a new schema file if something happened to your schema file.
So, you can do this, even though you shouldn't waste your time.
Here would be my order of operations:
Keep writing migrations for all these changes you want to make. Yes, write one or more migrations to make all your changes. Seriously. Let Rails handle versioning and timestamping your schema file as you go. And, this way you can do this in stages and test things out.
For changing your table column orders I'd do the following, it's messy, but it would work:
rename_table :users, :users_disorganized
create_table :users do |t|
...
end
# write some complicated SQL to copy 'users_disorganized' data into 'users'
# safety catch in case copying things over didn't work
drop_table :users_disorganized unless User.all.size.zero?
Use SQL to map and copy the contents of users_disorganized into users.
Here's a good SO post on the topic of SQL inside migrations
SQL is really your only choice here since you don't have an ApplicationRecord Model for UserDisorganized.
Could you make a UserDisorganized model just to make copying the files over easier? Yes, then you'd have to remember delete that file after your production deploy.
Starting to see how time-intensive this is going to be?
Now repeat this process for every table where you want to reorder columns.
Once you are all done writing migrations, your pristine schema now has the latest timestamp and version and everything so don't mess with these values.
(you can change them, it'll probably be fine, but if you decide to set them too far in the future... then forget about doing that... then write a small migration just to patch a bug or add a feature... then try to run that migration... then spend 4 hours trying to figure out why nothing happens when you run rake db:migrate... all just to realize the schema timestamp is later than your migration timestamp... and so on)
Now you can gulp delete all your migrations. Yeah. You don't want them? Here's your chance to prove it. You still serious about this?
How do you then initialize your production database? Tell Heroku to run rake db:schema:load instead of rake db:migrate.
Good luck.
(and don't bother)
I have a very high-traffic Rails app. We use an older version of PostgreSQL as the backend database which we need to upgrade. We cannot use either the data-directory copy method because the formats of data files have changed too much between our existing releases and the current PostgreSQL release (10.x at the time of writing). We also cannot use the dump-restore processes for migration because we would either incur downtime of several hours or lose important customer data. Replication would not be possible as the two DB versions are incompatible for that.
The strategy so far is to have two databases and copy all the data (and functions) from existing to a new installation. However, while the copy is happening, we need data arriving at the backend to reach both servers so that once the data migration is complete, the switch becomes a matter of redeploying the code.
I have figured out the other parts of the puzzle but am unable to determine how to send all writes happening on the Rails app to both DB servers.
I am not bothered if both installations get queried for displaying data to the user (I can discard the data coming out of the new installation); so, if it is possible on driver level, or adding a line somewhere in the ActiveRecord, I am fine with it.
PS: Rails version is 4.1 and the company is not planning to upgrade that.
you can have multiple database by adding an env for the database.yml file. After that you can have a seperate class Like ActiveRecordBase and connect that to the new env.
have a look at this post
However, as I can see, that will not solve your problem. Redirecting new data to the new DB while copying from the old one can lead to data inconsistencies.
For and example, ID of a record can be changed due to two data source feeds.
If you are upgrading the DB, I would recommend define a schedule downtime and let your users know in advance. I would say, having a small downtime is far better than fixing inconstant data down the line.
When you have a downtime,
Let the customers know well in advance
Keep the downtime minimal
Have a backup procedure, in an even the new site takes longer than you think, rollback to the old site.
I'm using MongoDB with mongo_mapper gem in Rails and project is big enough. Is there any way I can migrate data from Mongoid to Postgresql?
You should look into some of the automated migration/replication tools like MoSQL: https://stripe.com/blog/announcing-mosql
One risky strategy for this migration would be to convert your codebase to using postgres and all of your models, put your site into maintenance mode, migrate your databases, deploy the new code, and bring it back up. However, this requires significant downtime and development risk of bugs or data loss.
A safer, but much more involved strategy would be to setup automatic migration of data to a new database to sync up your databases. Then, every action in the application writes to both databases. After each transaction, you verify the data is in sync between both databases and read from Mongo. This allows you to fix errors as you find them and highlight any inconsistencies. Once you are no longer finding discrepancies, you can turn off writing to mongo and retire that database, remove the mongo models/code, and move on.
I have a simple rails-backed app running 2-3 million pageviews a day off a Heroku Ronin database. The load on the database is pretty light, though, and it could handle a lot more than we're throwing at it.
Is it safe for me to run a migration to add tables to this database without going into maintenance mode? Also, would it be safe to run a migration to add a few columns to the core table responsible for almost all of the reads and writes?
Downtime is not acceptable, even for a few minutes.
If running migrations live isn't advisable, what I'll probably do is set up a new database, run the migrations on that, write a script to sync the two databases, and then point the app at the new one.
But I'd rather avoid that if possible. :)
Sounds like your migration includes:
adding new tables (perhaps indexes? If so, that could take a bit longer than you might expect)
adding new columns (default values and/or nullable?)
wrapping your changes in a transaction (?)
Suggest you gauge the impact that your changes will have on your Prod environment by:
taking a backup of Prod (with all the Prod data within)
running your change scripts against that. Time each operation
Balance the 2 points above against the typical read & write load at the time you're expecting to run this (02:00, right?).
Consider a 'soft' downtime by disabling (somehow) write operations to the tables being effected.
Overall (or in general), adding n tables and new nullable columns to an existing table would/could likely be done without any downtime or performance impact.
Always measure the impact your changes will have on a copy of Prod. Measure 'responsiveness' at the time you apply your changes to this copy. Of course this means deploying another copy of your Prod app as well, but the effort would be worthwhile.
Assuming it's a pg database (which it should be for Heroku).
http://www.postgresql.org/docs/current/static/explicit-locking.html
alter table will acquire an access exclusive lock. So, the table will be locked.
On top of this, you will be required to restart the Rails application in order for it to be aware of any new models. If you are going to be adding tables to the application or modifying model code in any way.
As for pointing to a new app with a freshly modified database, how are you going to do the sync of the data and also sync the changes in data between the two databases in the time that the sync takes?
Adding tables shouldn't be a concern, as your application won't be aware of them until proper upgrades are done. As for adding columns to a core table, I'm not so sure. If you really need to prevent downtime, perhaps it's better to add a secondary table that (linked by an ID with the core table) adds your extra columns.
Just my two cents.
I'm fairly new to both Rails and Heroku but I'm seriously thinking of using it as a platform to deploy my Ruby/Rails applications.
I want to use all the power of Heroku, so I prefer the "embedded" PostgreSQL managed by Heroku instead of the addon for Amazon RDS for MySQL, but I'm not so confident without the possibility to access my data in a SQL client...
I know that in a well made app you have no need to access DB, but there are some situations (add rows to a config table, see data not mapped in a view, update some columns for debugging issues, performance monitoring, running queries for reporting, etc.) when this can be good...
How do you solve this problem? What's you experience in a real life app powered by Heroku?
Thanks!
I have been using it for a about a year. I love the workflow that it provides but I find not having access to the data is a real bother. Your options for working with database are:
Taps: In theory you create your database however you want locally and use taps to copy both schema and data to Heroku. In practice, most of the time its amazingly great. However I am currently dealing with the cleanup after taps translated some of my columns poorly and corrupted my data.
Heroku console: Totally fine for all the usual ActiveRecord stuff, but closest you can get to the database is ActiveRecord::Base.connection.execute "some sql". When you find yourself wondering about doing alter table commands like that you will know you're in trouble.
They also provide a "bundle" as a method for backing up your app. This lets you download all your code plus a sql dump of the database. The difficulty is that since there is no direct database access there is no way of loading that same sql dump back into the database so you can recover from dataloss, which, to me, is the point of having those dump files to begin with. All you can use the bundle for is to create a new application (heroku bundles:animate), not restore a current app.
I would love to be wrong about any/all of these. This seems like a curious rough spot in the best thought out service that I know of. Database access normally doesn't feel like much to give up when most of what you do is made so easy.
To me database access is like a fire extinguisher. Usually not a big deal, but when it matters, it matters a lot.
I use admin_data to give me some insight as to what is going on. I've successfully used it on Heroku, as well as other hosting providers.
Firstly let me start off by saying that heroku is awesome. I've had a great experience deploying my application and integrating with their other services such as websolr.
With that said, your questions:
Getting at your data
If you want to be able to get to your data you can use taps to pull your remote database down locally. This can be useful for debugging.
Performance monitoring
Use new relic RPM. This comes as part of heroku, you can enable it from the add-ons menu.
Add-hoc database queries
You could write a controller which allows you to execute arbitrary sql and view the results, but this isn't something I'd recommend. As suggest admin_data is a good solution for managing your data, but if you want to do anything more complicated you'll have to resort to writing the code yourself.