Looking for advice on developing a Rails app using Postgres on multiple machines. Default sqlite3 has your development database in the app folder. I leave my app folders in dropbox, so whether I'm using my desktop at work or my laptop at home, code and development databases are synced with sqlite. Further, there are no database "users" in sqlite so that's another layer I don't need to worry about.
With Postgres I always have problems; the database doesn't exist, the user doesn't exist, etc. Postgres stores its files separate from the app, which makes it hard. So, how do you develop with Postgres across multiple machines and keep everything in sync?
Use pg_dump to dump your database to a file, then transfer that file to the other machine, and use pg_restore to load it into your other database. Either that or don't depend on having identical data in your various development environments; your migrations should take care of keeping the database structure synchronized and the should patch up existing data as needed.
Related
We have infrastructure which contains about 30 host for 10 microservices and 4 hosts for databases ( SQL (PostgreSQL) and NoSQL (Cassandra, Riak, Neo4j), replicated and sharded ). Periodically we need to modify DB-structure (add some tables, fields and trigers).
For some time we was using python scripts for making migrations/patches to DBs (and propogate these changes to isolated develop/analytic clusters). Next, we moved to Rails migrations. Rails is the better way (a little higher-level) for our migrations (in opposite to python scripts), but in same cases (DB specific patches, like creating postgres trigers, scharding) it requires sql commands (no-DSL).
Our purpose is to go to high-level approach for migrations (generating), like IDE (PgAdmin, DataGrip or some similar), which could:
generate migrations/patches for DBs (which could be applied to DB cluster).
the generated migrations/patches must be sequenced (like in Rails).
it would be good to have the approach for SQL and NoSQL migrations/patches.
And after generating these migrations/paches we need to apply it (from commad line, CLI) on the cluster side.
There is also very specific actions for making virtual connections between some table in NoSQL and Relational DB.
So, is there any solutions for deploying such migration-ecosystem?
How do you solve such problem in you projects?
I have a very high-traffic Rails app. We use an older version of PostgreSQL as the backend database which we need to upgrade. We cannot use either the data-directory copy method because the formats of data files have changed too much between our existing releases and the current PostgreSQL release (10.x at the time of writing). We also cannot use the dump-restore processes for migration because we would either incur downtime of several hours or lose important customer data. Replication would not be possible as the two DB versions are incompatible for that.
The strategy so far is to have two databases and copy all the data (and functions) from existing to a new installation. However, while the copy is happening, we need data arriving at the backend to reach both servers so that once the data migration is complete, the switch becomes a matter of redeploying the code.
I have figured out the other parts of the puzzle but am unable to determine how to send all writes happening on the Rails app to both DB servers.
I am not bothered if both installations get queried for displaying data to the user (I can discard the data coming out of the new installation); so, if it is possible on driver level, or adding a line somewhere in the ActiveRecord, I am fine with it.
PS: Rails version is 4.1 and the company is not planning to upgrade that.
you can have multiple database by adding an env for the database.yml file. After that you can have a seperate class Like ActiveRecordBase and connect that to the new env.
have a look at this post
However, as I can see, that will not solve your problem. Redirecting new data to the new DB while copying from the old one can lead to data inconsistencies.
For and example, ID of a record can be changed due to two data source feeds.
If you are upgrading the DB, I would recommend define a schedule downtime and let your users know in advance. I would say, having a small downtime is far better than fixing inconstant data down the line.
When you have a downtime,
Let the customers know well in advance
Keep the downtime minimal
Have a backup procedure, in an even the new site takes longer than you think, rollback to the old site.
I'm using MongoDB with mongo_mapper gem in Rails and project is big enough. Is there any way I can migrate data from Mongoid to Postgresql?
You should look into some of the automated migration/replication tools like MoSQL: https://stripe.com/blog/announcing-mosql
One risky strategy for this migration would be to convert your codebase to using postgres and all of your models, put your site into maintenance mode, migrate your databases, deploy the new code, and bring it back up. However, this requires significant downtime and development risk of bugs or data loss.
A safer, but much more involved strategy would be to setup automatic migration of data to a new database to sync up your databases. Then, every action in the application writes to both databases. After each transaction, you verify the data is in sync between both databases and read from Mongo. This allows you to fix errors as you find them and highlight any inconsistencies. Once you are no longer finding discrepancies, you can turn off writing to mongo and retire that database, remove the mongo models/code, and move on.
I am working on a team of 4 developers building a Rails application deployed on Heroku. We are currently using Sqlite on our development machines and Postgres for staging (on Heroku). Because of how a certain part of our application works it is necessary for all developers to have the same data while developing.
In our staging application we create these records that are then pushed up to a test Solr cluster. The problem then comes when we are developing locally we hit the same test cluster and the IDs that exist there don't match the IDs of the records in our local Sqlite tables and we need to be able to tie a Solr result back to a table record.
One thought we had was to convert over to Postgres for development (which we should have done already anyway), export data from Heroku and import into our local DB tables. While I believe this approach will work just fine. I was wondering if anyone else has encountered a similar problem and what approach they took to solve it.
Thanks for any and all help.
I actually just encountered this issue last night; I did actually convert my local database to be postgres as well. Specifically with Heroku, using PG Backups and importing into my local copy is almost trivial. This was my first time using PostGres, so coming from a new experience, I couldn't imagine it being any easier. My co-worker and I have now can use identical data which, like yours, is a requirement.
I am deploying my first ROR app with MongoDB. I've noticed that if the machine that I have it deployed on is reset then usually the database will be partially or fully gone. This was also happening before I had it explicitly stated in my Rails app that I was using MongoDB without active record, although it worked before I did.
I looked in the /data/db/ folder and there appears to be numerous database files: app-production.1, app-production.2, etc. In my mongo_config.rb initializer file I have the database specified as
MongoMapper.database = "app-#{Rails.env}"
I also launched the interative mongo console and noticed that there were 5+ app databases. So my question is, does MongoDB or Rails automatically create a new instance of the database when it shuts down? Does MongoDB use multiple files to span the database? Thanks!
MongoDB database is spanned across multiple files (app-production.1, app-production.2, ...). Each new database resides in its own set of files.
MongoDB offers lazy database creation. If you write to a database and it doesn't exist, it will be created. If a database with this name exists, it will be used.
if the machine is reset then database will be partially or fully gone
It depends on your definition of "reset". If you unplug power cord, then mongodb won't have the time to shutdown properly. Recent versions ship with journaling turned on by default, so no major data loss should occur - in case of hard crash, mongodb will try to replay journal. Note: if you're not running with journaling enabled, you should repair database after any unclean shutdown, or risk facing imminent database corruption.
If you reinstall the machine instead, then it's no wonder that data is gone :)