I am deploying my first ROR app with MongoDB. I've noticed that if the machine that I have it deployed on is reset then usually the database will be partially or fully gone. This was also happening before I had it explicitly stated in my Rails app that I was using MongoDB without active record, although it worked before I did.
I looked in the /data/db/ folder and there appears to be numerous database files: app-production.1, app-production.2, etc. In my mongo_config.rb initializer file I have the database specified as
MongoMapper.database = "app-#{Rails.env}"
I also launched the interative mongo console and noticed that there were 5+ app databases. So my question is, does MongoDB or Rails automatically create a new instance of the database when it shuts down? Does MongoDB use multiple files to span the database? Thanks!
MongoDB database is spanned across multiple files (app-production.1, app-production.2, ...). Each new database resides in its own set of files.
MongoDB offers lazy database creation. If you write to a database and it doesn't exist, it will be created. If a database with this name exists, it will be used.
if the machine is reset then database will be partially or fully gone
It depends on your definition of "reset". If you unplug power cord, then mongodb won't have the time to shutdown properly. Recent versions ship with journaling turned on by default, so no major data loss should occur - in case of hard crash, mongodb will try to replay journal. Note: if you're not running with journaling enabled, you should repair database after any unclean shutdown, or risk facing imminent database corruption.
If you reinstall the machine instead, then it's no wonder that data is gone :)
Related
I have a very high-traffic Rails app. We use an older version of PostgreSQL as the backend database which we need to upgrade. We cannot use either the data-directory copy method because the formats of data files have changed too much between our existing releases and the current PostgreSQL release (10.x at the time of writing). We also cannot use the dump-restore processes for migration because we would either incur downtime of several hours or lose important customer data. Replication would not be possible as the two DB versions are incompatible for that.
The strategy so far is to have two databases and copy all the data (and functions) from existing to a new installation. However, while the copy is happening, we need data arriving at the backend to reach both servers so that once the data migration is complete, the switch becomes a matter of redeploying the code.
I have figured out the other parts of the puzzle but am unable to determine how to send all writes happening on the Rails app to both DB servers.
I am not bothered if both installations get queried for displaying data to the user (I can discard the data coming out of the new installation); so, if it is possible on driver level, or adding a line somewhere in the ActiveRecord, I am fine with it.
PS: Rails version is 4.1 and the company is not planning to upgrade that.
you can have multiple database by adding an env for the database.yml file. After that you can have a seperate class Like ActiveRecordBase and connect that to the new env.
have a look at this post
However, as I can see, that will not solve your problem. Redirecting new data to the new DB while copying from the old one can lead to data inconsistencies.
For and example, ID of a record can be changed due to two data source feeds.
If you are upgrading the DB, I would recommend define a schedule downtime and let your users know in advance. I would say, having a small downtime is far better than fixing inconstant data down the line.
When you have a downtime,
Let the customers know well in advance
Keep the downtime minimal
Have a backup procedure, in an even the new site takes longer than you think, rollback to the old site.
Looking for advice on developing a Rails app using Postgres on multiple machines. Default sqlite3 has your development database in the app folder. I leave my app folders in dropbox, so whether I'm using my desktop at work or my laptop at home, code and development databases are synced with sqlite. Further, there are no database "users" in sqlite so that's another layer I don't need to worry about.
With Postgres I always have problems; the database doesn't exist, the user doesn't exist, etc. Postgres stores its files separate from the app, which makes it hard. So, how do you develop with Postgres across multiple machines and keep everything in sync?
Use pg_dump to dump your database to a file, then transfer that file to the other machine, and use pg_restore to load it into your other database. Either that or don't depend on having identical data in your various development environments; your migrations should take care of keeping the database structure synchronized and the should patch up existing data as needed.
I'm using MongoDB with mongo_mapper gem in Rails and project is big enough. Is there any way I can migrate data from Mongoid to Postgresql?
You should look into some of the automated migration/replication tools like MoSQL: https://stripe.com/blog/announcing-mosql
One risky strategy for this migration would be to convert your codebase to using postgres and all of your models, put your site into maintenance mode, migrate your databases, deploy the new code, and bring it back up. However, this requires significant downtime and development risk of bugs or data loss.
A safer, but much more involved strategy would be to setup automatic migration of data to a new database to sync up your databases. Then, every action in the application writes to both databases. After each transaction, you verify the data is in sync between both databases and read from Mongo. This allows you to fix errors as you find them and highlight any inconsistencies. Once you are no longer finding discrepancies, you can turn off writing to mongo and retire that database, remove the mongo models/code, and move on.
I got the above error message running Heroku Postgres Basic (as per this question) and have been trying to diagnose the problem.
One of the suggestions is to use connection pooling but it seems Rails has this built in. Another suggestion is that the app is configured improperly and opens too many connections.
My app manages all it's connections through Active Record, and I had one direct connection to the database from Navicat (or at least I thought I had).
How would I debug this?
RESOLUTION
Turns out it was an Heroku issue. From Heroku support:
We've detected an issue on the server running your Basic database.
While we pinpoint this and address it, we would recommend you
provision a new Basic database and migrate over with PGBackups as
detailed here:
https://devcenter.heroku.com/articles/upgrade-heroku-postgres-with-pgbackups
. That should put your database on a new server. I apologize for this
disruption – we're working to fix this issue and prevent it from
occurring in the future.
This has happened a few times on my app -- somehow there is a connection leak, then all of a sudden the database is getting 10 times as many connections as it should. If it is the case that you are getting swamped by an error like this, not traffic, try running this:
heroku pg:killall
That will terminate all connections to the database. If it is dangerous for your situation to possibly cut off queries be careful. I just have a rails app, and if it goes down, losing a couple queries is not a big deal, because the browser requests will have looooooong since timed out anyway.
You might be able to find why you have so many connections by inspecting view pg_stat_activity:
SELECT * FROM pg_stat_activity
Most likely, you have some stray loop that opens new connection(s) without closing it.
To save you the support call, here's the response I got from Heroku Support for a similar issue:
Hello,
One of the limitations of the hobby tier databases is unannounced maintenance. Many hobby databases run on a single shared server, and we will occasionally need to restart that server for hardware maintenance purposes, or migrate databases to another server for load balancing. When that happens, you'll see an error in your logs or have problems connecting. If the server is restarting, it might take 15 minutes or more for the database to come back online.
Most apps that maintain a connection pool (like ActiveRecord in Rails) can just open a new connection to the database. However, in some cases an app won't be able to reconnect. If that happens, you can heroku restart your app to bring it back online.
This is one of the reasons we recommend against running hobby databases for critical production applications. Standard and Premium databases include notifications for downtime events, and are much more performant and stable in general. You can use pg:copy to migrate to a standard or premium plan.
If this continues, you can try provisioning a new database (on a different server) with heroku addons:add, then use pg:copy to move the data. Keep in mind that hobby tier rules apply to the $9 basic plan as well as the free database.
Thanks,
Bradley
I am working on a team of 4 developers building a Rails application deployed on Heroku. We are currently using Sqlite on our development machines and Postgres for staging (on Heroku). Because of how a certain part of our application works it is necessary for all developers to have the same data while developing.
In our staging application we create these records that are then pushed up to a test Solr cluster. The problem then comes when we are developing locally we hit the same test cluster and the IDs that exist there don't match the IDs of the records in our local Sqlite tables and we need to be able to tie a Solr result back to a table record.
One thought we had was to convert over to Postgres for development (which we should have done already anyway), export data from Heroku and import into our local DB tables. While I believe this approach will work just fine. I was wondering if anyone else has encountered a similar problem and what approach they took to solve it.
Thanks for any and all help.
I actually just encountered this issue last night; I did actually convert my local database to be postgres as well. Specifically with Heroku, using PG Backups and importing into my local copy is almost trivial. This was my first time using PostGres, so coming from a new experience, I couldn't imagine it being any easier. My co-worker and I have now can use identical data which, like yours, is a requirement.