I am working on a team of 4 developers building a Rails application deployed on Heroku. We are currently using Sqlite on our development machines and Postgres for staging (on Heroku). Because of how a certain part of our application works it is necessary for all developers to have the same data while developing.
In our staging application we create these records that are then pushed up to a test Solr cluster. The problem then comes when we are developing locally we hit the same test cluster and the IDs that exist there don't match the IDs of the records in our local Sqlite tables and we need to be able to tie a Solr result back to a table record.
One thought we had was to convert over to Postgres for development (which we should have done already anyway), export data from Heroku and import into our local DB tables. While I believe this approach will work just fine. I was wondering if anyone else has encountered a similar problem and what approach they took to solve it.
Thanks for any and all help.
I actually just encountered this issue last night; I did actually convert my local database to be postgres as well. Specifically with Heroku, using PG Backups and importing into my local copy is almost trivial. This was my first time using PostGres, so coming from a new experience, I couldn't imagine it being any easier. My co-worker and I have now can use identical data which, like yours, is a requirement.
Related
I have a very high-traffic Rails app. We use an older version of PostgreSQL as the backend database which we need to upgrade. We cannot use either the data-directory copy method because the formats of data files have changed too much between our existing releases and the current PostgreSQL release (10.x at the time of writing). We also cannot use the dump-restore processes for migration because we would either incur downtime of several hours or lose important customer data. Replication would not be possible as the two DB versions are incompatible for that.
The strategy so far is to have two databases and copy all the data (and functions) from existing to a new installation. However, while the copy is happening, we need data arriving at the backend to reach both servers so that once the data migration is complete, the switch becomes a matter of redeploying the code.
I have figured out the other parts of the puzzle but am unable to determine how to send all writes happening on the Rails app to both DB servers.
I am not bothered if both installations get queried for displaying data to the user (I can discard the data coming out of the new installation); so, if it is possible on driver level, or adding a line somewhere in the ActiveRecord, I am fine with it.
PS: Rails version is 4.1 and the company is not planning to upgrade that.
you can have multiple database by adding an env for the database.yml file. After that you can have a seperate class Like ActiveRecordBase and connect that to the new env.
have a look at this post
However, as I can see, that will not solve your problem. Redirecting new data to the new DB while copying from the old one can lead to data inconsistencies.
For and example, ID of a record can be changed due to two data source feeds.
If you are upgrading the DB, I would recommend define a schedule downtime and let your users know in advance. I would say, having a small downtime is far better than fixing inconstant data down the line.
When you have a downtime,
Let the customers know well in advance
Keep the downtime minimal
Have a backup procedure, in an even the new site takes longer than you think, rollback to the old site.
I'm using MongoDB with mongo_mapper gem in Rails and project is big enough. Is there any way I can migrate data from Mongoid to Postgresql?
You should look into some of the automated migration/replication tools like MoSQL: https://stripe.com/blog/announcing-mosql
One risky strategy for this migration would be to convert your codebase to using postgres and all of your models, put your site into maintenance mode, migrate your databases, deploy the new code, and bring it back up. However, this requires significant downtime and development risk of bugs or data loss.
A safer, but much more involved strategy would be to setup automatic migration of data to a new database to sync up your databases. Then, every action in the application writes to both databases. After each transaction, you verify the data is in sync between both databases and read from Mongo. This allows you to fix errors as you find them and highlight any inconsistencies. Once you are no longer finding discrepancies, you can turn off writing to mongo and retire that database, remove the mongo models/code, and move on.
I am deploying my first ROR app with MongoDB. I've noticed that if the machine that I have it deployed on is reset then usually the database will be partially or fully gone. This was also happening before I had it explicitly stated in my Rails app that I was using MongoDB without active record, although it worked before I did.
I looked in the /data/db/ folder and there appears to be numerous database files: app-production.1, app-production.2, etc. In my mongo_config.rb initializer file I have the database specified as
MongoMapper.database = "app-#{Rails.env}"
I also launched the interative mongo console and noticed that there were 5+ app databases. So my question is, does MongoDB or Rails automatically create a new instance of the database when it shuts down? Does MongoDB use multiple files to span the database? Thanks!
MongoDB database is spanned across multiple files (app-production.1, app-production.2, ...). Each new database resides in its own set of files.
MongoDB offers lazy database creation. If you write to a database and it doesn't exist, it will be created. If a database with this name exists, it will be used.
if the machine is reset then database will be partially or fully gone
It depends on your definition of "reset". If you unplug power cord, then mongodb won't have the time to shutdown properly. Recent versions ship with journaling turned on by default, so no major data loss should occur - in case of hard crash, mongodb will try to replay journal. Note: if you're not running with journaling enabled, you should repair database after any unclean shutdown, or risk facing imminent database corruption.
If you reinstall the machine instead, then it's no wonder that data is gone :)
It is my first time working with a Rails app on Heroku. It uses a mysql db hosted on Amazon RDS. At some point I want to move it from development to production. I want to keep developing and adding features. What is the best way to accomplish this?
I see Heroku has some kind of staging app feature. Is that the best option for me to keep a separate app to test? And what about the database? I'm guessing I need to create a separate db on Amazon RDS for development and for production?
I am on a budget so I don't want to have to pay for 2 apps on Heroku and 2 db's on Amazon. Can I create both on the fly each time I do development work and then destroy them when I'm done, or is that too much? If so can I then copy the production data over to the development data? I would do local testing but I feel like I need to make sure it's working on Heroku as well.
I'm just trying to get a general idea of what workflow is best practice or most commonly used. Any comments are appreciated.
Unfortunately, as you're on RDS you're going to have to pay for two. If you were using Heroku Postgres you'd be able to get a simple small PG database for free.
Regarding applications - Heroku apps are free if you use less than 750 'dyno' hours a month (which is a little bit more than one dyno for an entire month), which is normally fine for staging small apps as long as you don't have masses of workers required.
You certainly could create the staging env whenever you need it, but only you know how complex this application is and what sort of overhead that would provide.
I'm fairly new to both Rails and Heroku but I'm seriously thinking of using it as a platform to deploy my Ruby/Rails applications.
I want to use all the power of Heroku, so I prefer the "embedded" PostgreSQL managed by Heroku instead of the addon for Amazon RDS for MySQL, but I'm not so confident without the possibility to access my data in a SQL client...
I know that in a well made app you have no need to access DB, but there are some situations (add rows to a config table, see data not mapped in a view, update some columns for debugging issues, performance monitoring, running queries for reporting, etc.) when this can be good...
How do you solve this problem? What's you experience in a real life app powered by Heroku?
Thanks!
I have been using it for a about a year. I love the workflow that it provides but I find not having access to the data is a real bother. Your options for working with database are:
Taps: In theory you create your database however you want locally and use taps to copy both schema and data to Heroku. In practice, most of the time its amazingly great. However I am currently dealing with the cleanup after taps translated some of my columns poorly and corrupted my data.
Heroku console: Totally fine for all the usual ActiveRecord stuff, but closest you can get to the database is ActiveRecord::Base.connection.execute "some sql". When you find yourself wondering about doing alter table commands like that you will know you're in trouble.
They also provide a "bundle" as a method for backing up your app. This lets you download all your code plus a sql dump of the database. The difficulty is that since there is no direct database access there is no way of loading that same sql dump back into the database so you can recover from dataloss, which, to me, is the point of having those dump files to begin with. All you can use the bundle for is to create a new application (heroku bundles:animate), not restore a current app.
I would love to be wrong about any/all of these. This seems like a curious rough spot in the best thought out service that I know of. Database access normally doesn't feel like much to give up when most of what you do is made so easy.
To me database access is like a fire extinguisher. Usually not a big deal, but when it matters, it matters a lot.
I use admin_data to give me some insight as to what is going on. I've successfully used it on Heroku, as well as other hosting providers.
Firstly let me start off by saying that heroku is awesome. I've had a great experience deploying my application and integrating with their other services such as websolr.
With that said, your questions:
Getting at your data
If you want to be able to get to your data you can use taps to pull your remote database down locally. This can be useful for debugging.
Performance monitoring
Use new relic RPM. This comes as part of heroku, you can enable it from the add-ons menu.
Add-hoc database queries
You could write a controller which allows you to execute arbitrary sql and view the results, but this isn't something I'd recommend. As suggest admin_data is a good solution for managing your data, but if you want to do anything more complicated you'll have to resort to writing the code yourself.