I have a very basic application deployed on Heroku that has gathered data over the last week and a half. I now want to do some more development on the site and would like to copy the data from my production application database across to the database being used for my development.
I added pgbackups using the following command:
heroku addons:add pgbackups
I then captured the production database using
heroku pgbackups:capture
When I run heroku pgbackups I can see the following:
ID Backup Time Status Size Database
---- ------------------------- ------------------------------------ ------ --------- ---------------------------------
b001 2014/06/19 13:31.08 +0000 Finished # 2014/06/19 13:31.10 +0000 27.7KB HEROKU_POSTGRESQL_AMBER_URL (DATABASE_URL)
I'm now tring to use the following command to restore this backup to my development database on Heroku (orange)
heroku pgbackups:restore HEROKU_POSTGRESQL_ORANGE b001
but I'm getting the following message:
! Unknown database: HEROKU_POSTGRESQL_563f103f. Valid options are: DATABASE_URL, HEROKU_POSTGRESQL_AMBER_URL
Is it possible to do what I want and if so how do I do it? I've been trying to find an answer the last couple of hours but am getting confused with the concept of development-staging-production. Everything I'm finding refers to copying a db from production to staging but as far as I can see I'm trying to copy from production to development. Thanks for looking
Are the databases in two separate apps, i.e. a development app and a production app? If so, try using the pgbackups:transfer command to transfer data between applications. Docs available here.
Basically try:
heroku pgbackups:transfer HEROKU_POSTGRESQL_AMBER_URL <development app name>::HEROKU_POSTGRESQL_ORANGE -a <production app name>
Related
When I log onto the website (example.herokuapp.com) and create a user, the user is created in the Heroku Postgres::Database but when I go into Cloud9 and enter Rails Console, the user does not exist in my production database. How can I merge the changes made in production (Heroku) with development (local Rails) so that these users show up in Rails Console? Thanks.
If you need the users to show up in the console you can run
heroku run console --app APP_NAME
If you'd like to "merge" the database you have to export your heroku Postgres database and convert the table to SQL format.
For more information about exporting PG Backups: https://devcenter.heroku.com/articles/heroku-postgres-import-export
Another stackoverflow post that might help: How to I merge two databases with same schema that are on Heroku?
I've been given access to Heroku application with rather strange setup. It has one database but when I run heroku config, I get different DATABASE_URL and HEROKU_POSTGRESQL_BRONZE_URL.
When I run heroku pg:info I get the following result:
=== HEROKU_POSTGRESQL_BRONZE_URL
Plan: Dev
Status: available
Connections: 1
PG Version: 9.2.4
Created: 2013-09-05 11:02 UTC
Data Size: 6.5 MB
Tables: 0
Rows: 0/10000 (In compliance)
Fork/Follow: Unsupported
I realised that my database is at DATABASE_URL, but I can't access that database, only through heroku run console. All heroku pg commands fail with this message:
! Unknown database: DATABASE_URL. Valid options are: HEROKU_POSTGRESQL_BRONZE_URL
If I run heroku pg HEROKU_POSTGRESQL_BRONZE_URL, I get access to empty database from above.
Since I have some issues with running migrations, I think my database might be full, and I'd like to check it. Any ideas how I can do that?
Here's the error after I run heroku run rake db:migrate:
PG::Error: ERROR: permission denied for relation schema_migrations
: INSERT INTO "schema_migrations" ("version") VALUES ('20130918114202')
More information about the setup:
rails 3.2.12
RAILS_ENV: staging (I don't have access to production, but I know this is "dev" server and there's also real "staging" from which this app was forked).
I have same problem and i fixed it:
Just Keep a Backup from your database and restore it back again, here is the steps:
heroku pg:info <-- to get the Database Name
heroku addons:add pgbackups <-- make sure you have the addons for backup
heroku pgbackups:capture <-- Capture the backup
heroku pgbackups <-- check your backups and make sure its there
heroku pg:reset DATABASE_NAME <-- Reset your database don't worry we have a backup, replace DATABASE_NAME with database name
heroku pgbackups:restore DATABASE_NAME b001 <-- Restore the backup again, replace DATABASE_NAME with database name and b001 with your Database version you can see this version number in heroku pgbackups step
heroku run rake db:migrate <-- Now you can run your migration and Operate in normal mode.
This seems like something screwy on the Heroku side. Have you tried submitting a ticket with them? I've always had good luck with their support.
I just got an update from my client. Before, I couldn't drop the database, because of the data in it. At the end, we decided to drop the database since the data can be added back (it's dev server, doesn't matter if we loose some dummy data).
I didn't find a solution for the problem above, but promoting HEROKU_POSTGRESQL_BRONZE_URL and restoring from backup solved the issue about not being able to access the database.
I'm considering paying the $50/ mo for a production database so I can use PostGIS with my rails 4 app. My issue is that I would like to continue developing with a 'staging' environment, but can't yet justify paying for two production databases. Staging is obviously just a clone of production, so just pointing both apps to the same DB url (post) would likely cause some major headaches.
Is there any way to partition the database, or another strategy you'd recommend?
Initial answer from heroku support is no, but I'm hoping there's a scrappy workaround.
Thanks!
First off, I would highly suggest just paying the extra $50/mo. For that, you get all kinds of cool stuff like forking and pipelines, as well as the fact that this is kind of hacky. I honestly don't know if this might end up wiping your production data when you clear the staging database. Please make backups before trying this.
I'll view this answer as an answer to a technical challenge rather than a business decision. Without further ado,
Setting up multiple environments on the same database with Heroku Postgres schemas
First, I deleted the existing dev database and added a production database to my production app.
heroku addons:add heroku-postgresql:crane
> Adding heroku-postgresql:crane on test-shared-app... done, v# ($50/mo)
> Attached as HEROKU_POSTGRESQL_RED_URL
This attached a database under the RED color, so replace HEROKU_POSTGRESQL_RED_URL with the appropriate color for your app.
This attaches a database to the production app, but we need to connect to the same app for staging. First, create the staging schema
heroku run "bundle exec rails runner 'ActiveRecord::Base.connection.execute(%q{CREATE SCHEMA staging})'"
Next, create the staging app. See Managing Multiple Environments for an App for more information.
heroku create --remote staging test-shared-app-staging
heroku config:set RACK_ENV=staging RAILS_ENV=staging --remote staging
Then, copy the environment data from your existing app. Add ?schema_search_path=staging to the end of the URL.
heroku config --remote heroku --shell
# make note of your database URLs
heroku config:set --remote staging \
DATABASE_URL=postgres://...?schema_search_path=staging \
HEROKU_POSTGRESQL_RED_URL=postgres://...?schema_search_path=staging
And push to staging
git push staging master
Now, run migrations on staging
heroku run --remote staging bundle exec rake db:migrate
And try it out.
My "app" is running at http://test-shared-app.herokuapp.com/posts and http://test-shared-app-staging.herokuapp.com/posts. You can see the source at https://github.com/benmanns/heroku-shared-app.
If I were you, I would use production DB until the right moment (start making money or getting more users, etc). I would replicate all the tables in production DB and give the new tables new names by prepending something like "stag_{original_table_name}. So, you would have two different sets of the same tables. In your models, make them use the new table on staging environment:
class Foo < ActiveRecord::Base
self.table_name = "staging_#{self.class.name}" if Rails.env.staging?
I am pretty cheap... and this may be an ugly solution in the eyes of the true Rails masters.
Heroku Schemas is another approach to this; it's a Heroku plugin that basically lets you run a single command to apply Benjamin Manns's solution of multiple schemas.
I've been having problems yesterday with Heroku shared Postgres DB. It looks like my db is not correctly 'pushed' to my heroku DB.
Configuration
Using 'pg' gem for both development and production environments.
Using Heroku's 'shared-database' add-on.
Running Rails 3.1.1.
What I'm trying to do
Push my source code to Heroku and then synchronize my postgres DB with Heroku's one. No problem for the source itself, but I can't get my DB up and running.
What I've tried so far
1: Source code push : git push heroku master. OK. This works.
2: Then I try to run : heroku db:push. Seems to be running normally, no error, the schema is sent to Heroku, the different tables are correctly detected and seem to be sent as well :
3: Let's check it out, is everything allright ? heroku info :
Doh ! The database is still empty (and therefore I get DB-related errors when accessing dynamic pages). Can't understand what's happening, I've spent time on this issue yesterday evening but only been mpessing with Heroku for a little while and never encountered this issue. Any clues ?
I'm not convinced that the Database size figure is a live figure. Here's a clean deployment of an app to Heroku and the output of heroku info at each stage. The first run is post application creation, the second after a push of the code, the third after a heroku db:push.
The database was uploaded before the third output and the application is running however the DB:size is not reporting the figure.
Is your application actually erroring because the database isn't present?
The problem is in the commit process.
You should run manually after each commit from your console :
heroku run rake db:migrate
Enjoy
daniel
Is there a faster way to transfer my production database to a test app?
Currently I'm doing a heroku db:pull to my local machine then heroku db:push --app testapp but this is becoming time consuming. I have some seed data but it is not nearly as accurate as simply testing with my real-world data. And since they're both stored on a neighboring AWS cloud, there must be a faster way to move the data?
I thought about using a heroku bundle, but I noticed the animate command is gone?
bundles:animate <bundle> # animate a bundle into a new app
It's quite common to migrate databases between staging, testing and production environments for Rails Apps. And heroku db:pull/push is painfully slow. The best way I have found so far is using Heroku PG Backups add-on and it's free. I followed following steps to migrate
production database to staging server:
1) Create the backup for the production-app db
heroku pg:backups capture --app production-app
This will generate b001 backup file from the main database (usually production db in database.yml)
2) To view all the backups (OPTIONAL)
heroku pg:backups --app production-app
3) Now use the pg:backups restore command to populate staging server database from the last backup file on production server
heroku pg:backups restore $(heroku pg:backups public-url --app production-app) DATABASE_URL --app staging-app
Remember that restore is a destructive operation, it will delete existing data before replacing it with the contents of the backup file.
So things are even easier now .. checkout the transfer command as part of pgbackups
heroku pgbackups:transfer HEROKU_POSTGRESQL_PINK sushi-staging::HEROKU_POSTGRESQL_OLIVE -a sushi
https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#4b-alternative-transfer-data-between-applications
This has worked beautifully for me taking production code back to my staging site.
The correct answer has changed again as of March 11, 2015.
heroku pg:backups restore $(heroku pg:backups public-url --app myapp-production) DATABASE_URL --app myapp-staging
Note specifically that the argument is now public-url.
https://blog.heroku.com/archives/2015/3/11/pgbackups-levels-up
Update for mid-2015...
The pgbackups add-on has been deprecated. No more pgbackups:transfer. pg:copy is ideal for this scenario.
To copy a database from yourapp (example db name: HEROKU_POSTGRESQL_PINK_URL to yourapp_staging (example db name: HEROKU_POSTGRESQL_WHITE_URL)
# turn off the web dynos in staging
heroku maintenance:on -a yourapp-staging
# if you have non-web-dynos, do them too
heroku ps:scale worker=0 -a yourapp-staging
# backup the staging database if you are paranoid like me (optional)
heroku pg:backups capture -a yourapp-staging
# execute the copy to splat over the top of the staging database
heroku pg:copy yourapp::HEROKU_POSTGRESQL_PINK_URL HEROKU_POSTGRESQL_WHITE_URL -a yourapp-staging
Then when it's complete, turn staging back on:
# this is if you have workers, change '1' to whatever
heroku ps:scale worker=1 -a yourapp-staging
heroku maintenance:off -a yourapp-staging
Reminder: you can use heroku pg:info -a yourapp-staging (and yourapp) to get the database constants.
(source: https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrade-with-pg-copy-default)
psql -h test_host -c 'drop database test_db_name; create database test_db_name;'
pg_dump -h production_host production_db_name | psql -h test_host test_db_name`
This can be done on production_host or on test_host — will work both ways.
Have not tested this, but it might work.
Do this to get the URL of your source database:
heroku console "ENV['DATABASE_URL']" --app mysourceapp
Then try executing db:push with that.
heroku db:push database_url_from_before --app mytargetapp
This might not work if Heroku doesn't allow access to the DB machines from outside their network, which is probably the case. You could, perhaps, try using taps (gem that heroku db commands use internally) from within your app code somewhere (maybe a rake task). This would be even faster than the above approach because everything stays completely within AWS.
Edit:
Here's an (admittedly hacky) way to do what I described above:
Grab the database URL as in the first code snippet above. Then from a rake task (you could do it on console but you risk running into the 30 second timeout limit on console commands), execute a shell command to taps (couldn't easily determine whether it's possible to use taps directly from Ruby; all docs show use of the CLI):
`taps pull database_url_from_source_app #{ENV['DATABASE_URL']}`
The backticks are important; this is how Ruby denotes a shell command, which taps is. Hopefully the taps command is accessible from the app. This avoids the problem of accessing the database machine from outside Heroku, since you're running this command from within your app.
Heroku enables you to fork existing applications in production. Use heroku fork to copy an existing application, including add-ons, config vars, and Heroku Postgres data.
Follow the instructions on Heroku: https://devcenter.heroku.com/articles/fork-app
Update for mid-2016...
Heroku now have a --fast flag when creating forks, however they will be up to 30 hours out-of-date.
$ heroku addons:create heroku-postgresql:standard-4 --fork HEROKU_POSTGRESQL_CHARCOAL --fast --app sushi
https://devcenter.heroku.com/articles/heroku-postgres-fork#fork-fast-option