I'm trying to clone my development postgres db to Heroku with the command:
heroku pg:transfer -t postgres://localhost/MentorConnect_development -f postgres://mbarwrandompn:DaVXj1_AVytIgLU3lsyEKjU8eC#ec2-23-21-129-229.compute-1.amazonaws.com:5432/d4ahirandom
(changed the url for privacy)
It seems to work -- I get a lot of lines going with my database columns and everything with no errors. However when I go into the Heroku rails console, the database is empty. What am I doing wrong?
Your command there is transferring from Heroku to localhost, not development > heroku as per your question.
Also, it's much easier to use the colour of the database rather than the full URL (get it from heroku config output) eg;
heroku pg:transfer -f postgres://localhost/mydb -t COLOUR
where COLOUR is from the HEROKU_POSTGRESQL_COLOUR_URL from the heroku config output.
Related
I want to create the new application from the existing one using Heroku.
I cloned the old app to the new one and changed old name to the new one on Heroku.
I’ve transferred the environmental variables using as below:
$ heroku config -s -a existing-heroku-app > config.txt
$ cat config.txt | tr '\n' ' ' | xargs heroku config:set -a new-heroku-app
DATABASE_URL for the new application was created automatically when I've cloned the existing application and I could not change it using edit option on Heroku.
How to transfer database from old application to the new one?
Updated:
I tried as below
$ heroku pg:copy old::postgres://cbnedduwickhsw:c30d4bf09f3f0e756e6360b9331e8b001c70b6dc5c2c4cee899e71ad972f3f23#ec2-55-165-254-49.compute-1.amazonaws.com:5432/d9egrorbiba5at postgres://gjducqczpmibkn:0c7e9eecead030222792675a076c8980e8bd6edf2f68ba38465fbe978097445a#ec2-184-76-153-63.compute-1.amazonaws.com:5432/dfa8rqk1n47ec0--app new-staging
But have error
-bash: postgres://gjducqczpmibkn:0c7e9eecead030222792675a076c8980e8bd6edf2f68ba38465fbe978097445a#ec2-184-76-153-63.compute-1.amazonaws.com:5432/dfa8rqk1n47ec0--app: No such file or directory
From Heroku documentation
You can also transfer directly from a database on another app:
$ heroku pg:copy sushi::ORANGE GREEN --app sushi-staging
This would copy data from the ORANGE database of the sushi app to the GREEN database in sushi-staging. This could be used to copy production data into a staging app for testing purposes.
Usually there just one database per one Heroku app.
So you can run it as
$ heroku pg:copy old-app::DATABASE_URL DATABASE_URL --app new-app
Replace only old-app and new-app with your data.
I'm trying to export my db from production app to local.
As instructed by Heroku, I'm trying to do this with the Heroku plugin heroku-pg-transfer.
But I (really) need help understanding how to use it..
Do I need to specify the URL of my local db? And if so how do I find out the URL?
The documentation is quite clear.
$ heroku help pg:transfer
Usage: heroku pg:transfer
transfer data between databases
-f, --from DATABASE # source database, defaults to DATABASE_URL on the app
-t, --to DATABASE # target database, defaults to local $DATABASE_URL
If you want to transfer from production to local, you need to swap --to to --from. Run $ heroku config and get the URL of your remote database. It will probably be under DATABASE_URL or similar.
Then run
$ heroku pg:transfer --from THE_DATABASE_URL --to YOUR_LOCAL_DATABASE
where database connection details are found in your database.yml file. You need to merge them in a connection string, with the format
postgres://username:password#localhost:5432/database-name
Here's an example
$ heroku pg:transfer --from postgres://heroku#someheroku:5432/database-heroku --to postgres://username:password#localhost:5432/database-name
I am attempting to use Heroku's newer pgbackups:transfer command to transfer data from production to staging databases. The documentation is difficult to understand, and S.O. answers seem to redirect people to use pgbackups:restore, which I know how to use.
Documentation states
$ heroku pgbackups:transfer HEROKU_POSTGRESQL_PINK sushi-staging::HEROKU_POSTGRESQL_OLIVE -a sushi
When I run this as:
$ heroku pgbackups:transfer DATABASE_URL HEROKU_POSTGRESQL_IVORY sushi-staging::HEROKU_POSTGRESQL_JADE -a warm-brushlands-XXXX
I get this error, along with several different possible syntaxes, all different from the first example:
! Invalid argument: "sushi-staging::HEROKU_POSTGRESQL_JADE"
Usage: heroku pgbackups:transfer [SOURCE DATABASE] DESTINATION DATABASE
direct database-to-database transfer
If no DATABASE is specified, defaults to DATABASE_URL.
The pgbackups add-on is required to use direct transfers
Example:
$ heroku pgbackups:transfer green teal --app example
note that both the FROM and TO database must be accessible to the pgbackups service
$ heroku pgbackups:transfer DATABASE postgres://user:password#host/dbname --app example
Variations on this theme produce the same error.
I am looking for a clear syntax example for pgbackups:transfer, something like:
$ heroku pgbackups:transfer -from source-app-name-XXXX -to destination-app-name-XXXX -confirm destination-app-name-XXXX
Thank you.
After clarification from the Heroku support team, my successful syntax looks like this:
syntax
heroku pgbackups:transfer source-color destination-app-name-XXXX::destination-color -a source-app-name-XXXX
key
source-color = each database on your app gets a color. if your database color is: HEROKU_POSTGRESQL_PINK, you will simply put 'pink' here, with no quote marks.
destination-color = the simple color of your destination database
source-app-name = the name of the source app, such as warm-brushlands-1111
destination-app-name = the name of the destination app
Example
Source app: young-refuge-1111.herokuapp.com
Source database: HEROKU_POSTGRESQL_PINK
Destination app: warm-springs-1111.herokuapp.com
Destination database: HEROKU POSTGRESQL_IVORY
heroku pgbackups:transfer pink warm-springs-1111::ivory -a young-refuge-1111
I think you want to change this
heroku pgbackups:transfer DATABASE_URL HEROKU_POSTGRESQL_IVORY sushi-staging::HEROKU_POSTGRESQL_JADE -a warm-brushlands-XXXX
To this
heroku pgbackups:transfer HEROKU_POSTGRESQL_IVORY sushi-staging::HEROKU_POSTGRESQL_JADE -a warm-brushlands-XXXX
i.e. remove the DATABASE_URL. That should transfer HEROKU_POSTGRESQL_IVORY (assuming on prod) to HEROKU_POSTGRESQL_JADE (on staging).
I have a staging app full of data that I want to use to populate my currently empty production database. Currently I am trying to use pg:transfer. What is the correct syntax to use?
Addresses:
Staging app: afternoon-oasis-XXXX
Production app: warm-springs-XXXX, or postgres://long-database-url.compute-1.amazonaws.com:XXX/XXXXXXXX
The documentation states:
#documentation
$ heroku pg:transfer --to `heroku config:get DATABASE_URL -a app-staging` --confirm someapp
I have tried
$ heroku pg:transfer -t postgres://long-database-url.compute-1.amazonaws.com:XXX/XXXXXXXX -f JADE
This should be pulling from JADE, but the confirm message that appears to indicate that JADE, my afternoon-oasis app, my intended source, is going to be altered:
WARNING: Destructive Action
! This command will affect the app: afternoon-oasis-XXXX
! To proceed, type "afternoon-oasis-XXXX" or re-run this command with --confirm afternoon-oasis-XXXX
Why would Heroku be altering the source database? Or am I getting the syntax wrong?
Thanks in advance.
Second Update
You need the pgbackups addon for this but it is free. Sorry forgot that.
This addon will backup your postgres database every so often which is great if you ever need to recover data.
To add it just run heroku addons:add pgbackups:auto-week -a warm-springs-XXXX
Also add pgbackups to staging app
heroku addons:add pgbackups:auto-week -a afternoon-oasis-XXXX
Then run
heroku pgbackups:capture -a afternoon-oasis-XXXX to backups the latest
Finally you can run
heroku pgbackups:restore DATABASE_URL `heroku pgbackups:url -a afternoon-oasis-XXXX` -a warm-springs-XXXX
This command first gets the url of your staging app db backup and the pulls it to your production.
Original
Instead of using pg:transfer try doing something like this:
heroku pgbackups:restore DATABASE_URL 'INPUT YOUR STAGING DATABASE URL' -a warm-springs-XXXX
Let me know if you've got any questions. Then you can just type the confirm message with prompted or add the --confirm warm-springs-XXXX to the end of the command above.
Is there a faster way to transfer my production database to a test app?
Currently I'm doing a heroku db:pull to my local machine then heroku db:push --app testapp but this is becoming time consuming. I have some seed data but it is not nearly as accurate as simply testing with my real-world data. And since they're both stored on a neighboring AWS cloud, there must be a faster way to move the data?
I thought about using a heroku bundle, but I noticed the animate command is gone?
bundles:animate <bundle> # animate a bundle into a new app
It's quite common to migrate databases between staging, testing and production environments for Rails Apps. And heroku db:pull/push is painfully slow. The best way I have found so far is using Heroku PG Backups add-on and it's free. I followed following steps to migrate
production database to staging server:
1) Create the backup for the production-app db
heroku pg:backups capture --app production-app
This will generate b001 backup file from the main database (usually production db in database.yml)
2) To view all the backups (OPTIONAL)
heroku pg:backups --app production-app
3) Now use the pg:backups restore command to populate staging server database from the last backup file on production server
heroku pg:backups restore $(heroku pg:backups public-url --app production-app) DATABASE_URL --app staging-app
Remember that restore is a destructive operation, it will delete existing data before replacing it with the contents of the backup file.
So things are even easier now .. checkout the transfer command as part of pgbackups
heroku pgbackups:transfer HEROKU_POSTGRESQL_PINK sushi-staging::HEROKU_POSTGRESQL_OLIVE -a sushi
https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#4b-alternative-transfer-data-between-applications
This has worked beautifully for me taking production code back to my staging site.
The correct answer has changed again as of March 11, 2015.
heroku pg:backups restore $(heroku pg:backups public-url --app myapp-production) DATABASE_URL --app myapp-staging
Note specifically that the argument is now public-url.
https://blog.heroku.com/archives/2015/3/11/pgbackups-levels-up
Update for mid-2015...
The pgbackups add-on has been deprecated. No more pgbackups:transfer. pg:copy is ideal for this scenario.
To copy a database from yourapp (example db name: HEROKU_POSTGRESQL_PINK_URL to yourapp_staging (example db name: HEROKU_POSTGRESQL_WHITE_URL)
# turn off the web dynos in staging
heroku maintenance:on -a yourapp-staging
# if you have non-web-dynos, do them too
heroku ps:scale worker=0 -a yourapp-staging
# backup the staging database if you are paranoid like me (optional)
heroku pg:backups capture -a yourapp-staging
# execute the copy to splat over the top of the staging database
heroku pg:copy yourapp::HEROKU_POSTGRESQL_PINK_URL HEROKU_POSTGRESQL_WHITE_URL -a yourapp-staging
Then when it's complete, turn staging back on:
# this is if you have workers, change '1' to whatever
heroku ps:scale worker=1 -a yourapp-staging
heroku maintenance:off -a yourapp-staging
Reminder: you can use heroku pg:info -a yourapp-staging (and yourapp) to get the database constants.
(source: https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrade-with-pg-copy-default)
psql -h test_host -c 'drop database test_db_name; create database test_db_name;'
pg_dump -h production_host production_db_name | psql -h test_host test_db_name`
This can be done on production_host or on test_host — will work both ways.
Have not tested this, but it might work.
Do this to get the URL of your source database:
heroku console "ENV['DATABASE_URL']" --app mysourceapp
Then try executing db:push with that.
heroku db:push database_url_from_before --app mytargetapp
This might not work if Heroku doesn't allow access to the DB machines from outside their network, which is probably the case. You could, perhaps, try using taps (gem that heroku db commands use internally) from within your app code somewhere (maybe a rake task). This would be even faster than the above approach because everything stays completely within AWS.
Edit:
Here's an (admittedly hacky) way to do what I described above:
Grab the database URL as in the first code snippet above. Then from a rake task (you could do it on console but you risk running into the 30 second timeout limit on console commands), execute a shell command to taps (couldn't easily determine whether it's possible to use taps directly from Ruby; all docs show use of the CLI):
`taps pull database_url_from_source_app #{ENV['DATABASE_URL']}`
The backticks are important; this is how Ruby denotes a shell command, which taps is. Hopefully the taps command is accessible from the app. This avoids the problem of accessing the database machine from outside Heroku, since you're running this command from within your app.
Heroku enables you to fork existing applications in production. Use heroku fork to copy an existing application, including add-ons, config vars, and Heroku Postgres data.
Follow the instructions on Heroku: https://devcenter.heroku.com/articles/fork-app
Update for mid-2016...
Heroku now have a --fast flag when creating forks, however they will be up to 30 hours out-of-date.
$ heroku addons:create heroku-postgresql:standard-4 --fork HEROKU_POSTGRESQL_CHARCOAL --fast --app sushi
https://devcenter.heroku.com/articles/heroku-postgres-fork#fork-fast-option