Download PG database as a file from Heroku - ruby-on-rails

I'm trying to download a file (a copy) of our staging DB. I'm using pgbackups following these docs.
I created a backup, than got the public-url using heroku pgbackups:url b462 --app staging-appname, which worked.
Than I run the command curl -o latest.dump heroku pg:backups <public-url> b462 -a staging-appname and get curl: no URL specified!
Where it says <public-url> I've copied the actual long public-url, as in https://s3.amazonaws.com/reallly-long-url-with-acess-key-ect.
What am I missing here?

The following will create the backup:
heroku pg:backups capture --app name-of-the-app
Using the following information printed in terminal
---backup---> b001
In the following command
heroku pg:backups public-url b001 -a name-of-the-app
Will give you the url of the dump

Try just the following to get the latest file immediately after running heroku pgbackups:capture --app=staging-appname:
curl -o latest.dump `heroku pgbackups:url --app=staging-appname`
The b.. is required only if you are looking for a specific older file.

I was also struggling with this doc, and Prakash's suggestion didn't work for me (error complained about not being able to resolve 'heroku' which isn't part of the URL, so the syntax is seems to be bad, at least for the version of the pg or pgbackup gem I'm using.)
I also tried wget, as suggested in the heroku doc, to no avail.
When I'm stumped I try to think it down to the most simple elements, which led me to a far simpler solution than is provided by the doc...
I used the command in Rafael's answer to generate a temporary URL to retrieve the dump file:
heroku pg:backups public-url b001 -a name-of-the-app
Then, instead of plugging the resulting URL into the curl command, I just pasted that URL into Chrome's address bar and downloaded the file using my web browser.
The download was successful, the db file is valid.

With a little more sugar, here an example of shell script to download mysql dump from heroku and to restore it as a local database
backup_name=$1
if [ -z "$backup_name" ]
then
echo "You must supplied heroku backup name (ex: b001)"
return
fi
echo "Loading '$backup_name' database from heroku"
heroku pg:backups public-url $backup_name -a your-app-name > last_url.txt
curl -o latest.dump `cat last_url.txt`
dropdb local-db-name
createdb local-db-name
pg_restore -c -d local-db-name latest.dump
You need to give the heroku backup name as input.

Related

heroku pgbackups:url command is no longer working?

How do I download my dump directly from Amazon AWS S3 if heroku pgbackups:url b004 isn't working?
Specifically, when I run this command it returns:
! Please add the pgbackups addon first via:
! heroku addons:add pgbackups
And then when I run this command I get:
! No such add-on plan.
What to do? I'd like to get the link/URL to my pgdump file b004, and then use that link to import the postgresql data onto my mirrored heroku app.. But I am having trouble getting the URL, which used to be easily done by running heroku pgbackups:url b004
Any help here would be great!
To download backup b004 use the following syntax:
curl -o b004.dump `heroku pg:backups public-url b004`
That will download the backup as b004.dump in the current directory.

heroku pgbackups:restore not working on rails app

For some reason heroku pgbackups:restore won't work on my rails app,
I'm creating my dumpfile using the command:
pg_dump -F c -v -U john -h localhost project_development > -f dumpfile.dump
I then upload it to dropbox, as the heroku documentation says it needs to be needs to uploaded somewhere with an HTTP-accessible URL. After I've uploaded the file to dropbox, I then run the command
heroku pgbackups:restore HEROKU_POSTGRESQL_RED_URL 'https://dl.dropboxusercontent.com/content_link/JgHkmg0iv8vIMKWqlN6medJqjClPeWYl2AQqJWV8IeEiujnfYvPdtvq7xh26dvCQ'
the output of which is
Retrieving... done
Restoring... done
HEROKU_POSTGRESQL_RED_URL is my database url and the line in the single quotes is a raw version of my dumpfile(the dropbox documentation showed how to turn the dl file into a raw file).
But whenever go to my apps homepage I get an error, I can see from running heroku logs that the error is a database error
PG::UndefinedTable: ERROR: relation "listings" does not exist
LINE 1: SELECT "listings".* FROM "listings" ORDER BY listings.pri..
Also, when I run heroku pg:info I can see that nothing has actually been inserted into the database
=== HEROKU_POSTGRESQL_RED_URL (DATABASE_URL)
Plan: Hobby-dev
Status: Available
Connections: 2/20
PG Version: 9.3.4
Created: 2014-10-21 15:24 UTC
Data Size: 7.0 MB
Tables: 0
Rows: 0/10000 (In compliance)
Fork/Follow: Unsupported
Rollback: Unsupported
Can anyone tell me why my database isn't being updated? Thanks.
I tested this on a blank hobby dev database with heroku. You don't need to include the the no-owner and no-acl options (-x -O) but it's considered good practice since those alter statements will fail.
The problem with your dump command is the stray >
pg_dump -F c -v -O -x -U john -h localhost project_development -f dumpfile.dump
Upload the resulting file to dropbox or s3 and try the restore again. If there are errors you can run
heroku logs -n 100 --ps pgbackups
To see the log output from the pg_restore command.

Heroku transfer data between remote databases, clear syntax example

I have a staging app full of data that I want to use to populate my currently empty production database. Currently I am trying to use pg:transfer. What is the correct syntax to use?
Addresses:
Staging app: afternoon-oasis-XXXX
Production app: warm-springs-XXXX, or postgres://long-database-url.compute-1.amazonaws.com:XXX/XXXXXXXX
The documentation states:
#documentation
$ heroku pg:transfer --to `heroku config:get DATABASE_URL -a app-staging` --confirm someapp
I have tried
$ heroku pg:transfer -t postgres://long-database-url.compute-1.amazonaws.com:XXX/XXXXXXXX -f JADE
This should be pulling from JADE, but the confirm message that appears to indicate that JADE, my afternoon-oasis app, my intended source, is going to be altered:
WARNING: Destructive Action
! This command will affect the app: afternoon-oasis-XXXX
! To proceed, type "afternoon-oasis-XXXX" or re-run this command with --confirm afternoon-oasis-XXXX
Why would Heroku be altering the source database? Or am I getting the syntax wrong?
Thanks in advance.
Second Update
You need the pgbackups addon for this but it is free. Sorry forgot that.
This addon will backup your postgres database every so often which is great if you ever need to recover data.
To add it just run heroku addons:add pgbackups:auto-week -a warm-springs-XXXX
Also add pgbackups to staging app
heroku addons:add pgbackups:auto-week -a afternoon-oasis-XXXX
Then run
heroku pgbackups:capture -a afternoon-oasis-XXXX to backups the latest
Finally you can run
heroku pgbackups:restore DATABASE_URL `heroku pgbackups:url -a afternoon-oasis-XXXX` -a warm-springs-XXXX
This command first gets the url of your staging app db backup and the pulls it to your production.
Original
Instead of using pg:transfer try doing something like this:
heroku pgbackups:restore DATABASE_URL 'INPUT YOUR STAGING DATABASE URL' -a warm-springs-XXXX
Let me know if you've got any questions. Then you can just type the confirm message with prompted or add the --confirm warm-springs-XXXX to the end of the command above.

Is it possible to run Heroku toolbelt commands from the server via cron/scheduler (e.g. in a rake task or shell script)

I would like to run Heroku toolbelt command on the server in a scheduled way:
e.g.
heroku maintenance:on
# do some other stuff
heroku maintenance:off
As far as I could find, there does not seem to be a way to do this, not even a workaround?
You can use Heroku Scheduler and configure the following command:
curl -s https://s3.amazonaws.com/assets.heroku.com/heroku-client/heroku-client.tgz \
| tar xz && ./heroku-client/bin/heroku maintenance:on -a you-app-name-here
For this to work, you need to add a Config Variable named HEROKU_API_KEY and set its value to the "API Key" value from your Accounts page.

transfer db from one heroku app to another faster

Is there a faster way to transfer my production database to a test app?
Currently I'm doing a heroku db:pull to my local machine then heroku db:push --app testapp but this is becoming time consuming. I have some seed data but it is not nearly as accurate as simply testing with my real-world data. And since they're both stored on a neighboring AWS cloud, there must be a faster way to move the data?
I thought about using a heroku bundle, but I noticed the animate command is gone?
bundles:animate <bundle> # animate a bundle into a new app
It's quite common to migrate databases between staging, testing and production environments for Rails Apps. And heroku db:pull/push is painfully slow. The best way I have found so far is using Heroku PG Backups add-on and it's free. I followed following steps to migrate
production database to staging server:
1) Create the backup for the production-app db
heroku pg:backups capture --app production-app
This will generate b001 backup file from the main database (usually production db in database.yml)
2) To view all the backups (OPTIONAL)
heroku pg:backups --app production-app
3) Now use the pg:backups restore command to populate staging server database from the last backup file on production server
heroku pg:backups restore $(heroku pg:backups public-url --app production-app) DATABASE_URL --app staging-app
Remember that restore is a destructive operation, it will delete existing data before replacing it with the contents of the backup file.
So things are even easier now .. checkout the transfer command as part of pgbackups
heroku pgbackups:transfer HEROKU_POSTGRESQL_PINK sushi-staging::HEROKU_POSTGRESQL_OLIVE -a sushi
https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#4b-alternative-transfer-data-between-applications
This has worked beautifully for me taking production code back to my staging site.
The correct answer has changed again as of March 11, 2015.
heroku pg:backups restore $(heroku pg:backups public-url --app myapp-production) DATABASE_URL --app myapp-staging
Note specifically that the argument is now public-url.
https://blog.heroku.com/archives/2015/3/11/pgbackups-levels-up
Update for mid-2015...
The pgbackups add-on has been deprecated. No more pgbackups:transfer. pg:copy is ideal for this scenario.
To copy a database from yourapp (example db name: HEROKU_POSTGRESQL_PINK_URL to yourapp_staging (example db name: HEROKU_POSTGRESQL_WHITE_URL)
# turn off the web dynos in staging
heroku maintenance:on -a yourapp-staging
# if you have non-web-dynos, do them too
heroku ps:scale worker=0 -a yourapp-staging
# backup the staging database if you are paranoid like me (optional)
heroku pg:backups capture -a yourapp-staging
# execute the copy to splat over the top of the staging database
heroku pg:copy yourapp::HEROKU_POSTGRESQL_PINK_URL HEROKU_POSTGRESQL_WHITE_URL -a yourapp-staging
Then when it's complete, turn staging back on:
# this is if you have workers, change '1' to whatever
heroku ps:scale worker=1 -a yourapp-staging
heroku maintenance:off -a yourapp-staging
Reminder: you can use heroku pg:info -a yourapp-staging (and yourapp) to get the database constants.
(source: https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrade-with-pg-copy-default)
psql -h test_host -c 'drop database test_db_name; create database test_db_name;'
pg_dump -h production_host production_db_name | psql -h test_host test_db_name`
This can be done on production_host or on test_host — will work both ways.
Have not tested this, but it might work.
Do this to get the URL of your source database:
heroku console "ENV['DATABASE_URL']" --app mysourceapp
Then try executing db:push with that.
heroku db:push database_url_from_before --app mytargetapp
This might not work if Heroku doesn't allow access to the DB machines from outside their network, which is probably the case. You could, perhaps, try using taps (gem that heroku db commands use internally) from within your app code somewhere (maybe a rake task). This would be even faster than the above approach because everything stays completely within AWS.
Edit:
Here's an (admittedly hacky) way to do what I described above:
Grab the database URL as in the first code snippet above. Then from a rake task (you could do it on console but you risk running into the 30 second timeout limit on console commands), execute a shell command to taps (couldn't easily determine whether it's possible to use taps directly from Ruby; all docs show use of the CLI):
`taps pull database_url_from_source_app #{ENV['DATABASE_URL']}`
The backticks are important; this is how Ruby denotes a shell command, which taps is. Hopefully the taps command is accessible from the app. This avoids the problem of accessing the database machine from outside Heroku, since you're running this command from within your app.
Heroku enables you to fork existing applications in production. Use heroku fork to copy an existing application, including add-ons, config vars, and Heroku Postgres data.
Follow the instructions on Heroku: https://devcenter.heroku.com/articles/fork-app
Update for mid-2016...
Heroku now have a --fast flag when creating forks, however they will be up to 30 hours out-of-date.
$ heroku addons:create heroku-postgresql:standard-4 --fork HEROKU_POSTGRESQL_CHARCOAL --fast --app sushi
https://devcenter.heroku.com/articles/heroku-postgres-fork#fork-fast-option

Resources