I have an initial set of production data stored locally in the development database that I'd like to migrate to production for a starting point for data. What's the best way to transfer this data?
It does not seem evident if there is a way to use pgbackups as per the instructions. Perhaps I have to run a manual backup of some sort locally and then push it over with pgbackups and if that is the case, I'd appreciate some specific instructions on accomplishing this.
After some additional digging and an answer from Heroku, the answer for importation of initial data is to:
1) If using PGSQL locally, first dump the data:
pg_dump -U your_username your_database_name -f backup.sql
2) Then follow the instructions found here for importation to Heroku's database:
http://devcenter.heroku.com/articles/pgbackups#importing_from_a_backup
First dump your local database using pg_dump:
pg_dump -Fc --no-acl --no-owner -h ... mydb > mydb.dump
and then use heroku pgbackups:restore:
heroku pgbackups:restore heroku-database-url url-to-your-mydb.dump-file
Note that the mydb.dump file needs to be accessible by the Heroku servers.
The Heroku Dev Center page has detailed instructions:
https://devcenter.heroku.com/articles/heroku-postgres-import-export
Related
I'm trying to move a postgres database between machines as I move from one development platform to another. I have yaml_db gem installed on both machines.
On my old platform I do:
rake db:scheme:dump
rake db:data:dump
When I go to reload the database on my new machine I've discovered that my 2 dozen foreign_keys are preventing me for loading my data. What are my options?
You're copying a database, Rails really shouldn't have anything to do with the process (and as you're seeing, it just gets in the way).
Instead, put on your DBA hat and copy the database without bothering with Rails. Dump the data using pg_dump and then restore the data with pg_restore. The database's backup/restore tools know all about foreign keys, triggers, extensions, and anything else that Railsy tools don't understand.
you can use pg_dump command to dump your database using:
eg:
pg_dump -U <user-name> -h <host> <database> > <file-name>.sql
pg_dump -U postgres -h 127.0.0.1 database1 > database1.sql
Then copy file to other machine and run following command to restore database
psql <database-name> < path/to/sql_dump_file
psql database1 < database1.sql
I just try to learn RoR from this TUTORIAL and in 25th video i didn't get the idea how to restore my data. Example codes are deprecated.
What I have done :
1 - heroku pg:backups capture
2 - pg_dump -Fc --no-acl --no-owner -h localhost -U postgres myrubyblog > myrubyblog.dump
3 - on this point i didn't get the idea how to restore my data from dump
heroku pg:backups restore
'https://www.dropbox.com/home?preview=myrubyblog.dump' DATABASE_URL
What does it mean DATABASE_URL? the database form my local or heroku's database URL ?
Hello there If you had backed up you data with heroku then you can easily restore it there is a good documentation on heroku dev centre Heroku pg data restore
I wanted to use an external Database with my heroku application. But I'm unable to edit the configuration cariables. I tried using GUI, Which says, Cannot overwrite attachment values DATABASE_URL. While I tried using CLI as well. I used the command: heroku config:addDATABASE_URL="postgresql://username:password#IP:PORT". However, this throws an error ... is not a heroku command.
After trying out most these answers, I came across an update in 2016, here:
the database needs to be detached first, then update the variable of the DATABASE_URL.
heroku addons:attach heroku-postgresql -a <app_name> --as HEROKU_DATABASE
heroku addons:detach DATABASE -a <app_name>
heroku config:add DATABASE_URL=
An alternative method which does not require detaching (which may not be a desired outcome of the switch) is to simply attach the new database and then promote it, which the Heroku Documents explicitly states as a way to set the DATABASE_URL.
heroku addons:attach heroku-postgresql -a <app_name>
heroku pg:promote heroku-postgresql -a <app_name>
I got the very same situation today when I need to change postgres to postgis. Detach doesn't work for me so I done this to database.yml:
production:
url: <%= ENV['DATABASE_URL'].sub(/^postgres/, "postgis") %>
https://github.com/rgeo/activerecord-postgis-adapter/issues/214.
SQLAlchemy 1.4.x has removed support for the postgres:// URI scheme, which is used by Heroku Postgres (https://github.com/sqlalchemy/sqlalchemy/issues/6083). To maintain compatibility, perform the following before setting up connections with SQLAlchemy:
import os
import re
uri = os.getenv("DATABASE_URL") # or other relevant config var
if uri.startswith("postgres://"):
uri = uri.replace("postgres://", "postgresql://", 1)
# rest of connection code using the connection string `uri`
This will allow you to connect to Heroku Postgres services using SQLAlchemy >= 1.4.x
As explained in this article, the correct syntax to set/add a configuration variable is
$ heroku config:set DATABASE_URL="postgresql://username:password#IP:PORT"
However, it looks like (see the comments) the DATABASE_URL has been deprecated and trying to update it will trigger an error.
Based on the Heroku docs this is how you would share a database with multiple apps.
heroku addons:attach my-originating-app::DATABASE --app sushi
Solved it. Just for the reference of the users who have the same issue or want to have a similar implementation. Here's the workaround which worked for me.
Heroku no more overwrites databse.yml, so I just modified the DATBASE_URL in the database.yml and pushed it :)
It worked too!
Source
In my case, I needed to launch an java spring boot application with my personal data base (postgres). I have an instance on AWS, and when loading the app, an error was occurring because it would connect without ssl.
Considering this documentation (https://devcenter.heroku.com/articles/connecting-to-relational-databases-on-heroku-with-java#using-ssl-with-postgresql), it says:
We used to suggest adding the URL parameter sslmode=disable to JDBC URLs. We now require use of SSL for all new Heroku Postgres databases. We will be enforcing use of SSL on all Heroku Postgres databases from March 2018. Please do not disable SSL for your database or your applications may break.
So, resuming, step 1, I deleted my addon Heroku Postgres on Resources tab.
Step 2, I changed my application.yml
from:
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://<url>:<port>/<dataBaseName>?createDatabaseIfNotExist=true&useSSL=false
username: <user>
password: <pass>
to
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://<url>:<port>/<dataBaseName>?createDatabaseIfNotExist=true&useSSL=false&sslmode=disable
username: <user>
password: <pass>
I added "&sslmode=disable" at the end of url string.
And finally, rebuild/deploy (which in my case is automatic after pushing into my repo on github).
I hope this would help someone.
Peace...
One way to edit the DATABASE_URL will be to create another app and add the heroku_postgres add-on there and then grab the url of that database and use that in your main app by configuring the environment variables and set the value of DATABASE_URL to that url of the database.
Now you can easily change the DATABASE_URL as that is not attached with the app.
I can dump a heroku database with $ heroku pgbackups:capture. Also, this SO post shows that there are tools for taking a development database and dumping it to seed.rb.
I'm wondering if there is an easy way to combine the two processes, effectively dumping the data from a production Heroku database into my local seeds.rb for more realistic development testing.
If this is possible, what's the cleanest way to do this?
Update:
Based on the insightful answer from dB', I may consider using PGSQL locally. I am still interested, however, in the seed.rb aspect of the question if there is a way to do that easily.
There are a couple ways to accomplish such a thing. #dB' has outlined one of them - using the PG Backups add-on to export your database. It's a great options but involves a few (trivial) manual commands.
I would recommend using the pg:transfer Heroku CLI plugin to transfer the data in a single step. Under the covers it's still very much the same thing happening as with using PG Backups, but it's packaged a bit nicer and has some useful defaults.
From your app's directory, copy your production database locally (assuming a local PG db), by installing the plugin and execute the pg:transfer command.
$ heroku plugins:install https://github.com/ddollar/heroku-pg-transfer
$ heroku pg:transfer
There are a couple options you can set as well. See my writeup for more details.
Hope that helps! And yes, please do use the same database during development as you do in production.
Not sure if it's what you're looking for, but have you tried copying the database to your local machine using pgbackups:capture and pg_restore? This approach doesn't use seeds.rb, but still recreates your production database on your local machine. It looks something like this.
$ heroku pgbackups:capture
$ curl -o latest.dump `heroku pgbackups:url`
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
(This code is copied liberally from the explanation at the Heroku dev center.)
Is there a faster way to transfer my production database to a test app?
Currently I'm doing a heroku db:pull to my local machine then heroku db:push --app testapp but this is becoming time consuming. I have some seed data but it is not nearly as accurate as simply testing with my real-world data. And since they're both stored on a neighboring AWS cloud, there must be a faster way to move the data?
I thought about using a heroku bundle, but I noticed the animate command is gone?
bundles:animate <bundle> # animate a bundle into a new app
It's quite common to migrate databases between staging, testing and production environments for Rails Apps. And heroku db:pull/push is painfully slow. The best way I have found so far is using Heroku PG Backups add-on and it's free. I followed following steps to migrate
production database to staging server:
1) Create the backup for the production-app db
heroku pg:backups capture --app production-app
This will generate b001 backup file from the main database (usually production db in database.yml)
2) To view all the backups (OPTIONAL)
heroku pg:backups --app production-app
3) Now use the pg:backups restore command to populate staging server database from the last backup file on production server
heroku pg:backups restore $(heroku pg:backups public-url --app production-app) DATABASE_URL --app staging-app
Remember that restore is a destructive operation, it will delete existing data before replacing it with the contents of the backup file.
So things are even easier now .. checkout the transfer command as part of pgbackups
heroku pgbackups:transfer HEROKU_POSTGRESQL_PINK sushi-staging::HEROKU_POSTGRESQL_OLIVE -a sushi
https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#4b-alternative-transfer-data-between-applications
This has worked beautifully for me taking production code back to my staging site.
The correct answer has changed again as of March 11, 2015.
heroku pg:backups restore $(heroku pg:backups public-url --app myapp-production) DATABASE_URL --app myapp-staging
Note specifically that the argument is now public-url.
https://blog.heroku.com/archives/2015/3/11/pgbackups-levels-up
Update for mid-2015...
The pgbackups add-on has been deprecated. No more pgbackups:transfer. pg:copy is ideal for this scenario.
To copy a database from yourapp (example db name: HEROKU_POSTGRESQL_PINK_URL to yourapp_staging (example db name: HEROKU_POSTGRESQL_WHITE_URL)
# turn off the web dynos in staging
heroku maintenance:on -a yourapp-staging
# if you have non-web-dynos, do them too
heroku ps:scale worker=0 -a yourapp-staging
# backup the staging database if you are paranoid like me (optional)
heroku pg:backups capture -a yourapp-staging
# execute the copy to splat over the top of the staging database
heroku pg:copy yourapp::HEROKU_POSTGRESQL_PINK_URL HEROKU_POSTGRESQL_WHITE_URL -a yourapp-staging
Then when it's complete, turn staging back on:
# this is if you have workers, change '1' to whatever
heroku ps:scale worker=1 -a yourapp-staging
heroku maintenance:off -a yourapp-staging
Reminder: you can use heroku pg:info -a yourapp-staging (and yourapp) to get the database constants.
(source: https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrade-with-pg-copy-default)
psql -h test_host -c 'drop database test_db_name; create database test_db_name;'
pg_dump -h production_host production_db_name | psql -h test_host test_db_name`
This can be done on production_host or on test_host — will work both ways.
Have not tested this, but it might work.
Do this to get the URL of your source database:
heroku console "ENV['DATABASE_URL']" --app mysourceapp
Then try executing db:push with that.
heroku db:push database_url_from_before --app mytargetapp
This might not work if Heroku doesn't allow access to the DB machines from outside their network, which is probably the case. You could, perhaps, try using taps (gem that heroku db commands use internally) from within your app code somewhere (maybe a rake task). This would be even faster than the above approach because everything stays completely within AWS.
Edit:
Here's an (admittedly hacky) way to do what I described above:
Grab the database URL as in the first code snippet above. Then from a rake task (you could do it on console but you risk running into the 30 second timeout limit on console commands), execute a shell command to taps (couldn't easily determine whether it's possible to use taps directly from Ruby; all docs show use of the CLI):
`taps pull database_url_from_source_app #{ENV['DATABASE_URL']}`
The backticks are important; this is how Ruby denotes a shell command, which taps is. Hopefully the taps command is accessible from the app. This avoids the problem of accessing the database machine from outside Heroku, since you're running this command from within your app.
Heroku enables you to fork existing applications in production. Use heroku fork to copy an existing application, including add-ons, config vars, and Heroku Postgres data.
Follow the instructions on Heroku: https://devcenter.heroku.com/articles/fork-app
Update for mid-2016...
Heroku now have a --fast flag when creating forks, however they will be up to 30 hours out-of-date.
$ heroku addons:create heroku-postgresql:standard-4 --fork HEROKU_POSTGRESQL_CHARCOAL --fast --app sushi
https://devcenter.heroku.com/articles/heroku-postgres-fork#fork-fast-option