I am writing a script for periodically dumping the database and uploading it to S3
I have this method call utilizes the Heroku gem and that gets called from a rake task at various intervals
def dump_database
Rails.logger.info "Dumping database into temporary file."
# Stamp the filename
datestamp = Time.now.strftime("%d-%m-%Y_%H-%M-%S")
# Drop it in the db/backups directory temporarily
file_path = "#{Rails.root}/db/backups/APPNAME_#{Rails.env}_#{datestamp}_dump.sql.gz"
# Dump and zip the backup file
sh "heroku pgbackups:capture --expire -a APPNAME-#{Rails.env}"
sh "curl -o #{file_path} `heroku pgbackups:url -a APPNAME-#{Rails.env}` | gzip > #{file_path}"
return file_path
end
Problem is that when I call the
sh "heroku pgbackups:capture --expire -a APPNAME-#{Rails.env}"
Heroku asks for credentials (email/password) to me entered into the terminal
Cant this be done without entering credentials?
Related
I am getting following Error while creating Elastic Beanstalk Environment
Command failed on instance. Return code: 1 Output: (TRUNCATED)... ^
/var/app/ondeck/config/environment.rb:5:in <top (required)>'
/opt/rubies/ruby-2.4.3/bin/bundle:23:inload'
/opt/rubies/ruby-2.4.3/bin/bundle:23:in `' Tasks: TOP =>
db:migrate => environment (See full trace by running task with
--trace). Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/12_db_migration.sh failed.
For more detail, check /var/log/eb-activity.log using console or EB
CLI.
In /var/log/eb-activity.log file, I found following Errors -
Tasks: TOP => db:migrate => environment (See full trace by running
task with --trace) (Executor::NonZeroExitStatus)
AppDeployStage0/AppDeployPreHook/12_db_migration.sh] : Activity failed.
AppDeployStage0/AppDeployPreHook] : Activity failed.
AppDeployStage0] : Activity failed.
Application update - CommandCMD-AppDeploy failed
I encountered this same problem when using Elastic Beanstalk with an external Amazon RDS database. Basically, the problem is that the Elastic Beanstalk pre-deployment scripts will attempt to migrate the database before it even exists.
There are two ways I discovered for how to solve this.
The first way is to set the RAILS_SKIP_MIGRATIONS=true environment variable on your app configuration. This should allow you to at least get the codebase deployed. After that, you can use eb ssh to shell into the app, browse to the /var/app/current/ folder, and manually run bundle exec rails db:create and bundle exec rails db:migrate.
Another way to solve the problem is to create an app pre-deploy shell script hook file in the /opt/elasticbeanstalk/hooks/appdeploy/pre/ folder.
I used the /opt/elasticbeanstalk/hooks/appdeploy/pre/12_db_migration.sh file as reference, and here's what I came up with.
Create a file in your project called /.ebextensions/0001_rails_db_create.config, with the following contents:
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/11_create_db.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
set -xe
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_APP_STAGING_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_staging_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
. $EB_SUPPORT_DIR/envvars
RAKE_TASK="db:create"
. $EB_SCRIPT_DIR/use-app-ruby.sh
cd $EB_APP_STAGING_DIR
if su -s /bin/bash -c "bundle exec $EB_SCRIPT_DIR/check-for-rake-task.rb $RAKE_TASK" $EB_APP_USER; then
if [ "$RAILS_SKIP_DB_CREATE" = "true" ]; then
echo "Skipping database creation (RAILS_SKIP_DB_CREATE=true)."
else
su -s /bin/bash -c "leader_only bundle exec rake db:create" $EB_APP_USER
fi
else
echo "No $RAKE_TASK task in Rakefile, skipping database creation."
fi
Commit that file to your git repo and then run eb deploy.
This should create the shell script hook file which will create the rails db if it doesn't exist. The database migration shell script hook file should run immediately afterwards, since its name starts with the number 12.
Once this script is in place, if you ever want to bypass it, you can set the RAILS_SKIP_DB_CREATE=true environment variable on your app.
I'm trying to create a rake task so that I can simply type "rake db:sync" in order to update my local DB to match production.
This solution leverages code provided by the Heroku team here:
Importing and Exporting Heroku Postgres Databases with PG Backups
When I use curl --output /tmp/latest.dump #{url} I'm getting the following error in my latest.dump file:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.</Message><RequestId>421FEFF763870123</RequestId><HostId>vlVr/ihmQiDgYIpdFFkuCgEP8Smvr2ks0wRkf89fJ8NfHfsBb92EVv40Q0NZuQIC</HostId></Error>
Here is the code I'm using.
#lib/tasks/db_sync.rake
namespace :db do
desc 'Pull production db to development'
task :sync => [:backup, :dump, :restore]
task :backup do
Bundler.with_clean_env {
puts 'Backup started...'
system "heroku pg:backups capture --app YOUR_APP_NAME"
puts 'Backup complete!'
}
end
task :dump do
dumpfile = "#{Rails.root}/tmp/latest.dump"
puts 'Fetching url and file...'
Bundler.with_clean_env {
url = `heroku pg:backups public-url --app YOUR_APP_NAME | cat`
system "curl --output #{dumpfile} #{url}"
}
puts 'Fetching complete!'
end
task :restore do
dev = Rails.application.config.database_configuration['development']
dumpfile = "#{Rails.root}/tmp/latest.dump"
puts 'PG_RESTORE on development database...'
system "pg_restore --verbose --clean --no-acl --no-owner -h localhost -U #{dev['username']} -d #{dev['database']} #{dumpfile}"
puts 'PG_RESTORE Complete!'
end
end
Check out the Parity gem. It offers several commands to do the following Heroku Rails tasks easily -
Backup DB's
Restore DB's
Run rails console
Tail logs
Run migrations
Deploy
You're of course primarily looking for the first two.
After installation, it expects that you have two git remote values set named staging and production. development isn't needed as it is assumed to be your local machine.
You can get the git url for the other two environments from your Heroku dashboard -> (your app) -> Settings -> Info
After you have that set up, it's as simple as
production backup
development restore production
The code is pretty simple, so I encourage you to read it. But it's essentially doing exactly what your rake code attempts to do by getting a public URL and restoring it.
I'm trying to download a file (a copy) of our staging DB. I'm using pgbackups following these docs.
I created a backup, than got the public-url using heroku pgbackups:url b462 --app staging-appname, which worked.
Than I run the command curl -o latest.dump heroku pg:backups <public-url> b462 -a staging-appname and get curl: no URL specified!
Where it says <public-url> I've copied the actual long public-url, as in https://s3.amazonaws.com/reallly-long-url-with-acess-key-ect.
What am I missing here?
The following will create the backup:
heroku pg:backups capture --app name-of-the-app
Using the following information printed in terminal
---backup---> b001
In the following command
heroku pg:backups public-url b001 -a name-of-the-app
Will give you the url of the dump
Try just the following to get the latest file immediately after running heroku pgbackups:capture --app=staging-appname:
curl -o latest.dump `heroku pgbackups:url --app=staging-appname`
The b.. is required only if you are looking for a specific older file.
I was also struggling with this doc, and Prakash's suggestion didn't work for me (error complained about not being able to resolve 'heroku' which isn't part of the URL, so the syntax is seems to be bad, at least for the version of the pg or pgbackup gem I'm using.)
I also tried wget, as suggested in the heroku doc, to no avail.
When I'm stumped I try to think it down to the most simple elements, which led me to a far simpler solution than is provided by the doc...
I used the command in Rafael's answer to generate a temporary URL to retrieve the dump file:
heroku pg:backups public-url b001 -a name-of-the-app
Then, instead of plugging the resulting URL into the curl command, I just pasted that URL into Chrome's address bar and downloaded the file using my web browser.
The download was successful, the db file is valid.
With a little more sugar, here an example of shell script to download mysql dump from heroku and to restore it as a local database
backup_name=$1
if [ -z "$backup_name" ]
then
echo "You must supplied heroku backup name (ex: b001)"
return
fi
echo "Loading '$backup_name' database from heroku"
heroku pg:backups public-url $backup_name -a your-app-name > last_url.txt
curl -o latest.dump `cat last_url.txt`
dropdb local-db-name
createdb local-db-name
pg_restore -c -d local-db-name latest.dump
You need to give the heroku backup name as input.
I am aware of the heroku pg:push command which pushes an entire database up to Heroku.
Now that I am launching my product, I would like to be able to push up only a specific table that contains information collected locally without overwriting existing tables (such as users).
Is there a command that enables me to only push specific tables to heroku?
My suggestion is to use PostgreSQL dump/restore capabilities directly using the pg_dump and psql commands.
With pg_dump you can dump a specific table from your local database
$ pg_dump --data-only --table=products sourcedb > products.sql
Then grab the Heroku PostgreSQL connection string from the configs
$ heroku config | grep HEROKU_POSTGRESQL
# example
# postgres://user3123:passkja83kd8#ec2-117-21-174-214.compute-1.amazonaws.com:6212/db982398
and restore the table in the remote database, using the information retrieved from Heroku.
$ psql -h ec2-117-21-174-214.compute-1.amazonaws.com -p 6212 -U user3123 db982398 < products.sql
You will need to customize the -p, -h and -U parameters, as well as the database name. The password will be prompted by psql.
You can also use the pg_restore to filter a dump and restore the table, but I personally prefer psql.
Note that Heroku is recommending the use of PostgreSQL tools in several documentations, such as Importing and Exporting for large data, or whenever the provided CLI commands don't cover specific cases like the one in this question.
I wrote script which extracts DB url from heroku. Then it dumps single tables from production and restores them on development/localhost. Run it like this:
rake production_to_development:run\['users;news;third_table',my-sushi-app\]
Code:
namespace :production_to_development do
task :run, [:tables, :app] => [:environment] do |t, args|
tables = args["tables"].split(';')
database_url = nil
Bundler.with_clean_env { database_url = `heroku config:get DATABASE_URL --app=#{args["app"]}` }
require 'addressable/uri'
uri = Addressable::URI.parse(database_url)
remote_database = uri.path[1,uri.path.length-2] # there is \n at the end of the path!
tables.each do |table|
backup_file = "tmp/#{table}.backup"
#bin_dir = "/Applications/Postgres.app/Contents/Versions/latest/bin"
bin_dir = ""
dump_command = "PGPASSWORD=#{uri.password} #{bin_dir}/pg_dump --file \"#{backup_file}\" --host \"#{uri.host}\" --port \"#{uri.port}\" --username \"#{uri.user}\" --no-password --verbose --format=c --blobs --table \"public.#{table}\" \"#{remote_database}\""
`#{dump_command}`
`psql -U 'root' -d my_table -c 'drop table if exists #{table}'`
`pg_restore -d my_table --no-owner #{backup_file}`
end
end
end
If I understand correctly, you just need a single database table with its locally created data pushed to your Rails production app. Maybe this is a simplistic approach, but you could create a migration for your table and then populate using db/seeds.rb.
After you've populated the seeds.rb file and pushed your repo to heroku:
heroku run rake db:migrate
heroku run rake db:seed
Also, if your local table has a ton of data and you're using Rails 4, check out the seed dump gem: https://github.com/rroblak/seed_dump. This will take your existing db data and map it to the seed format.
I would like to run Heroku toolbelt command on the server in a scheduled way:
e.g.
heroku maintenance:on
# do some other stuff
heroku maintenance:off
As far as I could find, there does not seem to be a way to do this, not even a workaround?
You can use Heroku Scheduler and configure the following command:
curl -s https://s3.amazonaws.com/assets.heroku.com/heroku-client/heroku-client.tgz \
| tar xz && ./heroku-client/bin/heroku maintenance:on -a you-app-name-here
For this to work, you need to add a Config Variable named HEROKU_API_KEY and set its value to the "API Key" value from your Accounts page.