Sometimes we do a COPY from our Production Database to our Quality Database. When this happens I just create a new database and link it in the Rails Application.
Question: Can you empty an entire postgresql Database including the Relationships and import the new DB?
Problem: When importing the database it does not overwrite the current data / structure / relations ...
Info:
I export the Production database like this:
## Dump without user privileges
pg_dump -x -U database1 -h localhost -O database1 > database1.sql
And normally I import the exported database like this:
## Import
psql -U database2 database2 < database1.sql
I've used the following rake task for a number of years and it's worked well for me. The local:db:abort_if_active_connections dependency isn't strictly necessary but it's nice as otherwise the backup fails since you can't drop a database currently in use.
You'll want to tweak the local:backup:production task's system commands to be whatever you need to get a copy of the database. Then you can just run:
bin/rake local:backup:production
lib/tasks/local/backup.rake
namespace :local do
namespace :backup do
desc 'Backup production and restore to development'
task :production => ['production:db']
namespace :production do
desc 'Backup production database and restore to development'
task :db => ['local:db:abort_if_active_connections', 'db:drop', 'db:create'] do
config = ActiveRecord::Base.configurations[Rails.env]
gz_file = "#{ActiveRecord::Base.configurations['production']['database']}.gz"
puts "Copying production database to temporary file on this machine..."
system "scp user#example.com:/tmp/#{gz_file} /tmp/#{gz_file}"
puts "Recreating local database..."
system "gunzip -c /tmp/#{gz_file} | psql #{config['database']}"
puts "Removing temporary file..."
File.unlink "/tmp/#{gz_file}"
end
end
end
end
lib/tasks/local.rake
namespace :local do
namespace :db do
task :abort_if_active_connections => ['db:load_config'] do
config = ActiveRecord::Base.configurations[Rails.env]
ActiveRecord::Base.establish_connection(config.merge('database' => 'postgres', 'schema_search_path' => 'public'))
version = ActiveRecord::Base.connection.send(:postgresql_version)
if version >= 90200
pid = 'pid'
else
pid = 'procpid'
end
connections = ActiveRecord::Base.connection.select_all("SELECT * FROM pg_stat_activity WHERE pg_stat_activity.datname = '#{config['database']}' AND #{pid} <> #{$$}")
unless connections.empty?
puts
puts "There are active connections to the database '#{config['database']}':"
puts
puts "%-7s %-20s %-16s %-20s %s" % %w[pid usename client_addr application_name backend_start]
connections.each do |c|
puts "%-7s %-20s %-16s %-20s %s" % [c[pid], c['usename'], c['client_addr'], c['application_name'], c['backend_start']]
end
puts
exit 1
end
ActiveRecord::Base.clear_all_connections!
end
end
end
Related
I am familiar with Rails but this is my first time uploading to production. I am able to successfully upload my app to AWS and deploy it. However, every time I do that, I have to ssh into my server and run the necessary rake tasks to clean up my models and fully prep my website. Is there a file like production.rb where you can write a script to be run on every production upload. For instance run all tests and rake tests ? Is there a simple example of a script someone. This is the example of my rake file.
Note: I am using AWS Beanstalk, super easy to deploy, just want to run some post production ready scripts.
This is the rake file I want to run commands of post deployment.
require "#{Rails.root}/app/helpers/application_helper"
include ApplicationHelper
namespace :db do
desc "Generate a new blog post markdown"
task new_article: :environment do
cp 'lib/assets/articles/template.md', "lib/assets/articles/NEW_ARTICLE#{Time.now.strftime("%s")}.md"
puts 'new article created!'
end
task populate: :environment do
Article.destroy_all
if User.count == 0
User.create!(name: "AJ", email: "aj#psychowarfare.com")
end
puts Dir.pwd
a = File.join("lib", "assets", "articles", "*.md")
Dir.glob(a).reject { |name| /.*(template|NEW_ARTICLE).*/ =~ name }.each do |file|
File.open(file, "r") do |f|
contents = f.read
mkdown = Metadown.render(contents)
md = mkdown.metadata
unrendered_content = contents.sub(/^---(\n|.)*---/, '')
#puts unrendered_content
article = Article.create!(title: md["title"],
content: markdown(unrendered_content),
header_image: md["header_image"],
published: md["published"],
useful_links: md["useful_links"],
people_mentioned: md["people_mentioned"],
written_at_date: md["written_at_date"],
timestamp: md["timestamp"],
embedded_link: md["embedded_link"],
user: User.first)
article.add_tag(md["tags"])
puts article.useful_links
puts article.people_mentioned
puts article.header_image
puts article.tags
end
end
puts "Article Count: #{Article.count}"
end
end
For post deployment, you can try the following way.
Create a file in .ebextensions/01_build.config
commands:
create_post_dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_build_app.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
cd /var/app/current/app/
Your-Post-Deploy-Command1
Your-Post-Deploy-Command2
Your-Post-Deploy-Command3
What this config does is:
create the “post” directory if it doesn’t already exist (it won’t by
default) – ignore any errors (such as if the directory already
existed)
deploy the shell script with the appropriate permissions into the right directory
For more details look at the following references: Blog-Article & Stackoverflow-Question
I was wondering how to write a rake task for backing up the default rails database. I tried the following. However, nothing seems to be getting written in the file.
namespace :mockdb do
desc "Back up the database"
task :backup => :environment do
puts "Backing up the database.."
system "sqlite3 .dump > #{dump_path}"
puts "Phew! All data has been backed up!"
end
def dump_path
Rails.root.join('db/mock.sql').to_path
end
end
Apparently the system was not able to find sqlite3. I had to give the installation path. Following is the final snippet
namespace :mockdb do
sqlite_path = "/usr/bin/sqlite3"
sql_file = "db/#{Rails.env}.sqlite3"
desc "Back up the database"
task :backup => :environment do
puts "Backing up the database.."
system "#{sqlite_path} #{sql_file} .dump > #{dump_path}"
puts "All data has been backed up!"
end
def dump_path
Rails.root.join('db/mock.sql').to_path
end
end
Seeking to move all my shared models to an engine which can be included in each of my micro apps.
This engine should provide a model layer to all our legacy data, including:
Model files
Schema files
Migrations (we're following Pivotal Labs' pattern, this isn't the issue)
Model files are being patched in automatically, that's fine.
Schema files are being monkey-patched in using Nikolay Strum's db.rake:
namespace :db do
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
filename = "#{Rails.root}/db/foo_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("foo_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
# like db:test:purge
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs['foo_test']['database'], mysql_creation_options(abcs['foo_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection('foo_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/foo_schema.rb")
end
end
end
We need rake db:create and rake db:schema:load to work,
The db.rake patches only affect db:schema:dump and db:test:load_schema (part of tests_prepare, I assume). I've attempted to patch them into db:schema:load using:
namespace :db do
# Helpers
def mysql_creation_options(config)
#charset = ENV['CHARSET'] || 'utf8'
#collation = ENV['COLLATION'] || 'utf8_unicode_ci'
{:charset => (config['charset'] || #charset), :collation => (config['collation'] || #collation)}
end
def load_schema(schema_name)
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs[schema_name+'_test']['database'], mysql_creation_options(abcs[schema_name+'_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection(schema_name+'_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/#{schema_name}_schema.rb")
end
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
dump_schema = -> (schema_name) {
filename = "#{Rails.root}/db/#{schema_name}_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("#{schema_name}_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
}
dump_schema.call('kiddom')
dump_schema.call('kiddom_warehouse')
end
# When loading from schema, load these files, too
task :load => [:environment, :load_config] do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
end
But this gives me the error NoMethodError: undefined method 'recreate_database' for #<ActiveRecord::ConnectionAdapters::SQLite3Adapter:0x007feb6bb43558>. Apparently, this only works on Oracle-type databases?
What are the Rails commands for the underlying DROP and CREATE DATABASE sql I'm trying to affect for the extra schema.rb's?
You are using SQLite as your database engine. Hope that is what you want to do.
Since you are creating SQLite Database , things differ a bit from other database adapters like MySQLAdpter or Postgress.
In case of MySQL, database has to be created prior to establish a connection by spending "CREATE DATABASE ... " SQL commands. So you must create the database before establishing connection.
But in case of SQLite, since the database reside in a file and one file can contain only one database, there is no separate step to create database. An attempt to establishing a connection to the database itself will cause the database file to be created.
Hence create_database method won't work when using SQLiteAdapter. You can simply remove that line from your code.
You may have a look at the source code for the Rake task db:create
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake
Also, the source code for 'create' method in SQLiteDatabaseTasks. As you can see, it simply calls the establish_connection method
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake
I run task for importing dates to DB Mysql. After running task rake db:restore displayed error:
cannot open %=ENV[C9_USER]%: No such file
What wrong?
task:
require 'yaml'
namespace :db do
def backup_prep
#directory = File.join(Rails.root, 'db', 'backup')
#db = YAML::load( File.open( File.join(Rails.root, 'config', 'database.yml') ) )[ Rails.env ]
#db_params = "-u #{#db['username']} #{#db['database']}"
#db_params = "-p#{#db['password']} #{#db_params}" unless #db['password'].blank?
end
desc 'Backup database by mysqldump'
task :backup => :environment do
backup_prep
FileUtils.mkdir #directory unless File.exists?(#directory)
file = File.join( #directory, "#{RAILS_ENV}_#{DateTime.now.to_s}.sql" )
command = "mysqldump #{#db_params} | gzip > #{file}.gz" #--opt --skip-add-locks
puts "dumping to #{file}..."
# p command
exec command
end
desc "restore most recent mysqldump (from db/backup/*.sql.*) into the current environment's database."
task :restore => :environment do
unless ENV['RAILS_ENV']=='development'
puts "Are you sure you want to import into #{ENV['RAILS_ENV']}?! [y/N]"
return unless STDIN.gets =~ /^y/i
end
backup_prep
wildcard = File.join( #directory, ENV['FILE'] || "#{ENV['FROM']}*.sql*" )
puts file = `ls -t #{wildcard} | head -1`.chomp # default to file, or most recent ENV['FROM'] or just plain most recent
if file =~ /\.gz(ip)?$/
command = "gunzip < #{file} | mysql #{#db_params}"
else
command = "mysql #{#db_params} < #{file}"
end
p command
puts "please wait, this may take a minute or two..."
exec command
end
end
dump db store in path workspace/db/backup/db.sql
This comment says:
db:drop can run without failure when db does not exist
This is exactly what I need: I need to run db:drop but without throwing an exception or halt my whole process if the database doesn't exist, just delete the database if it exists or do nothing.
How can I do that? how can I tell db:drop not to destroy my life if the database doesn't exist?
This is the code I'm experiencing the problem with (it it help):
namespace :db do
task import: :environment do
Rake::Task["db:drop"].invoke # If the database doesn't already exist, the whole import process terminates!
Rake::Task["db:create"].invoke
Rake::Task["db:migrate"].invoke
database_config = Rails.configuration.database_configuration[Rails.env]
system "psql --username=#{database_config['username']} #{database_config['database']} < PostgreSQL.sql"
end
end
Why can't you go for simple exception handling
namespace :db do
task import: :environment do
begin
Rake::Task["db:drop"].invoke # If the database doesn't already exist, the whole import process terminates!
Rake::Task["db:create"].invoke
Rake::Task["db:migrate"].invoke
database_config = Rails.configuration.database_configuration[Rails.env]
system "psql --username=#{database_config['username']} #{database_config['database']} < PostgreSQL.sql"
rescue Exception => e
p "The Exception is #{e.message}"
end
end
end