How to load multiple schemas into a Rails engine or app? - ruby-on-rails

Seeking to move all my shared models to an engine which can be included in each of my micro apps.
This engine should provide a model layer to all our legacy data, including:
Model files
Schema files
Migrations (we're following Pivotal Labs' pattern, this isn't the issue)
Model files are being patched in automatically, that's fine.
Schema files are being monkey-patched in using Nikolay Strum's db.rake:
namespace :db do
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
filename = "#{Rails.root}/db/foo_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("foo_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
# like db:test:purge
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs['foo_test']['database'], mysql_creation_options(abcs['foo_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection('foo_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/foo_schema.rb")
end
end
end
We need rake db:create and rake db:schema:load to work,
The db.rake patches only affect db:schema:dump and db:test:load_schema (part of tests_prepare, I assume). I've attempted to patch them into db:schema:load using:
namespace :db do
# Helpers
def mysql_creation_options(config)
#charset = ENV['CHARSET'] || 'utf8'
#collation = ENV['COLLATION'] || 'utf8_unicode_ci'
{:charset => (config['charset'] || #charset), :collation => (config['collation'] || #collation)}
end
def load_schema(schema_name)
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs[schema_name+'_test']['database'], mysql_creation_options(abcs[schema_name+'_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection(schema_name+'_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/#{schema_name}_schema.rb")
end
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
dump_schema = -> (schema_name) {
filename = "#{Rails.root}/db/#{schema_name}_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("#{schema_name}_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
}
dump_schema.call('kiddom')
dump_schema.call('kiddom_warehouse')
end
# When loading from schema, load these files, too
task :load => [:environment, :load_config] do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
end
But this gives me the error NoMethodError: undefined method 'recreate_database' for #<ActiveRecord::ConnectionAdapters::SQLite3Adapter:0x007feb6bb43558>. Apparently, this only works on Oracle-type databases?
What are the Rails commands for the underlying DROP and CREATE DATABASE sql I'm trying to affect for the extra schema.rb's?

You are using SQLite as your database engine. Hope that is what you want to do.
Since you are creating SQLite Database , things differ a bit from other database adapters like MySQLAdpter or Postgress.
In case of MySQL, database has to be created prior to establish a connection by spending "CREATE DATABASE ... " SQL commands. So you must create the database before establishing connection.
But in case of SQLite, since the database reside in a file and one file can contain only one database, there is no separate step to create database. An attempt to establishing a connection to the database itself will cause the database file to be created.
Hence create_database method won't work when using SQLiteAdapter. You can simply remove that line from your code.
You may have a look at the source code for the Rake task db:create
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake
Also, the source code for 'create' method in SQLiteDatabaseTasks. As you can see, it simply calls the establish_connection method
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake

Related

Capistrano 3.0 Rails 5.0.0 database custom task

I have deployed a Rails app on an Ubuntu server with Capistrano. However I am trying to run a custom task I had created and is in 'lib/tasks'. I tried to make it work by executing
cap production db:views
As if it was a custom task, but obviously it did not worked
cap aborted!
Don't know how to build task 'db:views'
The file is sql_views.rake, this task is for create views in the database
namespace :db do
desc "Update and create SQL views"
task :views => :environment do
Dir["#{Rails.root}/db/sql_views/*.sql"].each do |file_name|
STDERR.puts "Applying the SQL view at #{file_name}"
source_file = File.new(file_name, 'r')
if source_file and (sql_content = source_file.read)
ActiveRecord::Base.transaction do
# Each statement ends with a semicolon followed by a newline.
sql_lines = sql_content.split(/;[ \t]*$/)
if sql_lines.respond_to?(:each)
sql_lines.each do |line|
ActiveRecord::Base.connection.execute "#{line};"
end
end
end # transaction
end
end # Dir.each
end # task
end
If you used cap install to generate your Capfile, the following lines should be present:
# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob("lib/capistrano/tasks/*.rake").each { |r| import r }
So it's looking in lib/capistrano/tasks for custom tasks.
Move lib/tasks/sql_views.rake to lib/capistrano/tasks/sql_views.rake
Or just import each rake task:
import 'lib/tasks/sql_views.rake'

Is it possible to use Rails rake tasks on secondary databases?

If you are using multiple databases (e.g. to shard tables across databases) for a given environment is it possible to use the built in Rails rake tasks to manipulate the databases beyond the primary one for said environment?
e.g. if I use specific connection details for a set of models, can I use rake db:create to create said database?
If so, how?
I've actually overwritten the built-in Rails rake tasks in order to manipulate a second database. The neat thing is that it totally works with the original rake tasks as well.
This is what I did... credit to Nikolay Sturm, as I based my rake task off of his blog.
I added new database settings in my database.yml, as detailed in the blog I linked.
You also create a base model that has these database settings:
class BaseModel < ActiveRecord::Base
establish_connection "database_connection_name"
self.abstract_class = true #you want to make this an abstract class because you don't have a table for this
end
You then have any model that connects to this secondary database inherit from your BaseModel.
In your rake task, you set it up as follows (keep in mind that this is for a Postgres connection.)
namespace :db do
desc 'create database'
task create: :environment do
connection_config = BaseModel.connection_config #rename BaseModel to name of your model
ActiveRecord::Base.establish_connection connection_config.merge('database => 'postgres') #you don't need this bit if you're not using Postgres
ActiveRecord::Base.connection.create_database connection_config[:database], connection_config
ActiveRecord::Base.establish_connection connection_config
end
desc 'migrate database'
task migrate: :environment do
ActiveRecord::Base.establish_connection BaseModel.connection_config
ActiveRecord::Migrator.migrate('db/migrate/')
end
desc 'drop database'
task drop: :environment do
connection_config = BaseModel.connection_config
ActiveRecord::Base.establish_connection connection_config.merge('database' => 'postgres')
ActiveRecord::Base.connection.drop_database connection_config[:database]
end
Now, if you do a rake -T in your project, you should be able to see the your descriptions for the rake tasks instead of the default. When you run them, it will run the db:create, db:migrate, and db:drop for both your default and your secondary database. Hope this helps!

Rake task create not getting Rails.env in db:create for ActiveRecord>=4.05

For ActiveRecord 3.2.18, in /lib/active_record/railties/databases.rake:
task :load_config do
ActiveRecord::Base.configurations = Rails.application.config.database_configuration
ActiveRecord::Migrator.migrations_paths = Rails.application.paths['db/migrate'].to_a
if defined?(ENGINE_PATH) && engine = Rails::Engine.find(ENGINE_PATH)
if engine.paths['db/migrate'].existent
ActiveRecord::Migrator.migrations_paths += engine.paths['db/migrate'].to_a
end
end
end
desc 'Create the database from DATABASE_URL or config/database.yml for the current Rails.env (use db:create:all to create all dbs in the config)'
task :create => [:load_config, :rails_env] do
if ENV['DATABASE_URL']
create_database(database_url_config)
else
configs_for_environment.each { |config| create_database(config) }
ActiveRecord::Base.establish_connection(configs_for_environment.first)
end
end
For ActiveRecord 4.0.5+, in /lib/active_record/railties/databases.rake:
task :load_config do
ActiveRecord::Base.configurations = ActiveRecord::Tasks::DatabaseTasks.database_configuration || {}
ActiveRecord::Migrator.migrations_paths = ActiveRecord::Tasks::DatabaseTasks.migrations_paths
end
desc 'Create the database from DATABASE_URL or config/database.yml for the current Rails.env (use db:create:all to create all dbs in the config)'
task :create => [:load_config] do
if ENV['DATABASE_URL']
ActiveRecord::Tasks::DatabaseTasks.create_database_url
else
ActiveRecord::Tasks::DatabaseTasks.create_current
end
end
When I call bundle exec rake db:create from a Rakefile (for a gem I'm testing), ActiveRecord 3.2.18, ActiveRecord::Base.configurations gets the information it needs from my test/config/database.yml file via Rails.application.config.database_configuration. But when calling db:create using ActiveRecord 4.0.5+, ActiveRecord::AdapterNotSpecified: database configuration does not specify adapter error. It doesn't matter whether I call it with RAILS_ENV=some_environment. How do I give ActiveRecord 4.0.5+ the database configuration it needs without monkey patching it? The ideal solution is to somehow do it my Rakefile.
I had the same problem, and I "fixed" with a monkey patch:
in your Rakefile, after you imported the active_record, I did the following:
# hack to make it works with sqlite3
module Rails
def self.root
File.dirname(__FILE__)
end
def self.env
"development"
end
end
for sure you can do whatever you want inside your "self.env" method.

Geocoder ruby gem on demand?

I have a large database with 4+ million addresses/records.
The rake command (below) worked fine when the database was a small test set, but now with the large database it just simply stalls.
rake geocode:all CLASS=YourModel
2 questions:
1. Is there any simple method to have geocoder code a null/nil lat and long when the records are called (on the fly). I have a feeling that this would be hard.
2. Anyone else have problems with geocode-ing a large dataset and using the rake command?
Thanks!
Update:
I create pull request based on this answer and now you can use batch in geocoder:
rake geocode:all CLASS=YourModel SLEEP=0.25 BATCH=100
I would use this solution for large database, i take it from geocoder gem rake task:
You can refine this for your needs.
Some example create rake task:
namespace :geocode_my_data do
desc "Geocode all objects in my databse."
task all: :environment do
klass = User
klass.where(geocoded: false).find_each(limit: 100) do |obj|
obj.geocode; obj.save
end
end
end
$> rake geocode_my_data:all
Used below code and put it into my lib/tasks folder as geocode_my_data.rake
To run:
rake geocode:all CLASS=YourModel
Works great!
namespace :geocode_my_data do
desc "Geocode all objects without coordinates."
task :all => :environment do
class_name = ENV['CLASS'] || ENV['class']
sleep_timer = ENV['SLEEP'] || ENV['sleep']
raise "Please specify a CLASS (model)" unless class_name
klass = class_from_string(class_name)
klass.not_geocoded.find_each(batch_size: 100) do |obj|
obj.geocode; obj.save
sleep(sleep_timer.to_f) unless sleep_timer.nil?
end
end
end
##
# Get a class object from the string given in the shell environment.
# Similar to ActiveSupport's +constantize+ method.
#
def class_from_string(class_name)
parts = class_name.split("::")
constant = Object
parts.each do |part|
constant = constant.const_get(part)
end
constant
end

schema.sql not creating even after setting schema_format = :sql

I want to create schema.sql instead of schema.rb. After googling around I found that it can be done by setting sql schema format in application.rb. So I set following in application.rb
config.active_record.schema_format = :sql
But if I set schema_format to :sql, schema.rb/schema.sql is not created at all. If I comment the line above it creates schema.rb but I need schema.sql. I am assuming that it will have database structure dumped in it and
I know that the database structure can be dumped using
rake db:structure:dump
But I want it to be done automatically when database is migrated.
Is there anything I am missing or assuming wrong ?
Five months after the original question the problem still exists. The answer is that you did everything correctly, but there is a bug in Rails.
Even in the guides it looks like all you need is to change the format from :ruby to :sql, but the migrate task is defined like this (activerecord/lib/active_record/railties/databases.rake line 155):
task :migrate => [:environment, :load_config] do
ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
ActiveRecord::Migrator.migrate(ActiveRecord::Migrator.migrations_paths, ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
db_namespace["schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
end
As you can see, nothing happens unless the schema_format equals :ruby.
Automatic dumping of the schema in SQL format was working in Rails 1.x. Something has changed in Rails 2, and has not been fixed.
The problem is that even if you manage to create the schema in SQL format, there is no task to load this into the database, and the task rake db:setup will ignore your database structure.
The bug has been noticed recently: https://github.com/rails/rails/issues/715 (and issues/715), and there is a patch at https://gist.github.com/971720
You may want to wait until the patch is applied to Rails (the edge version still has this bug), or apply the patch yourself (you may need to do it manually, since line numbers have changed a little).
Workaround:
With bundler it's relatively hard to patch the libraries (upgrades are so easy, that they are done very often and the paths are polluted with strange numbers - at least if you use edge rails ;-), so, instead of patching the file directly, you may want to create two files in your lib/tasks folder:
lib/tasks/schema_format.rake:
import File.expand_path(File.dirname(__FILE__)+"/schema_format.rb")
# Loads the *_structure.sql file into current environment's database.
# This is a slightly modified copy of the 'test:clone_structure' task.
def db_load_structure(filename)
abcs = ActiveRecord::Base.configurations
case abcs[Rails.env]['adapter']
when /mysql/
ActiveRecord::Base.establish_connection(Rails.env)
ActiveRecord::Base.connection.execute('SET foreign_key_checks = 0')
IO.readlines(filename).join.split("\n\n").each do |table|
ActiveRecord::Base.connection.execute(table)
end
when /postgresql/
ENV['PGHOST'] = abcs[Rails.env]['host'] if abcs[Rails.env]['host']
ENV['PGPORT'] = abcs[Rails.env]['port'].to_s if abcs[Rails.env]['port']
ENV['PGPASSWORD'] = abcs[Rails.env]['password'].to_s if abcs[Rails.env]['password']
`psql -U "#{abcs[Rails.env]['username']}" -f #{filename} #{abcs[Rails.env]['database']} #{abcs[Rails.env]['template']}`
when /sqlite/
dbfile = abcs[Rails.env]['database'] || abcs[Rails.env]['dbfile']
`sqlite3 #{dbfile} < #{filename}`
when 'sqlserver'
`osql -E -S #{abcs[Rails.env]['host']} -d #{abcs[Rails.env]['database']} -i #{filename}`
# There was a relative path. Is that important? : db\\#{Rails.env}_structure.sql`
when 'oci', 'oracle'
ActiveRecord::Base.establish_connection(Rails.env)
IO.readlines(filename).join.split(";\n\n").each do |ddl|
ActiveRecord::Base.connection.execute(ddl)
end
when 'firebird'
set_firebird_env(abcs[Rails.env])
db_string = firebird_db_string(abcs[Rails.env])
sh "isql -i #{filename} #{db_string}"
else
raise "Task not supported by '#{abcs[Rails.env]['adapter']}'"
end
end
namespace :db do
namespace :structure do
desc "Load development_structure.sql file into the current environment's database"
task :load => :environment do
file_env = 'development' # From which environment you want the structure?
# You may use a parameter or define different tasks.
db_load_structure "#{Rails.root}/db/#{file_env}_structure.sql"
end
end
end
and lib/tasks/schema_format.rb:
def dump_structure_if_sql
Rake::Task['db:structure:dump'].invoke if ActiveRecord::Base.schema_format == :sql
end
Rake::Task['db:migrate' ].enhance do dump_structure_if_sql end
Rake::Task['db:migrate:up' ].enhance do dump_structure_if_sql end
Rake::Task['db:migrate:down'].enhance do dump_structure_if_sql end
Rake::Task['db:rollback' ].enhance do dump_structure_if_sql end
Rake::Task['db:forward' ].enhance do dump_structure_if_sql end
Rake::Task['db:structure:dump'].enhance do
# If not reenabled, then in db:migrate:redo task the dump would be called only once,
# and would contain only the state after the down-migration.
Rake::Task['db:structure:dump'].reenable
end
# The 'db:setup' task needs to be rewritten.
Rake::Task['db:setup'].clear.enhance(['environment']) do # see the .clear method invoked?
Rake::Task['db:create'].invoke
Rake::Task['db:schema:load'].invoke if ActiveRecord::Base.schema_format == :ruby
Rake::Task['db:structure:load'].invoke if ActiveRecord::Base.schema_format == :sql
Rake::Task['db:seed'].invoke
end
Having these files, you have monkeypatched rake tasks, and you still can easily upgrade Rails. Of course, you should monitor the changes introduced in the file activerecord/lib/active_record/railties/databases.rake and decide whether the modifications are still necessary.
I'm using rails 2.3.5 but this may apply to 3.0 as well:
rake db:structure:dump does the trick for me.
It's possible you need to delete the schema.rb for the schema.sql to be created.

Resources