I have a Rails 5 application that uses three different existing databases. No migrations were needed in this application. I want to build db/schema.rb to include all three databases, not just the main database. Executing rake db:schema:dump rebuilds the schema using the main database. I believe there is a way to do this but for some reason the way I've been searching I cannot find anything about this. All of the posts I have found discuss how to use models from different databases, not how to rebuild the schema to include all databases.
I started looking at this again about a month ago. After reviewing several options I found a solution created by #ostinelli that worked really well and easy to implement. This solution creates separate migrations for each database being maintained instead of putting them all in a single schema file. My goal mainly was to have a schema for each database which this accomplishes for me. There are some parts I did not implement because they were not needed.
http://www.ostinelli.net/setting-multiple-databases-rails-definitive-guide/
I created separate database yml files for each database like this:
config/database_stats.yml
development:
adapter: postgresql
encoding: utf8
host: localhost
pool: 10
database: myapp_stats_development
username: postgres
password:
test:
adapter: postgresql
encoding: utf8
host: localhost
pool: 10
database: myapp_stats_test
username: postgres
password:
production:
adapter: postgresql
encoding: utf8
url: <%= ENV["DATABASE_STATS_URL"] %>
pool: <%= ENV["DB_POOL"] || 5 %>
Then I created separate database rake tasks for each database:
lib/tasks/db_stats.rake
task spec: ["stats:db:test:prepare"]
namespace :stats do
namespace :db do |ns|
task :drop do
Rake::Task["db:drop"].invoke
end
task :create do
Rake::Task["db:create"].invoke
end
task :setup do
Rake::Task["db:setup"].invoke
end
task :migrate do
Rake::Task["db:migrate"].invoke
end
task :rollback do
Rake::Task["db:rollback"].invoke
end
task :seed do
Rake::Task["db:seed"].invoke
end
task :version do
Rake::Task["db:version"].invoke
end
namespace :schema do
task :load do
Rake::Task["db:schema:load"].invoke
end
task :dump do
Rake::Task["db:schema:dump"].invoke
end
end
namespace :test do
task :prepare do
Rake::Task["db:test:prepare"].invoke
end
end
# append and prepend proper tasks to all the tasks defined here above
ns.tasks.each do |task|
task.enhance ["stats:set_custom_config"] do
Rake::Task["stats:revert_to_original_config"].invoke
end
end
end
task :set_custom_config do
# save current vars
#original_config = {
env_schema: ENV['SCHEMA'],
config: Rails.application.config.dup
}
# set config variables for custom database
ENV['SCHEMA'] = "db_stats/schema.rb"
Rails.application.config.paths['db'] = ["db_stats"]
Rails.application.config.paths['db/migrate'] = ["db_stats/migrate"]
Rails.application.config.paths['db/seeds'] = ["db_stats/seeds.rb"]
Rails.application.config.paths['config/database'] = ["config/database_stats.yml"]
end
task :revert_to_original_config do
# reset config variables to original values
ENV['SCHEMA'] = #original_config[:env_schema]
Rails.application.config = #original_config[:config]
end
end
Then I created custom migration generators for the additional databases.
lib/generators/stats_migration_generator.rb
require 'rails/generators/active_record/migration/migration_generator'
class StatsMigrationGenerator < ActiveRecord::Generators::MigrationGenerator
source_root File.join(File.dirname(ActiveRecord::Generators::MigrationGenerator.instance_method(:create_migration_file).source_location.first), "templates")
def create_migration_file
set_local_assigns!
validate_file_name!
migration_template #migration_template, "db_stats/migrate/#{file_name}.rb"
end
end
I created database migrations for each database similar to:
rails g stats_migration create_clicks
create db_stats/migrate/20151201191642_create_clicks.rb
rake stats:db:create
rake stats:db:migrate
Then created separate database initializers for the additional databases then modified each model.
config/initializers/db_stats.rb
DB_STATS = YAML::load(ERB.new(File.read(Rails.root.join("config","database_stats.yml"))).result)[Rails.env]
class Click < ActiveRecord::Base
establish_connection DB_STATS
end
I had three databases in my application when I first wrote this question. I'm now adding another one database. I'm so grateful that #ostinelli wrote this. It made my life easier.
Related
how are you? I've been having some troubles when I try to establish a connection to two databases in postres. I will try to describe the scenario:
This is my pg.yml:
default: &default
adapter: postgresql
encoding: unicode
pool: 5
host: localhost
user: xxxxx
password: xxxxx
development:
<<: *default
database: not_legacy
legacy:
<<: *default
database: legacy
This is my Legacy::Base class:
#/models/legacy/base.rb
require 'active_record'
require 'erb'
module Legacy
class Base < ActiveRecord::Base
self.abstract_class = true
conf_contents = File.read('config/pg.yml')
conf_evaluated = ::ERB.new(conf_contents).result
conf = YAML.load(conf_evaluated)
ActiveRecord::Base.establish_connection conf['legacy']
end
end
and this is my NotLegacy::Base class:
#/models/not_legacy/base.rb
require 'active_record'
require 'erb'
module NotLegacy
class Base < ActiveRecord::Base
self.abstract_class = true
conf_contents = File.read('config/pg.yml')
conf_evaluated = ::ERB.new(conf_contents).result
conf = YAML.load(conf_evaluated)
ActiveRecord::Base.establish_connection conf['not_legacy']
end
end
Also, I have two classes that inherit from the classes previously described.
This is Legacy::Company:
#/models/legacy/company.rb
require_relative 'base'
module Legacy
class Company < Base
self.table_name = "company"
end
end
and NotLegacy::Company:
#/models/not_legacy/company.rb
require_relative 'base'
module NotLegacy
class Company < Base
self.table_name = "company"
end
end
Now, if I go to the console and do something like(I'm printing conf value):
irb(main):001:0> load 'app/models/legacy/company.rb'
CONFS: {"adapter"=>"postgresql", "encoding"=>"unicode", "pool"=>5, "host"=>"localhost", "user"=>"xxxxx", "password"=>"xxxxx", "database"=>"legacy"}
=> true
irb(main):002:0> Legacy::Company.count
=> 8
irb(main):003:0> load 'app/models/not_legacy/company.rb'
CONFS: {"adapter"=>"postgresql", "encoding"=>"unicode", "pool"=>5, "host"=>"localhost", "user"=>"xxxxx", "password"=>"xxxxx", "database"=>"not_legacy"}
=> true
irb(main):004:0> NotLegacy::Company.count
=> 1
At this point everything seems to work correctly since in the legacy database there are 8 records for company and in the not_legacy database there is only 1 record. But if I call Legacy::Company again, I get:
irb(main):005:0> Legacy::Company.count
=> 1
irb(main):005:0> NotLegacy::Company.count
=> 1
It seems that the second connection (made to the not_legacy database) is overwriting the first one (the one made to the legacy database).
If anyone of you can explain me why this is happening and how to fix it I will be immensely grateful
Thanks.
First I would like to go with basic about how you are going through code flow,
When you load any class, its code is executed & methods defined, your connection is created through static code written & establish_connection is executed.
Single instant of rails app can have connection to single database
You better to write code to switch through different database to have access for different database connectivity.
Legacy::Company or NotLegacy::Company both models have same table name companies from different databases. But when you connect particular database both model (which got loaded) will point to companies table in current database.
So what is current database mean to both model ?
-> The one which is loaded last time by establish_connection.
So this is little tedious but it is working that if you deal with different databases, you have to keep switching from one database to another one.
on config/database.yml
other_base:
adapter: postgresql
encoding: unicode
pool: 5
host: localhost
user: xxxxx
password: xxxxx
on your model
class YourModel < ActiveRecord::Base
self.establish_connection :other_base
end
Seeking to move all my shared models to an engine which can be included in each of my micro apps.
This engine should provide a model layer to all our legacy data, including:
Model files
Schema files
Migrations (we're following Pivotal Labs' pattern, this isn't the issue)
Model files are being patched in automatically, that's fine.
Schema files are being monkey-patched in using Nikolay Strum's db.rake:
namespace :db do
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
filename = "#{Rails.root}/db/foo_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("foo_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
# like db:test:purge
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs['foo_test']['database'], mysql_creation_options(abcs['foo_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection('foo_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/foo_schema.rb")
end
end
end
We need rake db:create and rake db:schema:load to work,
The db.rake patches only affect db:schema:dump and db:test:load_schema (part of tests_prepare, I assume). I've attempted to patch them into db:schema:load using:
namespace :db do
# Helpers
def mysql_creation_options(config)
#charset = ENV['CHARSET'] || 'utf8'
#collation = ENV['COLLATION'] || 'utf8_unicode_ci'
{:charset => (config['charset'] || #charset), :collation => (config['collation'] || #collation)}
end
def load_schema(schema_name)
abcs = ActiveRecord::Base.configurations
ActiveRecord::Base.connection.recreate_database(abcs[schema_name+'_test']['database'], mysql_creation_options(abcs[schema_name+'_test']))
# like db:test:load_schema
ActiveRecord::Base.establish_connection(schema_name+'_test')
ActiveRecord::Schema.verbose = false
load("#{Rails.root}/db/#{schema_name}_schema.rb")
end
namespace :schema do
# desc 'Dump additional database schema'
task :dump => [:environment, :load_config] do
dump_schema = -> (schema_name) {
filename = "#{Rails.root}/db/#{schema_name}_schema.rb"
File.open(filename, 'w:utf-8') do |file|
ActiveRecord::Base.establish_connection("#{schema_name}_#{Rails.env}")
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
end
}
dump_schema.call('kiddom')
dump_schema.call('kiddom_warehouse')
end
# When loading from schema, load these files, too
task :load => [:environment, :load_config] do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
namespace :test do
# desc 'Purge and load foo_test schema'
task :load_schema do
load_schema('kiddom')
load_schema('kiddom_warehouse')
end
end
end
But this gives me the error NoMethodError: undefined method 'recreate_database' for #<ActiveRecord::ConnectionAdapters::SQLite3Adapter:0x007feb6bb43558>. Apparently, this only works on Oracle-type databases?
What are the Rails commands for the underlying DROP and CREATE DATABASE sql I'm trying to affect for the extra schema.rb's?
You are using SQLite as your database engine. Hope that is what you want to do.
Since you are creating SQLite Database , things differ a bit from other database adapters like MySQLAdpter or Postgress.
In case of MySQL, database has to be created prior to establish a connection by spending "CREATE DATABASE ... " SQL commands. So you must create the database before establishing connection.
But in case of SQLite, since the database reside in a file and one file can contain only one database, there is no separate step to create database. An attempt to establishing a connection to the database itself will cause the database file to be created.
Hence create_database method won't work when using SQLiteAdapter. You can simply remove that line from your code.
You may have a look at the source code for the Rake task db:create
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake
Also, the source code for 'create' method in SQLiteDatabaseTasks. As you can see, it simply calls the establish_connection method
https://github.com/rails/rails/blob/f47b4236e089b07cb683ee9b7ff8b06111a0ec10/activerecord/lib/active_record/railties/databases.rake
is it possible to use 2 types of database in a Rails application?
I am using 2 databases - Postgres to store data which might not change much, and MongoDB
to store data which change dynamically.
is this valid approach? and is it possible to connect between these 2 databases and operate
in a single Rails application?
Please correct, if i am wrong.
regards,
Balan
yes this is possible here is an example from my old code (but here we use mysql for both the DB, but I think you can get an idea)
in database.yml file define two databases
development: &defaults
adapter: mysql
encoding: utf8
database: DB1
enable_call: true
username:
password:
host:
portal_development: &defaults
adapter: mysql
encoding: utf8
database: DB2
enable_call: true
username:
password:
host:
in your models have to base models coupled with above databases
portal_base.rb
class PortalBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "portal_#{Rails.env}"
def self.table_name_prefix
"DB1."
end
end
default_base.rb
class DefaultBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "#{Rails.env}"
def self.table_name_prefix
"DB2."
end
end
and derive your models accordingly
class Client < PortalBase
#code
end
Hope you have an idea now :)
Found solution :
We can use mongoid gem to achieve this.
Steps to install
1) Install mongoid by adding "gem mongoid" in Gemfile and running bundle command
2) Generate mongoid configuration by typing "rails g mongoid:config", this will create a
config file mongoid.yml near database.yml file, you can add the configuration to Mongo server
in this file.
Note : After adding Mongoid, all the models created will be created for MongoDB by default, you can specify --orm option to generate Models for Postgres.
I'm currently trying to perform some tests on a rails app that has two database connections.
As of course I wouldn't like to delete the 2nd database each time I perform a test, I have set up a mechanism to connect to another 2nd-DB if the environment is "test" (see below).
My question: How could I tell rake test to also perform a rake db:test:prepare on the 2nd DB?
This query already touched my problem but couldn't help me completely.
Some code to explain:
My ActiveRecord-Classes that connect to the 2nd db inherit from the following class InformixConnect:
class InformixConnect < ActiveRecord::Base
self.abstract_class = true
case Rails.env
when 'production', 'development'
establish_connection :development_informix
when 'test'
establish_connection :test_informix_dummy
else
raise "Please specify a correct informix Environment."
end
end
like
class RenewalNotify < InformixConnect
set_table_name :renewal_notify
set_primary_key :renewal_notify_id
end
(yes, I know... The schema does not follow the rails conventions. It's a legacy one)
database.yml contains
...
development:
adapter: postgresql
database: rails_development
host: 127.0.0.1
reconnect: true
username: rails
password: guessone
...
development_informix:
adapter: informix
database: SOMETHING
username: yyy
password: yyy
test_informix_dummy:
adapter: sqlite3
database: db/test_informix_dummy.sqlite3
...
# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test: &TEST
adapter: sqlite3
database: db/test.db
verbosity: silent
timeout: 5000
So I'd need to setup the "test_informix_dummy" db each time I perform a rake test
Is it possible to append two databases in rails application? For example, SQLite database is portable. I can download SQLite database from another server. When rails application starts, it mount the database. Can i append all the data from another database to existing database? May be SQLite provide a way for merging databases?
I'm not sure sure what did you mean saying "append databases in application". But you can use 2 different (with different schemes) databases in your application. For example:
config/database.yml
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: project_dev
pool: 5
username: project
password:
test:
adapter: mysql2
encoding: utf8
reconnect: false
database: project_test
pool: 5
username: project
password:
sqlite:
development:
adapter: sqlite3
database: db/development.project.db
pool: 5
timeout: 5000
test:
adapter: sqlite3
database: db/test.project.db
pool: 5
timeout: 5000
Models:
Abstract model for all sqlite models;
uses connection ninja gem
class SqliteModel < ActiveRecord::Base
self.abstract_class = true
use_connection_ninja(:sqlite)
end
Sqlite model
class Book < SqliteModel
set_table_name :Books
set_primary_key :BookID
belongs_to :volume, :foreign_key => :VolumeID
has_many :chapters, :foreign_key => :BookID
end
Mysql model
class Highlight < ActiveRecord::Base
# ...
end
You can even use associations between tables in different databases.
But if you was asking about using 2 databases with the same scheme(i.e. just different data), then my answer is no, it's not possible(I can be wrong though). I think that is a question about replication, synchronization, backups or something similar - DB layer, not application.
Of course you can have 2 same tables in both databases, 2 models - one per database, and then just copy records from one to another. But Rails won't do it automatically.
If you want to "seed" your database with data from an external source, like a central location, your best bet is to:
Provide such data into a easy to iterate format (like CSV, JSON or YAML);
Merge this data into your database on your app's initialization, providing it does not exist.
I do something like this on some projects. I would not, however, perform the merge automatically: I'd rather use the db/seeds.rb file and the rake db:seed task:
if State.count == 0
State.transaction do
CSV.foreach("#{::Rails.root}/db/seed_data/states.csv", 'r') do |row|
code, acronym, name = *row
State.create! code: code, acronym: acronym, nane: name
end
end
end
In this code I load the data from a local file, but you can easily change to a remote file using Net::HTTP.