I followed the
multiple databases with ActiveRecord guide to configure my development environment. I have 3 databases, 2 databases are primary and one is supposed to be a read-only replica.
My issue is that the read replica database is not being populated I have verified that data is correctly split among the 2 primary databases, I have 2 database schema files, and I do not have any errors in the application when doing get requests. The read replica database is empty however.
My thought was maybe something was wrong with my config for the automatic connection switching between read and write databases? The guide says that you need to setup the read-only replica database manually but doesn't give any instructions. I'm using Postgresql and Rails 7.0.3.
I had previously assumed that Rails would copy the data from the primary database to the replica but now that I'm typing this I'm wondering if that data replication is to be manually setup in Postgres somehow?
I'm not sure if both the primary_abstract_class and self.abstract_class = true are needed in application_record? Rails 7 adds the primary_abstract_class by default and the multi-database guide tells you to add self.abstract_class = true
app/models/application_record.rb
class ApplicationRecord < ActiveRecord::Base
primary_abstract_class
self.abstract_class = true
connects_to database: { writing: :primary, reading: :primary }
end
app/models/offices_record.rb
class OfficesRecord < ApplicationRecord
self.abstract_class = true
# this isn't the name of the actual databases, but rather the primary/replica key in database.yml
connects_to database: { writing: :offices, reading: :offices_replica }
end
config/database.yml
development:
primary:
<<: *default
database: myapp_development
migrations_paths: db/migrate
offices:
<<: *default
database: myapp_offices_development
migrations_paths: db/offices_migrate
# replica user's database permissions should be set to read-only.
offices_replica:
<<: *default
database: myapp_offices_replica_development
username: read_only_user
password: R34dOnlyUs3rP#ssw0rd
replica: true
config/initializers/multi_db.rb
Rails.application.configure do
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver
config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session
end
The office class inherits from OfficesRecord instead of ApplicationRecord because it is in the 2nd primary database.
app/models/office.rb
class Office < OfficesRecord
end
How I configured Postgres
In Postgres a database backup of the myapp_offices_development database was taken (with no data) and restored over a new database to create the replica called myapp_offices_replica_development. I then ran the below to create a read-only user to access the read-only replica.
CREATE ROLE readaccess;
GRANT CONNECT ON DATABASE myapp_offices_replica_development TO readaccess;
GRANT USAGE ON SCHEMA public TO readaccess;
CREATE USER read_only_user WITH PASSWORD ‘R34dOnlyUs3rP#ssw0rd';
GRANT readaccess TO read_only_user;
I'm trying to figure out how to populate the database myapp_offices_replica_development with data from myapp_offices_development. Does Rails auto-magically handle this?
Related
I am trying to build an app which will use one of latest Rails 6.1 features: multiple databases - you can read in sources at the bottom.
I would like to implement multitenancy by being able to switch between databases based on request domain.
Each club would have his own database, and one primary database would store app specific data like Users, Clubs.
I created two types of records: GlobalRecord using primary database and ShardRecord using horizontal shards databases.
And also tried to use around_action to select current club database.
# config/database.yml
default: &default
adapter: postgresql
encoding: unicode
development:
primary:
<<: *default
database: primary
migrations_paths: db/migrate
club_1:
<<: *default
database: club_1
migrations_paths: db/shard_migrate
club_2:
<<: *default
database: club_2
host: 1.1.1.1
username: deploy
password: pass
migrations_paths: db/shard_migrate
class Club < GlobalRecord; end
class GlobalRecord < ActiveRecord::Base
self.abstract_class = true
connects_to shards: {
club_1: { writing: :primary, reading: :primary },
club_2: { writing: :primary, reading: :primary }
}
end
class MemberRecord < ShardRecord; end
class ShardRecord < ActiveRecord::Base
self.abstract_class = true
connects_to shards: {
club_1: { writing: :club_1, reading: :club_1 },
club_2: { writing: :club_2, reading: :club_2 }
}
end
class ApplicationController < ActionController::API
before_action :set_club
around_action :connect_to_shard
private
def set_club
#club = SelectClubByDomain.new(request).slug
end
def connect_to_shard
ActiveRecord::Base.connected_to(role: :writing, shard: #club.slug) do
yield
end
end
end
I have few design questions/issues I would like to ask YOU about:
I would like to set GlobalRecord with connects_to database: { writing: :primary } but because I added around_action
I have to set it as above.
Is it bad design?
Can I refactor it so I could always call GlobalRecord without ActiveRecord::Base.connected_to(role: :writing, shard: :club_1) block?
Tried to use ActiveRecord::Base.connected_to_many(GlobalRecord, ShardRecord, role: :writing, shard: #club_slug.to_sym) in around_action but on request(clubs#index => [Club.all]) I get an error:
ActiveRecord::ConnectionNotEstablished (No connection pool for 'GlobalRecord' found for the 'club_1' shard.)
On production server on start I see this warning, how to fix it?
=> Run `bin/rails server --help` for more startup options
Failed to define attribute methods because of ActiveRecord::ConnectionNotEstablished: ActiveRecord::ConnectionNotEstablished
Sources:
https://guides.rubyonrails.org/active_record_multiple_databases.html
https://www.freshworks.com/horizontal-sharding-in-a-multi-tenant-app-with-rails-61-blog/
https://api.rubyonrails.org/classes/ActiveRecord/ConnectionHandling.html#method-i-connected_to_many
In general it's absolutely fine to switch connections with an around filter, we have been doing this for years (even before Rails 6) in my previous company.
Btw: did you see that the guide is mentioning your posted exception?
Note that connected_to with a role will look up an existing connection and switch using the connection specification name. This means that if you pass an unknown role like connected_to(role: :nonexistent) you will get an error that says ActiveRecord::ConnectionNotEstablished (No connection pool for 'ActiveRecord::Base' found for the 'nonexistent' role.)
https://guides.rubyonrails.org/active_record_multiple_databases.html#using-manual-connection-switching
I am building a rails app to store many different, user-supplied database connections and execute arbitrary sql against each of them. I am representing each database connection string as an instance of a 'connection' model and want to be able to write a method to query the database represented by each connection, ideally using the activerecord ORM. However, my code as written overwrites the database connection for the entire connections table when I use the establish_connection method in the following code. How would you advise me to change the code to query an arbitrary database, without overwriting the connection for the whole Connections table?
class Connection < ActiveRecord::Base
validates_presence_of :host, :port, :db_name, :db_user, :db_password, :db_type
def connect
self.connection = ActiveRecord::Base.establish_connection(
adapter: self.db_type,
host: self.host,
database: self.db_name,
username: self.db_user,
password: self.db_password
)
end
end
A good way to do that is making a model for each database connection you need, and then make other models as subclasses of them. So, for example:
Define all the needed connections in database.yml (per-environment)
# DB 1
development:
adapter: mysql2
encoding: utf8
database: db_1
username: ****
password: ****
host: ********
pool: 5
...
# DB 2
db2_development:
adapter: mysql2
encoding: utf8
database: db_2
username: ****
password: ****
host: ********
pool: 5
...
# Same for production (and/or other environments)
production:
...
db2_production:
...
Define a "master" model for each database, which inherits from ActiveRecord::Base
# DB1
class DB1 < ActiveRecord::Base
self.abstract_class = true
end
# DB2
class DB2 < ActiveRecord::Base
self.abstract_class = true
establish_connection "db2_#{Rails.env}"
end
...
Now define all the database-specific models as subclasses of the models defined above, in this way:
# DB1 specific model
class DB1_model < DB1
# model logic here
end
# DB2 specific model
class DB2_model < DB2
# model logic here
end
...
And you're good to go.
In this way, you can connect to N databases in different environments (usually development, staging, preprod and production, but they may be different in your case).
Also, remeber that Rails will manage a pool of SQL connections for each database.
In the example above, Rails will open a maximum of 5 connections for each database, so total will be 10 (for a single instance of your application).
If you use Phusion Passenger or Unicorn, and spawn 8 application instances, total SQL connections will be (at maximum) 10*8 = 80.
I have a ruby on rails application working fine and connected to a database. Now i want to connect to a different database from the same application. The data model can be exactly the same. In fact if i connect to the different database the application works fine. However I want to connect to two different databases. Is it possible in ruby on rails?
For multiple database connection, you need to add the following codes to the database.yml file. Here, I am giving the example of connecting two databases from a rails application
config/database.yml
development:
adapter: mysql2
database: db1_dev
username: root
password: xyz
host: localhost
development_sec:
adapter: mysql2
database: db2_dev
username: root
password: xyz
host: localhost
production:
adapter: mysql2
database: db1_prod
username: root
password: xyz
host: your-production-ip
production_sec:
adapter: mysql2
database: db2_prod
username: root
password: xyz
host: your-production-ip
Here I have used two databases for the development and production environment.
Now we need to connect the model to databases. When you are running your application in development and production mode, all the models will be mapped through the development and production db parameters those been mentioned in your database.yml. So for some model we need to connect to other database.
Lets assume that, we have two models User and Category. The users table is in db1_dev and db1_prod, the categories table in db2_dev and db2_prod.
Category model
class Category < ActiveRecord::Base
establish_connection "#{Rails.env}_sec".to_sym
end
Similarly, when you adding the new migration for the second database, need to add following code to it.
class CreateRewards < ActiveRecord::Migration
def connection
ActiveRecord::Base.establish_connection("#{Rails.env}_sec".to_sym).connection
end
def change
# your code goes here.
end
end
Hope it will work for you :) .
Use establish_connection to switch to a different database:
ActiveRecord::Base.establish_connection(
:adapter => "mysql",
:host => "localhost",
:username => "myuser",
:password => "mypass",
:database => "somedatabase"
)
You can also pass a preconfigured environment from database.yml like so:
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['other_env'])
You can also set it for a specific model:
MyClass.establish_connection(...)
You might like to note that as of Rails 6 (2019), Rails has support for multiple primary databases!
https://guides.rubyonrails.org/active_record_multiple_databases.html
The database.yml file will now look something like this:
development:
primary:
database: primary_db
user: root
primary_replica:
database: primary_db
user: ro_user
replica: true
animals:
database: my_animals_db
user: root
migrations_path: db/animals_migrate
animals_replica:
database: my_animals_db
user: ro_user
replica: true
And then it's as simple as specifying in your model files:
class AnimalsModel < ApplicationRecord
self.abstract_class = true
connects_to database: { writing: :animals_primary, reading: :animals_replica }
end
class Dog < AnimalsModel
# connected to both the animals_primary db for writing and the animals_replica for reading
end
(These examples were taken from this helpful tutorial.)
is it possible to use 2 types of database in a Rails application?
I am using 2 databases - Postgres to store data which might not change much, and MongoDB
to store data which change dynamically.
is this valid approach? and is it possible to connect between these 2 databases and operate
in a single Rails application?
Please correct, if i am wrong.
regards,
Balan
yes this is possible here is an example from my old code (but here we use mysql for both the DB, but I think you can get an idea)
in database.yml file define two databases
development: &defaults
adapter: mysql
encoding: utf8
database: DB1
enable_call: true
username:
password:
host:
portal_development: &defaults
adapter: mysql
encoding: utf8
database: DB2
enable_call: true
username:
password:
host:
in your models have to base models coupled with above databases
portal_base.rb
class PortalBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "portal_#{Rails.env}"
def self.table_name_prefix
"DB1."
end
end
default_base.rb
class DefaultBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "#{Rails.env}"
def self.table_name_prefix
"DB2."
end
end
and derive your models accordingly
class Client < PortalBase
#code
end
Hope you have an idea now :)
Found solution :
We can use mongoid gem to achieve this.
Steps to install
1) Install mongoid by adding "gem mongoid" in Gemfile and running bundle command
2) Generate mongoid configuration by typing "rails g mongoid:config", this will create a
config file mongoid.yml near database.yml file, you can add the configuration to Mongo server
in this file.
Note : After adding Mongoid, all the models created will be created for MongoDB by default, you can specify --orm option to generate Models for Postgres.
Is it possible to append two databases in rails application? For example, SQLite database is portable. I can download SQLite database from another server. When rails application starts, it mount the database. Can i append all the data from another database to existing database? May be SQLite provide a way for merging databases?
I'm not sure sure what did you mean saying "append databases in application". But you can use 2 different (with different schemes) databases in your application. For example:
config/database.yml
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: project_dev
pool: 5
username: project
password:
test:
adapter: mysql2
encoding: utf8
reconnect: false
database: project_test
pool: 5
username: project
password:
sqlite:
development:
adapter: sqlite3
database: db/development.project.db
pool: 5
timeout: 5000
test:
adapter: sqlite3
database: db/test.project.db
pool: 5
timeout: 5000
Models:
Abstract model for all sqlite models;
uses connection ninja gem
class SqliteModel < ActiveRecord::Base
self.abstract_class = true
use_connection_ninja(:sqlite)
end
Sqlite model
class Book < SqliteModel
set_table_name :Books
set_primary_key :BookID
belongs_to :volume, :foreign_key => :VolumeID
has_many :chapters, :foreign_key => :BookID
end
Mysql model
class Highlight < ActiveRecord::Base
# ...
end
You can even use associations between tables in different databases.
But if you was asking about using 2 databases with the same scheme(i.e. just different data), then my answer is no, it's not possible(I can be wrong though). I think that is a question about replication, synchronization, backups or something similar - DB layer, not application.
Of course you can have 2 same tables in both databases, 2 models - one per database, and then just copy records from one to another. But Rails won't do it automatically.
If you want to "seed" your database with data from an external source, like a central location, your best bet is to:
Provide such data into a easy to iterate format (like CSV, JSON or YAML);
Merge this data into your database on your app's initialization, providing it does not exist.
I do something like this on some projects. I would not, however, perform the merge automatically: I'd rather use the db/seeds.rb file and the rake db:seed task:
if State.count == 0
State.transaction do
CSV.foreach("#{::Rails.root}/db/seed_data/states.csv", 'r') do |row|
code, acronym, name = *row
State.create! code: code, acronym: acronym, nane: name
end
end
end
In this code I load the data from a local file, but you can easily change to a remote file using Net::HTTP.