In an effort to move from distribute_reads to the native Rails 6 multiple databases support I came across a weird behavior, I'm still unsure if it's a bug or a configuration problem.
// database.yml
default: &default
adapter: postgresql
encoding: unicode
pool: 10
reaping_frequency: 10
timeout: 5000
username: my_username
password: my_password
database: my_database
schema_search_path: 'public,utils'
development: &development
primary:
<<: *default
database: my_database
host: 'localhost'
primary_replica:
<<: *default
database: my_database
host: 'localhost'
replica: true
third_replica:
<<: *default
database: my_database
host: 'localhost'
replica: true
test: &test
primary:
<<: *default
database: 'my_database_test'
host: 'localhost'
primary_replica:
<<: *default
database: 'my_database_test'
host:'localhost'
replica: true
third_replica:
<<: *default
database: 'my_database_test'
host: 'localhost'
replica: true
I also changed the ApplicationRecord that connects to all the records of this codebase
# application_record.rb
connects_to shards: {
default: { writing: :primary, reading: :primary_replica },
admin: { writing: :primary, reading: :third_replica }
}
Now when I'm in development it acts exactly as it should
> ActiveRecord::Base.connected_to(role: :reading, shard: :default, prevent_writes: true) do
EventType.count
end
(72.3ms) SELECT COUNT(*) FROM "event_types" /*line:(pry):2:in `block in <main>'*/
=> 2
This is the correct number of records. When I run a test and use the same lines, the EventType.count outside of the wrapper will count the correct number as well, but connected_to will result in 0 no matter what record I'm checking, in any model.
> EventType.count
=> 1
> ActiveRecord::Base.connected_to(role: :reading, shard: :default, prevent_writes: true) do
EventType.count
end
=> 0
The replica in development/test is actually the same database, so it shouldn't be possible to get 0 in my case.
When using the writing role it works though
> ActiveRecord::Base.connected_to(role: :writing, shard: :default) do
EventType.count
end
=> 1
Be aware the shards: part doesn't seem to matter because it also breaks with a simplified version through database: and no third_replica
Is there something I'm missing about this Rails functionality? Is there something related to tests that aren't compatible with this? I couldn't find any documentation related to this.
In the end, the problem came from the transaction we have throughout our tests setup
DatabaseCleaner.strategy = :transaction
And it put in evidence it's a much bigger problem throughout our codebase. When having queries wrapped into transactions and using connected_to you basically go out of the said transaction and it creates lots of consistency problems in our specific case. It's purely logical, but adding this into your stack may be incompatible with what was there in the first place.
The conclusion is to be careful when using transactions and dealing with read-replica.
Related
The following main/replica database structure
development:
primary:
<<: *default
database: users_development
username: deploy_root
password: password
host: "localhost"
migrations_paths: db/user_migrate
primary_replica:
<<: *default
database: users_development
username: deploy_readonly
password: password
host: "localhost"
replica: true
has defined as its main AR defined as:
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
connects_to database: { writing: :primary, reading: :primary_replica }
end
However, when The following commands are run following the suggested syntax by the rails guides
rails generate devise User nick avatar --database users
bin/rails db:migrate
the only response is a prompt. Two problems arise:
the migration is created but not in the proper directory migrations_paths: db/user_migrate
Logging into the database users_development=# \dt returns, consistently with the prompt reply Did not find any relations. In other words the table was not created (which is confirmed by the schema.rb file being unaltered
is specifying migrations_paths with a sub directory a mistake for the primary database connection?
Or should rails generate devise User nick avatar --database users invoke primary in lieu of users?
The primary database is assumed to have its migrations in the migrate directory, not a sub-directory.
development:
primary:
<<: *default
database: users_development
username: deploy_root
password: password
host: "localhost"
primary_replica:
<<: *default
database: users_development
username: deploy_readonly
password: password
host: "localhost"
replica: true
Running a rails generate with --database name_of_primary_database will proceed as in single database application.
Although, this observer would have enjoyed that all migrations be organised in a similar manner, migrations_paths is not allowed for a primary database.
I have read the documentation about MultiDb connection switching of Rails 6.0.0.beta3 and implemented this way:
database.yml
default: &default
adapter: mysql2
encoding: utf8mb4
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
username: root
password: root
socket: /var/run/mysqld/mysqld.sock
development:
main:
<<: *default
database: r6_multidb_development
main_replica:
<<: *default
database: r6_multidb_development_copy
replica: true
Article Model
class Article < ApplicationRecord
connect_to database: { writing: :main, reading: :main_replica }
end
Both the databases r6_multidb_development and r6_multidb_development_copy have different records, i was expecting when record is created records should inserted on r6_multidb_development Database and when retrieving record it should from r6_multidb_development_copy database. But in both cases when record is inserted and retrieved it is happening from main configuration that is r6_multidb_development database.
I believe write to the database should happen from r6_multidb_development and read should happen from r6_multidb_development_copy. I would appreciate if anybody figure out this issue.
Add following options to your environment config:
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver
config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session
I created two rails apps sample and test. There are two databases. users table in sample app with the field report_id refers to reports table in test app.
I want to display the test app data in sample app by fetching the unique_id field. I want to display reports data for a particular user by connecting these two databases.
How can I achieve this in the simplest way?
For rails6, you can provide two connections and specify databases for each connection,
eg
adapter: postgresql
encoding: unicode
username: username
password: password
pool: 5
host: localhost
development:
primary:
<<: *default
database: database1
adapter: postgresql
secondary:
<<: *default
database: database2
adapter: postgresql
and for production environment, you can use database url's like
primary:
url: <%= ENV['DATABASE_URL'] %>
secondary:
url: <%= ENV['SECONDARY_DATABASE_URL'] %>
You can connect to two databases from each rails project:
#config/sabple_database.yml
default: &default
encoding: utf8
adapter: mysql2
port: 5500
development:
<<: *default
database: sample_db
host:
username:
password:
#config/initializers/sample_database.rb
SAMPLE_DB = YAML.load_file(File.join(Rails.root, "config", "sample_database.yml"))[Rails.env.to_s]
#models
class SampleDbBase < ActiveRecord::Base
self.abstract_class = true
establish_connection SAMPLE_DB
end
#models/my_model.rb
class MyModel < SampleDbBase
end
Read more about that. So, then you can create the same models without migration(allows using Reports.find(report_id)).
Also, you can write a query to some database using ActiveRecord::Base.establish_connection and execute.
Thanks to rails 6. multi DB connection will come default from rails 6.
Please use GitHub version of rails. and configure the database.yml:
development:
primary:
<<: *default
database: multiple_databases_development
animals:
<<: *default
database: multiple_databases_development_animals
migrations_paths: "db/animals_migrate"
For more detail please have a look at https://github.com/eileencodes/multiple_databases_demo
I have an app which which uses different database based on the subdomain. So essentially, the schema would be the same, but the data would differ for each databases. But when I release some new features and it would require some schema changes, I would need to run a command that would run on all databases configured in the shards.yml.
database.yml
default: &default
adapter: postgresql
encoding: unicode
pool: 15
host: localhost
port: 5432
username: postgres
password:
development:
<<: *default
database: app_default
production:
<<: *default
database: app_default
username: <%= ENV['BACKEND_DATABASE_USERNAME'] %>
password: <%= ENV['BACKEND_DATABASE_PASSWORD'] %>
shards.yml
shared: &shared
adapter: postgresql
encoding: unicode
pool: 15
host: localhost
username: postgres
password:
port: 5432
octopus:
environments:
- development
- test
- production
development:
default:
<<: *shared
database: app
first:
<<: *shared
database: first
second:
<<: *shared
database: second
....
test:
test:
host: postgres
adapter: postgresql
database: app_test
production:
default:
<<: *shared
database: app
first:
<<: *shared
database: first
second:
<<: *shared
database: second
....
I am using Octopus to set the shard based on subdomain, which works fine. The problems I have are:
I cannot do rails db:reset . Getting the error ActiveRecord::StatementInvalid: PG::ObjectInUse: ERROR: cannot drop the currently open database
I cannot do rails db:migrate that would migrate on all databases
You have to add using to your migrations
class CreateComments < ActiveRecord::Migration
using :first, :second
def change
create_table :comments do |t|
t.belongs_to :page, index: true
t.text :body
t.timestamps
end
end
end
the above migration will run on both first and second
I am working on existing rails project and just finished writing some basic RSpec tests for that.
In our database.yml file we have connection to 3-4 different databases and all are either pointing to either production database or staging database.
For testing environment I created a different yml file called database.test.yml and then symlinked it to database.yml file, so that the correct testing databases connections are created.
My database.test.yml looks something like this:
abc:
adapter: mysql2
host: localhost
reconnect: true
username: monty
password: python
database: abc
pqr:
adapter: mysql2
host: localhost
reconnect: true
username: monty
password: python
database: pqr
test:
adapter: mysql2
host: localhost
reconnect: true
username: monty
password: python
pool: 50
database: testing
My problem now is that how can I enforce that other developers/testers use this database.test.yml and not accidentally run the main database.yml file which contains connections to staging and production databases.
EDIT:
To further clarify my question, I have establish_connection in various models that connect to different databases. So I am not sure the earlier suggestions will solve my problem.
Example:
class Abc < ActiveRecord::Base
establish_connection :abc
end
One way I can think of for solving my problem is doing something like this:
class Abc < ActiveRecord::Base
establish_connection "abc_#{Rails.env}"
end
and in the database.yml:
defaults: &defaults
adapter: mysql2
encoding: utf8
username: root
password: secret
abc_production:
database: abc
<<: *defaults
abc_development:
database: abc_devlopment
<<: *defaults
abc_test:
database: abc_test
<<: *defaults
But I am not sure if this the best practice and I really don't feel to change the application code to setup test environment is good practice.
This idea is based on: http://blog.nistu.de/2012/03/25/multi-database-setup-with-rails-and-rspec
Any nudge or pointers towards the correction direction will be greatly helpful.
You could have something, like:
default: &default
adapter: mysql2
encoding: utf8
database: my_db_name
username: root
password: my_password
host: 127.0.0.1
port: 3306
development:
<<: *default
database: xxxx_development
# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
<<: *default
database: xxxx_test
production:
<<: *default
database: xxxx_production