My app is set up in such a way that we use a different database connection per subdomain, using different environments. Delayed Job does what you'd expect (I guess) and uses the current environment of that request (and thus DB connection) when inserting the job to it's delayed_jobs table.
Problem is that DJ can't process jobs from all these different tables, so I'm trying to force DJ to use just one database, especially set up for it. I have tried this but it just won't work and I've no idea what to try next.
Any pointers/suggestions would be VERY much appreciated, really at my wits end with this.
Attempted code:
Delayed::Job.class_eval do
establish_connection ActiveRecord::Base.configurations["delayed_job"]
end
Connection to the DB is done in a before_filter in the ApplicationController.
The code in ApplicationController to establish the connection per the domain will happen only in your application server on each request.
Add a :domain attribute to your Job class and set it when you queue the job. In Job#perform, establish your DB connection.
Related
In our application, we have several models which need to connect to different external databases that hold the same tables and columns, but are each separate and cannot be unified.
Currently, the application runs on separate servers, in which each is connected only to a specific external database. However, these are 10+ servers, all serving the exact same application, with the only difference being the external database they connect to.
The goal is to have a single server running the application and have the application decide which database to query based on a certain parameter passed into the controller.
Our current approach is the following. We have an abstract class from which relevant models inherit, with a method to reconnect it to the specific database:
class AbstractRecord < ApplicationRecord
self.abstract_class = true
def self.reconnect
database = Thread.current[:database_name].constantize
self.establish_connection database
end
end
Then, we have every controller inherit from a controller class with a before_action that sets the current database name in Thread.current and calls that method:
class AccessController < ApplicationController
before_action :set_current_database
private
def set_current_database
Thread.current[:database_name] = current_user.database_name
AbstractRecord.reconnect
end
end
Each user has the information on which database they need to connect to, and so the application reconnects the database based on the current user.
This application also serves an API, with controllers inheriting from a similar controller that also reconnects the database based on the current API user.
We know all of the databases we need to connect to and keep them in yml files, and all of them are loaded into constants inside an initializer.
This approach works for the most part. Whenever a request is made, the database is successfully reconnected to the appropiate database, and the application functions as normal.
However, issues arise when a request is sent at the same time that another request is being processed, both in development and production:
ActiveRecord::ConnectionNotEstablished (No connection pool with 'AbstractRecord' found.)
This error is raised whenever any model that needs to query the AbstractRecord database does so after a new connection has been initiated in a different request.
Given enough time to finish, requests don't seem to interfere with each other and the database reconnections work fine.
It is my understanding that Rails handles requests on individual threads for each of them, and each thread uses a different database connection, which raises the question: Why is establish_connection causing other requests to lose their connection? Is there a major misunderstanding on how threads and database connections work in Rails in this case?
Back to the main question: How can I dynamically connect my models to a specific database during a single request in this version of Rails? Is this approach correct, or is there a more adequate solution?
Rails version: 5.2.4.3
Ruby version: 2.6.3p62
#Joaquin for me this is clearly a case of multi-tenancy, where I must have a central database with a customer table and their respective database connection. There are some libraries that do this elegantly, with the ar-octopus gem.
In your case, there is a concorrency failure, as the key you are using in Thread.current is probably being used in two or more simultaneous executions. A change I would make would be to make your Thread.current key more specific as
Thread.current[:"#{current-table-name}_database_name"] = current_user.database_name
where the Person class would have the key Thread.current[:"#{Person.table_name}_database_name"], but this approach is not a silver bullet and is certainly has flaws.
I suggest looking at gem ar-octopus, it will bring you many benefits.
I have a rails 4 app I am hosting on heroku. They give some specific advice about how to manage your DB connection pool when using a multi-threaded server (puma)https://devcenter.heroku.com/articles/concurrency-and-database-connections
When I ran load testing for my app I got an error- can't connect to the db. when each page was being hit, rails initializes active record, even if I'm not making any queries on that page, or referencing any models.
My question is:
How can I make a sort of whitelist (or blacklist) so that active record is not initialized with a db connection for these specific controller actions? In an initializer?
Ideally I would run the cheaper postgres service on heroku (40 connections) because I know my app doesnt use the db very often. If traffic hits higher that the 40 connections things will start to error, which seems silly for an app that wasn't going to use the db on those requests.
I read about how to disable active record for an entire app: Disable ActiveRecord for Rails 4
But how do I selectively enable it? Are there any other different performance considerations here (by not eager loading these things or any other gotchas)
In you application_controller.rb
before_filter :maybe_disconnect_db
def maybe_disconnect_db
ActiveRecord::Base.remove_connection() if ActiveRecord::Base.connected?
end
def maybe_connect_db
ActiveRecord::Base.establish_connection() unless ActiveRecord::Base.connected?
end
Then for each controller/action that needs the db connection add
skip_before_filter :maybe_disconnect_db, #only or unless filter here
before_filter :maybe_connect_db, #only or unless filter here
This should establish the connection for any specific db request, and disconnect for any request that doesn't need it, while also handling multiple db requests in a row without action, and multiple non db requests in a row without action.
ActiveRecord::Base.remove_connection
ActiveRecord::Base.establish_connection
I'm trying to understand how ActiveRecord connects to a database on a per model basis. So lets assume we have 2 models and one database. When you do a simple Model.find on both models does this create a new database connection for each model so it has its own defined connection or is there a single connection to the database that has been initialised that is shared across the 2 models. Does anyone know where it's doing this in the code?
I am trying to understand how ActiveRecord achieves this.
ActiveRecord uses a connection pool. When you try to interact with the database, a connection is acquired from the pool, query is made and connection is returned to pool after the query. All this is handled by ActiveRecord in the background.
You can adjust the number of connections in the pool in your database configuration
http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
You can test this easily. Execute a query in controller action that takes couple of minutes. Issue multiple requests to the action and you would be able to see out of connection errors
We are using request store gem in our app. It is used for storing global data. But the problem is If I try to access request store variable in the delayed job It is not accessible. Is there anything extra which needs to be done in order for the request store data to be available in delayed job ?
Delayed Job Code
class CustomersCreateJob < Struct.new()
def perform
puts "Request Data =====> #{RequestStore.store[:current_user] }"
end
end
In general, current_user by default is only available in controllers for reason.
You did not mention you method or running jobs, but in any way by the time when job starts, even if it happens to be in same process and thread - request is already finished and there's no current_user. So pass user's id to job explicitly (this depends on how you run them)
delayed_job workers won't get the request_store normally, because they are outside of the request/response cycle.
However this frequently isn't the desired behaviour, given the typical uses of request_store.
You can always extend ApplicationJob yourself with such functionality, (e.g. around_enqueue and around_perform), and I do recall having to do something similar at a previous role.
I have a Rails controller that needs to write some data to my MongoDB. This is what it looks like at the moment.
def index
data = self.getCheckinData
dbCollection = self.getCheckinsCollection
dbCollection.insert(data)
render(:json => data[:_id].to_s())
end
protected
def getCheckinsCollection
connection = Mongo::Connection.new('192.168.1.2', 27017)
db = connection['local']
db.authenticate('arman', 'arman')
return db['checkins']
end
Is it okay to authenticate with MongoDB per request?
It is probably unnecessarily expensive and creating a lot more connections than needed.
Take a look at the documentation:
http://www.mongodb.org/display/DOCS/Rails+3+-+Getting+Started
They connect inside an initializer. It does some connection pooling so that connections are re-used.
Is there only one user in the database?
I'd say: don't do the db authentication. If MongoDB server is behind a good firewall, it's pretty secure. And it never ever should be exposed to the internet (unless you know what you're doing).
Also, don't establish a new connection per request. This is expensive. Initialize one on startup and reuse it.
In general, this should be avoided.
If you authenticate per request and you get many requests concurrently, you could have a problem where all connections to the database are taken. Moreover, creating and destroying database connections can use up resources within your database server -- it will add a load to the server that you can easily avoid.
Finally, this approach to programming can result in problems when database connections aren't released -- eventually your database server can run out of connections.