I have a Rails API with a PostgreSQL database.
Some requests to the API show a strange behavior that
doesn't depend on the endpoint.
These requests (around 5-10% of total requests) start with
the same 7 database queries :
SET client_min_messages TO ?
SET standard_conforming_strings = on
SET SESSION timezone TO ?
SELECT t.oid, t.typname FROM pg_type WHERE t.typname IN ( ? )
...
The request also takes a long time to start before the 7 queries are executed.
It seems to be the database adapter initiating a connection.
ActiveRecord::ConnectionAdapters::PostgreSQLAdapter
This significantly slows down the query.
I am using a PostegreSQL 11.6 AWS RDS instance, with default parameters.
Here is my database.yml config :
default: &default
adapter: postgresql
encoding: unicode
username: *****
password: *****
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
production:
<<: *default
database: *****
username: *****
password: *****
pool: 50
How do I reduce the number of connections initiating ?
Is there a way to cache the queries ?
Thank you,
Ran into the same thing and here's what I think is happening:
Every time a new connection is instantiated it performs the bootstrapping queries you mention above. Assuming a new process is not spawned, a new connection would need to be instantiated because existing connections have been reaped by ActiveRecord.
By default, the ConnectionPool::Reaper will disconnect any connection that has been idle for over 5 minutes.
See: https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
If your API does not receive any requests for a period of 5 minutes and all the connections are reaped the next request will need to instantiate a new connection and therefore run the queries.
How do I reduce the number of connections initiating ?
You could set an idle_timeout of 0 in database.yml. This would prevent ActiveRecord from reaping the connections but could potentially cause issues depending on how many processes are running and what your PG max_connections value is.
Is there a way to cache the queries ?
There's a closed issue that talks about this but it doesn't look like it's possible to cache these today.
https://github.com/rails/rails/issues/35311
Related
Summary: I'm using a single Redis instance for the Rails cache, actioncable and (non-cache) use in my rails code. Should all these uses share a single connection pool and if so how can I config this to make it happen since there seem to be totally different ways to setup the pooling for each?
Details follow since people seem to like to see them.
I'm using redis as the adapter for rails cache using the following config.
config.cache_store = :redis_cache_store, {
url: "redis://XXX.net:6379/0",
pool_size: ENV.fetch('RAILS_MAX_THREADS') { 5 },
password: Rails.application.credentials.dig(:redis, :password),
expires_in: 24.hours,
pool_timeout: 5
}
I've set the expires_in option so that I can set the option in my redis config to evict keys with expiration set so I can use the same redis instance for both cache and non-cache data. Now, I want to also access REDIS directly for non-cache related tasks via something like the example config below
pool_size = ENV.fetch("RAILS_MAX_THREADS", 5)
redis_pool = ConnectionPool.new(size: pool_size) do
Redis.new(
url: "redis://XXX.net:6379/0",
)
end
But I'm not sure if that is correct. Shouldn't I be sharing a connection pool between the cache_store connections and the other connections to Redis? If so, how can I do this?
To complicate matters further I'm also using Redis for actioncable via a config like
production:
adapter: redis
url: <%= ENV.fetch("REDIS_URL") { "redis://XXX.net:6379/0" } %>
password: <%= Rails.application.credentials.dig(:redis, :password) %>
I've seen suggestions that actioncable will automatically handle connection pooling with Redis if I'm using the connection_pool gem (is this right?) but I feel like all these connections should be drawing from the same pool. If so how can I make that happen?
I created a method called connect_and_do_something like below;
What this method does is:
①to connect SampleModel with various databases passed as an argument db_name
②do something by using SampleModel
def connect_and_do_something(db_name)
config = {
adapter: XXXX,
host: XXXX,
username: XXXX,
password: XXXX,
db_name: db_name
}
# connect with database dynamically
SampleModel.establish_connection(config)
### do something here with SampleModel and DB(MySQL) ####
end
This is working fine in most cases, but sometimes I receive connection pool error when it's requested by a lot of users at the same time (multiple thread).
No connection pool for SampleModel
This is different from ActiveRecord::ConnectionTimeoutError, which appears we cannot obtain connection from connection pool within the time limit.
Now, I'm so confused why this can happen and why this happen occasionally.
Why does this happen occasionally? and How can I resolve it?
Thank you in advance.
I'm learning & doing SSE for the first time in rails! My controller code:
def update
response.headers['Content-Type'] = 'text/event-stream'
sse = SSE.new(response.stream, event: 'notice')
begin
User.listen_to_creation do |user_id|
sse.write({id: user_id})
end
rescue ClientDisconnected
ensure
sse.close
end
end
Front end:
var source = new EventSource('/site_update');
source.addEventListener('notice', function(event) {
var data = JSON.parse(event.data)
console.log(data)
});
Model pub/sub
class User
after_commit :notify_creation, on: :create
def notify_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
self.class.execute_query(connection, ["NOTIFY user_created, '?'", id])
end
end
def self.listen_to_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
begin
execute_query(connection, ["LISTEN user_created"])
connection.raw_connection.wait_for_notify do |event, pid, id|
yield id
end
ensure
execute_query(connection, ["UNLISTEN user_created"])
end
end
end
def self.clean_sql(query)
sanitize_sql(query)
end
private
def self.execute_query(connection, query)
sql = self.clean_sql(query)
connection.execute(sql)
end
end
I've noticed that if I'm writing to SSE, something trivial like in a tutorial like... sse.write({time_now: Time.now}), everything works great. In command line, CTRL+C successfully shuts down the local server.
However, whenever I need to write something that requires some kind of database action, for example when I'm doing a postgres pub/sub as in this tutorial, then CTRL+C doesn't shut down the local server, it's just stuck and hangs and requires me to manually kill the PID.
On the actual spun up server, sometimes a page refresh will hang forever as well. Other times, it will throw a timeout error:
ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use):
Unfortunately this issue persists in production as well, where i'm using Heroku. I just get lots of timeout errors. But I think I have Heroku properly configured, and also local settings... my understanding is I just need to have a sizable pool (I have 5) to pull connections from and allow multiple threads. Below you'll find some config code.
THERE ARE NO ENV VARIABLES, DEFAULTS USED!
# config/database.yml
default: &default
adapter: postgresql
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
timeout: 5000
development:
<<: *default
database: proper_development
# config/puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 1)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
If it's helpful here's the output when I run rails s
=> Booting Puma
=> Rails 5.0.2 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 4.3.3 (ruby 2.4.0-p0), codename: Mysterious Traveller
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://127.0.0.1:3000
* Listening on tcp://[::1]:3000
Use Ctrl-C to stop
The issue here seems to be the lack of consistency between the puma threads and the database connections. If some connection was initiated by middleware etc through AR, the code you have written can lead to two connections being held in the same request cycle until you receive a notification and the thread finishes its job. AR caches connections per thread, so if a request was made and connection was checked out from the pool it will be held by that. Look at this issue for more details. If you end up using the connection pool to check out one more connection and make that connection wait till you get a notification from Postgres, potentially two connections can be held by the same Puma thread that is waiting.
To see this in action, start a new Rails server instance in development and make a request to your SSE endpoint. If you were getting timeouts before you might see two connections to Postgres while you made just one request to a newly launched server. So, even though your number of threads and connection pool size was same, you might run out of free connections from the pool. An easier way might be to just add this line in development after you checkout a connection to see how many cached connections are being held right now.
def self.listen_to_creation
ActiveRecord::Base.connection_pool.with_connection do |connection|
# Print or log this array
p ActiveRecord::Base.connection_pool.instance_variable_get(:#thread_cached_conns).keys.map(&:object_id)
begin
execute_query(connection, ["LISTEN user_created"])
.........
.........
Also, the snippets you have posted seem to indicate you are running up to 16 threads on a connection pool of size 5 in development environment, so that is a different issue.
To fix this, you would want to investigate which thread is holding the connection and if you can reuse it for your notification or just increase the DB pool size.
Now, coming to SSE itself here. Since a SSE connection blocks and reserves a thread in your current setup. If you have multiple requests to this endpoint you might quickly starve out of Puma threads itself making requests wait. This might work in case you are not expecting a lot of requests to this endpoint but if you are, you would need more free threads so you might even want to increase the Puma thread count. Ideally, though a non blocking server would work better here.
Edit:
Also, forgot to add that SSE in rails has an issue of keeping alive dead connections if it doesn't know the connection is dead. You might have threads endlessly waiting this way until some data comes and they realize the connection is no longer valid.
I'm currently trying to upgrade a Rails 3.2 app to Rails 4.0.13 (ruby 2.3.5, PostgreSQL 9.4.13) and I'm getting this error when running my integration tests;
: PG::UnableToSend at /companies/get_current_firm
: ===============================================
:
: > socket not open
:
:
: activerecord (4.0.13) lib/active_record/connection_adapters/postgresql_adapter.rb, line 798
: -------------------------------------------------------------------------------------------
:
: ``` ruby
: 793 end
: 794
: 795 FEATURE_NOT_SUPPORTED = "0A000" #:nodoc:
: 796
: 797 def exec_no_cache(sql, binds)
: > 798 #connection.async_exec(sql, [])
: 799 end
: 800
: 801 def exec_cache(sql, binds)
: 802 stmt_key = prepare_statement sql
: 803
: ```
This integration test logs in and then there is a number of calls from the frontend (AngularJS) to get details like the current firm selected, etc. Each one of these calls gets the same message, it's like the connection has been shutdown. I'm using the chrome browser through selenium & capybara in LIVE mode so I can see what is happening.
Here is my database.yml file for test
test: &test
adapter: postgresql
encoding: unicode
database: test_<%= ENV['USER'] %><%= ENV['TEST_ENV_NUMBER'] %>
pool: 5
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
Here is the call it is making from the frontend
AngularJS
current_firm: (scope) ->
$http({method: 'GET', url: '/companies/get_current_firm'}).success (data) ->
scope.current_firm = data['firm']
Rails Controller
def get_current_firm
render json: {firm: current_firm.organisation}
end
There are three separate calls to get different data when a user logs in, sometimes in the tests it will get 1 or 2 of them but never three.
I tried the work around from http://www.ruby-railings.com/en/rails/postgresql/2014/01/11/connection-problems-in-rails-4.html. This didn't fix the problem at all.
Update
When the user logs in there are three http calls fired off from AngularJS to the Rails backend. If I remark out two of the calls then that call works every time. If I remark out only one, so we have two calls then it fails one or both of the calls. I get a 'pending' message on one of the calls.
I've worked that this was an AngularJS problem sending too many requests to Rails 4 at once. To fix the problem I put each call in a promise.
I'm using central_logger to store logs from our Rails app in mongodb. When the mongo server went down recently our app started timing out on mongo inserts. How can I prevent Rails from timing out if the mongo server goes down?
The ruby driver supports timeouts like so
#conn = Connection.new("localhost", 27017, :pool_size => 5, :timeout => 5)
But the central_logger gem isn't using that. So you can either fork it to add that in there, or monkey-path the CentralLogger::MongoLogger.connect method
It currently has
def connect
#mongo_connection ||= Mongo::Connection.new(#db_configuration['host'],
#db_configuration['port'],
:auto_reconnect => true).db(#db_configuration['database'])
if #db_configuration['username'] && #db_configuration['password']
# the driver stores credentials in case reconnection is required
#authenticated = #mongo_connection.authenticate(#db_configuration['username'],
#db_configuration['password'])
end
end
You would need to monkey-path in :timeout=>5 (or whatever) to the Mongo::Connection.new
I would bet the author of central-logger would like to have this in there, so a fork and pull request would likely be welcome.
You could use replica sets - so if the master goes down, it can failover automatically to one of the replicas.
Usually the database insert should be fast, so you could work with the ruby timeout:
require 'timeout'
Timeout::timeout(0.2) do
... write to log server
end
this code will timeout and continue after 200 milliseconds in any case.