I have a rails application (with puma) and few postgresql databases. As a DB driver gem Sequel is used. Here is the code for multiple DB connections when application starting:
require "sequel"
module MyApp
class Databases
class << self
attr_accessor :first_db, :second_db
def start_connections
#first_db = Sequel.connect(ENV.fetch("FIRST_DB_CREDENTIONALS"))
#second_db = Sequel.connect(ENV.fetch("SECOND_DB_CREDENTIONALS"))
end
def disconnect_all
first_db.disconnect
second_db.disconnect
end
end
end
end
MyApp::Databases.start_connections
But, from time to time, some requests (which fetches records from first or second DB) to application fails with error: "PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly. This probably means the server terminated abnormally before or while processing the request: ..."
How to fix this error? Is there a problem with connection timeout settings or what?
After lookup at documentation, I found this moment and add following code to my puma config:
before_fork do
MyApp::Databases.disconnect_all
end
But problem still persists. Also tried to close connection manually on end of each request, but the error, mentioned above, raises from time to time.
Related
I'm currently working on a project to enable database backed configurations in the frontend of our application. These need to be loaded after application initialization, so I created a module to load them and added a call to it in environment.rb, after Rails.application.initialize!.
The problem is that when this code is enabled, my console gets flooded with listen loop errors with bad file descriptors like:
2020-01-24 09:18:16 -0500: Listen loop error: #<Errno::EBADF: Bad file descriptor>
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:383:in `select'
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:383:in `handle_servers'
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:356:in `block in run'
When I disable either the call to the ConfigurationLoader or the methods I'm calling on the model, I no longer get these errors.
The rub is that I can't reproduce this issue on another machine, or in specs. I've tried on two other laptops and on one of our staging servers and they work perfectly with the ConfigurationLoader enabled.
I've tried restarting my computer, working from a freshly cloned repository, and setting all the file permissions for the application to 777. Nothing has worked so far.
Here's the ConfigurationLoader module:
module ConfigurationLoader
# Overrides client default configurations if frontend configurations exist
def self.call
Configurations::ImportRowMapping.override_configurations
rescue ActiveRecord::NoDatabaseError => e
log_no_database_error(e)
rescue ActiveRecord::StatementInvalid => e
log_statement_invalid_error(e)
rescue Mysql2::Error::ConnectionError => e
log_connection_error(e)
end
def self.log_no_database_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, database does '\
'not exist'
)
Rails.logger.warn(error.message)
end
def self.log_statement_invalid_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, table does '\
'not exist'
)
Rails.logger.warn(error.message)
end
def self.log_connection_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, could not '\
'connect to database'
)
Rails.logger.warn(error.message)
end
end
The call in environment.rb:
# Load the Rails application.
require_relative 'application'
require_relative 'configuration_loader'
# Initialize the Rails application.
Rails.application.initialize!
ConfigurationLoader.call
And the model method being called:
def self.override_configurations
return unless any?
Rails.application.client.payroll_service_file.payroll_service_file
.mappings = all.to_a
end
I'll note here that I get the errors when either the guard clause or the assignment are enabled.
Anyone have any ideas about what's going on? I'm about at my wits' end.
So I'm still not sure on the exact cause of the problem, but the solution was to move the configuration loader call out of environment.rb and into an after_initialize block in application.rb.
Major edit: Since originally finding this issue I have whittled it down to the below. I think this is now a marginally more precise description of the problem. Comments on the OP may therefore not correlate entirely.
Edit lightly modified version posted in rails/puma projects: https://github.com/rails/rails/issues/21209, https://github.com/puma/puma/issues/758
Edit Now reproduced with OS X and Rainbows
Summary: When using Puma and running long-running connections I am consistently receiving errors related to ActiveRecord connections crossing threads. This manifests itself in message like message type 0x## arrived from server while idle and a locked (crashed) server.
The set up:
Ubuntu 15 / OSX Yosemite
PostgreSQL (9.4) / MySQL (mysqld 5.6.25-0ubuntu0.15.04.1)
Ruby - MRI 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] / Rubinius rbx-2.5.8
Rails (4.2.3, 4.2.1)
Puma (2.12.2, 2.11)
pg (pg-0.18.2) / mysql2
Note, not all combinations of the above versions have been tried. First listed version is what I'm currently testing against.
rails new issue-test
Add a route get 'events' => 'streaming#events'
Add a controller streaming_controller.rb
Set up database stuff (pool: 2, but seen with different pool sizes)
Code:
class StreamingController < ApplicationController
include ActionController::Live
def events
begin
response.headers["Content-Type"] = "text/event-stream"
sse = SSE.new(response.stream)
sse.write( {:data => 'starting'} , {:event => :version_heartbeat})
ActiveRecord::Base.connection_pool.release_connection
while true do
ActiveRecord::Base.connection_pool.with_connection do |conn|
ActiveRecord::Base.connection.query_cache.clear
logger.info 'START'
conn.execute 'SELECT pg_sleep(3)'
logger.info 'FINISH'
sse.write( {:data => 'continuing'}, {:event => :version_heartbeat})
sleep 0.5
end
end
rescue IOError
rescue ClientDisconnected
ensure
logger.info 'Ensuring event stream is closed'
sse.close
end
render nothing: true
end
end
Puma configuration:
workers 1
threads 2, 2
#...
bind "tcp://0.0.0.0:9292"
#...
activate_control_app
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end
Run the server puma -e production -C path/to/puma/config/production.rb
Test script:
#!/bin/bash
timeout 30 curl -vS http://0.0.0.0/events &
timeout 5 curl -vS http://0.0.0.0/events &
timeout 30 curl -vS http://0.0.0.0/events
This reasonably consistently results in a complete lock of the application server (in PostgreSQL, see notes). The scary message comes from libpq:
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x54 arrived from server while idle
In the 'real-world' I have quite a few extra elements and the issue presents itself at random. My research indicates that this message comes from libpq and is subtext for 'communication problem, possibly using connection in different threads'. Finally, while writing this up, I had the server lock up without a single message in any log.
So, the question(s):
Is the pattern I'm following not legal in some way? What have I mis[sed|understood]?
What is the 'standard' for working with database connections here that should avoid these problems?
Can you see a way to reliably reproduce this?
or
What is the underlying issue here and how can I solve it?
MySQL
If running MySQL, the message is a bit different, and the application recovers (though I'm not sure if it is then in some undefined state):
F, [2015-07-30T14:12:07.078215 #15606] FATAL -- :
ActiveRecord::StatementInvalid (Mysql2::Error: This connection is in use by: #<Thread:0x007f563b2faa88#/home/dev/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.3/lib/action_controller/metal/live.rb:269 sleep>: SELECT `tasks`.* FROM `tasks` ORDER BY `tasks`.`id` ASC LIMIT 1):
Warning: read 'answer' as 'seems to make a difference'
I don't see the issue happen if I change the controller block to look like:
begin
#...
while true do
t = Thread.new do #<<<<<<<<<<<<<<<<<
ActiveRecord::Base.connection_pool.with_connection do |conn|
#...
end
end
t.join #<<<<<<<<<<<<<<<<<
end
#...
rescue IOError
#...
But I don't know whether this has actually solved the problem or just made it extremely unlikely. Nor can I really fathom why this would make a difference.
Posting this as a solution in case it helps, but still digging on the issue.
I'm having an issue trying to get a timeout when connecting via TCPSocket to a remote resource that isn't available. It just hangs indefinitely without timing out. Ideally I'd want it to try reconnect every 2 minutes or so, but the TCPSocket.new call seems to block. I've tried using timeout() but that doesn't do anything either. Trying the same call in an IRB instance works perfectly fine, but when it's in Rails, it fails. Anyone have a work around for this?
My code looks something as follows:
def self.connect!
##connection = TCPSocket.new IP, 4449
end
def self.send(cmd)
puts "send "
unless ##connection
self.connect!
end
loop do
begin
##connection.puts(cmd)
return
rescue IOError
sleep(self.get_reconnect_delay)
self.connect!
end
end
end
Unfortunately, there is currently no way to set timeouts on TCPSocket directly.
See http://bugs.ruby-lang.org/issues/5101 for the feature request. You will have use the basic Socket class and set socket options.
I understand that when we fork a process the child process inherits a copy of the parents open file descriptors and offsets. According to the man pages this refers to the same file descriptors used by the parent. Based on that theory in the following program
puts "Process #{Process.pid}"
file = File.open('sample', 'w')
forked_pid = fork do
sleep(10)
puts "Writing to file now..."
file.puts("Hello World. #{Time.now}")
end
file.puts("Welcome to winter of my discontent #{Time.now}")
file.close
file = nil
Question 1:
Shouldn't the forked process which is sleeping for 10 seconds lose its file descriptor and not be able to write to the file as the parent process completes and closes the file and exits.
Question 2: But for whatever reason if this works then how does ActiveRecord lose its connection in this scenario. It only works if I set :reconnect => true on ActiveRecord connect can it actually connect, which means its losing connection.
require "rubygems"
require "redis"
require 'active_record'
require 'mysql2'
connection = ActiveRecord::Base.establish_connection({
:adapter => 'mysql2',
:username => 'root_user',
:password => 'Pi',
:host => 'localhost',
:database => 'list_development',
:socket => '/var/lib/mysql/mysql.sock'
})
class User < ActiveRecord::Base
end
u = User.first
puts u.inspect
fork do
sleep 3
puts "*" * 50
puts User.first.inspect
puts "*" * 50
end
puts User.first.inspect
However, the same is not true with Redis (v2.4.8) which does not lose connection on a fork, again. Does the it try to reconnect internally on a fork?
If thats the case then why isn't the write file program not throwing an error.
Could somebody explain whats going on here. Thanks
If you close a file descriptor in one process it stays valid in the other process, this is why your file example works fine.
The mysql case is different because it's a socket with another process at the end. When you call close on the mysql adapter (or when the adapter gets garbage collected when ruby exits) it actually sends a "QUIT" command to the server saying that you're disconnecting, so the server tears down its side of the socket. In general you really don't want to share a mysql connection between two processes - you'll get weird errors depending on whether the two processes are trying to use the socket at the same time.
If closing a redis connection just closes the socket (as opposed to sending a "I'm going away " message to the server) then the child connection should continue to work because the socket won't actually have been closed
CORRECTION:
Specifically I'm looking to start and stop an EventMachine (EM) from a Ruby on Rails controller.
(I've been told that 'Thin' would work well for this.)
NOTE:
This EM server is SEPARATE from the Mongrel server running the Ruby on Rails application. (The EM server is accepting connections from an Arduino microcontroller.)
Running 'Thin' as the server .. I SHOULD be able to accept both HTTP requests and the Arduino's connections.
# Starts Server
def start_control_server
EventMachine::run {
#EchoServer = EventMachine::start_server "0.0.0.0", 40013, EchoServer
}
end
# Attempts ( fails ) to stop server
def stop_control_server
EventMachine.stop_server(#EchoServer)
end
If you're recommending other servers OTHER than a straight-up EventMachine please provide code to execute the above code.
Complete controller code available here: http://pastie.org/1698383
I'm guessing you're not calling start_control_server and stop_control_server inside another controller method. That means your instance variable (#EchoServer) isn't going to exist when you call stop.
One solution might be to store the id returned from start_server in the session. As in
def start_control_server
session[:em_server_id] = EventMachine::start_server "0.0.0.0", 4000, EchoServer
end
def stop_control_server
EventMachine.stop_server(session[:em_server_id]) if session[:em_server_id]
session[:em_server_id] = nil
end
Also, if you're running your rails app using thin then you are already inside an eventmachine loop, so you don't need to call EventMachine::run. Calling EventMachine.stop_server doesn't seem to disconnect anything that is already connected but stops any further connections from being established to the specified port.
Hope thats of some use!