I have 2 databases defined in my YAML: primary and datawarehouse.
With the following code:
pp ActiveRecord::Base.connection_config
ActiveRecord::Base.connected_to(database: :datawarehouse) do
ActiveRecord::Base.connection.execute('select 1')
end
pp ActiveRecord::Base.connection_config
I would expect the output to be the same before and after the block and let it get back to the default.
However the output before gives me:
=> {:adapter=>"postgresql", :host=>"postgres", :encoding=>"unicode", :migrations_paths=>"db/migrate", :database=>"rails_devise_production"}
And the output after the block keeps the connection to the datawarehouse.
=> {:adapter=>"postgresql", :host=>"postgres-archive", :encoding=>"unicode", :migrations_paths=>"db/datawarehouse_migrate", :database=>"archive" }
How is this possible? I would expect only the code in the block to be run against the other database connection.
There's a similar question in Rails repo issues. In short, connected_to with database key is specifically for one-off connections, and you need to use the role key in your application.
Related
I have a rails application (with puma) and few postgresql databases. As a DB driver gem Sequel is used. Here is the code for multiple DB connections when application starting:
require "sequel"
module MyApp
class Databases
class << self
attr_accessor :first_db, :second_db
def start_connections
#first_db = Sequel.connect(ENV.fetch("FIRST_DB_CREDENTIONALS"))
#second_db = Sequel.connect(ENV.fetch("SECOND_DB_CREDENTIONALS"))
end
def disconnect_all
first_db.disconnect
second_db.disconnect
end
end
end
end
MyApp::Databases.start_connections
But, from time to time, some requests (which fetches records from first or second DB) to application fails with error: "PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly. This probably means the server terminated abnormally before or while processing the request: ..."
How to fix this error? Is there a problem with connection timeout settings or what?
After lookup at documentation, I found this moment and add following code to my puma config:
before_fork do
MyApp::Databases.disconnect_all
end
But problem still persists. Also tried to close connection manually on end of each request, but the error, mentioned above, raises from time to time.
I'm currently working on a project to enable database backed configurations in the frontend of our application. These need to be loaded after application initialization, so I created a module to load them and added a call to it in environment.rb, after Rails.application.initialize!.
The problem is that when this code is enabled, my console gets flooded with listen loop errors with bad file descriptors like:
2020-01-24 09:18:16 -0500: Listen loop error: #<Errno::EBADF: Bad file descriptor>
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:383:in `select'
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:383:in `handle_servers'
/Users/fionadurgin/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/puma-4.3.1/lib/puma/server.rb:356:in `block in run'
When I disable either the call to the ConfigurationLoader or the methods I'm calling on the model, I no longer get these errors.
The rub is that I can't reproduce this issue on another machine, or in specs. I've tried on two other laptops and on one of our staging servers and they work perfectly with the ConfigurationLoader enabled.
I've tried restarting my computer, working from a freshly cloned repository, and setting all the file permissions for the application to 777. Nothing has worked so far.
Here's the ConfigurationLoader module:
module ConfigurationLoader
# Overrides client default configurations if frontend configurations exist
def self.call
Configurations::ImportRowMapping.override_configurations
rescue ActiveRecord::NoDatabaseError => e
log_no_database_error(e)
rescue ActiveRecord::StatementInvalid => e
log_statement_invalid_error(e)
rescue Mysql2::Error::ConnectionError => e
log_connection_error(e)
end
def self.log_no_database_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, database does '\
'not exist'
)
Rails.logger.warn(error.message)
end
def self.log_statement_invalid_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, table does '\
'not exist'
)
Rails.logger.warn(error.message)
end
def self.log_connection_error(error)
Rails.logger.warn(
'Could not initialize database backed configurations, could not '\
'connect to database'
)
Rails.logger.warn(error.message)
end
end
The call in environment.rb:
# Load the Rails application.
require_relative 'application'
require_relative 'configuration_loader'
# Initialize the Rails application.
Rails.application.initialize!
ConfigurationLoader.call
And the model method being called:
def self.override_configurations
return unless any?
Rails.application.client.payroll_service_file.payroll_service_file
.mappings = all.to_a
end
I'll note here that I get the errors when either the guard clause or the assignment are enabled.
Anyone have any ideas about what's going on? I'm about at my wits' end.
So I'm still not sure on the exact cause of the problem, but the solution was to move the configuration loader call out of environment.rb and into an after_initialize block in application.rb.
I'd like to start over with errbit - there are millions of records in our mongodb database and we hadn't been cleaning them up. I'd like to start over, but I don't want to lose my user accounts.
I've tried to run these routines (https://mensfeld.pl/2015/01/making-errbit-work-faster-by-keeping-it-clean-and-tidy/):
bundle exec rake errbit:clear_resolved
desc 'Resolves problems that didnt occur for 2 weeks'
task :cleanup => :environment do
offset = 2.weeks.ago
Problem.where(:updated_at.lt => offset).map(&:resolve!)
Notice.where(:updated_at.lt => offset).destroy_all
end
but the second one (deleting problems and notices over 2 weeks old), just seems to run forever.
Querying problems and notices collections via mongo shell doesn't seem to show any being deleted... we're using errbit V 0.7.0-dev and mongodb version 3.2.22.
Fastest way would be to get a mongo console and drop most of the collections. I'd say stop your errbit server, get a mongo console, connect to the db you use and run:
> db.errs.drop()
true
> db.problems.drop()
true
> db.backtraces.drop()
true
> db.notices.drop()
true
> db.comments.drop()
Problem.where(:updated_at.lt => 2.months.ago).destroy_all
runs too long because of N+1 problem with recursive deletion of Err, Notice and Comment, also mongoid does not support nested eager loading, so only way to delete faster - is to manually take these ids and delete directly, without callbacks:
problem_ids = Problem.where(:updated_at.lt => 2.months.ago).pluck(:id)
err_ids = Err.where(problem_id: {:$in => problem_ids}).pluck(:id)
Notice.where(err_id:{:$in => err_ids}).delete_all
Err.where(id:{:$in => err_ids}).delete_all
Comment.where(problem_id: {:$in => problem_ids}).delete_all
Problem.where(id: {:$in => problem_ids}).delete_all
Major edit: Since originally finding this issue I have whittled it down to the below. I think this is now a marginally more precise description of the problem. Comments on the OP may therefore not correlate entirely.
Edit lightly modified version posted in rails/puma projects: https://github.com/rails/rails/issues/21209, https://github.com/puma/puma/issues/758
Edit Now reproduced with OS X and Rainbows
Summary: When using Puma and running long-running connections I am consistently receiving errors related to ActiveRecord connections crossing threads. This manifests itself in message like message type 0x## arrived from server while idle and a locked (crashed) server.
The set up:
Ubuntu 15 / OSX Yosemite
PostgreSQL (9.4) / MySQL (mysqld 5.6.25-0ubuntu0.15.04.1)
Ruby - MRI 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] / Rubinius rbx-2.5.8
Rails (4.2.3, 4.2.1)
Puma (2.12.2, 2.11)
pg (pg-0.18.2) / mysql2
Note, not all combinations of the above versions have been tried. First listed version is what I'm currently testing against.
rails new issue-test
Add a route get 'events' => 'streaming#events'
Add a controller streaming_controller.rb
Set up database stuff (pool: 2, but seen with different pool sizes)
Code:
class StreamingController < ApplicationController
include ActionController::Live
def events
begin
response.headers["Content-Type"] = "text/event-stream"
sse = SSE.new(response.stream)
sse.write( {:data => 'starting'} , {:event => :version_heartbeat})
ActiveRecord::Base.connection_pool.release_connection
while true do
ActiveRecord::Base.connection_pool.with_connection do |conn|
ActiveRecord::Base.connection.query_cache.clear
logger.info 'START'
conn.execute 'SELECT pg_sleep(3)'
logger.info 'FINISH'
sse.write( {:data => 'continuing'}, {:event => :version_heartbeat})
sleep 0.5
end
end
rescue IOError
rescue ClientDisconnected
ensure
logger.info 'Ensuring event stream is closed'
sse.close
end
render nothing: true
end
end
Puma configuration:
workers 1
threads 2, 2
#...
bind "tcp://0.0.0.0:9292"
#...
activate_control_app
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end
Run the server puma -e production -C path/to/puma/config/production.rb
Test script:
#!/bin/bash
timeout 30 curl -vS http://0.0.0.0/events &
timeout 5 curl -vS http://0.0.0.0/events &
timeout 30 curl -vS http://0.0.0.0/events
This reasonably consistently results in a complete lock of the application server (in PostgreSQL, see notes). The scary message comes from libpq:
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x54 arrived from server while idle
In the 'real-world' I have quite a few extra elements and the issue presents itself at random. My research indicates that this message comes from libpq and is subtext for 'communication problem, possibly using connection in different threads'. Finally, while writing this up, I had the server lock up without a single message in any log.
So, the question(s):
Is the pattern I'm following not legal in some way? What have I mis[sed|understood]?
What is the 'standard' for working with database connections here that should avoid these problems?
Can you see a way to reliably reproduce this?
or
What is the underlying issue here and how can I solve it?
MySQL
If running MySQL, the message is a bit different, and the application recovers (though I'm not sure if it is then in some undefined state):
F, [2015-07-30T14:12:07.078215 #15606] FATAL -- :
ActiveRecord::StatementInvalid (Mysql2::Error: This connection is in use by: #<Thread:0x007f563b2faa88#/home/dev/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.3/lib/action_controller/metal/live.rb:269 sleep>: SELECT `tasks`.* FROM `tasks` ORDER BY `tasks`.`id` ASC LIMIT 1):
Warning: read 'answer' as 'seems to make a difference'
I don't see the issue happen if I change the controller block to look like:
begin
#...
while true do
t = Thread.new do #<<<<<<<<<<<<<<<<<
ActiveRecord::Base.connection_pool.with_connection do |conn|
#...
end
end
t.join #<<<<<<<<<<<<<<<<<
end
#...
rescue IOError
#...
But I don't know whether this has actually solved the problem or just made it extremely unlikely. Nor can I really fathom why this would make a difference.
Posting this as a solution in case it helps, but still digging on the issue.
I understand that when we fork a process the child process inherits a copy of the parents open file descriptors and offsets. According to the man pages this refers to the same file descriptors used by the parent. Based on that theory in the following program
puts "Process #{Process.pid}"
file = File.open('sample', 'w')
forked_pid = fork do
sleep(10)
puts "Writing to file now..."
file.puts("Hello World. #{Time.now}")
end
file.puts("Welcome to winter of my discontent #{Time.now}")
file.close
file = nil
Question 1:
Shouldn't the forked process which is sleeping for 10 seconds lose its file descriptor and not be able to write to the file as the parent process completes and closes the file and exits.
Question 2: But for whatever reason if this works then how does ActiveRecord lose its connection in this scenario. It only works if I set :reconnect => true on ActiveRecord connect can it actually connect, which means its losing connection.
require "rubygems"
require "redis"
require 'active_record'
require 'mysql2'
connection = ActiveRecord::Base.establish_connection({
:adapter => 'mysql2',
:username => 'root_user',
:password => 'Pi',
:host => 'localhost',
:database => 'list_development',
:socket => '/var/lib/mysql/mysql.sock'
})
class User < ActiveRecord::Base
end
u = User.first
puts u.inspect
fork do
sleep 3
puts "*" * 50
puts User.first.inspect
puts "*" * 50
end
puts User.first.inspect
However, the same is not true with Redis (v2.4.8) which does not lose connection on a fork, again. Does the it try to reconnect internally on a fork?
If thats the case then why isn't the write file program not throwing an error.
Could somebody explain whats going on here. Thanks
If you close a file descriptor in one process it stays valid in the other process, this is why your file example works fine.
The mysql case is different because it's a socket with another process at the end. When you call close on the mysql adapter (or when the adapter gets garbage collected when ruby exits) it actually sends a "QUIT" command to the server saying that you're disconnecting, so the server tears down its side of the socket. In general you really don't want to share a mysql connection between two processes - you'll get weird errors depending on whether the two processes are trying to use the socket at the same time.
If closing a redis connection just closes the socket (as opposed to sending a "I'm going away " message to the server) then the child connection should continue to work because the socket won't actually have been closed