Simple rails app on Puma throws segfault, cannot handle concurrency - ruby-on-rails

Question

I have a fairly simple Rails app. It listens for requests in the form
example.com/items?access_key=my_secret_key
My application controller looks at the secret key to determine which user is making the call, looks up their database credentials, and connects to the appropriate database to get that person's items.
However we need to have this support multiple requests at a time, and Puma seems like everyone's favorite / the fastest server for us to use. We started running into problems when benchmarking it with ApacheBench. FYI, puma is configured to have 3 workers and min=1, max=16 threads.
If I were to run
ab -n 100 -c 10 127.0.0.1:3000/items?access_key=my_key
then this error is thrown with a whole lot of stack trace after it:
/home/user/.gem/ruby/2.0.0/gems/mysql2-0.3.16/lib/mysql2/client.rb:70: [BUG] Segmentation fault
ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
Edit: This line also appears in the enormous amount of info that the error contains:
*** glibc detected *** puma: cluster worker 1: 17088: corrupted double-linked list: 0x00007fb671ddbd60
And it looks to me like that's tripping multiple times. I have been unable to determine exactly when (on which requests) it trips.
The benchmarking seems to still finish, but it seems quite slow (from ab):
Concurrency Level: 10
Time taken for tests: 21.085 seconds
Complete requests: 100
Total transferred: 3620724 bytes
21 seconds for 3 megabytes? Even if mysql was being slow, that's... bad. But I think it's worse than that - the amount of data isn't high enough. There are no segfaults when I run concurrency 1, and the amount of data for -n 10 -c 1 is 17 megabytes. So puma is responding with some error page that I can't see - running 'curl address' gives me the expected data, and I can't manually do concurrency.
It gets worse when I run more requests or higher concurrency.
ab -n 1000 -c 10 127.0.0.1:3000/items?access_key=my_key
yields
apr_socket_recv: Connection reset by peer (104)
Total of 199 requests completed
and
ab -n 100 -c 50 127.0.0.1:3000/items?access_key=my_key
yields
apr_socket_recv: Connection reset by peer (104)
Total of 6 requests completed
Running top in another putty window shows me that very often (most times I try to benchmark) only one of the three workers puma created is performing any work. Rarely, all three do.
Because it seems like the error might be somewhere in here, I'll show you my application_controller. It's short, but the bulk of the application (which, like I said, is fairly simple).
class ApplicationController < ActionController::Base
# Prevent CSRF attacks by raising an exception.
# For APIs, you may want to use :null_session instead.
protect_from_forgery with: :exception
def get_yaml_params
YAML.load(File.read("#{APP_ROOT}/config/ecommerce_config.yml"))
end
def access_key_login
access_key = params[:access_key]
unless access_key
show_error("missing access_key parameter")
return false
end
access_info = get_yaml_params
unless client_login = access_info[access_key]
show_error("invalid access_key")
return false
end
status = ActiveRecord::Base.establish_connection(
:adapter => "mysql2",
:host => client_login["host"],
:username => client_login["username"],
:password => client_login["password"],
:database => client_login["database"]
)
end
def generate_json (columns, footer)
// config/application.rb includes the line
// require 'json'
query = "select"
columns.each do |column, name|
query += " #{column}"
query += " AS #{name}" unless column == name
query += ","
end
query = query[0..-2] # trim ','
query += " #{footer}"
dbh = ActiveRecord::Base.connection
results = dbh.select_all(query).to_hash
data = results.map do |result|
columns.map {|column, name| result[name]}
end
({"fields" => columns.values, "values" => data}).to_json
end
def show_error(msg)
render(:text => "Error: #{msg}\n")
nil
end
end
And an example of a controller that uses it
class CategoriesController < ApplicationController
def index
access_key_login or return
columns = {
"prd_type" => "prd_type",
"prd_type_description" => "description"
}
footer = "from cin_desc;"
json = generate_json(columns, footer)
render(:json => json)
end
end
That's pretty much it as far as custom code goes. I can't find anything making this not threadsafe, so I don't know what the cause of the segfaults is. I don't know why not all of the workers spin up when requests are made. I don't know what error is getting returned to ApacheBench. Thanks for helping, I can post more information as you need it.

Answer 1

It appears that the stable version of mysql2 library, 0.3.17, is NOT threadsafe. Until it is updated to be threadsafe, using it with multithreaded puma will be impossible. An alternative would be to use Unicorn.