I am having an issue deploying an JRuby Rails application into JBoss, using JNDI to manage database connections.
After the first request I have this error:
[CachedConnectionManager] Closing a connection for you. Please close them yourself
I think this is because JBoss uses a connection pool and expect that rails (jruby) release the connection after each use, what is wrong, because rails (ActiveRecord) has its own connection pool.
I've tried to call
ActiveRecord::Base.clear_active_connections!
after each request, in a after_filter, but this haven't worked.
Does somebody have some idea?
Thanks in advance.
I am also having connection pool problems trying to run a Sinatra/ActiveRecord app in multithreaded mode in a Glassfishv3 container. The app includes a "before" block with ActiveRecord::Base.connection to force acquisition of a thread-owned connection, and an "after" block with ActiveRecord::Base.clear_active_connections! to release that connection.
Multi threaded mode
I have tried a great many variations of ActiveRecord 3.0.12 and 3.2.3, JNDI with a Glassfish connection pool, simply using ActiveRecord's connection pool, or even monkey patching the ActiveRecord connection pool to bypass it completely, using the Glassfish pool directly.
When testing with a simple multi-threaded HTTP fetcher, all variations I have tried have resulted errors, with a greater percentage of failing requests as I increase the worker threads in the HTTP fetcher.
With AR 3.0.12, the typical error is the Glassfish pool throwing a timeout exception (for what it's worth, my AR connection pool is larger than my Glassfish pool; I understand AR will pool the connection adapter object, and arjdbc will acquire and release actual connections behind the scenes).
With AR 3.2.3, I get a more ominous error. The error below came from a test not using JNDI but simply using the ActiveRecord connection pool. This is one of the better configurations in that about 95% of requests complete OK. The error requests fail with this exception:
org.jruby.exceptions.RaiseException: (ConcurrencyError) Detected invalid hash contents due to unsynchronized modifications with concurrent users
at org.jruby.RubyHash.keys(org/jruby/RubyHash.java:1356)
at ActiveRecord::ConnectionAdapters::ConnectionPool.release(/Users/pat/app/glassfish/glassfish3/glassfish/domains/domain1/applications/lookup_service/WEB-INF/gems/gems/activerecord-3.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:294)
at ActiveRecord::ConnectionAdapters::ConnectionPool.checkin(/Users/pat/app/glassfish/glassfish3/glassfish/domains/domain1/applications/lookup_service/WEB-INF/gems/gems/activerecord-3.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:282)
at MonitorMixin.mon_synchronize(classpath:/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:201)
at MonitorMixin.mon_synchronize(classpath:/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:200)
at ActiveRecord::ConnectionAdapters::ConnectionPool.checkin(/Users/pat/app/glassfish/glassfish3/glassfish/domains/domain1/applications/lookup_service/WEB-INF/gems/gems/activerecord-3.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:276)
at ActiveRecord::ConnectionAdapters::ConnectionPool.release_connection(/Users/pat/apps/glassfish/glassfish3/glassfish/domains/domain1/applications/lookup_service/WEB-INF/gems/gems/activerecord-3.2.3/lib/active_record/connection_adapters/abstrac/connection_pool.rb:110)
at ActiveRecord::ConnectionAdapters::ConnectionHandler.clear_active_connections!(/Users/pat/apps/glassfish/glassfish3/glassfish/domains/domain1/applications/lookup_service/WEB-INF/gems/gems/activerecord-3.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:375)
...
Single threaded mode
Losing confidence in the general thread safety of ActiveRecord (or arjdbc?), I gave up on using ActiveRecord multithreaded and configured warbler and JRuby-Rack do JRuby runtime pooling, emulating multiple single-threaded Ruby processes much like Unicorn, Thin, and other typical Ruby servers.
In config/warble.rb:
config.webxml.jruby.min.runtimes = 10
config.webxml.jruby.max.runtimes = 10
I reran my test, using JNDI this time, and all requests completed with no errors!
My Glassfish connection pool is size 5. Note the number of jruby runtimes is greater than the connection pool size. There is probably little point in making the JRuby runtime pool larger than the database connection pool since in this application, each request consumes a database connection, but I just wanted to make sure that even with contention for database connections I did not get the time-out errors I had seen in multithreaded mode.
These concurrency problems are disappointing to say the least. Is anyone successfully using ActiveRecord under moderate concurrency? I know that in a Rails app, one must call config.threadsafe! to avoid the global lock. Looking at the code, it does not seem to modify any ActiveRecord setting; is there some configuration of ActiveRecord I'm not doing?
The Rails connection pool does hold connections open, even after they are returned to the pool, so the underlying JDBC connection does not get closed. Are you using a JNDI connection with activerecord-jdbc-adapter (adapter: jndi or jndi: location in database.yml)?
If so, this should happen automatically, and would be a bug.
If not, you can use the code in ar-jdbc and apply the appropriate connection pool callbacks yourself.
https://github.com/jruby/activerecord-jdbc-adapter/blob/master/lib/arjdbc/jdbc/callbacks.rb
class ActiveRecord::ConnectionAdapters::JdbcAdapter
# This should force every connection to be closed when it gets checked back
# into the connection pool
include ActiveRecord::ConnectionAdapters::JndiConnectionPoolCallbacks
end
Related
Background: I have a Ruby/Rails + Nginx/Unicorn web app with connections to multiple Redis DBs (i.e. I am not using Redis.current and am instead using global variables for my different connections). I understanding that I need to create a new connection in the after_fork block when a new Unicorn worker is created, as explained here and here.
My question is about the need for connection pooling. According to this SO thread, "In Unicorn each process establishes its own connection pool, so you if your db pool setting is 5 and you have 5 Unicorn workers then you can have up to 25 connections. However, since each unicorn worker can handle only one connection at a time, then unless your app uses threading internally each worker will only actually use one db connection... Having a pool size greater than 1 means each Unicorn worker has access to connections it can't use, but it won't actually open the connections, so that doesn't matter."
Since I am NOT using Sidekiq, do I even need to use connection pools for my Redis connections? Is there any benefit of a connection pool with a pool size of 1? Or should I simply use variables with single connections -- e.g. Redis.new(url: ENV["MY_CACHE"])?
Connection pool is only used when ActiveRecord talks to the SQL databases defined in your databases.yml config file. It is not related to Redis at all and the SO answer that you cite is actually not relevant for Redis.
So, unless you wanted to use some custom connection pool solution for Redis, you don't have to deal with it at all, as there is no pool for Redis in Rails by default. I guess the custom pool might be suitable if you had multiple threads in your applications which is not your case.
Update: Does building a connection pool make sense in your scenario? I doubt it. Connection pool is a way to reuse open connections (typically among multiple threads / requests). But you say that you:
use unicorn, the workers of which are separate, independent processes, not threads,
open a stable connection (or two) during after_fork, a connection which is then open all the time the unicorn worker lives
do not use threads in your application anywhere (I'd check if this is true again - it's not only Sidekiq but it might be any gem that tends to do things in the background).
In such scenario, pooling connection to Redis makes no sense to me as there seems to be no code that would benefit from reusing the connection - it is open all the time anyway.
I have a Rails application that I want to connect to a Redis data structure server. I'm wondering how I should proceed. I'm using a global variable $redis locate at config/initializers/redis.rb to make queries across the entire application.
I believe this approach it is not suitable for a application with 80+ simultaneous connections, because it uses one single global variable to handle the Redis connection.
What should I do to overcome this problem? am I missing something about Rails internals?
Tutorial I'm following
http://jimneath.org/2011/03/24/using-redis-with-ruby-on-rails.html
This depends on the application server you will use. If you're using Unicorn which is a popular choice you should be fine.
Unicorn forks it's workers and each one will establish it's own database connection. And since each worker can only handle one request at a time it will only need one connection at a time. Adding more connections won't increase performance, it just will open more (useless) connections.
ActiveRecord (which is the DB-part of Rails) or DataMapper support connection pooling which is a common solution to overcome the problem you've mentioned. Connection pooling however only make sense in a threaded environment.
On top of that Redis is mainly single threaded (search for "Single threaded nature of Redis") so there might be no advantages anyway. There was an request to add connection pooling but it got closed, you might get more information from there.
I am getting
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds (waited 5.000798 seconds). The max pool size is currently 1; consider increasing it.)
when I try to run WEBrick (rails server) with the pool size 1, and no problems with higher pool sizes.
What does WEBrick use the first connection on, and what is the best pool size for a single-threaded application? Is this a WEBrick-specific issue, or it applies to any other servers (like Unicorn)?
Rails version is 3.2.13
Update. Just verified this with unicorn, it works fine with single connection.
If I recall correctly, Rails reserves a connection to the database when it boots up and uses the remainder of the connections available in the connection pool to handle requests. Even if you never touch ActiveRecord objects during the life of the request, Rails will still try to reserve one connection from the pool for each request, or will block until one is available up until the timeout limit.
The default pool size is 5 connections: 1 reserved for Rails + 4 available for requests.
Rails does this to maintain thread-safety in the application.
If your application is single-threaded and only processes one request at a time with no regard for concurrency, the number of connections in the pool should be set to 2 at an absolute minimum. I'd still recommend the default of 5 though so you have some breathing room in case you need to utilize more than one connection per request.
This is not specific to WEBrick. The connection pool limit affects the application the same regardless of what application server is running.
What are the benefits to using an external connection pool?
I've heard that most other applications will open up a connection for each unit of work. In Rails, for example, I'd take that to mean that each request could open a new connection. I'm assuming a connection pool would make that possible.
The only benefit I can think of is that it allows you to have 1,000 frontend processes without having 1,000 postgres processes running.
Are there any other benefits?
Rails has connection pooling built in:
Simply use ActiveRecord::Base.connection as with Active Record 2.1 and earlier (pre-connection-pooling). Eventually, when you’re done with the connection(s) and wish it to be returned to the pool, you call ActiveRecord::Base.clear_active_connections!. This will be the default behavior for Active Record when used in conjunction with Action Pack’s request handling cycle.
Manually check out a connection from the pool with ActiveRecord::Base.connection_pool.checkout. You are responsible for returning this connection to the pool when finished by calling ActiveRecord::Base.connection_pool.checkin(connection).
Use ActiveRecord::Base.connection_pool.with_connection(&block), which obtains a connection, yields it as the sole argument to the block, and returns it to the pool after the block completes.
This has been available since version 2.2. You'll see a pool parameter in your database.yml for controlling it:
pool: number indicating size of connection pool (default 5)
I don't think there would be much point in layering another pooling system underneath it and it could even confuse AR's pooling if you tried it.
We're running a JRuby on Rails application on Jetty, and having reached the staging server prior to launch have suddenly hit a problem with our JDBC connections being abandoned. Here's a lovely stacktrace to illustrate:
Last packet sent to the server was 12 ms ago.
STACKTRACE:
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1913)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2304)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803)
From reading around my understanding is that MySQL is killing our connection pool over a period of time due to inactivity (which makes sense as staging is under very light load right now). It's running under JRuby 1.3.1 with the following gems:
activerecord-jdbc-adapter (0.9.1)
activerecord-jdbcmysql-adapter (0.9.1)
jdbc-mysql (5.0.4)
I'm assuming that I probably need to set some JDBC configuration somehow to ensure the connections are kept alive or recycled properly, but I need some help finding out where to look. Can anyone furnish me with the details?
Thanks,
Steve
This is probably due to the wait_timeout setting. You could try increasing it to something very large, but that assumes you have administrative access on the database server.
I've never used JRuby or Rails. But in "regular" Java, the way I would solve this is to use a connection pool which automatically recycles idle connections. For example, c3p0 has a maxIdleTime setting that controls this.
EDIT: Just for fun, I did a Google search on "activerecord idle connection" and I got a few hits. Here's one: http://groups.google.com/group/sinatrarb/browse_thread/thread/54138bfedac59849
Apparently there is a method called ActiveRecord::Base.verify_active_connections! that you can use. I make no guarantees whatsoever about this solution :-). IANARP (I am not a Ruby programmer).
Either Rails or our ActiveRecord-JDBC code should probably provide for a periodic connection ping or idle-time teardown. Connections being culled by the server is a standard case any connection pooling implementation should be able to handle.
I'd say file a bug with ActiveRecord-JDBC on kenai.com, but first ask on JRuby ML if anyone else has found a solid solution for this.