I installed redmine on the apache and used mod_ruby first, which was incredible slow... now i switched to phusion passenger but the response time is still really slow ( talking about 5-6 seconds here, even using a wget to localhost from the server itself.. )
i just removed the "old" mods from the apache dir, but it's still slow... anyway, the logfile at least shows, that the passenger is used:
127.0.0.1 - - [15/Nov/2009:10:38:25 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.9
(Debian) Phusion_Passenger/2.2.5 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_ssl/2.2.9
OpenSSL/0.9.8g mod_perl/2.0.4 Perl/v5.10.0 (internal dummy connection)"
I have no idea why this happens, the server should be fast enough.. apache-log isn't showing anything suspicious..
EDIT:
Thanks for the hint..
The "passenger-status" is "empty":
----------- General information -----------
max = 6
count = 0
active = 0
inactive = 0
Waiting on global queue: 0
Any advice? Thanks!
Try increasing the PoolIdleTime setting (which is 2 Minutes by default I think). Setting it to 0 helped speed the startup of my Redmine stack a lot. Check out this question on Serverfault on which values to set.
You can use the config option PassengerMinInstances, avaliable since Passenger 3.0.0. This setting allows you to tell Apache how many instances of your deployment must stay alive, even when your applications have been idle for a logner period than the defined by PoolIdleTime. Have a look at the Phusion Passenger docs. There are some otrher usefull options to improve your deoployment's performance.
This answer may be a bit outdated --I'm quite sure almost everyone know the new features of Passenger, but I wasn't, and this question helped a lot.
I found a tool ( http://www.wekkars.com ) that keeps my application alive. I just updated the PoolIdleTime to 30 minutes and the tool does the rest...
Related
I have a Rails system in which every half hour, the following is done:
There are 15 clients somewhere else on the network
The server creates a record called Measurement for each of these clients
The measurement records are configured, and then they are run asynchronously via Sidekiq, using MeasurementWorker.perform_async(m.id)
The connection to the client is done with Celluloid actors and a WebSocket client
Each measurement, when run, creates a number of event records that are stored in the database
The system has been running well with 5 clients, but now I am at 15, and many of the measurements don't run anymore when I start them at the same time, with the following error:
2015-02-04T07:30:10.410Z 35519 TID-owd4683iw MeasurementWorker JID-15f6b396ae9e3e3cb2ee3f66 INFO: fail: 5.001 sec
2015-02-04T07:30:10.412Z 35519 TID-owd4683iw WARN: {"retry"=>false, "queue"=>"default", "backtrace"=>true, "class"=>"MeasurementWorker", "ar
gs"=>[6504], "jid"=>"15f6b396ae9e3e3cb2ee3f66", "enqueued_at"=>1423035005.4078047}
2015-02-04T07:30:10.412Z 35519 TID-owd4683iw WARN: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2015-02-04T07:30:10.412Z 35519 TID-owd4683iw WARN: /home/webtv/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.1.4/lib/active_
record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
....
Now, my production environment looks like this:
config/sidekiq.yml
production:
:verbose: false
:logfile: ./log/sidekiq.log
:poll_interval: 5
:concurrency: 50
config/unicorn.rb
...
worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 60
...
config/database.yml
production:
adapter: postgresql
database: ***
username: ***
password: ***
host: 127.0.0.1
pool: 50
postgresql.conf
max_connections = 100 # default
As you see, I've already increased the concurrency of Sidekiq to 50, to cater for a high number of possible concurrent measurements. I've set the database pool to 50, which already looks like overkill to me.
I should add that the server itself is quite powerful, with 8 GB RAM and a quad-core Xeon E5-2403 1.8 GHz.
What should these values ideally be set to? What formula can I use to calculate them? (E.g. number of maximum DB connections = Unicorn workers × Sidekiq concurrency × N)
It looks to me like your pool configuration of 100 is not taking affect. Each process will need a max of 50 so change 100 to 50. I don't know if you are using Heroku but it is notoriously tough to configure the pool size.
Inside mysql, your max connection count should look like this:
((Unicorn processes) * 1) + ((sidekiq processes) * 50)
Unicorn is single threaded and never needs more than one connection unless you are spinning up your own threads in your Rails app for some reason.
I'm sure the creator of sidekiq #MikePerham is more than suited to the task of fixing your sidekiq issues but as a ruby dev two things stand out.
If you're doing a lot of database operations via ruby can you push some of them into the database as triggers? You could still start them on the appside with a sidekiq process of course. :)
Second every half hour screams to me of a rake task run via cron. Hope you're doing that too. FWIW I usually use the Whenever gem to create the cron line I have to drop into the crontab of the user running the app. Note its designed to autocreate the crontask in a scripted deploy but in a non-scripted one you can still leverage it to give you the lines you have to paste into your crontab though via the whenever command.
Also you mention this is for measurements.
Have you considered leveraging something like elasticsearch and the searchkick gem? This is a little more of a complex setup, be sure to firewall the server you install ES on. But this might make your code a lot more manageable as you grow. Also it gives you a good search mechanism almost for free and its distributed and more language agnostic, e.g. Bloodhound, Java. :) Plus kibana gives you a nice window into the ES records
We have been trying to make use of the RabbitMQ Service Bus (v3.3.4) but the central bus keeps crashing. At the moment we are not using any clustering and its hosted on Windows Server 2008 R2. We'd like to isolate the root cause but the below error is the only one we can find. Can anyone shed some light on what; if anything; we can do to find the root cause of this?
Note: There are roughly 20 consumers with roughly the same number of Topic subscriptions. Also, all the clients are .NET 4.5 using the 3.3.4 Rabbit client libraries.
Version=1
EventType=APPCRASH
EventTime=130658038736577295
ReportType=2
Consent=1
ReportIdentifier=7f93ccd8-9cbe-11e4-ae00-000c29c08139
IntegratorReportIdentifier=7f93ccd7-9cbe-11e4-ae00-000c29c08139
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=erl.exe
Sig[1].Name=Application Version
Sig[1].Value=0.0.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=5343035d
Sig[3].Name=Fault Module Name
Sig[3].Value=MSVCR100.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=10.0.30319.1
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=4ba220dc
Sig[6].Name=Exception Code
Sig[6].Value=40000015
Sig[7].Name=Exception Offset
Sig[7].Value=00000000000760d9
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=8d79
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=8d79a00078e92d9c3d5d79d4324254fe
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=9af5
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=9af5b20633c279dbf44b04a614c6a1f6
UI[2]=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=erl.exe stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\MSVCR100.dll
LoadedModule[5]=C:\Program Files\erl6.0\erts-6.0\bin\erlexec.dll
LoadedModule[6]=C:\Windows\system32\USER32.dll
LoadedModule[7]=C:\Windows\system32\GDI32.dll
LoadedModule[8]=C:\Windows\system32\LPK.dll
LoadedModule[9]=C:\Windows\system32\USP10.dll
LoadedModule[10]=C:\Windows\system32\msvcrt.dll
LoadedModule[11]=C:\Windows\system32\IMM32.DLL
LoadedModule[12]=C:\Windows\system32\MSCTF.dll
LoadedModule[13]=C:\Windows\system32\apphelp.dll
LoadedModule[14]=C:\Program Files\erl6.0\erts-6.0\bin\beam.dll
LoadedModule[15]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[16]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[17]=C:\Windows\system32\RPCRT4.dll
LoadedModule[18]=C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7600.16661_none_fa62ad231704eab7\COMCTL32.dll
LoadedModule[19]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[20]=C:\Windows\system32\COMDLG32.dll
LoadedModule[21]=C:\Windows\system32\SHELL32.dll
LoadedModule[22]=C:\Windows\system32\WS2_32.dll
LoadedModule[23]=C:\Windows\system32\NSI.dll
LoadedModule[24]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[25]=C:\Windows\system32\WINNSI.DLL
LoadedModule[26]=C:\Windows\system32\mswsock.dll
LoadedModule[27]=C:\Windows\System32\wshtcpip.dll
LoadedModule[28]=C:\Windows\system32\NLAapi.dll
LoadedModule[29]=C:\Windows\system32\DNSAPI.dll
LoadedModule[30]=C:\Windows\System32\winrnr.dll
LoadedModule[31]=C:\Windows\system32\napinsp.dll
LoadedModule[32]=C:\Windows\System32\wship6.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=erl.exe
AppPath=C:\Program Files\erl6.0\erts-6.0\bin\erl.exe
So I'm getting page load times in the range of 30-45 seconds.
Some history:
This was not always the case for this project. This project is in production so I haven't really touched the code in a while. I noticed it started happening the last time I was updating the code. I don't recall anything specific that I changed that should have anything to do with the problem. I have other projects that are running with the same Grails versions with no problem.
I think it started happening in 2.2.3. I am now running 2.2.4.
I am using x64 JDK 1.7.0_25, Windows 7 x64.
I'm not sure what else to put here that would be relevant. Any assistance is appreciated!
Edit: running with -noreloading has no effect.
Edit2: I've tried deleting my .grails folder entirely, running clean, and deleting my target folder and stacktrace log.
Edit3: It does seem that the amount of time it takes is dependent on the amount of data displayed/read. Small pages take 3-4 seconds. Medium pages 10-12 seconds...
Edit4: I'm running it via IntelliJ IDEA 12.1.4 x64 (idea64.exe). I've also tried it outside of IntelliJ with the same results.
Edit5: The database is Oracle enterprise that supports the entire company. It is managed by full time adminstrators. This isn't a MySQL server on my local machine.
Edit6: The application also functions normally when deployed in TEST (test war), but still is slow when ran with test run-app.
Starting to get somewhere:
I downloaded JDK 1.7.21 and ran the app with that and it started working no problem! I then ran clean which triggered a recompile and it stopped working... grr
Now with 1.7.21 still active, I tried -noreloading and it works!
Annnd... now it works even if I don't use -noreloading..........
I've gone back to 1.7.25.. ran clean, and it works. Sooooooo yeah... explain that.
And now it doesn't anymore.
This is under Linux but will maybe useful:
If you are running the code within an IDE:
ps auwx|grep java
-Dgrails.console.class=grails.build.logging.GrailsEclipseConsole -Dosgi.requiredJavaVersion=1.6 -Xms40m -Xmx768m -XX:MaxPermSize=256m -
As you can see the memory settings Xms and Xmx are quite low...
In your IDE there should be an INI file:
more STS.ini
1 -vm
2 /usr/lib/jvm/java-6-openjdk-amd64/bin/java
3 -startup
4 plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
5 --launcher.library
6 plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.200.v20120913-144807
7 -product
8 org.springsource.sts.ide
9 --launcher.defaultAction
10 openFile
11 -vmargs
12 -Dgrails.console.enable.interactive=false
13 -Dgrails.console.enable.terminal=false
14 -Djline.terminal=jline.UnsupportedTerminal
15 -Dgrails.console.class=grails.build.logging.GrailsEclipseConsole
16 -Dosgi.requiredJavaVersion=1.6
17 -Xms40m
18 -Xmx768m
19 -XX:MaxPermSize=256m
You can up these value and try restarting your IDE...
I would also suggest you run something like nmon before/during and monitor whilst the code is running and monitor disk/cpu/network throughputs.
You may find you are hammering your dev box which is causing the issue.
If the production is fine I really don't see what the problem is..
E2A Ahhh forgot it was under windows so no nmon for windblows but hey not that I tried it - http://sourceforge.net/projects/jnmonanalyser/
E2A again:
1. Enable DataSource.groovy debugging:
dataSource {
pooled = true
driverClassName ="com.mysql.jdbc.Driver"
username = "aaa"
password = "aaaa"
//SQL Logging - refer to Config.groovy at hibernate.sql now
logSql=true
...
config.groovy - this will stop your app from running if you have issues with lets say records you are trying to add in your BootStrap
// Return error when it fails
//grails.gorm.failOnError=true
Enable log4j and use this or part of it:
// log4j configuration
log4j {
appender.stdout = "org.apache.log4j.ConsoleAppender"
appender.'stdout.layout'="org.apache.log4j.PatternLayout"
appender.'stdout.layout.ConversionPattern'='[%r] %c{2} %m%n'
appender.stacktraceLog = "org.apache.log4j.FileAppender"
appender.'stacktraceLog.layout'="org.apache.log4j.PatternLayout"
appender.'stacktraceLog.layout.ConversionPattern'='[%r] %c{2} %m%n'
appender.'stacktraceLog.File'="stacktrace.log"
appender.'stacktraceLog.MaxFileSize'="1MB"
rootLogger="error,stdout"
logger {
grails="error"
StackTrace="error,stacktraceLog"
org {
codehaus.groovy.grails.web.servlet="error" // controllers
codehaus.groovy.grails.web.pages="error" // GSP
codehaus.groovy.grails.web.sitemesh="error" // layouts
codehaus.groovy.grails."web.mapping.filter"="error" // URL mapping
codehaus.groovy.grails."web.mapping"="error" // URL mapping
codehaus.groovy.grails.commons="info" // core / classloading
codehaus.groovy.grails.plugins="error" // plugins
codehaus.groovy.grails.orm.hibernate="error" // hibernate integration
// Hibernate should be on - if you want to catch sql logs
springframework="off"
hibernate="on"
//hibernate.SQL = 'debug'
//hibernate.type = 'trace'
//hibernate.SQL = 'info,hibernate'
//hibernate.type = 'info,hibernate'
//hibernate = 'info,hibernate'
//apache.commons.digester.Digester = 'debug,javaclasses'
}
}
additivity.StackTrace=false
}
try and capture what it is doing, it is also worth running developer tools on your browser whether its firefox of chrome and trying to figure out on what elements it is taking that time - but between the logs and the browser developer tools should lie your answer.
Usually you can fix this by doing
grails clean
on the grails command line (I open it via CRTL+ALT+G in IntelliJ IDEA).
This erases all compiled files and will recompile your project from scratch (afaik), which usually erases errors like that. This is not a real fix for the underlying problem, but it solves the problem. Grails is highly experimental and unstable if you ask me, i have a lot of weird error that usually disappear when doing a clean. Btw i'm using 2.1.5 on Windows 7 x64, too.
Delete stacktrace file in the target folder of your project. It can
get huge. (At present mine is 48 GB).
Check if there is enough space in your C directory.
If you are hot swapping code, then page loads can get slow. So in such cases, restart the dev server (grails app).
Sometimes, requests to the server can hang, where focusing (left or right clicking on the cmd) on the command prompt seems to skip the pause. (weird)
Increasing the JVM permgen, heap spaces depending on your memory might help as well.
Try running the server using command prompt rather within an IDE.
Better use methods for actions than closures.
For a system with 3GB RAM, my environment variable setting is:
JAVA_OPTS
-Xms512m -Xmx1g
The STS.ini settings:
-startup
plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.1.100.v20110502
-product
com.springsource.sts.ide
--launcher.defaultAction
openFile
--launcher.XXMaxPermSize
384M
-vmargs
-Dosgi.requiredJavaVersion=1.5
-Xmn128m
-Xms1024m
-Xmx1024m
-Xss2m
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:MaxGCPauseMillis=10
-XX:MaxHeapFreeRatio=50
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:+UserParallelGC
8) Maybe the problem is with the JDK and grails versions combination. There seems to be an error with OpenJDK 1.7u25 and spring loaded. Okay, you are not using OpenJDK, but try with other version anyway. Try with JDK1.7u03.
9) Try JVM with -server flag, and see if it improves runtime performance.
grails run-app -server
So the reason why this was happening:
JDK 1.7.25
I am running a Rails 3.1.0 app and I have an odd problem. On our staging server, with VERY little activity we have 5 ruby processes CONSTANTLY pinging mySQL with the following:
poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(12, "\f\0\0\0\3SHOW TABLES", 16) = 16
select(13, [12], NULL, NULL, NULL) = 1 (in [12])
read(12, "\1\0\0\1\1D\0\0\2\3def\0\vTABLE_NAMES\0\31Tabl"..., 16384) = 637
poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(12, "\f\0\0\0\3SHOW TABLES", 16) = 16
select(13, [12], NULL, NULL, NULL)
That last line is incomplete, but we're talking a few times every single second (x5/6 processes). The server is a beast, it has 32GB of RAM and has been optimised somewhat (the mySQL setup that is) but its killing the server.
Like I say, the server has very little activity, so its not users, or a task.
(For admins thinking of moving this away from this forum, I believe this is a ruby/rails issue, I'm not sure if it was in a server forum it would have a good compatibility with answerers)
I would be incredibly grateful for any advice, I fear it might be a bit over my head. I'm not such a Linux/mySQL pro.
Thanks
I would look at the connection pool for your database. Does running this help?
ActiveRecord::Base.clear_active_connections!
Specifically, in your config/database.yml for this environment, try setting pool: 50 and restart rails, then see if this affects the result. The next question if your pool is exhausted would be to get to the specifics of why the database connection pool is getting used up (this command, or something running in resque). I think the pool default size is 4 or 5
This question already has an answer here:
Overriding/Modifying Rails Class (ActiveResource)
(1 answer)
Closed 3 years ago.
I'm trying to contact a REST API using ActiveResource on Rails 2.3.2.
I'm attempting to use the timeout functionality so that if the resource I'm contacting is down I can fail quickly - I'm doing this with the following:
class WorkspaceResource < ActiveResource::Base
self.timeout = 5
self.site = "http://mysite.com/restAPI"
end
However, when I try to contact the service when I know it isn't available, the class only times out after the default 60 seconds. I can see from the error stack that the timeout error does indeed come from an ActiveResource class in my gem folder that has the proper functions to allow timeout settings, but my set timeout never seems to work.
Any thoughts?
So apparently the issue is not that timeout is not functioning. I can run a server locally, make it not return a response within the timeout limit, and see that timeout works.
The issue is in fact that if the server does not accept the connection, timeout does not function as I expected it to - it doesn't function at all. It appears as though timeout only works when the server accepts the connection but takes too long to respond.
To me, this seems like an issue - shouldn't timeout also work when the server I'm contacting is down? If not, there should be another mechanism to stop a bunch of requests from hanging...anyone know of a quick way to do this?
The problem
If you're running on Ruby 1.8.x then the problem is its lack of real system threads.
As you can read first hereand then here, there are systemic problems with timeouts in Ruby. An interesting discussion but for you in particular some comments suggest that the timeout is effectively ignored and defaults to 60 seconds - exactly what you are seeing.
Solutions ...
I have a similar issue with our own product when trying to send emails - if the email server is down the thread blocks. For me the solution was to spin the request off on a separate thread and therefore my main request-processing thread doesn't block.
There are non-blocking libraries out there for Ruby but perhaps you could take a look first at this System Timeout Gem.
An option open to anyone using Rails behind a proxy like nginx would be to set the upstream timeout to a lower number - that way you'll get notified if the server is taking too long. I'd only do this if I were really stuck for a solution.
Last but not least, it's possible that running Rails 2.3.2 on top of Ruby 1.9.1 will fix the issue.
Alternatively, you could try to catch these connection errors and retry once (after certain period of time) just to make sure the connection is really out.
retried = false
begin
#businesses = Business.find(:all, :params => { :shop_domain => #shop.domain })
retried = false
rescue ActiveResource::TimeoutError => ex
#raise ex
rescue ActiveResource::ConnectionError, ActiveResource::ServerError, ActiveResource::ClientError => ex
unless retried
sleep(((ex.respond_to?(:response) && ex.response['Retry-After']) || 5).to_i)
retried = true
retry
else
# raise ex
end
end
Inspired by this solution from Shopify for paginating a large number of records. https://ecommerce.shopify.com/c/shopify-apis-and-technology/t/paginate-api-results-113066