We have a Java app that uses Groovy over Grails framework. We recently upgraded from Grails 3.x to Grails 5.x, and seeing performance degradation under load. The application server's CPU utilization has increased by 20%, and upon taking thread dumps, we see that multiple threads are blocked that are running the above method.
Although this method was present in Grails 3.x, the number of threads blocked were less and block time was also almost negligible. I am not able to find the documentation on this method. Appreciate any help.
Related
I have this working grails app, and i want to determine how many calls various parts of it's API gets per time unit.
This app offers 100+ web services, and 10k+ open connections at any time, so it is pretty busy.
It's written in grails 1.3.6 (developed over 10 years).
My overall target is to gain knowledge about which interfaces are actually used and how often.
I see several poor possibilities:
Log everything, parse the logs and 'grep -c'.... (Bad idea)
tcpdump... (Bad idea)
Increment global variables (thread safe) where i want to instrument it (least bad idea)
How do i get grails to tell me which services are called and how often?
...without killing performance?
Any ideas/pointers will be appreciated :-)
I'm trying to speedup the startup of a grails 2.3.7 application.
Part of this has been to move stuff over to a RamDrive and start the project and inteli-j from there.
I have noticed though that grails tries to read many files, and in many cases these files are not there or the path is not there.
It seems very hectic and disorganized.
Does anyone have any idea how to improve and avoid these redundant and inefficient system calls as well as how to speedup startup?
Is it a matter of grails itself or the specific plugins being included?
Picture available at screenshot as well.
Additionally, below please find the graphs for the various operations performed during startup. Unfortunately the CPU usage is never up to 100% meaning grails startup may not be optimized to use all cores.
I am expericencing heavy performance problems with generating PDFs while using Jasper Reports in my grails application. I am invoking the jasperService:
def reportDef = jasperService.buildReportDefinition(parameter, LocaleContextHolder.getLocale(), [data: emptyData])
Running in Jboss several times, performance is good. After X hours, performance is 100+ times worse than after the start of Jboss... Response time is changing from 7-12 seconds to several minutes for creating a PDF with one single page. I am sure, that the performance lag is within this invocation, because I have added time measurements around it. As the report data is passed within the parameters, I can exclude also data base connection issues.
I have analyzed the HEAP, but it is used ~50% and not changing much during PDF creation. Overall memory is also not fully used.
I have analyzed the PermGen, but it is also far from being full.
The CPU ist permanently at 100% during creation, which is ok, knowing that PDF creation is very CPU consuming. I have ensured that no other process is holding the PDF creation up, 1st by restarting the process several times and measuring no difference, so I can exclude external interruption and 2nd) knowing that performance is much better if JBoss is restarted.
Due to the facts, I have started to analyze the JBoss itself by analyzing the Thread dumps while running the PDF creation thread. I see that nothing else is running (except the thread dumping thread), neither when it is slow nor fast after restart. I can just see that in several Thread dumps Groovy is making several AST transformations which is not strange for Groovy...
Now, I am despaired. HEAP/PermGen is ok, CPU is ok. What the hell is Jasper Reports / Grails doing?
Maybe someone has made similar experiences or an idea for the root cause? Is there something which needs/should to be cleaned up in Jasper Reports?
EDIT: My further analysis yield to the unproofed but certain outcome that JBoss 7.1.1 (latest stable) is the root cause. After installing the app on a Tomcat, everything runs smoothly, also after several days. I'll keep this open. Maybe someone has made same experience and likes to post it...? Otherwise, I will close it with this solution. I will maybe test my app on earlier versions of Jboss or 7.2/7.3.
The solution was that we haven't perceived that JBoss was partially ignoring our Log4J configuration and was massively logging into the server.log which we were not monitoring. Jasper and Grails plugins were writing dozens of MB for each PDF generation into the log file. After removing these log inserts, performance was good again.
in my Grails application using the Spring Security Core plugin for authentication. I am facing a serious problem with that because my application took 21 seconds to lift the Tomcat was carrying 43/2 after installation.
So far so good, but began to occur error 'PermGen Error' memory error Tomcat server. Before it was 64 and Aug is 256 so that the error does not crash my app so often.
I wonder whether you know some plugin configuration in order to reduce the incidence of this error or some method to effect the release of this cache because the number of users is increasing and if you can not solve it unfortunately have to leave the plugin I seems to be an excellent choice for application security.
Someone could tell me if the amount of plugins used in an application interference has this memory?
PermGen is a part of memory to store the static components of your app, mostly classes. Literally it will not be affected by either the amount of users or logs associated with user activities, which consumes heap space instead.
To reduce PermGen storage, you have to check your code, redesign those algorithms which contains unnecessary/redundant objects and operations, and consolidate variables and functions if possible. Generally speaking, simplified code will produce smaller executable files. That's the way you save the PermGen space.
Some versions of Tomcat permgen more than others. There was a minor version in the 6 line that I couldn't every get to reliably stay running. And even with the latest versions you still need to tweak your memory settings. I use the following and it works best for me. I still get them now and again, especially if I'm doing a lot of runtime compiling. In production, it is a non-issue because all the development overhead of grails isn't there.
-XX:MaxPermSize=512m -XX:PermSize=512m -Xms256m -Xmx1024m
I have a deployment using:
rails 2.3.2
ruby 1.8.7
mysql db
and
3 mongrel instances (windows services) with apache as load balancer
[I know it is due for upgrade...]
OS: Windows2003
We have many CPU intensive tasks and when these occur on the 4 core machine the mongrel process is able to only use a max 25% cpu power on the core the task was scheduled.
After running many tests we noticed that it is only able to use the power of a single core and therefore there is time lag in finishing tasks.
There is a suggestion to virtualize... which is difficult to do on the client server.
Has anyone got any suggestion on how the situation can be improved? Memory does reach 250MB to 1GB for this process but this not such a big issue.
Thanks in advance
Linus
The typically used Ruby versions (MRI or YARV, i.e 1.8 or 1.9) are not able to use more than one core at a time. MRI is single-threaded and just provides green threads internally. YARV uses real OS threads but has a GIL (global interpreter lock) that ensures that only one thread is running at a time.
Thus, your mongrels are unable to use more than one core each (even if you would have coded your Rails app to be threadsafe). There are alternative Ruby implementations, like JRuby or Rubinius that provide native threads without a global interpreter lock and thus allow your app to use more than one core, but you'd probably need to adapt your app a bit.
That said, even then a single request would run in a single thread and thus only use a single core. But that is something that is hard to come by without you handling your own threads (or at least fibers in 1.9) which is most probably not worth the hassle.
So generally, the recommendation is to start multiple app server processes (mongrels in your case). I personally use about 1.5 - 3 per core (depending on the app). That way, you are able to answer that many parallel requests and fully use your available CPU power shared between them.