Can someone suggest a good tomcat memory profiler? (I am trying to profile the memory usage of a JRuby on Rails app deployed on a Tomcat instance in a Windows machine).
You can use VisualVM which is part of the JDK - more recent versions can be downloaded from visualvm.java.net/.
Or you can use the integrated profiler from NetBeans which is essentially the same thing as VisualVM just with some additional features.
I personally use YourKit. You can download trial version and see if it is good for you.
Last time I checked VisualVM does not show aggregated numbers. This means that you can see that, for example, your app has 10000 allocations of int[] array but you have no clue where these come from. YourKit has Biggest objects (dominators) screen which is quite helpful; you can see which objects take the most memory overall.
Related
I've been tasked with analyzing some memory consumption issues with one of our web apps. I'd made myself passibly familiar with tools like Mission Control and VisualVM and used them to resolve a number of leaks, but in doing so came across behavior for which I can't account.
Setup
JBoss 7.1.1 AS
Java 1.7.0_67
Specifically, I've found that even when I run only JBoss 7 by itself (that is, I turn off the deployer and just let the server itself run) I can see regular allocations (followed by garbage collection) of about 1MB/3 seconds or so.
On a whim, I took heap dumps immediately after doing a GC and then once the allocations had been going on. It seems like the majority of the objects I'm seeing have to do with modules, either Xerces activity (reading the module XML, I guess?) or objects associated with ModuleLoader. The majority of the objects I see all have 'References' that look something like this:
http://i.stack.imgur.com/LlUmv.png (sorry, I can't mark up images)
My thinking (which may be entirely off base) was that JBoss scans for new modules to support hot deploys? The thing is though, that use case isn't one I ever use: new deployments always involve just shutting down the server, so dynamically scanning for modules is really unnecessary.
I guess my questions are:
Does my belief about module loading have any merit?
If so, is there any way to get JBoss to stop scanning?
If not, does anyone have any suggestions about what else I can investigate?
Thanks for reading!
I previously posted on the neo4j mailing list (https://groups.google.com/forum/#!topic/neo4j/zn-7lKHVvNI) but haven't received any response from the community, so I'm x-posting here...
I've noticed what appears to be memory leak behavior from neo4j community edition. Running this test code (https://gist.github.com/mlaldrid/85a03fc022170561b807) against 2.1.2 (also tested against 2.0.3) and a 512MB heap results in GC churn after a few hundred thousand cypher queries. Eventually I either get an OutOfMemory error or jetty times out.
However, when I run the same test code against an eval copy of the neo4j enterprise edition it proceeds though 3.5M queries with no signs of bumping up against the 512MB heap limit. I killed the test after that, satisfied that the behavior was sufficiently different from the community edition.
My questions are thus: Why is this memory leak behavior different in the community and enterprise editions? Is it something that the enterprise edition's "advanced caching" feature solves? Is it a known but opaque limitation of the community edition?
Thanks for any insight on this issue.
This is a recently discovered memory leak in 2 of the 4 cache types available for the community edition (weak and soft caches). It does not affect enterprise as enterprise uses the 'hpc' cache per default.
It only affects deployments where you are unlikely to read from the existing data in the db, or where the majority of load on the system is writes.
We've got a fix for this which will go out in subsequent releases. For now, if your use case is unfortunate enough to trigger this issue, you'll need to use either the 'strong' cache or 'none' in community, or switch to enterprise until the next patch release.
I'm posting output of Sampler of jvisualvm
I guess this answer the question as the leak is still there in 2.2.0
Edit:
The problem was using ExecutionEngine. I used execute method on GraphDatabaseService instead and it solved my problem
Or rather, why aren't there better tools for profiling memory in ruby, specifically rails apps?
Recently our rails app (hosted on heroku) has started seeing lots of R14 errors in the worker dynos. This means we're running out of memory. Bumping the dynos to 2x (512mb -> 1GB) only alleviated the problem temporarily, leading me to believe there is a memory leak somewhere. Naturally, my next step was to find a good profiling gem that can help me discover the source of the leak.
Maybe I'm just ignorant of the tools available, or maybe I just don't know how to use the ones I have. My wish is that I could install a gem and then run reports on the memory usage statistics. Hitting an endpoint to get a report is not really viable as my memory issues are isolated to worker dynos running delayed jobs.
I've looked at memprof, but it's 1.8 only.
I've looked at ruby-prof (awesome), but the memory profiling requires a patched ruby intepreter.
I've looked at GC::Profiler, but I don't understand how to find memory leaks with it.
So, is it just plain difficult to find memory leaks in ruby? Or am I missing the point somehow?
Depending on your "type" of leak, You can run valgrind against ruby. Might require recompilation again though. In general it's hard because ruby does method allocation without firing any events, by default, so it's tricky to track. See also perftools.rb project, which somewhat works around this limitation.
Recently I've been having some success with Skylight to profile web and background worker methods and then hunt down opportunities for optimisation. It probably wasn't around when you posted this question. The downside is that it only really helps you debug in staging or production, not development environments, so the dev loop can be very slow.
Make sure you install both skylight-ruby and sidekiq-skylight to get profiling on both your web server and background workers if you're using Sidekiq.
Good luck!
There is a nice way if your application is running on OS with dtrace or similar technology like SystemTap. In my case, we use RHEL/CentOS which has the latter:
https://lukas.zapletalovi.com/2016/08/probing-ruby-20-apps-with-systemtap-in-rhel7.html
You can easily connect to production application and "inject" profiling code for a moment and track calls, memory, CPU time or I/O and then "disconnect" at any time. It's very efficient so you will not likely notice any drastice slowdown (unless you screw your script up).
I disagree that memory profiling in Ruby is hard. The JVM has some of the best memory profiling tools on the planet, and you can run your Ruby programs on the JVM. Don't reinvent the wheel.
Browsing Memory the JRuby Way
Finding Leaks in Ruby Apps with Eclipse Memory Analyzer
Monitoring Memory with JRuby, Part 1: jhat and VisualVM
Monitoring Memory with JRuby, Part 2: Eclipse Memory Analyzer
We are running 2 rails application on server with 4GB of ram. Both servers use rails 3.2.1 and when run in either development or production mode, the servers eat away ram at incredible speed consuming up-to 1.07GB ram each day. Keeping the server running for just 4 days triggered all memory alarms in monitoring and we had just 98MB ram free.
We tried active-record optimization related to bloating but still no effect. Please help us figure out how can we trace the issue that which of the controller is at fault.
Using mysql database and webrick server.
Thanks!
This is incredibly hard to answer, without looking into the project details itself. Though I am quite sure you won't be using Webrick in your target production build(right?), so check if it behaves the same under Passenger or whatever is your choice.
Also without knowing the details of the project I would suggest looking at features like generating pdfs, csv parsing, etc. Seen a case, where generating pdf files have been eating resources in a similar fashion, leaving like 5mb of not garbage collected memory for each run.
Good luck.
I have noticed that after my Grails app has been deployed for about 2 weeks, performance degrades significantly, and I have to redeploy. I am using the Spring Security plugin and caching users. My first inclination is that it has something to do with this and the session cache size, but I'm not sure how to go about verifying this.
Does it sound like I'm on the right track? Has anyone else experienced this and narrowed down the problem? Any help would be great.
Thanks!
Never guess where to optimize, it's going to be wrong.
Get a heap dump and a profile it a little (VisualVM worked fine for me).
It might be a memory leak, like it happened to me. What is your environment - OS, webserver, Grails?
I would recommend getting YourKit (VisualVM has limited information) and use this to profile your application in production (if possible).
Alternatively you could create a performance test (with JMeter for example) and performance test the pieces of your application that you suspect is causing the performance degredation.
Monitoring memory,cpu,threads,gc and such while running some simple JMeter performance tests will definitely find the culprit. This way you can easily re-test your system over time and see if you have incorporated new "performance killing" bugs.
Performance testing tools/services:
JMeter
Grinder
Selenium (Can performance test with selenium grid, need hw though)
Browsermob (Commercial, which uses Selenium + Selenium-Grid)
NeoLoad by NeoTys (Commercial, trial version available)
HP Loadrunner (Commercial, The big fish on the market, trial version available)
I'd also look into installing the app-info plugin and turning on a bunch of the options (especially around hibernate) to see if things are getting out of control there. Could be something that's filling the hibernate session but never closing a transaction.
Another area to look at is if you're doing anything with the groovy template engine. That has a known memory leak, that's sort of unfixable if you're not caching the class/results. Recently fixed a problem around this on our app. If you're seeing any perm gen errors, this could be the case.
Try to install Javamelody plugin. In our case it helped to find problem with GC.