[Building Sakai] Production Problems with Garbage Collection

[Joseph Dane]

On Thu, Jan 20, 2011 at 7:13 AM, Beth Kirschner <bkirschn at umich.edu> wrote:

>   On a related note, I'm wondering if anyone has specific diagnostic tools
> that they use for production Sakai systems, to allow for better analysis of:
> heap dumps, thread dumps, tomcat monitoring (e.g. psi-probe, jstack,
> jconsole, visualvm), and log file analysis (e.g. swatch)?
>
>
Several years ago, I wrote a tool to analyze Java thread dumps.  It was ad
hoc, and not at all "user friendly", but I used it to track down a number of
serious performance issues in our production portal (not sakai -- it was a
commercial uPortal-based thing -- some of you probably know what I'm talking
about).  I even did a presentation on the tool at a JA-SIG conference a few
years back.  It looks like my slides are still available:


http://web.princeton.edu/sites/isapps/jasig/2005WinterAustin/presentations/portalpostmortems.ppt

I just looked over the slides for the first time in years.  I didn't do a
terrible job with the slides, if I do say so myself.  When I gave the
presentation, I remember I was coming down with pneumonia and could barely
communicate coherently, or so I remember it.  So maybe the presentation
itself was lacking.  Still, they gave me (as a presenter) a nice thermos,
which I use to this day to make coffee, every morning.

I also notice that, although I promised to make my little tool available, I
apparently never did so.

I haven't used the tool in some time (I no longer work with the portal), but
I'm pretty sure I still have it, and could make it available if there was
interest.  It's very short (maybe 60 or so lines of Perl) and not pretty.
It's also possible (since I haven't been paying attention in the meantime)
that there are now other tools that do the same thing.  Certainly, more
important than the tool itself is that an institution have someone on hand
who understands how to extract information from the very large thread dumps
(i.e., containing several hundred stack traces) produced by production
systems under stress.

-- 

joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110121/7e6e4b0a/attachment.html