[Deploying Sakai] [Building Sakai] Production Problems with Garbage Collection

Mike De Simone michael.desimone at rsmart.com
Thu Jan 20 09:46:57 PST 2011


Hi Beth,

I found some mentions of long GC times in java 1.5, so don't know if it's
applicable still in 1.6.  See
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6433335 and
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6459113

the gist of the recommendations is to turn off the parallel collector:
-XX:-UseParNewGC
and to set -Xmx and -Xms to the same value to try & avoid some of the
allocation fragmentation.

I don't know that we've seen this behavior though.

I did some quick research a couple of months ago and came across a JMX
client called jmxterm that can call into the JVM and return some information
like what you're mentioning.  http://www.cyclopsgroup.org/projects/jmxterm.
You have to pass -Dcom.sun.management.jmxremote to java on startup to enable
this to work.

Then you can issues commands like the following to see stats on memory:
get -b java.lang:type=Memory * and get -d java.lang -b java.lang:name=Perm\
Gen,type=MemoryPool Usage

Thanks,

-------------------------------
Mike DeSimone
Lead Systems Engineer
*r**Smart* | 602-490-0473


On Thu, Jan 20, 2011 at 10:13, Beth Kirschner <bkirschn at umich.edu> wrote:

> Hi all,
>
>   We've been looking into some serious production problems which have
> required frequent restarting of our production servers due to the
> performance impact of the server being mired in continuous Garbage
> Collection. The only clue we have from the logs is "Concurrent Mode Failure"
> messages in logs. These problems started shortly after updating to Java 1.6
> and Sakai 2.7.1.
>
>   I'm looking for help and advice, specifically: Has anyone seen these
> types of issues in Sakai production?
>
>   On a related note, I'm wondering if anyone has specific diagnostic tools
> that they use for production Sakai systems, to allow for better analysis of:
> heap dumps, thread dumps, tomcat monitoring (e.g. psi-probe, jstack,
> jconsole, visualvm), and log file analysis (e.g. swatch)?
>
> Thanks!
> - Beth
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110120/17f96cec/attachment.html 


More information about the production mailing list