[Deploying Sakai] [Building Sakai] Sakai 2.9 performance issue

Sat Mar 22 12:45:25 PDT 2014

There is no magic fix for an unresponsive Tomcat.  You need to debug the
issue to try to find the root of the issue.  The most common problem is
Tomcat running out of memory.  To better understand what is in the JVM's
memory, you need to get a heap dump of the unresponsive Tomcat instance and
to analyze that heap dump in a tool like Eclipse MAT.  MAT will provide you
visual reports and lists of memory "dominators" that will most likely point
to a specific problem.  Once you have narrowed down where the problem is,
members of this list can most likely help with possible fixes or
suggestions where to look next.

On Sat, Mar 22, 2014 at 11:39 AM, Sanghyun Jeon <euksa99 at gmail.com> wrote:

> Hello all
> We plan to upgrade sakai2.9.3 this summer and I think I have a similar
> problem: unresponsive tomcats
> We have tomcat 7.0.42 and java 1.7.0_40
> JVM setting is below.
> JAVA_OPTS="-server -d64 -Xmx6g -XX:MaxPermSize=512m
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote
> -Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
> -Dsakai.security=/opt/sakai-cfg -Dsakai.home=/opt/sakai-cfg/tc2A"
>
> I am wondering David found the root cause of this symptom and its solution
> can be shared.
> Thank you.
>
> S
>
>
>
>  On Tue, May 28, 2013 at 5:49 AM, David Horwitz <david.horwitz at uct.ac.za>wrote:
>
>>  Hi David,
>>
>>
>> On Tue, 2013-05-28 at 08:15 -0400, David Haines wrote:
>>
>> David,
>>
>>
>>
>>       At Michigan we've found that examining a heap dump usually makes
>> the memory problem clear.  YourKit does this well.
>>
>>
>>
>>  Unfortunately my attempt to grab a production heap dump failed :( I got
>> an error jdk error - we have upgraded to the latest 1.6 jdk and I will try
>> get one if we see the issue resurface.
>>
>>
>>
>>
>>
>>      If you reset all the caches from the memory tool does performance
>> get better?  That's not a long term solution but can help diagnose (and
>> just might provide a short term bandaid).
>>
>>
>>
>>  We only thought of that one after the fact - again something we will try
>> if we can reproduce.
>>
>>
>> D
>>
>>
>> - Dave
>>
>>
>>
>>  On Mon, May 27, 2013 at 9:24 AM, Aaron Zeckoski <azeckoski at unicon.net>
>> wrote:
>>
>> I suggest you try changing one of your nodes JVM settings and drop the
>> following from it.
>>
>> -Xms5000m -Xmx5000m -XX:+UseConcMarkSweepGC
>>
>>  -XX:+ExplicitGCInvokesConcurrent -XX:+CMSIncrementalMode -verbose:gc
>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
>>
>>   -Dsun.lang.ClassLoader.allowArraySyntax=true
>>
>> Putting those in severely limits the JVM autotuning (also known as
>> ergonomics).
>> http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
>> (plus the slowdown from all the GC monitoring)
>> Also, "-Dsun.lang.ClassLoader.allowArraySyntax=true" is no longer needed.
>>
>> Then monitor the node using jstat or something similar and compare it
>> to the performance of the other nodes. You can tweak things from there
>> but in most cases you will find the tuner handles things better on its
>> own.
>> -AZ
>>
>>
>>
>> On Mon, May 27, 2013 at 8:58 AM, David Horwitz <david.horwitz at uct.ac.za>
>> wrote:
>> > Hi All,
>> >
>> > We are/did experience a performance issue last week we would like to
>> seek
>> > community input on.
>> >
>> > Some back ground:
>> >
>> > Kernel 1,3,x based (1.3.2 msub with KNL-1068 applied)
>> > Java 1.6.0_30
>> > mysql 5.1.68
>> >
>> > Javaopts:
>> >
>> > -server -d64 -Xms5000m -Xmx5000m -XX:MaxPermSize=640m
>> > -Dhttp.proxyHost=campusnet.uct.ac.za -Dhttp.proxyPort=8080
>> > -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent
>> > -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCApplicationStoppedTime
>> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
>> > -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true
>> > -Dnetworkaddress.cache.ttl=60
>> > -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
>> > -Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
>> > -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true
>> >
>> >
>> > Symptoms:
>> > Randomly nodes enter a state where they become increasingly
>> unresponsive due
>> > to longer and more frequent full GC operations. Until Friday we seemed
>> to
>> > see this randomly with a higher incidence after the end of the lectures,
>> > during extensive online testing on Friday we experienced this across all
>> > nodes. It seems to be unrelated to the node running times as we have
>> seen it
>> > on nodes that have been restarted in the last 24 hours as well as others
>> > that have been running for some time.
>> >
>> > Any one seen similar issues? As a preventative we have upgraded to the
>> > latest 1.6 jdk (1.6.0_45) but are far from convinced this is the
>> problem.
>> >
>> >
>> > Thanks
>> >
>> > David
>> > ________________________________
>> > UNIVERSITY OF CAPE TOWN
>> >
>> > This e-mail is subject to the UCT ICT policies and e-mail disclaimer
>> > published on our website at
>> > http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable
>> from +27
>> > 21 650 9111. This e-mail is intended only for the person(s) to whom it
>> is
>> > addressed. If the e-mail has reached you in error, please notify the
>> author.
>> > If you are not the intended recipient of the e-mail you may not use,
>> > disclose, copy, redirect or print the content. If this e-mail is not
>> related
>> > to the business of UCT it is sent by the sender in the sender's
>> individual
>> > capacity.
>> >
>> >
>>
>>   > _______________________________________________
>> > production mailing list
>> > production at collab.sakaiproject.org
>> > http://collab.sakaiproject.org/mailman/listinfo/production
>> >
>> > TO UNSUBSCRIBE: send email to
>> production-unsubscribe at collab.sakaiproject.org
>> > with a subject of "unsubscribe"
>>
>>
>>
>> --
>> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>>
>> TO UNSUBSCRIBE: send email to
>> production-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>>
>>
>> ------------------------------
>> UNIVERSITY OF CAPE TOWN
>>
>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
>> published on our website at
>> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27
>> 21 650 9111. This e-mail is intended only for the person(s) to whom it
>> is addressed. If the e-mail has reached you in error, please notify the
>> author. If you are not the intended recipient of the e-mail you may not
>> use, disclose, copy, redirect or print the content. If this e-mail is not
>> related to the business of UCT it is sent by the sender in the sender's
>> individual capacity.
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20140322/46903a11/attachment.html 