[Deploying Sakai] [Building Sakai] Sakai 2.9 performance issue

Sanghyun Jeon euksa99 at gmail.com
Sat Mar 22 08:39:45 PDT 2014


Hello all
We plan to upgrade sakai2.9.3 this summer and I think I have a similar
problem: unresponsive tomcats
We have tomcat 7.0.42 and java 1.7.0_40
JVM setting is below.
JAVA_OPTS="-server -d64 -Xmx6g -XX:MaxPermSize=512m
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
-Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
-Dsakai.security=/opt/sakai-cfg -Dsakai.home=/opt/sakai-cfg/tc2A"

I am wondering David found the root cause of this symptom and its solution
can be shared.
Thank you.

S



On Tue, May 28, 2013 at 5:49 AM, David Horwitz <david.horwitz at uct.ac.za>wrote:

>  Hi David,
>
>
> On Tue, 2013-05-28 at 08:15 -0400, David Haines wrote:
>
> David,
>
>
>
>       At Michigan we've found that examining a heap dump usually makes
> the memory problem clear.  YourKit does this well.
>
>
>
>  Unfortunately my attempt to grab a production heap dump failed :( I got
> an error jdk error - we have upgraded to the latest 1.6 jdk and I will try
> get one if we see the issue resurface.
>
>
>
>
>
>      If you reset all the caches from the memory tool does performance get
> better?  That's not a long term solution but can help diagnose (and just
> might provide a short term bandaid).
>
>
>
>  We only thought of that one after the fact - again something we will try
> if we can reproduce.
>
>
> D
>
>
> - Dave
>
>
>
>  On Mon, May 27, 2013 at 9:24 AM, Aaron Zeckoski <azeckoski at unicon.net>
> wrote:
>
> I suggest you try changing one of your nodes JVM settings and drop the
> following from it.
>
> -Xms5000m -Xmx5000m -XX:+UseConcMarkSweepGC
>
>  -XX:+ExplicitGCInvokesConcurrent -XX:+CMSIncrementalMode -verbose:gc
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
>
>   -Dsun.lang.ClassLoader.allowArraySyntax=true
>
> Putting those in severely limits the JVM autotuning (also known as
> ergonomics).
> http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
> (plus the slowdown from all the GC monitoring)
> Also, "-Dsun.lang.ClassLoader.allowArraySyntax=true" is no longer needed.
>
> Then monitor the node using jstat or something similar and compare it
> to the performance of the other nodes. You can tweak things from there
> but in most cases you will find the tuner handles things better on its
> own.
> -AZ
>
>
>
> On Mon, May 27, 2013 at 8:58 AM, David Horwitz <david.horwitz at uct.ac.za>
> wrote:
> > Hi All,
> >
> > We are/did experience a performance issue last week we would like to seek
> > community input on.
> >
> > Some back ground:
> >
> > Kernel 1,3,x based (1.3.2 msub with KNL-1068 applied)
> > Java 1.6.0_30
> > mysql 5.1.68
> >
> > Javaopts:
> >
> > -server -d64 -Xms5000m -Xmx5000m -XX:MaxPermSize=640m
> > -Dhttp.proxyHost=campusnet.uct.ac.za -Dhttp.proxyPort=8080
> > -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent
> > -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCApplicationStoppedTime
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> > -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true
> > -Dnetworkaddress.cache.ttl=60
> > -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> > -Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
> > -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true
> >
> >
> > Symptoms:
> > Randomly nodes enter a state where they become increasingly unresponsive
> due
> > to longer and more frequent full GC operations. Until Friday we seemed to
> > see this randomly with a higher incidence after the end of the lectures,
> > during extensive online testing on Friday we experienced this across all
> > nodes. It seems to be unrelated to the node running times as we have
> seen it
> > on nodes that have been restarted in the last 24 hours as well as others
> > that have been running for some time.
> >
> > Any one seen similar issues? As a preventative we have upgraded to the
> > latest 1.6 jdk (1.6.0_45) but are far from convinced this is the problem.
> >
> >
> > Thanks
> >
> > David
> > ________________________________
> > UNIVERSITY OF CAPE TOWN
> >
> > This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> > published on our website at
> > http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from
> +27
> > 21 650 9111. This e-mail is intended only for the person(s) to whom it is
> > addressed. If the e-mail has reached you in error, please notify the
> author.
> > If you are not the intended recipient of the e-mail you may not use,
> > disclose, copy, redirect or print the content. If this e-mail is not
> related
> > to the business of UCT it is sent by the sender in the sender's
> individual
> > capacity.
> >
> >
>
>   > _______________________________________________
> > production mailing list
> > production at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/production
> >
> > TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org
> > with a subject of "unsubscribe"
>
>
>
> --
> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
>
>
>
> ------------------------------
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27
> 21 650 9111. This e-mail is intended only for the person(s) to whom it is
> addressed. If the e-mail has reached you in error, please notify the
> author. If you are not the intended recipient of the e-mail you may not
> use, disclose, copy, redirect or print the content. If this e-mail is not
> related to the business of UCT it is sent by the sender in the sender's
> individual capacity.
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20140322/78594d43/attachment.html 


More information about the production mailing list