[Deploying Sakai] Sakai 2.9 performance issue

[Aaron Zeckoski]

I suggest you try changing one of your nodes JVM settings and drop the
following from it.

-Xms5000m -Xmx5000m -XX:+UseConcMarkSweepGC
-XX:+ExplicitGCInvokesConcurrent -XX:+CMSIncrementalMode -verbose:gc
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Dsun.lang.ClassLoader.allowArraySyntax=true

Putting those in severely limits the JVM autotuning (also known as ergonomics).
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
(plus the slowdown from all the GC monitoring)
Also, "-Dsun.lang.ClassLoader.allowArraySyntax=true" is no longer needed.

Then monitor the node using jstat or something similar and compare it
to the performance of the other nodes. You can tweak things from there
but in most cases you will find the tuner handles things better on its
own.
-AZ


On Mon, May 27, 2013 at 8:58 AM, David Horwitz <david.horwitz at uct.ac.za> wrote:
> Hi All,
>
> We are/did experience a performance issue last week we would like to seek
> community input on.
>
> Some back ground:
>
> Kernel 1,3,x based (1.3.2 msub with KNL-1068 applied)
> Java 1.6.0_30
> mysql 5.1.68
>
> Javaopts:
>
> -server -d64 -Xms5000m -Xmx5000m -XX:MaxPermSize=640m
> -Dhttp.proxyHost=campusnet.uct.ac.za -Dhttp.proxyPort=8080
> -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent
> -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true
> -Dnetworkaddress.cache.ttl=60
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> -Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
> -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true
>
>
> Symptoms:
> Randomly nodes enter a state where they become increasingly unresponsive due
> to longer and more frequent full GC operations. Until Friday we seemed to
> see this randomly with a higher incidence after the end of the lectures,
> during extensive online testing on Friday we experienced this across all
> nodes. It seems to be unrelated to the node running times as we have seen it
> on nodes that have been restarted in the last 24 hours as well as others
> that have been running for some time.
>
> Any one seen similar issues? As a preventative we have upgraded to the
> latest 1.6 jdk (1.6.0_45) but are far from convinced this is the problem.
>
>
> Thanks
>
> David
> ________________________________
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27
> 21 650 9111. This e-mail is intended only for the person(s) to whom it is
> addressed. If the e-mail has reached you in error, please notify the author.
> If you are not the intended recipient of the e-mail you may not use,
> disclose, copy, redirect or print the content. If this e-mail is not related
> to the business of UCT it is sent by the sender in the sender's individual
> capacity.
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"



-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile