[Deploying Sakai] Sakai 2.9 performance issue

David Horwitz david.horwitz at uct.ac.za
Tue May 28 05:49:48 PDT 2013


Hi David,

On Tue, 2013-05-28 at 08:15 -0400, David Haines wrote:
David,


     At Michigan we've found that examining a heap dump usually makes the memory problem clear.  YourKit does this well.


Unfortunately my attempt to grab a production heap dump failed :( I got an error jdk error - we have upgraded to the latest 1.6 jdk and I will try get one if we see the issue resurface.




     If you reset all the caches from the memory tool does performance get better?  That's not a long term solution but can help diagnose (and just might provide a short term bandaid).


We only thought of that one after the fact - again something we will try if we can reproduce.


D


- Dave


On Mon, May 27, 2013 at 9:24 AM, Aaron Zeckoski <azeckoski at unicon.net<mailto:azeckoski at unicon.net>> wrote:
I suggest you try changing one of your nodes JVM settings and drop the
following from it.

-Xms5000m -Xmx5000m -XX:+UseConcMarkSweepGC
-XX:+ExplicitGCInvokesConcurrent -XX:+CMSIncrementalMode -verbose:gc
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps

-Dsun.lang.ClassLoader.allowArraySyntax=true

Putting those in severely limits the JVM autotuning (also known as ergonomics).
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
(plus the slowdown from all the GC monitoring)
Also, "-Dsun.lang.ClassLoader.allowArraySyntax=true" is no longer needed.

Then monitor the node using jstat or something similar and compare it
to the performance of the other nodes. You can tweak things from there
but in most cases you will find the tuner handles things better on its
own.
-AZ


On Mon, May 27, 2013 at 8:58 AM, David Horwitz <david.horwitz at uct.ac.za<mailto:david.horwitz at uct.ac.za>> wrote:
> Hi All,
>
> We are/did experience a performance issue last week we would like to seek
> community input on.
>
> Some back ground:
>
> Kernel 1,3,x based (1.3.2 msub with KNL-1068 applied)
> Java 1.6.0_30
> mysql 5.1.68
>
> Javaopts:
>
> -server -d64 -Xms5000m -Xmx5000m -XX:MaxPermSize=640m
> -Dhttp.proxyHost=campusnet.uct.ac.za<http://campusnet.uct.ac.za> -Dhttp.proxyPort=8080
> -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent
> -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true
> -Dnetworkaddress.cache.ttl=60
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> -Dsun.lang.ClassLoader.allowArraySyntax=true -Dhttp.agent=Sakai
> -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true
>
>
> Symptoms:
> Randomly nodes enter a state where they become increasingly unresponsive due
> to longer and more frequent full GC operations. Until Friday we seemed to
> see this randomly with a higher incidence after the end of the lectures,
> during extensive online testing on Friday we experienced this across all
> nodes. It seems to be unrelated to the node running times as we have seen it
> on nodes that have been restarted in the last 24 hours as well as others
> that have been running for some time.
>
> Any one seen similar issues? As a preventative we have upgraded to the
> latest 1.6 jdk (1.6.0_45) but are far from convinced this is the problem.
>
>
> Thanks
>
> David
> ________________________________
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27
> 21 650 9111. This e-mail is intended only for the person(s) to whom it is
> addressed. If the e-mail has reached you in error, please notify the author.
> If you are not the intended recipient of the e-mail you may not use,
> disclose, copy, redirect or print the content. If this e-mail is not related
> to the business of UCT it is sent by the sender in the sender's individual
> capacity.
>
>

> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org<mailto:production at collab.sakaiproject.org>
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org<mailto:production-unsubscribe at collab.sakaiproject.org>
> with a subject of "unsubscribe"



--
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
_______________________________________________
production mailing list
production at collab.sakaiproject.org<mailto:production at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/production

TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org<mailto:production-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"



________________________________
UNIVERSITY OF CAPE TOWN

This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20130528/25f53e41/attachment-0001.html 


More information about the production mailing list