[Building Sakai] [Using Sakai] Sakai Slow Down

[Glen Parker]

If you want to use more than 3000m RAM, you'll need to switch to the
64bit JVM using the -d64 switch.

Additionally, you probably don't want to use the -XX:+UseParallelGC
with that much RAM.  The stop-the-world pauses would take minutes to
complete, minutes that the whole system is paused and no work gets
done.

My friend John Calvin at UToronto is a fellow Blackboard user, but
he's done a lot of research into JVM tuning and came up the the
following large-heap memory settings for his T5220.  The kep is
switching from the parallel collection to the mark&sweep collector.
-XX:+UseConcMarkSweepGC.    Standard SysAdmin warnings apply, please
don't switch willy-nily without understanding these first.

>From my friend John Calvin at UToronto

# This is a new option in the JVM 1.6
-XX:+AlwaysPreTouch

# Don't ever let anything trigger a GC event except the JVM
-XX:+DisableExplicitGC
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-d64

# Enable the jconsole and SNMP interfaces
-Dcom.sun.management.jmxremote
-Dcom.sun.management.snmp.interface=0.0.0.0
-Dcom.sun.management.snmp.port=8161
-Dcom.sun.management.snmp.acl=true
-Dcom.sun.management.snmp.acl.file=/usr/java/jre/lib/management/snmp.acl

# It seemed like a good idea at the time
-XX:+UseTLAB
-XX:+MaxFDLimit

# increase the stack size
-Xss256k

# Use a 20GB heap with NO survivor space!
-Xms20g
-Xmx20g
-XX:OldSize=16g
-XX:NewSize=4g
-XX:MaxNewSize=4g
-XX:NewRatio=4
-XX:SurvivorRatio=4096
-XX:MaxTenuringThreshold=0

# Increase the PermGen size
-XX:PermSize=256m
-XX:MaxPermSize=256m

# Increase the Code cache size
-XX:InitialCodeCacheSize=128m
-XX:ReservedCodeCacheSize=128m

# Use the CMS collector and the ParNewGC collector with 16 threads
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:ParallelGCThreads=16

# Tune the CMS collector!
-XX:CMSMarkStackSize=8M
-XX:CMSMarkStackSizeMax=8M
-XX:+CMSScavengeBeforeRemark
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:CMSInitiatingOccupancyFraction=60
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled



On Thu, Sep 10, 2009 at 11:33 PM, Mustansar Mehmood <mustansar at rice.edu> wrote:
> I am not so sure but looking at your JAVA_OPTS
>
> -Xms2500m -Xmx2500m
>
> seems like you are not using your RAM completely assuming you have 32G of
> it.
>
> here is a little reference
> http://www.mail-archive.com/java-linux@java.blackdown.org/msg13984.html
> Let us know how it goes.
> --mustansar
>
> Barb Kerns wrote:
>
> We have two SUN T5220s. One runs the Apache front-end and the MySQL
> database, each in a seperate zone. The other physical system runs Tomcat in
> a zone.
> The two servers are identical hardware:
> * Sun Fire T5220 servers
> * 1.2 GHz, 8 core
> * 32 Gb memory
> * Two 146 Gb disk drives (mirrored), plus disk space on the SAN
>
> On the production Sakai Tomcat server JAVA_OPTS are set to:
> JAVA_OPTS="-server -XX:+UseParallelGC -Xms2500m -Xmx2500m -XX:NewSize=512m
> -XX:MaxNewSize=512m -XX:PermSize=512m -XX:MaxPermSize=512m
> -Djava.awt.headless=true -Dsakai.demo=false"
>
> A quick look at top while the slowdown happens doesn't seem to indicate the
> problem is a hardware bottleneck.
>
> (Would one of you grant a favor and pass this on to the Production and
> Sakai-dev lists? I have registered to both the production and sakai-dev
> lists, but am still being denied the permissions to post to those lists.)
>
> Barb
>
>
> On 9/10/09 12:43 PM, "Stephen Marquard" <stephen.marquard at uct.ac.za> wrote:
>
>
>
> Hi Barb
>
> It's a bit hard to say without more details. Some generic suggestions are:
>
> - Give the app servers more memory
> - Add more app servers
> - Upgrade to the latest release and/or maintenance branch (2.5.4, 2.6.0,
> 2-5-x
> or 2-6-x).
>
> You could also look at the PROD category in JIRA to see production
> configurations from other sites of similar scale.
>
> I'd suggest you post some more detailed info (e.g. about your app servers
> and
> database and any information you have about where the bottlenecks are) to
> the
> production or sakai-dev lists, which are better places to get feedback than
> sakai-user.
>
> Regards
> Stephen
>
>
>
> Barb Kerns <brk at bumail.bradley.edu> 9/10/2009 7:35 PM >>>
>
>
> We are in the midst of moving from Blackboard to Sakai and have had high
> adoption this Fall by faculty.  However, this week Sakai web services
> stopped twice and at other times is slow to load pages.  We are self hosted
> Sakai 2.5.0 on two Sun servers and have about 650 course sites, 1000
> simultaneous sessions through most of the day/evening and about 1 million
> hits per day.  Where should we start in improving speed/performance?
> We would appreciate any guidance.
>
> Barb Kerns
> Bradley University
> Peoria, IL USA
> (309) 677-2332
>
>
>
>
> _______________________________________________
> sakai-user mailing list
> sakai-user at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-user
>
> TO UNSUBSCRIBE: send email to sakai-user-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
>
>
> --
> Mustansar Mehmood
> Educational System Developer
>
> Infromation Technology
> 6100 Main St MS 119
> Houston Texas 77005
>
> Phone:(713)348-2523
> Fax  :(713)348 6099
> email:mustansar at rice.edu
>
>
> Tell me and I forget. Teach me and I remember. Involve me and I learn.
>                                     -- Benjamin Franklin
>
> _______________________________________________
> sakai-user mailing list
> sakai-user at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-user
>
> TO UNSUBSCRIBE: send email to sakai-user-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>