[Building Sakai] JVM memory usage up after upgrading to 2.9.x

[Kusnetz, Jeremy]

We just upgraded from the rSmart 2.8 CLE to 2.9.x community (close to 2.9.3 rc01).

I'm seeing a significant amount of JVM memory usage that we were not seeing previously, and the GC doesn't seem to be happening as frequently.

We switched from Tomcat 5.5 to Tomcat 7, but we are still running on Java6, just a slightly newer version.

We pretty much kept the same tomcat startup options as before, but GC seems to be working very differently now.  Really the only thing changed is adding  -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false


-server -d64 -XX:+ForceTimeHighResolution -XX:+UseLargePages -Xms7g -Xmx7g -XX:PermSize=768m -XX:MaxPermSize=768m -XX:NewSize=1536m -XX:MaxNewSize=1536m -Djava.awt.headless=true -Dsakai.home=/etc/tomcat100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -Dnetworkaddress.cache.ttl=60 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dcom.sun.management.snmp.port=41003 -Dcom.sun.management.snmp.acl.file=/usr/local/apus/tomcat_servers/etc/snmp100.acl -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dhttp.agent=Sakai -XX:ParallelGCThreads=20 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:CMSMaxAbortablePrecleanTime=60000 -XX:+UseCMSCompactAtFullCollection -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/usr/local/apus/tomcat_servers/servers/tomcat100/logs/sakai.vgc -XX:TargetSurvivorRatio=90 -XX:+UseBiasedLocking -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSIncrementalSafetyFactor=75 -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false

Here is a 1 day graph.  You can see the CMS Old Gen is working very differently than before, it just seems to grow and  grow and then do a large GC, where before it was doing lots of GCs.

[https://edge.apus.edu/monitor/graphs/stsakai5/stsakai5-javamemused-day.png]

Here is a full week of JVM usage.  Sundays are our big night and we had about double the number of users on the node, although you can see our memory usage for today is much higher than normal.

[https://edge.apus.edu/monitor/graphs/stsakai5/stsakai5-javamemused-week.png]
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130730/b214a8e0/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 47095 bytes
Desc: image001.png
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130730/b214a8e0/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 50345 bytes
Desc: image002.png
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130730/b214a8e0/attachment-0001.png