[Building Sakai] JVM memory usage up after upgrading to 2.9.x

[Noah Botimer]

There is a ton of magic in those JAVA_OPTS. The general recommendation for JDK 6+ is to "let it be" and let the GC tune itself. Are those settings based on a lot of precise tuning or mostly just stuff that stuck?

To follow up with one more question on Sam's:

c) are you seeing performance problems/changes or full GC's as a result or just anticipating based on a different allocation profile?

For one possible contributor to more memory usage, various caches have been added/adjusted to account for typical production scenarios. I'd check there first.

Thanks,
-Noah

On Aug 1, 2013, at 10:25 AM, Sam Ottenhoff wrote:

> So I guess the relevant questions are
> 
> a) is your Tomcat running out of memory?
> 
> b) what is taking up all of the memory?  Heap dump => Eclipse MAT will give you a great idea of what is in your heap.
> 
> 
> On Tue, Jul 30, 2013 at 6:31 PM, Kusnetz, Jeremy <JKusnetz at apus.edu> wrote:
> We just upgraded from the rSmart 2.8 CLE to 2.9.x community (close to 2.9.3 rc01).
> 
>  
> 
> I’m seeing a significant amount of JVM memory usage that we were not seeing previously, and the GC doesn’t seem to be happening as frequently.
> 
>  
> 
> We switched from Tomcat 5.5 to Tomcat 7, but we are still running on Java6, just a slightly newer version.
> 
>  
> 
> We pretty much kept the same tomcat startup options as before, but GC seems to be working very differently now.  Really the only thing changed is adding  -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false
> 
>  
> 
>  
> 
> -server -d64 -XX:+ForceTimeHighResolution -XX:+UseLargePages -Xms7g -Xmx7g -XX:PermSize=768m -XX:MaxPermSize=768m -XX:NewSize=1536m -XX:MaxNewSize=1536m -Djava.awt.headless=true -Dsakai.home=/etc/tomcat100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -Dnetworkaddress.cache.ttl=60 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dcom.sun.management.snmp.port=41003 -Dcom.sun.management.snmp.acl.file=/usr/local/apus/tomcat_servers/etc/snmp100.acl -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dhttp.agent=Sakai -XX:ParallelGCThreads=20 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:CMSMaxAbortablePrecleanTime=60000 -XX:+UseCMSCompactAtFullCollection -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/usr/local/apus/tomcat_servers/servers/tomcat100/logs/sakai.vgc -XX:TargetSurvivorRatio=90 -XX:+UseBiasedLocking -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSIncrementalSafetyFactor=75 -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false
> 
>  
> 
> Here is a 1 day graph.  You can see the CMS Old Gen is working very differently than before, it just seems to grow and  grow and then do a large GC, where before it was doing lots of GCs.
> 
>  
> 
> <image001.png>
> 
>  
> 
> Here is a full week of JVM usage.  Sundays are our big night and we had about double the number of users on the node, although you can see our memory usage for today is much higher than normal.
> 
>  
> 
> <image002.png>
> 
> This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
> 
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130801/a840699c/attachment.html