[Deploying Sakai] out of memory exceptions after upgrade to Sakai CLE 2.9

Thu May 16 06:29:14 PDT 2013

Hi Sam,

Thanks for the JDK bug references. It seems very likely that we are encountering either this bug and a very similar one that is leaking semaphores.  I think our next step will be attempting to replicate this condition in our test infrastructure and see if a newer JDK version does indeed correct the semaphore leak.

We are running on VMware VMs.  Below is a listing of the JAVA_OPTS that we are using:

export JAVA_OPTS="$JAVA_OPTS -server -d64"

# only create a huge JVM if the operation is 'start'
if [[ "$1" == 'start' ]]; then
   export JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing"
   export JAVA_OPTS="$JAVA_OPTS -XX:+CMSParallelRemarkEnabled -XX:+UseAdaptiveSizePolicy -XX:+UseBiasedLocking"
   export JAVA_OPTS="$JAVA_OPTS -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=70 -XX:MinHeapFreeRatio=18"
   export JAVA_OPTS="$JAVA_OPTS -Xmx6144m -Xms6144m"
   export JAVA_OPTS="$JAVA_OPTS -XX:MaxNewSize=1280m  -XX:NewSize=1024m"
   export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -XX:PermSize=256m"

   # was "-XX:+CMSPermGenSweepingEnabled"
   export JAVA_OPTS="$JAVA_OPTS -XX:+CMSClassUnloadingEnabled"
fi

export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false"
export JAVA_OPTS="$JAVA_OPTS -Dsun.lang.ClassLoader.allowArraySyntax=true"

# TOMCAT 7 options - SAK-21265 and useLegacyMergeSort per taliesin
export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false"
export JAVA_OPTS="$JAVA_OPTS -Djava.util.Arrays.useLegacyMergeSort=true"

# other JVM settings (TTLs, network, etc)
# http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
export JAVA_OPTS="$JAVA_OPTS -Dsun.net.inetaddr.ttl=180"
export JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
export JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=utf-8"
export JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

# sakai-related props
export JAVA_OPTS="$JAVA_OPTS -Dsakai.home=/opt/tomcat/sakai"
export JAVA_OPTS="$JAVA_OPTS -Dsakai.security=/etc/sakai"

- Eric

From: Sam Ottenhoff [mailto:ottenhoff at longsight.com]
Sent: Wednesday, May 15, 2013 5:21 PM
To: Crossman, Eric
Cc: production at collab.sakaiproject.org
Subject: Re: [Deploying Sakai] out of memory exceptions after upgrade to Sakai CLE 2.9

Hi Eric,

Are you running on baremetal servers or on VMs? Can you post your JAVA_OPTS or CATALINA_OPTS?

Maybe a JDK upgrade is in order?  I see a few bugs related to leaking handlers that were reported in 1.7.0_04 and weren't fixed until 1.7.0_08:

  http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7166379

--Sam

On Wed, May 15, 2013 at 5:04 PM, Crossman, Eric <ecrossman at ufl.edu<mailto:ecrossman at ufl.edu>> wrote:
Hi All,

I'm writing to see if any other institutions have encountered this strange out of memory exception after their upgrade to CLE 2.9. We are encountering this exception after a given tomcat node has been running for 2-3 days.  The exception looks like the following:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
mmap failed for CEN and END part of zip file
Exception in thread "Thread-423" java.lang.OutOfMemoryError
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:214)
        at java.util.zip.ZipFile.<init>(ZipFile.java:144)
        at java.util.jar.JarFile.<init>(JarFile.java:152)
        at java.util.jar.JarFile.<init>(JarFile.java:116)
        at org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:2789
)
        at org.apache.catalina.loader.WebappClassLoader.findResources(WebappClassLoader.java
:1328)
        at java.lang.ClassLoader.getResources(ClassLoader.java:1176)
       at javax.mail.Session$5.run(Session.java:1231)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.mail.Session.getResources(Session.java:1225)
        at javax.mail.Session.loadAllResources(Session.java:1137)
        at javax.mail.Session.loadProviders(Session.java:902)
        at javax.mail.Session.<init>(Session.java:219)
        at javax.mail.Session.getInstance(Session.java:258)
        at org.sakaiproject.email.impl.BasicEmailService.sendToUsers(BasicEmailService.java:
767)
        at org.sakaiproject.util.EmailNotification$RunSendToUsers.run(EmailNotification.java
:186)
        at java.lang.Thread.run(Thread.java:722)

>From our research so far it seems that the JVM is not actually out of memory but has hit a Linux kernel limit for the number of memory mapped files for a single process. For our RHEL 5.x boxes, this is 65536. While we could raise this value, we would likely only delay the inevitable failure of the JVM once it hits that limit. We have observed that a freshly restarted Sakai/Tomcat instance will start with around 1100-1200 memory files and continuously grow upwards from there.

We have also observed that there is a large population of temporary /dev/shm files that have been memory mapped but later deleted which is contributed to this mmap count being so high. Here is an example from a box:

2aaabbeb4000-2aaabbeb5000 rw-s 00000000 00:13 9279287                    /dev/shm/sem.m1TZAN (deleted)
2aaabbeb5000-2aaabbeb6000 rw-s 00000000 00:13 9279288                    /dev/shm/sem.oU5lY9 (deleted)
2aaabbeb6000-2aaabbeb7000 rw-s 00000000 00:13 9279289                    /dev/shm/sem.bTTLlw (deleted)
2aaabbeb7000-2aaabbeb8000 rw-s 00000000 00:13 9279290                    /dev/shm/sem.RHXiKS (deleted)
2aaabbeb8000-2aaabbeb9000 rw-s 00000000 00:13 9272028                    /dev/shm/sem.epwwAM (deleted)
2aaabbeb9000-2aaabbeba000 rw-s 00000000 00:13 9272029                    /dev/shm/sem.yZCZxN (deleted)
2aaabbeba000-2aaabbebb000 rw-s 00000000 00:13 9272030                    /dev/shm/sem.46lwvO (deleted)
2aaabbebb000-2aaabbebc000 rw-s 00000000 00:13 9272031                    /dev/shm/sem.iRdcuP (deleted)
2aaabbebc000-2aaabbebd000 rw-s 00000000 00:13 9272032                    /dev/shm/sem.SyKWsQ (deleted)
2aaabbebd000-2aaabbebe000 rw-s 00000000 00:13 9272033                    /dev/shm/sem.eAEKrR (deleted)
2aaabbebe000-2aaabbebf000 rw-s 00000000 00:13 9272034                    /dev/shm/sem.NMFAqS (deleted)

For reference here are the versions of various software that we are using:
Apache 2.2.3
Tomcat 7.0.27
Java 1.7.0_05
Linux Kernel 2.6.18-348.4.1.el5
Redhat Enterprise Linux 5.8

Has anyone seen this scenario and might have some advice on how to further troubleshoot this problem?

Thanks in advance,

Eric Crossman
University of Florida

_______________________________________________
production mailing list
production at collab.sakaiproject.org<mailto:production at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/production

TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org<mailto:production-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20130516/0424ddf2/attachment-0001.html 