[Deploying Sakai] out of memory exceptions after upgrade to Sakai CLE 2.9

Sam Ottenhoff ottenhoff at longsight.com
Thu May 16 06:46:16 PDT 2013


Yeah, your JAVA_OPTS look great, so JDK bug is the line I would pursue.  I
saw a couple of bugs in u04,u05 that seemed like they could point to your
root cause.

Good luck,
Sam


On Thu, May 16, 2013 at 9:29 AM, Crossman, Eric <ecrossman at ufl.edu> wrote:

>  Hi Sam,****
>
> ** **
>
> Thanks for the JDK bug references. It seems very likely that we are
> encountering either this bug and a very similar one that is leaking
> semaphores.  I think our next step will be attempting to replicate this
> condition in our test infrastructure and see if a newer JDK version does
> indeed correct the semaphore leak.****
>
> ** **
>
> We are running on VMware VMs.  Below is a listing of the JAVA_OPTS that we
> are using:****
>
> ** **
>
> export JAVA_OPTS="$JAVA_OPTS -server -d64"****
>
> ** **
>
> # only create a huge JVM if the operation is 'start'****
>
> if [[ "$1" == 'start' ]]; then****
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing"****
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:+CMSParallelRemarkEnabled
> -XX:+UseAdaptiveSizePolicy -XX:+UseBiasedLocking"****
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:MaxTenuringThreshold=3
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MinHeapFreeRatio=18"****
>
>    export JAVA_OPTS="$JAVA_OPTS -Xmx6144m -Xms6144m"****
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:MaxNewSize=1280m  -XX:NewSize=1024m"**
> **
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -XX:PermSize=256m"***
> *
>
> ** **
>
>    # was "-XX:+CMSPermGenSweepingEnabled"****
>
>    export JAVA_OPTS="$JAVA_OPTS -XX:+CMSClassUnloadingEnabled"****
>
> fi****
>
> ** **
>
> export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"****
>
> export JAVA_OPTS="$JAVA_OPTS
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false" ****
>
> export JAVA_OPTS="$JAVA_OPTS -Dsun.lang.ClassLoader.allowArraySyntax=true"
> ****
>
> ** **
>
> # TOMCAT 7 options - SAK-21265 and useLegacyMergeSort per taliesin****
>
> export JAVA_OPTS="$JAVA_OPTS
> -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false"****
>
> export JAVA_OPTS="$JAVA_OPTS -Djava.util.Arrays.useLegacyMergeSort=true"**
> **
>
> ** **
>
> # other JVM settings (TTLs, network, etc)****
>
> # http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html****
>
> export JAVA_OPTS="$JAVA_OPTS -Dsun.net.inetaddr.ttl=180" ****
>
> export JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"****
>
> export JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=utf-8"****
>
> export JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps"****
>
> ** **
>
> # sakai-related props****
>
> export JAVA_OPTS="$JAVA_OPTS -Dsakai.home=/opt/tomcat/sakai"****
>
> export JAVA_OPTS="$JAVA_OPTS -Dsakai.security=/etc/sakai"****
>
> ** **
>
> - Eric****
>
> ** **
>
> *From:* Sam Ottenhoff [mailto:ottenhoff at longsight.com]
> *Sent:* Wednesday, May 15, 2013 5:21 PM
> *To:* Crossman, Eric
> *Cc:* production at collab.sakaiproject.org
> *Subject:* Re: [Deploying Sakai] out of memory exceptions after upgrade
> to Sakai CLE 2.9****
>
> ** **
>
> Hi Eric,****
>
> ** **
>
> Are you running on baremetal servers or on VMs? Can you post your
> JAVA_OPTS or CATALINA_OPTS?****
>
> ** **
>
> Maybe a JDK upgrade is in order?  I see a few bugs related to leaking
> handlers that were reported in 1.7.0_04 and weren't fixed until 1.7.0_08:*
> ***
>
> ** **
>
>   http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7166379****
>
> ** **
>
> --Sam****
>
> ** **
>
> On Wed, May 15, 2013 at 5:04 PM, Crossman, Eric <ecrossman at ufl.edu> wrote:
> ****
>
> Hi All,****
>
>  ****
>
> I’m writing to see if any other institutions have encountered this strange
> out of memory exception after their upgrade to CLE 2.9. We are encountering
> this exception after a given tomcat node has been running for 2-3 days.
> The exception looks like the following:****
>
>  ****
>
> Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
> pages failed.****
>
> mmap failed for CEN and END part of zip file****
>
> Exception in thread "Thread-423" java.lang.OutOfMemoryError****
>
>         at java.util.zip.ZipFile.open(Native Method)****
>
>         at java.util.zip.ZipFile.<init>(ZipFile.java:214)****
>
>         at java.util.zip.ZipFile.<init>(ZipFile.java:144)****
>
>         at java.util.jar.JarFile.<init>(JarFile.java:152)****
>
>         at java.util.jar.JarFile.<init>(JarFile.java:116)****
>
>         at
> org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:2789
> ****
>
> )****
>
>         at
> org.apache.catalina.loader.WebappClassLoader.findResources(WebappClassLoader.java
> ****
>
> :1328)****
>
>         at java.lang.ClassLoader.getResources(ClassLoader.java:1176)****
>
>        at javax.mail.Session$5.run(Session.java:1231)****
>
>         at java.security.AccessController.doPrivileged(Native Method)****
>
>         at javax.mail.Session.getResources(Session.java:1225)****
>
>         at javax.mail.Session.loadAllResources(Session.java:1137)****
>
>         at javax.mail.Session.loadProviders(Session.java:902)****
>
>         at javax.mail.Session.<init>(Session.java:219)****
>
>         at javax.mail.Session.getInstance(Session.java:258)****
>
>         at
> org.sakaiproject.email.impl.BasicEmailService.sendToUsers(BasicEmailService.java:
> ****
>
> 767)****
>
>         at
> org.sakaiproject.util.EmailNotification$RunSendToUsers.run(EmailNotification.java
> ****
>
> :186)****
>
>         at java.lang.Thread.run(Thread.java:722)****
>
>  ****
>
> From our research so far it seems that the JVM is not actually out of
> memory but has hit a Linux kernel limit for the number of memory mapped
> files for a single process. For our RHEL 5.x boxes, this is 65536. While we
> could raise this value, we would likely only delay the inevitable failure
> of the JVM once it hits that limit. We have observed that a freshly
> restarted Sakai/Tomcat instance will start with around 1100-1200 memory
> files and continuously grow upwards from there.****
>
>  ****
>
> We have also observed that there is a large population of temporary
> /dev/shm files that have been memory mapped but later deleted which is
> contributed to this mmap count being so high. Here is an example from a box:
> ****
>
>  ****
>
> 2aaabbeb4000-2aaabbeb5000 rw-s 00000000 00:13 9279287
> /dev/shm/sem.m1TZAN (deleted)****
>
> 2aaabbeb5000-2aaabbeb6000 rw-s 00000000 00:13 9279288
> /dev/shm/sem.oU5lY9 (deleted)****
>
> 2aaabbeb6000-2aaabbeb7000 rw-s 00000000 00:13 9279289
> /dev/shm/sem.bTTLlw (deleted)****
>
> 2aaabbeb7000-2aaabbeb8000 rw-s 00000000 00:13 9279290
> /dev/shm/sem.RHXiKS (deleted)****
>
> 2aaabbeb8000-2aaabbeb9000 rw-s 00000000 00:13 9272028
> /dev/shm/sem.epwwAM (deleted)****
>
> 2aaabbeb9000-2aaabbeba000 rw-s 00000000 00:13 9272029
> /dev/shm/sem.yZCZxN (deleted)****
>
> 2aaabbeba000-2aaabbebb000 rw-s 00000000 00:13 9272030
> /dev/shm/sem.46lwvO (deleted)****
>
> 2aaabbebb000-2aaabbebc000 rw-s 00000000 00:13 9272031
> /dev/shm/sem.iRdcuP (deleted)****
>
> 2aaabbebc000-2aaabbebd000 rw-s 00000000 00:13 9272032
> /dev/shm/sem.SyKWsQ (deleted)****
>
> 2aaabbebd000-2aaabbebe000 rw-s 00000000 00:13 9272033
> /dev/shm/sem.eAEKrR (deleted)****
>
> 2aaabbebe000-2aaabbebf000 rw-s 00000000 00:13 9272034
> /dev/shm/sem.NMFAqS (deleted)****
>
>  ****
>
> For reference here are the versions of various software that we are using:
> ****
>
> Apache 2.2.3****
>
> Tomcat 7.0.27****
>
> Java 1.7.0_05****
>
> Linux Kernel 2.6.18-348.4.1.el5****
>
> Redhat Enterprise Linux 5.8****
>
>  ****
>
> Has anyone seen this scenario and might have some advice on how to further
> troubleshoot this problem?****
>
>  ****
>
> Thanks in advance,****
>
>  ****
>
> Eric Crossman****
>
> University of Florida****
>
>  ****
>
>  ****
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20130516/262f2170/attachment-0001.html 


More information about the production mailing list