[Building Sakai] Java 6 seems to be a problem for us

[Charles Hedrick]

As I hinted last week, we've concluded that Java 6 is a problem. We'll  
work with Sun, but before doing so we need to fix out support  
contract, so it may take a while.

Java 6 actually works fine with Sakai, as long as you add - 
Dsun.lang.ClassLoader.allowArraySyntax=true. However since moving to  
Java 6 we started seeing pauses in the garbage collector of 20 sec to  
several minutes, several times a day. It was most often in GC's of the  
new space. Java 6 reports both CPU time and real time for a GC. The  
CPU time was reasonable. It was just the real time that was a problem.  
Until we can figure out what is going on, we've moved back to Java 5.  
We did that last week. We haven't seen any problems since, so I  
believe the Java change actually did make a difference. This could  
well be an interaction between Java and Solaris, so it might not occur  
with Linux or a different machine configuration.

Here are our current options for Java 5.

JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -Xmn3g -XX:+UseConcMarkSweepGC - 
XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 - 
XX:MaxTenuringThreshold=15 -XX:MaxPermSize=512m -XX:PermSize=512m -XX: 
+UseMembar -XX:-UseThreadPriorities -XX:+DisableExplicitGC "

-d64 - force 64 bit; may be the default. You'll also want -server if  
it isn't the default for your system

-Xmx12000m -Xms12000m - this sets the size of the heap. We have 16 GB  
machines. However there are advantage to not using more memory than  
you actually need. Among other things, it will tend to keep all of  
Java in memory. This seems to be the most we need. We may experiment  
with somewhat lower numbers. It is strongly recommended that you set - 
Xms to be the same as -Xmx. The fixes the size of the heap. For large  
heaps this is a good idea.

-Xmn3g - specifies the size of new. In Java 5 the default is too  
small. At least in older versions of Sakai, Sakai would sometimes  
bring all of a file into memory. So we needed new to be several times  
the size of the largest file that could ever be uploaded. Even without  
that, a fairly large new will keep the number of full GCs down. The  
best current tuning advice is that new should be significantly larger  
than the Java 5 default. Java 6 has changed to a larger default new.

  -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  - this specifies the low- 
pause GC. I tried the stop and copy GC, and it resulted in times that  
were too long.

  -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 - 
XX:MaxTenuringThreshold=15  - these tune the GC in new. These are  
recommended in a Sun tuning paper. SurvivorRatio gives the size of the  
two survivor spaces compared to eden (the space from which objects are  
initially allocated). Use 90% of the survivor space (recommended in  
the tuning paper). Keep up to 15 generations in the survivor space.  
Our generation sizes are fairly large, so 31 (another Sun  
recommendation) probably won't work.

-XX:MaxPermSize=512m -XX:PermSize=512m  - Sakai uses a lot of  
permanent memory. This is where generated object code is put. If you  
don't set a non-default value, Sakai will fail. I recommend setting  
initial size to maximum, because when Java expands permanent memory it  
often does a full GC. That's slow, so we want to avoid it.

  -XX:+UseMembar -XX:-UseThreadPriorities  - Solaris-specific hacks to  
work around possible bugs. The bugs are supposed to have been fixed,  
but some reports say they aren't.

-XX:+DisableExplicitGC  - there are a couple of places in the Sakai  
code (one particularly odd one is in Samigo) where Sakai requests a  
full GC. That results in a stop and copy GC, not a normal CMS  
concurrent GC, so it's a really bad idea.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2421 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20091013/8f7a9430/attachment.bin