[Building Sakai] Java 6 seems to be a problem for us

[Aaron Zeckoski]

Charles,
Have you tried running without the tuning params and seeing where the
auto tuning settles? You are currently forcing the JVM to run a series
of params that might be brutally inefficient and it can be eye-opening
to see how the JVM tunes itself. You may have already done this and
this might be how you settled on these numbers but I thought I would
ask anyway.

>  -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
>  - these tune the GC in new. These are recommended in a Sun tuning paper.
> SurvivorRatio gives the size of the two survivor spaces compared to eden
> (the space from which objects are initially allocated). Use 90% of the
> survivor space (recommended in the tuning paper). Keep up to 15 generations
> in the survivor space. Our generation sizes are fairly large, so 31 (another
> Sun recommendation) probably won't work.

This is actually enabling the survivor spaces (they are basically
disabled by using the collectors you chose). They are some pretty huge
spaces considering the 3g eden.

Have you tried the parallel collector (for old and new)?

Do you have any charts of the memory allocation over time on your
current servers (java 5 vs. java 6)? Maybe charts for a day or so
showing the various space usages?

-AZ


On Tue, Oct 13, 2009 at 2:35 PM, Charles Hedrick <hedrick at rutgers.edu> wrote:
> As I hinted last week, we've concluded that Java 6 is a problem. We'll work
> with Sun, but before doing so we need to fix out support contract, so it may
> take a while.
>
> Java 6 actually works fine with Sakai, as long as you add
> -Dsun.lang.ClassLoader.allowArraySyntax=true. However since moving to Java 6
> we started seeing pauses in the garbage collector of 20 sec to several
> minutes, several times a day. It was most often in GC's of the new space.
> Java 6 reports both CPU time and real time for a GC. The CPU time was
> reasonable. It was just the real time that was a problem. Until we can
> figure out what is going on, we've moved back to Java 5. We did that last
> week. We haven't seen any problems since, so I believe the Java change
> actually did make a difference. This could well be an interaction between
> Java and Solaris, so it might not occur with Linux or a different machine
> configuration.
>
> Here are our current options for Java 5.
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -Xmn3g -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=15 -XX:MaxPermSize=512m -XX:PermSize=512m
> -XX:+UseMembar -XX:-UseThreadPriorities -XX:+DisableExplicitGC "
>
> -d64 - force 64 bit; may be the default. You'll also want -server if it
> isn't the default for your system
>
> -Xmx12000m -Xms12000m - this sets the size of the heap. We have 16 GB
> machines. However there are advantage to not using more memory than you
> actually need. Among other things, it will tend to keep all of Java in
> memory. This seems to be the most we need. We may experiment with somewhat
> lower numbers. It is strongly recommended that you set -Xms to be the same
> as -Xmx. The fixes the size of the heap. For large heaps this is a good
> idea.
>
> -Xmn3g - specifies the size of new. In Java 5 the default is too small. At
> least in older versions of Sakai, Sakai would sometimes bring all of a file
> into memory. So we needed new to be several times the size of the largest
> file that could ever be uploaded. Even without that, a fairly large new will
> keep the number of full GCs down. The best current tuning advice is that new
> should be significantly larger than the Java 5 default. Java 6 has changed
> to a larger default new.
>
>  -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  - this specifies the low-pause
> GC. I tried the stop and copy GC, and it resulted in times that were too
> long.
>
>  -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
>  - these tune the GC in new. These are recommended in a Sun tuning paper.
> SurvivorRatio gives the size of the two survivor spaces compared to eden
> (the space from which objects are initially allocated). Use 90% of the
> survivor space (recommended in the tuning paper). Keep up to 15 generations
> in the survivor space. Our generation sizes are fairly large, so 31 (another
> Sun recommendation) probably won't work.
>
> -XX:MaxPermSize=512m -XX:PermSize=512m  - Sakai uses a lot of permanent
> memory. This is where generated object code is put. If you don't set a
> non-default value, Sakai will fail. I recommend setting initial size to
> maximum, because when Java expands permanent memory it often does a full GC.
> That's slow, so we want to avoid it.
>
>  -XX:+UseMembar -XX:-UseThreadPriorities  - Solaris-specific hacks to work
> around possible bugs. The bugs are supposed to have been fixed, but some
> reports say they aren't.
>
> -XX:+DisableExplicitGC  - there are a couple of places in the Sakai code
> (one particularly odd one is in Samigo) where Sakai requests a full GC. That
> results in a stop and copy GC, not a normal CMS concurrent GC, so it's a
> really bad idea.


-- 
Aaron Zeckoski (azeckoski (at) vt.edu)
Senior Research Engineer - CARET - University of Cambridge
https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile