[Building Sakai] Java 6 seems to be a problem for us

Tue Oct 13 08:18:37 PDT 2009

That is pretty subjective depending on the number of servers you are
running, whether they are 64 bit or not, and how heavy the load is. I
tend to recommend letting the JVM tune itself and running with params
like this:
-server -Xmx1500m -XX:MaxPermSize=250m -Djava.awt.headless=true

This is what we use at Cambridge plus a few to set the locale
correctly, add in JMX for monitoring, and add in debugging hooks. We
used to have something a lot more complex a couple years ago but our
servers would crash sometimes. They have been pretty stable since we
have been using these params though. We run 32 bit JVMs with java 5. I
run something similar for my dev box on java 6 but slightly lower Xmx.
That is hardly helpful though since a production environment is quite
different.

-AZ

On Tue, Oct 13, 2009 at 3:56 PM, Mustansar Mehmood <mustansar at rice.edu> wrote:
> Is there any place to find the complete set of  recommended JVM args for JSE
> 6 for sakai like we have for previous versions of sakai .
> On 10/13/2009 09:07 AM, Aaron Zeckoski wrote:
>
> Charles,
> Have you tried running without the tuning params and seeing where the
> auto tuning settles? You are currently forcing the JVM to run a series
> of params that might be brutally inefficient and it can be eye-opening
> to see how the JVM tunes itself. You may have already done this and
> this might be how you settled on these numbers but I thought I would
> ask anyway.
>
>
>
>  -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
>  - these tune the GC in new. These are recommended in a Sun tuning paper.
> SurvivorRatio gives the size of the two survivor spaces compared to eden
> (the space from which objects are initially allocated). Use 90% of the
> survivor space (recommended in the tuning paper). Keep up to 15 generations
> in the survivor space. Our generation sizes are fairly large, so 31 (another
> Sun recommendation) probably won't work.
>
>
> This is actually enabling the survivor spaces (they are basically
> disabled by using the collectors you chose). They are some pretty huge
> spaces considering the 3g eden.
>
> Have you tried the parallel collector (for old and new)?
>
> Do you have any charts of the memory allocation over time on your
> current servers (java 5 vs. java 6)? Maybe charts for a day or so
> showing the various space usages?
>
> -AZ
>
>
> On Tue, Oct 13, 2009 at 2:35 PM, Charles Hedrick <hedrick at rutgers.edu>
> wrote:
>
>
> As I hinted last week, we've concluded that Java 6 is a problem. We'll work
> with Sun, but before doing so we need to fix out support contract, so it may
> take a while.
>
> Java 6 actually works fine with Sakai, as long as you add
> -Dsun.lang.ClassLoader.allowArraySyntax=true. However since moving to Java 6
> we started seeing pauses in the garbage collector of 20 sec to several
> minutes, several times a day. It was most often in GC's of the new space.
> Java 6 reports both CPU time and real time for a GC. The CPU time was
> reasonable. It was just the real time that was a problem. Until we can
> figure out what is going on, we've moved back to Java 5. We did that last
> week. We haven't seen any problems since, so I believe the Java change
> actually did make a difference. This could well be an interaction between
> Java and Solaris, so it might not occur with Linux or a different machine
> configuration.
>
> Here are our current options for Java 5.
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -Xmn3g -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=15 -XX:MaxPermSize=512m -XX:PermSize=512m
> -XX:+UseMembar -XX:-UseThreadPriorities -XX:+DisableExplicitGC "
>
> -d64 - force 64 bit; may be the default. You'll also want -server if it
> isn't the default for your system
>
> -Xmx12000m -Xms12000m - this sets the size of the heap. We have 16 GB
> machines. However there are advantage to not using more memory than you
> actually need. Among other things, it will tend to keep all of Java in
> memory. This seems to be the most we need. We may experiment with somewhat
> lower numbers. It is strongly recommended that you set -Xms to be the same
> as -Xmx. The fixes the size of the heap. For large heaps this is a good
> idea.
>
> -Xmn3g - specifies the size of new. In Java 5 the default is too small. At
> least in older versions of Sakai, Sakai would sometimes bring all of a file
> into memory. So we needed new to be several times the size of the largest
> file that could ever be uploaded. Even without that, a fairly large new will
> keep the number of full GCs down. The best current tuning advice is that new
> should be significantly larger than the Java 5 default. Java 6 has changed
> to a larger default new.
>
>  -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  - this specifies the low-pause
> GC. I tried the stop and copy GC, and it resulted in times that were too
> long.
>
>  -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
>  - these tune the GC in new. These are recommended in a Sun tuning paper.
> SurvivorRatio gives the size of the two survivor spaces compared to eden
> (the space from which objects are initially allocated). Use 90% of the
> survivor space (recommended in the tuning paper). Keep up to 15 generations
> in the survivor space. Our generation sizes are fairly large, so 31 (another
> Sun recommendation) probably won't work.
>
> -XX:MaxPermSize=512m -XX:PermSize=512m  - Sakai uses a lot of permanent
> memory. This is where generated object code is put. If you don't set a
> non-default value, Sakai will fail. I recommend setting initial size to
> maximum, because when Java expands permanent memory it often does a full GC.
> That's slow, so we want to avoid it.
>
>  -XX:+UseMembar -XX:-UseThreadPriorities  - Solaris-specific hacks to work
> around possible bugs. The bugs are supposed to have been fixed, but some
> reports say they aren't.
>
> -XX:+DisableExplicitGC  - there are a couple of places in the Sakai code
> (one particularly odd one is in Samigo) where Sakai requests a full GC. That
> results in a stop and copy GC, not a normal CMS concurrent GC, so it's a
> really bad idea.
>
>
>
>
> --
> Mustansar Mehmood
> Educational System Developer& Integrator
>
> Infromation Technology
> 6100 Main St MS 119
> Houston Texas 77005
>
> Phone:(713)348-2523
> Fax  :(713)348 6099
> email:mustansar at rice.edu
>
>
>
>
>
> I have yet to see any problem, however complicated, which, when,you  looked
> at it in the right way, did not become still more complicated.
>     -- Poul Anderson
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
>

-- 
Aaron Zeckoski (azeckoski (at) vt.edu)
Senior Research Engineer - CARET - University of Cambridge
https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile